Introduction -
This is not as simple as it might sound from the headline!
I built a small lab for myself to test out the CI/CD flow. I had a Fedora VM running on Virtualbox on Xubuntu 16.04 LTS with following details, I don't like to use my base operating system for development work, it is a stable build Xubuntu which works great. All of my dev work happens on VM's, so that if I am on move, I can ship it to my Macbook Air and go from there.
Kernel - Linux localhost.localdomain 4.10.14-200.fc25.x86_64 #1 SMP Wed May 3 22:52:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
[root@localhost ~]# docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 1.12.6
Storage Driver: devicemapper
Pool Name: docker-docker--pool
Pool Blocksize: 524.3 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file:
Metadata file:
Data Space Used: 20.45 MB
Data Space Total: 21.45 GB
Data Space Available: 21.43 GB
Metadata Space Used: 49.15 kB
Metadata Space Total: 54.53 MB
Metadata Space Available: 54.48 MB
Thin Pool Minimum Free Space: 2.145 GB
Udev Sync Supported: true
Deferred Removal Enabled: true
Deferred Deletion Enabled: true
Deferred Deleted Device Count: 0
Library Version: 1.02.136 (2016-11-05)
Logging Driver: journald
Cgroup Driver: systemd
Plugins:
Volume: local
Network: null bridge host overlay
Swarm: inactive
Runtimes: oci runc
Default Runtime: oci
Security Options: seccomp
Kernel Version: 4.10.14-200.fc25.x86_64
Operating System: Fedora 25 (Server Edition)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 2
CPUs: 2
Total Memory: 3.858 GiB
Name: localhost.localdomain
ID: CC4Y:D42S:64XF:6FIU:AHTE:XXQ6:HBBZ:IEFE:QRFE:KCZ6:LJD2:IVLM
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
127.0.0.0/8
Registries: docker.io (secure)
The VM has Host Only networking to communicate to Host OS & Bridge mode to communicate to Internet. For newbies, VM's Host OS is Xubuntu, Docker Container's Host is Fedora VM.
For you, all you should care about is Fedora VM.
Initial Dockerfile for Sample Apache Tomcat App -
I fetched a Base OS image from Docker Hub for CentOS 7.3.1611. Funny thing, if you notice following error, just make sure your system has nameservers properly configured (no DNS issues). root@localhost prompt is my VM OS Shell prompt.
[root@localhost ~]# docker pull centos
Using default tag: latest
Trying to pull repository docker.io/library/centos ...
Pulling repository docker.io/library/centos
Error while pulling image: Get https://index.docker.io/v1/repositories/library/centos/images: dial tcp: lookup index.docker.io on [::1]:53: read udp [::1]:55267->[::1]:53: read: connection refused
I faced issues with running Tomcat via Systemctl due to Systemd issues inside Docker Container. Known issues
here &
here. Workaround found at Redhat Blog
here. I had to create a CentOS_Systemd container first, then I had to use that as a base image for Tomcat Image. I never faced this issue on RHEL7 images from Redhat Registry. Good summary of steps are present
here. To run Tomcat as a Service, this is a must. Since I ran a Systemd based OS, their analogy is different than the Docker Upstream, it advocates that every process should be traced back to PID 1, so a Container should have PID 1. There is a latest article on Redhat Blog by the same author
here. The best documentation I found was on CentOS Systemd Integration page
here. This was a good enough detour from my original goal, but I got it working finally. The most important note is "systemctl start tomcat" doesn't work at the time of Docker Build, only "systemctl enable tomcat" works. When inside the container, /usr/sbin/init doesn't work, the key is to run "exec /usr/sbin/init", this will invoke Systemd routines and will also start the Tomcat service.
centos_systemd Docker Build -
[root@localhost ~]# docker build -t centos_systemd .
Sending build context to Docker daemon 69.65 MB
<<Snipped>>
Removing intermediate container 0b86dda24fbe
Successfully built 43c0e6cb0bd0
[root@localhost ~]#
Dockerfile
here.
centos_systemd Dockerfile -
# A basic Apache Tomcat server to showcase the CI/CD flow using Jenkins/Docker
# Based on http://developers.redhat.com/blog/2014/05/05/running-systemd-within-docker-container/ to circumvent the SystemD inside Docker known issue
FROM centos
MAINTAINER Subodh Pachghare version: 0.1 <subodh.cyber@gmail.com>
ENV container docker
RUN yum install -y systemd
RUN yum install -y java-1.8.0-openjdk-headless
RUN (cd /lib/systemd/system/sysinit.target.wants/; for i in *; do [ $i == systemd-tmpfiles-setup.service ] || rm -f $i; done); \
rm -f /lib/systemd/system/multi-user.target.wants/*;\
rm -f /etc/systemd/system/*.wants/*;\
rm -f /lib/systemd/system/local-fs.target.wants/*; \
rm -f /lib/systemd/system/sockets.target.wants/*udev*; \
rm -f /lib/systemd/system/sockets.target.wants/*initctl*; \
rm -f /lib/systemd/system/basic.target.wants/*;\
rm -f /lib/systemd/system/anaconda.target.wants/*
VOLUME ["/sys/fs/cgroup"]
CMD ["/usr/sbin/init"]
mytomcat Docker Build -
[root@localhost test]# docker build -t mytomcat .
Sending build context to Docker daemon 2.048 kB
Step 1 : FROM 172.17.0.2:5000/centos_systemd:latest
<<Snipped>>
Removing intermediate container c8effb63c2dd
Successfully built eb652553a64e
Dockerfile
here. Tomcat sample app used from
here.
mytomcat Dockerfile -
# A basic Apache Tomcat server to showcase the CI/CD flow using Jenkins/Docker
# Based on http://developers.redhat.com/blog/2014/05/05/running-systemd-within-docker-container/ to circumvent the SystemD inside Docker known issue
FROM 172.17.0.2:5000/centos_systemd:latest
MAINTAINER Subodh Pachghare version: 0.1 <subodh.cyber@gmail.com>
RUN yum install -y tomcat tomcat-webapps tomcat-admin-webapps wget
RUN systemctl enable tomcat
RUN cd /usr/share/tomcat/webapps && wget https://tomcat.apache.org/tomcat-6.0-doc/appdev/sample/sample.war
EXPOSE 8080
CMD ["/usr/sbin/init"]
Docker Images -
[root@localhost test]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
mytomcat latest eb652553a64e 10 minutes ago 460.1 MB
centos_systemd latest 4c8d701144c0 23 minutes ago 416.4 MB
docker.io/centos latest 8140d0c64310 2 weeks ago 192.5 MB
Docker Run Command(s) -
docker run --privileged -ti -v /sys/fs/cgroup:/sys/fs/cgroup:ro -p 8080:8080 mytomcat
Or in some case, following worked too.
docker run -ti -v /sys/fs/cgroup:/sys/fs/cgroup:ro -p 8080:8080 mytomcat
Note - Do not use Bash as Entrypoint, it overrides the CMD ["/usr/sbin/init"] inside the Dockerfile, systemd never gets invoked. If you use it, make sure to do "exec /usr/sbin/init" inside Container.
Shell Output -
[root@localhost ~]# docker run --privileged -ti -v /sys/fs/cgroup:/sys/fs/cgroup:ro -p 8080:8080 mytomcat
systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN)
Detected virtualization docker.
Detected architecture x86-64.
Welcome to CentOS Linux 7 (Core)!
Set hostname to <b04d834d8c31>.
[ OK ] Reached target Swap.
[ OK ] Reached target Paths.
[ OK ] Created slice Root Slice.
[ OK ] Listening on Delayed Shutdown Socket.
[ OK ] Listening on Journal Socket.
[ OK ] Created slice System Slice.
[ OK ] Reached target Slices.
[ OK ] Reached target Local File Systems.
Starting Create Volatile Files and Directories...
Starting Journal Service...
[ OK ] Started Create Volatile Files and Directories.
[ INFO ] Update UTMP about System Boot/Shutdown is not active.
[DEPEND] Dependency failed for Update UTMP about System Runlevel Changes.
Job systemd-update-utmp-runlevel.service/start failed with result 'dependency'.
[ OK ] Started Journal Service.
[ OK ] Reached target System Initialization.
[ OK ] Listening on D-Bus System Message Bus Socket.
[ OK ] Reached target Sockets.
[ OK ] Reached target Basic System.
[ OK ] Started Apache Tomcat Web Application Container.
Starting Apache Tomcat Web Application Container...
[ OK ] Reached target Multi-User System.
[ OK ] Reached target Timers.
Notice the Tomcat server starting. Since we have the sample app in Webapps directory, it worked directly. Since the init is in progress, this Container cannot be stopped with CTRL^C. From other shell prompt, I needed to do "docker stop".
From my Xubuntu OS, I accessed 192.168.56.101:8080/sample, it worked.
Docker Registry -
I needed a Docker Registry running on Docker Host (VM), so that the images built by Jenkins are pushed to it & it can be pulled for Continuous Deployment Cycle. So I created a Registry using a simple command -
docker run -d -p 5000:5000 --restart=always --name registry registry:2
Now I can pull/push from 5000 port number from any interface on this VM (Host Only/Bridge).
Edited the /etc/sysconfig/docker and added the docker registry as unsecured one. The 172.17.0.2 is the internal IP address assigned to Registry Container, which can be found out by "docker network ls" or "docker inspect" command. Reload the daemon after adding this.
ADD_REGISTRY='--add-registry 172.17.0.2:5000'
INSECURE_REGISTRY='--insecure-registry 172.17.0.2:5000'
Push the centos_systemd image to this repo for future reference.
docker tag 4c8d701144c0 172.17.0.2:5000/centos_systemd:latest
docker push 172.17.0.2:5000/centos_systemd:latest
Verify by pulling the image, so that Jenkins image mytomcat will not face any issues.
Docker Host-Client Setup -
In next step, I was going to use Jenkins in Container, we need the Docker Host to be accessible via outside, in Docker Host and Docker Client Configuration. By default, Docker doesn't exposes to outside world.
Added following to /etc/sysconfig/docker & reload the Docker -
OPTIONS=-H unix:///var/run/docker.sock -H tcp://0.0.0.0
This told Docker daemon to listen on all interfaces of the Docker Host (VM). Then I cleared all firewall/iptables rules in the system, so that ingress connections can be accepted.
iptables -F
iptables -t nat -F
iptables -L
Verified using "ps -aef" that the socket options are accepted by Docker Daemon. From outside docker box (a complete different box), I validated the Host-Client connection.
root@ninja:~# docker -H tcp://192.168.56.101:2375 ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
6fb8530fbda4 jenkins "/bin/tini -- /usr/lo" 5 minutes ago Up 5 minutes 0.0.0.0:50000->50000/tcp, 0.0.0.0:8081->8080/tcp focused_bartik
5898d5176fac registry:2 "/entrypoint.sh /etc/" 6 hours ago Up 8 minutes 0.0.0.0:5000->5000/tcp registry
root@ninja:~#
192.168.56.101 is external facing IP address of VM.
Continuous Integration (CI) -
Jenkins Master (in Container) Setup -
I used Jenkins LTS image from Docker Hub to run Jenkins Master Server inside Docker Container as it doesn't do actual docker build of new images. Jenkins slave should be a Docker Host in a Corporate Environment, so that it is easier to build Docker Images. Here I am using Docker Host (VM) server as a Docker Builder connected via SSH. Then I made the Jenkins Master as offline, so it will not perform any builds as Jenkins Master in Container doesn't have Docker binary.
[root@localhost ~]# docker pull jenkins
Using default tag: latest
Trying to pull repository docker.io/library/jenkins ...
sha256:983472b40004d48fb2896ca8b0d9825707e65527dd8e78a152b7f543051f8b1e: Pulling from docker.io/library/jenkins
Digest: sha256:983472b40004d48fb2896ca8b0d9825707e65527dd8e78a152b7f543051f8b1e
Status: Downloaded newer image for docker.io/jenkins:latest
Launch Jenkins with Bind Mount Volume -
docker run -p 8081:8080 -p 50000:50000 -u root -v /root/jenkins:/var/jenkins_home jenkins
I created a directory at /root named jenkins, since it was owned by root, I gave -u parameter as root.
Successful Output -
May 25, 2017 9:06:13 PM jenkins.install.SetupWizard init
INFO:
*************************************************************
*************************************************************
*************************************************************
Jenkins initial setup is required. An admin user has been created and a password generated.
Please use the following password to proceed to installation:
70b11f7f3c4549e88f641ca90d9b4929
This may also be found at: /var/jenkins_home/secrets/initialAdminPassword
*************************************************************
*************************************************************
*************************************************************
--> setting agent port for jnlp
--> setting agent port for jnlp... done
May 25, 2017 9:06:30 PM hudson.model.UpdateSite updateData
INFO: Obtained the latest update center data file for UpdateSource default
May 25, 2017 9:06:31 PM hudson.model.DownloadService$Downloadable load
INFO: Obtained the updated data file for hudson.tasks.Maven.MavenInstaller
May 25, 2017 9:06:35 PM hudson.model.DownloadService$Downloadable load
INFO: Obtained the updated data file for hudson.tools.JDKInstaller
May 25, 2017 9:06:35 PM hudson.model.AsyncPeriodicWork$1 run
INFO: Finished Download metadata. 22,313 ms
May 25, 2017 9:06:38 PM hudson.model.UpdateSite updateData
INFO: Obtained the latest update center data file for UpdateSource default
May 25, 2017 9:06:38 PM hudson.WebAppMain$3 run
INFO: Jenkins is fully up and running
Then I accessed the Web Console using my Xubuntu OS at http://192.168.56.101:8081 (8080 of Jenkins is exposed on 8081 to outside world as Tomcat was supposed to run on 8080).
Following plugins were automatically installed.
All of the plugins and static data was getting stored at /root/jenkins on Docker Host (VM). So the state of cluster is preserved, even if the Container was restarted. After creating an user, I was on landing page.
Installed the Cloudbees Build-Publish Plugin for Dockerfile Build.
Gave all inputs related to Github public project. Notice the Github Webhook configuration. Monitored the Github Hook Log to see any messages. Since I am behind Dynamic IP, my Jenkins server doesn't have Ingress access to Payload URL. So I don't have a unique URL like "http://yourdomain.com/github-webhook/" to which Github can send Push Events. Mine is "http://192.168.56.101:8081/github-webhook/" which is within private region. If my Jenkins server was public facing, then this Webhook will go into the "Add Hook" section of Github. Right now, I am going to trigger a manual build.
Jenkins Slave Configuration -
I reduced Build Executors to 1, and made master offline.
Configured master settings.
172.17.0.1 is the Gateway interface for Docker Network, on which Docker Host is running.
Created a /root/jenkins-slave directory on VM, this will act as a Workspace home for all project. $WORKSPACE for new jobs.
Notice below that Master is running inside Container and Jenkins-Slave is Docker Host.
Jenkins Credentials had both the Jenkins Global Namespace creds for Jenkins itself and for SSH User/Password for Slave Node.
Created a new project MyTomcat Pipeline and configured it.
Before running the build/push job -
[root@localhost ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
172.17.0.2:5000/centos_systemd latest 4c8d701144c0 12 hours ago 416.4 MB
centos_systemd latest 4c8d701144c0 12 hours ago 416.4 MB
docker.io/jenkins latest 681ef98a247f 9 days ago 704.2 MB
docker.io/centos latest 8140d0c64310 2 weeks ago 192.5 MB
docker.io/registry 2 9d0c4eabab4d 2 weeks ago 33.17 MB
No content is $WORKSPACE - /root/jenkins-slave/workspace/MyTomcat Pipeline
I manually ran "Build Now" from Jenkins, If I had a Static IP & hostname, I just would had to do a Git Commit.
Populated workspace content on Jenkins Slave -
[root@localhost MyTomcat Pipeline]# pwd
/root/jenkins-slave/workspace/MyTomcat Pipeline
[root@localhost MyTomcat Pipeline]# ls
Dockerfile LICENSE README.md
[root@localhost MyTomcat Pipeline]#
Full successful Build Log
here.
Docker images after successful build and push to Registry.
REPOSITORY TAG IMAGE ID CREATED SIZE
172.17.0.2:5000/mytomcat jenkins 3eae7b04dd10 5 minutes ago 627.3 MB
172.17.0.2:5000/mytomcat latest 3eae7b04dd10 5 minutes ago 627.3 MB
centos_systemd latest 4c8d701144c0 13 hours ago 416.4 MB
172.17.0.2:5000/centos_systemd latest 4c8d701144c0 13 hours ago 416.4 MB
docker.io/jenkins latest 681ef98a247f 9 days ago 704.2 MB
docker.io/centos latest 8140d0c64310 2 weeks ago 192.5 MB
docker.io/registry 2 9d0c4eabab4d 2 weeks ago 33.17 MB
This is beautiful, the Dockerfile was downloaded from Github & a container was created for Tomcat and it was pushed to Registry for users to consume with a single click.
Continuous Deployment (CD) -
Now I tweaked the Jenkins Project furthermore to include Deployment logic so that Tomcat should be up and running.
I used another plugin named "Docker Build Step" which allows for heavy customization to deploy images with various command line parameters.
Additional configuration in Build section in addition to CI -
Two steps were involved ,first was to create the Container and other one was to start the Container.
[root@localhost MyTomcat Pipeline]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ad446046f8e5 172.17.0.2:5000/mytomcat:jenkins "/usr/sbin/init" 12 seconds ago Up 9 seconds 0.0.0.0:8080->8080/tcp mytomcat1
1226ab94235f jenkins "/bin/tini -- /usr/lo" About an hour ago Up About an hour 0.0.0.0:50000->50000/tcp, 0.0.0.0:8081->8080/tcp furious_shannon
5898d5176fac registry:2 "/entrypoint.sh /etc/" 11 hours ago Up About an hour 0.0.0.0:5000->5000/tcp registry
[root@localhost MyTomcat Pipeline]#
I had the Container running right away & serving the Sample Tomcat App at http://192.168.56.101:8080/sample/.
This was great! This is how I was able to achieve a Build/Deploy process via single "Build Now" click in Jenkins.
Ofcourse, this is not a Enterprise Architecture, but enough to understand the intricacies.
Cheers!
