This is not as simple as it might sound from the headline!
I built a small lab for myself to test out the CI/CD flow. I had a Fedora VM running on Virtualbox on Xubuntu 16.04 LTS with following details, I don't like to use my base operating system for development work, it is a stable build Xubuntu which works great. All of my dev work happens on VM's, so that if I am on move, I can ship it to my Macbook Air and go from there.
Kernel - Linux localhost.localdomain 4.10.14-200.fc25.x86_64 #1 SMP Wed May 3 22:52:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux [root@localhost ~]# docker info Containers: 0 Running: 0 Paused: 0 Stopped: 0 Images: 0 Server Version: 1.12.6 Storage Driver: devicemapper Pool Name: docker-docker--pool Pool Blocksize: 524.3 kB Base Device Size: 10.74 GB Backing Filesystem: xfs Data file: Metadata file: Data Space Used: 20.45 MB Data Space Total: 21.45 GB Data Space Available: 21.43 GB Metadata Space Used: 49.15 kB Metadata Space Total: 54.53 MB Metadata Space Available: 54.48 MB Thin Pool Minimum Free Space: 2.145 GB Udev Sync Supported: true Deferred Removal Enabled: true Deferred Deletion Enabled: true Deferred Deleted Device Count: 0 Library Version: 1.02.136 (2016-11-05) Logging Driver: journald Cgroup Driver: systemd Plugins: Volume: local Network: null bridge host overlay Swarm: inactive Runtimes: oci runc Default Runtime: oci Security Options: seccomp Kernel Version: 4.10.14-200.fc25.x86_64 Operating System: Fedora 25 (Server Edition) OSType: linux Architecture: x86_64 Number of Docker Hooks: 2 CPUs: 2 Total Memory: 3.858 GiB Name: localhost.localdomain ID: CC4Y:D42S:64XF:6FIU:AHTE:XXQ6:HBBZ:IEFE:QRFE:KCZ6:LJD2:IVLM Docker Root Dir: /var/lib/docker Debug Mode (client): false Debug Mode (server): false Registry: https://index.docker.io/v1/ WARNING: bridge-nf-call-iptables is disabled WARNING: bridge-nf-call-ip6tables is disabled Insecure Registries: 127.0.0.0/8 Registries: docker.io (secure)
The VM has Host Only networking to communicate to Host OS & Bridge mode to communicate to Internet. For newbies, VM's Host OS is Xubuntu, Docker Container's Host is Fedora VM.
For you, all you should care about is Fedora VM.
Initial Dockerfile for Sample Apache Tomcat App -
I fetched a Base OS image from Docker Hub for CentOS 7.3.1611. Funny thing, if you notice following error, just make sure your system has nameservers properly configured (no DNS issues). root@localhost prompt is my VM OS Shell prompt.
[root@localhost ~]# docker pull centos Using default tag: latest Trying to pull repository docker.io/library/centos ... Pulling repository docker.io/library/centos Error while pulling image: Get https://index.docker.io/v1/repositories/library/centos/images: dial tcp: lookup index.docker.io on [::1]:53: read udp [::1]:55267->[::1]:53: read: connection refused
I faced issues with running Tomcat via Systemctl due to Systemd issues inside Docker Container. Known issues here & here. Workaround found at Redhat Blog here. I had to create a CentOS_Systemd container first, then I had to use that as a base image for Tomcat Image. I never faced this issue on RHEL7 images from Redhat Registry. Good summary of steps are present here. To run Tomcat as a Service, this is a must. Since I ran a Systemd based OS, their analogy is different than the Docker Upstream, it advocates that every process should be traced back to PID 1, so a Container should have PID 1. There is a latest article on Redhat Blog by the same author here. The best documentation I found was on CentOS Systemd Integration page here. This was a good enough detour from my original goal, but I got it working finally. The most important note is "systemctl start tomcat" doesn't work at the time of Docker Build, only "systemctl enable tomcat" works. When inside the container, /usr/sbin/init doesn't work, the key is to run "exec /usr/sbin/init", this will invoke Systemd routines and will also start the Tomcat service.
centos_systemd Docker Build -
[root@localhost ~]# docker build -t centos_systemd . Sending build context to Docker daemon 69.65 MB <<Snipped>> Removing intermediate container 0b86dda24fbe Successfully built 43c0e6cb0bd0 [root@localhost ~]#
centos_systemd Dockerfile -
# A basic Apache Tomcat server to showcase the CI/CD flow using Jenkins/Docker # Based on http://developers.redhat.com/blog/2014/05/05/running-systemd-within-docker-container/ to circumvent the SystemD inside Docker known issue FROM centos MAINTAINER Subodh Pachghare version: 0.1 <firstname.lastname@example.org> ENV container docker RUN yum install -y systemd RUN yum install -y java-1.8.0-openjdk-headless RUN (cd /lib/systemd/system/sysinit.target.wants/; for i in *; do [ $i == systemd-tmpfiles-setup.service ] || rm -f $i; done); \ rm -f /lib/systemd/system/multi-user.target.wants/*;\ rm -f /etc/systemd/system/*.wants/*;\ rm -f /lib/systemd/system/local-fs.target.wants/*; \ rm -f /lib/systemd/system/sockets.target.wants/*udev*; \ rm -f /lib/systemd/system/sockets.target.wants/*initctl*; \ rm -f /lib/systemd/system/basic.target.wants/*;\ rm -f /lib/systemd/system/anaconda.target.wants/* VOLUME ["/sys/fs/cgroup"] CMD ["/usr/sbin/init"]
mytomcat Docker Build -
Dockerfile here. Tomcat sample app used from here.
[root@localhost test]# docker build -t mytomcat . Sending build context to Docker daemon 2.048 kB Step 1 : FROM
172.17.0.2:5000/centos_systemd:latest<<Snipped>> Removing intermediate container c8effb63c2dd Successfully built eb652553a64e
mytomcat Dockerfile -
# A basic Apache Tomcat server to showcase the CI/CD flow using Jenkins/Docker # Based on http://developers.redhat.com/blog/2014/05/05/running-systemd-within-docker-container/ to circumvent the SystemD inside Docker known issue FROM 172.17.0.2:5000/centos_systemd:latest MAINTAINER Subodh Pachghare version: 0.1 <email@example.com> RUN yum install -y tomcat tomcat-webapps tomcat-admin-webapps wget RUN systemctl enable tomcat RUN cd /usr/share/tomcat/webapps && wget https://tomcat.apache.org/tomcat-6.0-doc/appdev/sample/sample.war EXPOSE 8080 CMD ["/usr/sbin/init"]
Docker Images -
[root@localhost test]# docker images REPOSITORY TAG IMAGE ID CREATED SIZE mytomcat latest eb652553a64e 10 minutes ago 460.1 MB centos_systemd latest 4c8d701144c0 23 minutes ago 416.4 MB docker.io/centos latest 8140d0c64310 2 weeks ago 192.5 MB
Docker Run Command(s) -
Or in some case, following worked too.
docker run --privileged -ti -v /sys/fs/cgroup:/sys/fs/cgroup:ro -p 8080:8080 mytomcat
docker run -ti -v /sys/fs/cgroup:/sys/fs/cgroup:ro -p 8080:8080 mytomcat
Note - Do not use Bash as Entrypoint, it overrides the CMD ["/usr/sbin/init"] inside the Dockerfile, systemd never gets invoked. If you use it, make sure to do "exec /usr/sbin/init" inside Container.
Shell Output -
[root@localhost ~]# docker run --privileged -ti -v /sys/fs/cgroup:/sys/fs/cgroup:ro -p 8080:8080 mytomcat systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) Detected virtualization docker. Detected architecture x86-64. Welcome to CentOS Linux 7 (Core)! Set hostname to <b04d834d8c31>. [ OK ] Reached target Swap. [ OK ] Reached target Paths. [ OK ] Created slice Root Slice. [ OK ] Listening on Delayed Shutdown Socket. [ OK ] Listening on Journal Socket. [ OK ] Created slice System Slice. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Started Create Volatile Files and Directories. [ INFO ] Update UTMP about System Boot/Shutdown is not active. [DEPEND] Dependency failed for Update UTMP about System Runlevel Changes. Job systemd-update-utmp-runlevel.service/start failed with result 'dependency'. [ OK ] Started Journal Service. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Apache Tomcat Web Application Container. Starting Apache Tomcat Web Application Container... [ OK ] Reached target Multi-User System. [ OK ] Reached target Timers.
Notice the Tomcat server starting. Since we have the sample app in Webapps directory, it worked directly. Since the init is in progress, this Container cannot be stopped with CTRL^C. From other shell prompt, I needed to do "docker stop".
From my Xubuntu OS, I accessed 192.168.56.101:8080/sample, it worked.
Docker Registry -
I needed a Docker Registry running on Docker Host (VM), so that the images built by Jenkins are pushed to it & it can be pulled for Continuous Deployment Cycle. So I created a Registry using a simple command -
docker run -d -p 5000:5000 --restart=always --name registry registry:2
Now I can pull/push from 5000 port number from any interface on this VM (Host Only/Bridge).
Edited the /etc/sysconfig/docker and added the docker registry as unsecured one. The 172.17.0.2 is the internal IP address assigned to Registry Container, which can be found out by "docker network ls" or "docker inspect" command. Reload the daemon after adding this.
ADD_REGISTRY='--add-registry 172.17.0.2:5000' INSECURE_REGISTRY='--insecure-registry 172.17.0.2:5000'
Push the centos_systemd image to this repo for future reference.
docker tag 4c8d701144c0 172.17.0.2:5000/centos_systemd:latest docker push 172.17.0.2:5000/centos_systemd:latest
Verify by pulling the image, so that Jenkins image mytomcat will not face any issues.
Docker Host-Client Setup -
In next step, I was going to use Jenkins in Container, we need the Docker Host to be accessible via outside, in Docker Host and Docker Client Configuration. By default, Docker doesn't exposes to outside world.
Added following to /etc/sysconfig/docker & reload the Docker -
OPTIONS=-H unix:///var/run/docker.sock -H tcp://0.0.0.0
This told Docker daemon to listen on all interfaces of the Docker Host (VM). Then I cleared all firewall/iptables rules in the system, so that ingress connections can be accepted.
iptables -F iptables -t nat -F iptables -L
Verified using "ps -aef" that the socket options are accepted by Docker Daemon. From outside docker box (a complete different box), I validated the Host-Client connection.
root@ninja:~# docker -H tcp://192.168.56.101:2375 ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 6fb8530fbda4 jenkins "/bin/tini -- /usr/lo" 5 minutes ago Up 5 minutes 0.0.0.0:50000->50000/tcp, 0.0.0.0:8081->8080/tcp focused_bartik 5898d5176fac registry:2 "/entrypoint.sh /etc/" 6 hours ago Up 8 minutes 0.0.0.0:5000->5000/tcp registry root@ninja:~#
192.168.56.101 is external facing IP address of VM.
Continuous Integration (CI) -
Jenkins Master (in Container) Setup -
I used Jenkins LTS image from Docker Hub to run Jenkins Master Server inside Docker Container as it doesn't do actual docker build of new images. Jenkins slave should be a Docker Host in a Corporate Environment, so that it is easier to build Docker Images. Here I am using Docker Host (VM) server as a Docker Builder connected via SSH. Then I made the Jenkins Master as offline, so it will not perform any builds as Jenkins Master in Container doesn't have Docker binary.
[root@localhost ~]# docker pull jenkins Using default tag: latest Trying to pull repository docker.io/library/jenkins ... sha256:983472b40004d48fb2896ca8b0d9825707e65527dd8e78a152b7f543051f8b1e: Pulling from docker.io/library/jenkins Digest: sha256:983472b40004d48fb2896ca8b0d9825707e65527dd8e78a152b7f543051f8b1e Status: Downloaded newer image for docker.io/jenkins:latest
Launch Jenkins with Bind Mount Volume -
docker run -p 8081:8080 -p 50000:50000 -u root -v /root/jenkins:/var/jenkins_home jenkins
I created a directory at /root named jenkins, since it was owned by root, I gave -u parameter as root.
Successful Output -
May 25, 2017 9:06:13 PM jenkins.install.SetupWizard init INFO: ************************************************************* ************************************************************* ************************************************************* Jenkins initial setup is required. An admin user has been created and a password generated. Please use the following password to proceed to installation: 70b11f7f3c4549e88f641ca90d9b4929 This may also be found at: /var/jenkins_home/secrets/initialAdminPassword ************************************************************* ************************************************************* ************************************************************* --> setting agent port for jnlp --> setting agent port for jnlp... done May 25, 2017 9:06:30 PM hudson.model.UpdateSite updateData INFO: Obtained the latest update center data file for UpdateSource default May 25, 2017 9:06:31 PM hudson.model.DownloadService$Downloadable load INFO: Obtained the updated data file for hudson.tasks.Maven.MavenInstaller May 25, 2017 9:06:35 PM hudson.model.DownloadService$Downloadable load INFO: Obtained the updated data file for hudson.tools.JDKInstaller May 25, 2017 9:06:35 PM hudson.model.AsyncPeriodicWork$1 run INFO: Finished Download metadata. 22,313 ms May 25, 2017 9:06:38 PM hudson.model.UpdateSite updateData INFO: Obtained the latest update center data file for UpdateSource default May 25, 2017 9:06:38 PM hudson.WebAppMain$3 run INFO: Jenkins is fully up and running
Then I accessed the Web Console using my Xubuntu OS at http://192.168.56.101:8081 (8080 of Jenkins is exposed on 8081 to outside world as Tomcat was supposed to run on 8080).
Following plugins were automatically installed.
All of the plugins and static data was getting stored at /root/jenkins on Docker Host (VM). So the state of cluster is preserved, even if the Container was restarted. After creating an user, I was on landing page.
Installed the Cloudbees Build-Publish Plugin for Dockerfile Build.
Gave all inputs related to Github public project. Notice the Github Webhook configuration. Monitored the Github Hook Log to see any messages. Since I am behind Dynamic IP, my Jenkins server doesn't have Ingress access to Payload URL. So I don't have a unique URL like "http://yourdomain.com/github-webhook/" to which Github can send Push Events. Mine is "http://192.168.56.101:8081/github-webhook/" which is within private region. If my Jenkins server was public facing, then this Webhook will go into the "Add Hook" section of Github. Right now, I am going to trigger a manual build.
Jenkins Slave Configuration -
I reduced Build Executors to 1, and made master offline.
Configured master settings.
172.17.0.1 is the Gateway interface for Docker Network, on which Docker Host is running.
Created a /root/jenkins-slave directory on VM, this will act as a Workspace home for all project. $WORKSPACE for new jobs.
Notice below that Master is running inside Container and Jenkins-Slave is Docker Host.
Jenkins Credentials had both the Jenkins Global Namespace creds for Jenkins itself and for SSH User/Password for Slave Node.
Created a new project MyTomcat Pipeline and configured it.
Before running the build/push job -
[root@localhost ~]# docker images REPOSITORY TAG IMAGE ID CREATED SIZE 172.17.0.2:5000/centos_systemd latest 4c8d701144c0 12 hours ago 416.4 MB centos_systemd latest 4c8d701144c0 12 hours ago 416.4 MB docker.io/jenkins latest 681ef98a247f 9 days ago 704.2 MB docker.io/centos latest 8140d0c64310 2 weeks ago 192.5 MB docker.io/registry 2 9d0c4eabab4d 2 weeks ago 33.17 MB
No content is $WORKSPACE - /root/jenkins-slave/workspace/MyTomcat Pipeline
I manually ran "Build Now" from Jenkins, If I had a Static IP & hostname, I just would had to do a Git Commit.
Populated workspace content on Jenkins Slave -
[root@localhost MyTomcat Pipeline]# pwd /root/jenkins-slave/workspace/MyTomcat Pipeline [root@localhost MyTomcat Pipeline]# ls Dockerfile LICENSE README.md [root@localhost MyTomcat Pipeline]#
Full successful Build Log here.
Docker images after successful build and push to Registry.
REPOSITORY TAG IMAGE ID CREATED SIZE 172.17.0.2:5000/mytomcat jenkins 3eae7b04dd10 5 minutes ago 627.3 MB 172.17.0.2:5000/mytomcat latest 3eae7b04dd10 5 minutes ago 627.3 MB centos_systemd latest 4c8d701144c0 13 hours ago 416.4 MB 172.17.0.2:5000/centos_systemd latest 4c8d701144c0 13 hours ago 416.4 MB docker.io/jenkins latest 681ef98a247f 9 days ago 704.2 MB docker.io/centos latest 8140d0c64310 2 weeks ago 192.5 MB docker.io/registry 2 9d0c4eabab4d 2 weeks ago 33.17 MB
This is beautiful, the Dockerfile was downloaded from Github & a container was created for Tomcat and it was pushed to Registry for users to consume with a single click.
Continuous Deployment (CD) -
Now I tweaked the Jenkins Project furthermore to include Deployment logic so that Tomcat should be up and running.
I used another plugin named "Docker Build Step" which allows for heavy customization to deploy images with various command line parameters.
Plugin details here.
Additional configuration in Build section in addition to CI -
Two steps were involved ,first was to create the Container and other one was to start the Container.
Full Deploy Log here.
[root@localhost MyTomcat Pipeline]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES ad446046f8e5 172.17.0.2:5000/mytomcat:jenkins "/usr/sbin/init" 12 seconds ago Up 9 seconds 0.0.0.0:8080->8080/tcp mytomcat1 1226ab94235f jenkins "/bin/tini -- /usr/lo" About an hour ago Up About an hour 0.0.0.0:50000->50000/tcp, 0.0.0.0:8081->8080/tcp furious_shannon 5898d5176fac registry:2 "/entrypoint.sh /etc/" 11 hours ago Up About an hour 0.0.0.0:5000->5000/tcp registry [root@localhost MyTomcat Pipeline]#
I had the Container running right away & serving the Sample Tomcat App at http://192.168.56.101:8080/sample/.
This was great! This is how I was able to achieve a Build/Deploy process via single "Build Now" click in Jenkins.
Ofcourse, this is not a Enterprise Architecture, but enough to understand the intricacies.