Jenkins Case Study
Continuous Integration is the most important part of DevOps that is used to integrate various DevOps stages. Jenkins is the most famous Continuous Integration tool.
What is Jenkins?
Jenkins is an open source automation tool written in Java programming language that allows continuous integration.
Jenkins builds and tests our software projects which continuously making it easier for developers to integrate changes to the project, and making it easier for users to obtain a fresh build.
It also allows us to continuously deliver our software by integrating with a large number of testing and deployment technologies.
Jenkins offers a straightforward way to set up a continuous integration or continuous delivery environment for almost any combination of languages and source code repositories using pipelines, as well as automating other routine development tasks.
With the help of Jenkins, organizations can speed up the software development process through automation. Jenkins adds development life-cycle processes of all kinds, including build, document, test, package, stage, deploy static analysis and much more.
Jenkins achieves CI (Continuous Integration) with the help of plugins. Plugins is used to allow the integration of various DevOps stages. If you want to integrate a particular tool, you have to install the plugins for that tool. For example: Maven 2 Project, Git, HTML Publisher, Amazon EC2, etc.
For example: If any organization is developing a project, then Jenkins will continuously test your project builds and show you the errors in early stages of your development.
Possible steps executed by Jenkins are for example:
- Perform a software build using a build system like Gradle or Maven Apache
- Execute a shell script
- Archive a build result
- Running software tests
Work Flow:
History of Jenkins
Kohsuke Kawaguchi, who is a Java developer, working at SUN Microsystems, was tired of building the code and fixing errors repetitively. In 2004, he created an automation server called Hudson that automates build and test task.
In 2011, Oracle who owned Sun Microsystems had a dispute with Hudson open source community, so they forked Hudson and renamed it as Jenkins.
Both Hudson and Jenkins continued to operate independently. But in short span of time, Jenkins acquired a lot of contributors and projects while Hudson remained with only 32 projects. Then with time, Jenkins became more popular, and Hudson is not maintained anymore.
What is Continuous Integration?
Continuous Integration (CI) is a development practice in which the developers are needs to commit changes to the source code in a shared repository at regular intervals. Every commit made in the repository is then built. This allows the development teams to detect the problems early.
Continuous integration requires the developers to have regular builds. The general practice is that whenever a code commit occurs, a build should be triggered.
Continuous Integration with Jenkins
Let’s consider a scenario where the complete source code of the application was built and then deployed on test server for testing. It sounds like a perfect way to develop software, but this process has many problems.
- Developer teams have to wait till the complete software is developed for the test results.
- There is a high prospect that the test results might show multiple bugs. It was tough for developers to locate those bugs because they have to check the entire source code of the application.
- It slows the software delivery process.
- Continuous feedback pertaining to things like architectural or coding issues, build failures, test status and file release uploads was missing due to which the quality of software can go down.
- The whole process was manual which increases the threat of frequent failure.
It is obvious from the above stated problems that not only the software delivery process became slow but the quality of software also went down. This leads to customer dissatisfaction.
So to overcome such problem there was a need for a system to exist where developers can continuously trigger a build and test for every change made in the source code.
This is what Continuous Integration (CI) is all about. Jenkins is the most mature Continuous Integration tool available so let us see how Continuous Integration with Jenkins overcame the above shortcomings.
Generic Flow Diagram of Jenkins
Jenkins Tutorial
Jenkins Tutorial is designed for both beginners and professionals. Our Tutorial provides all the basic and advanced concepts of Jenkins, such as Jenkins installation, Jenkins Configuration, Jenkins Pipeline, etc.
Jenkins is an open source automation tool written in Java programming language that allows continuous integration.
Jenkins builds and tests our software projects, which continuously making it easier for developers to integrate changes to the project, and making it easier for users to obtain a fresh build.
What is Jenkins?
Jenkins is an open source automation tool written in Java programming language that allows continuous integration.
Jenkins builds and tests our software projects which continuously making it easier for developers to integrate changes to the project, and making it easier for users to obtain a fresh build.
It also allows us to continuously deliver our software by integrating with a large number of testing and deployment technologies.
Jenkins offers a straightforward way to set up a continuous integration or continuous delivery environment for almost any combination of languages and source code repositories using pipelines, as well as automating other routine development tasks.
With the help of Jenkins, organizations can speed up the software development process through automation. Jenkins adds development life-cycle processes of all kinds, including build, document, test, package, stage, deploy static analysis and much more.
Jenkins achieves CI (Continuous Integration) with the help of plugins. Plugins is used to allow the integration of various DevOps stages. If you want to integrate a particular tool, you have to install the plugins for that tool. For example: Maven 2 Project, Git, HTML Publisher, Amazon EC2, etc.
For example: If any organization is developing a project, then Jenkins will continuously test your project builds and show you the errors in early stages of your development.
Possible steps executed by Jenkins are for example:
- Perform a software build using a build system like Gradle or Maven Apache
- Execute a shell script
- Archive a build result
- Running software tests
Work Flow:
History of Jenkins
Kohsuke Kawaguchi, who is a Java developer, working at SUN Microsystems, was tired of building the code and fixing errors repetitively. In 2004, he created an automation server called Hudson that automates build and test task.
In 2011, Oracle who owned Sun Microsystems had a dispute with Hudson open source community, so they forked Hudson and renamed it as Jenkins.
Both Hudson and Jenkins continued to operate independently. But in short span of time, Jenkins acquired a lot of contributors and projects while Hudson remained with only 32 projects. Then with time, Jenkins became more popular, and Hudson is not maintained anymore.
What is Continuous Integration?
Continuous Integration (CI) is a development practice in which the developers are needs to commit changes to the source code in a shared repository at regular intervals. Every commit made in the repository is then built. This allows the development teams to detect the problems early.
Continuous integration requires the developers to have regular builds. The general practice is that whenever a code commit occurs, a build should be triggered.
Continuous Integration with Jenkins
Let’s consider a scenario where the complete source code of the application was built and then deployed on test server for testing. It sounds like a perfect way to develop software, but this process has many problems.
- Developer teams have to wait till the complete software is developed for the test results.
- There is a high prospect that the test results might show multiple bugs. It was tough for developers to locate those bugs because they have to check the entire source code of the application.
- It slows the software delivery process.
- Continuous feedback pertaining to things like architectural or coding issues, build failures, test status and file release uploads was missing due to which the quality of software can go down.
- The whole process was manual which increases the threat of frequent failure.
It is obvious from the above stated problems that not only the software delivery process became slow but the quality of software also went down. This leads to customer dissatisfaction.
So to overcome such problem there was a need for a system to exist where developers can continuously trigger a build and test for every change made in the source code.
This is what Continuous Integration (CI) is all about. Jenkins is the most mature Continuous Integration tool available so let us see how Continuous Integration with Jenkins overcame the above shortcomings.
Let’s see a generic flow diagram of Continuous Integration with Jenkins:
Let’s see how Jenkins works. The above diagram is representing the following functions:
- First of all, a developer commits the code to the source code repository. Meanwhile, the Jenkins checks the repository at regular intervals for changes.
- Soon after a commit occurs, the Jenkins server finds the changes that have occurred in the source code repository. Jenkins will draw those changes and will start preparing a new build.
- If the build fails, then the concerned team will be notified.
- If built is successful, then Jenkins server deploys the built in the test server.
- After testing, Jenkins server generates a feedback and then notifies the developers about the build and test results.
- It will continue to verify the source code repository for changes made in the source code and the whole process keeps on repeating.
Advantages and Disadvantages of using Jenkins
Advantages of Jenkins
- It is an open source tool.
- It is free of cost.
- It does not require additional installations or components. Means it is easy to install.
- Easily configurable.
- It supports 1000 or more plugins to ease your work. If a plugin does not exist, you can write the script for it and share with community.
- It is built in java and hence it is portable.
- It is platform independent. It is available for all platforms and different operating systems. Like OS X, Windows or Linux.
- Easy support, since it open source and widely used.
- Jenkins also supports cloud based architecture so that we can deploy Jenkins in cloud based platforms.
Disadvantages of Jenkins
- Its interface is out dated and not user friendly compared to current user interface trends.
- Not easy to maintain it because it runs on a server and requires some skills as server administrator to monitor its activity.
- CI regularly breaks due to some small setting changes. CI will be paused and therefore requires some developer’s team attention.
Jenkins Architecture
Jenkins follows Master-Slave architecture to manage distributed builds. In this architecture, slave and master communicate through TCP/IP protocol.
Jenkins architecture has two components:
- Jenkins Master/Server
- Jenkins Slave/Node/Build Server
Jenkins Master
The main server of Jenkins is the Jenkins Master. It is a web dashboard which is nothing but powered from a war file. By default it runs on 8080 port. With the help of Dashboard, we can configure the jobs/projects but the build takes place in Nodes/Slave. By default one node (slave) is configured and running in Jenkins server. We can add more nodes using IP address, user name and password using the ssh, jnlp or webstart methods.
The server’s job or master’s job is to handle:
- Scheduling build jobs.
- Dispatching builds to the nodes/slaves for the actual execution.
- Monitor the nodes/slaves (possibly taking them online and offline as required).
- Recording and presenting the build results.
- A Master/Server instance of Jenkins can also execute build jobs directly.
Jenkins Slave
Jenkins slave is used to execute the build jobs dispatched by the master. We can configure a project to always run on a particular slave machine, or particular type of slave machine, or simple let the Jenkins to pick the next available slave/node.
As we know Jenkins is developed using Java is platform independent thus Jenkins Master/Servers and Slave/nodes can be configured in any servers including Linux, Windows, and Mac.
What Is Jenkins Pipeline?
Pipeline in Jenkins is a group of jobs (or events) that are interlinked in a particular sequence. Jenkins Pipeline is a set or suite of plugins that provides support for implementation and integration of Continuous Delivery pipelines into Jenkins.
The Pipeline also provides a set of tools that are useful for modeling simple as well as complex delivery pipelines ‘as code’ through ‘Pipeline Domain-Specific Language (DSL)’ syntax.
Every job in the Jenkins pipeline has some dependency on one or more events. Continuous delivery pipeline in Jenkins consists of four states — Build, Deploy, Test, and Release. Each of these states consist of events that execute in a sequence.
What Is Jenkinsfile?
Now that you understand what is Jenkins pipeline, we can dive deeper into the concept. The entire definition of a Jenkins Pipeline is written into a text file called Jenkinsfile. It contains the steps required for running a Jenkins Pipeline. ‘Pipeline as code’ can be implemented using Jenkinsfile and Domain Specific Language (DSL) is used for defining the same.
Jenkinsfile can also be committed to the source control repository of the project. With Jenkinsfile, the CD Pipeline is also treated as a part of the application that is versioned, committed, and reviewed like any other piece of code.
Some of the major benefits of Jenkinsfile are:
- Single Jenkinsfile can be used for creating a Pipeline build process for all the branches and executing pull requests.
- The implementation in a Pipeline can be reviewed like normal source code.
- Audit trail of the Pipeline.
- Singular source for the Pipeline can be viewed as well as edited by multiple members associated with the project.
Though Pipeline can be defined either in web UI or with a Jenkinsfile, it is recommended to define the what is Jenkins Pipeline in a Jenkinsfile and maintain the same in a source control repository.
Important Concepts Of Jenkins Pipeline
Moving on with what is Jenkins pipeline, here are some basic concepts that need to be well-understood if you are planning to use the Jenkins Pipeline:
a. Pipeline
The Pipeline consists of a set of instructions written as code. It defines the entire build process, which ideally consists of different stages for building, testing, and delivering the application.
b. Node
Node is a machine that is a part of the Jenkins environment. The Jenkins Pipeline executes on a node block that is usually a part of the Scripted Pipeline syntax.
c. Stage
Stage in a Jenkins Pipeline consists of a unique subset of tasks such as Build, Test, Deploy, etc. The Stage block is used by many plugins for providing the visualization of Jenkins status (and progress).
d. Step
Step is a single task that tells Jenkins what exactly needs to be done. For example, setting an environment variable can be done in a step, executing a build command can also be a step. On the whole, a Jenkins Pipeline constitutes a series of steps.
Advantages Of Using Jenkins Pipeline
Jenkins Pipeline is instrumental in adding a rich set of automation tools onto Jenkins. Hence, Jenkins can be used for simple continuous integration as well as for comprehensive CD pipelines.
These are some of the primary reasons for using the Jenkins Pipeline:
- As the Jenkins Pipeline is implemented as code, it can be checked in the source control repository. Teams can view, edit, as well as iterate upon the delivery pipeline.
- Jenkins Pipelines are robust. A Pipeline is automatically resumed in case the server witnesses an unplanned restart.
- Pipeline process can pause and wait for input for any input from the user.
- Pipelines are versatile as they can be used for realizing complex CD requirements, including performing work in parallel.
- Jenkins Pipelines are extensible by plugin developers and users with Pipeline Shared Libraries.
Jenkins Case Study: D4Science
What do several EU-funded projects, the Food and Agriculture Organization of the United Nations, Mercator Ocean, the European Bioinformatics Institute, CINECA, Sorbonne University, and many other institutions across the world have in common?
Some of their research and virtual research environments (VREs) are fueled by D4Science, an organization led by the Institute of Information Science and Technologies, an institute of the National Research Council, (ISTI-CNR), located in Pisa, Italy.
Partially funded by the European Commission, D4Science has created, maintains and constantly enhances an advanced test-bed allowing scientific communities to share knowledge and collaborate in a secure, coordinated, dynamic and cost-effective way.
Over the years, gCube, D4Science’s software framework, has evolved into a platform where interdisciplinary tools and services can be shared to better serve the collaborative needs of thousands of users worldwide. In fact, nearly ten new EU-funded projects which utilize gCube are currently ongoing or have been approved and are about to start this year. With so many organizations relying on gCube, D4Science decided to migrate to an entirely new platform for controlling, integrating, versioning, and releasing the software.
From one to hundreds of repositories.
Before 2019, D4Science had been using SVN as a version control system with a single repository hosting all the gCube source code. Over time, the issues they were encountering managing their code and releases overcame the benefits SVN was able to provide.
“gCube is a very extended framework. Composed of hundreds of software components, it has several disparate teams working on various applications and services at the same time,” said Manuele Simi, Software Engineer. “Before introducing our new code management platform, each release required an ever-increasing amount of effort, time and resources to coordinate and build. On top of that, our daily work was continually disrupted by recurring problems in the Continuous Integration pipeline that were not under our control.”
A first step D4Science took was to migrate from SVN to Git to better manage and version their code. According to Simi, a crucial decision in this transition was how best to map their “huge” SVN repository to Git. After careful consideration, they opted to split the source tree into smaller units of work and create a Git repository for each. With hundreds of repositories to release and a build system that could no longer meet the needs of this demanding scenario, it was clear that they needed to switch to a new Continuous Integration and Continuous Delivery pipeline.
Empowering e-Science and virtual research communities with software released via Jenkins.
The Jenkins Declarative Pipelines implemented by D4Science provide full control of orchestration activities for their complex releases. Additionally, all build jobs are coordinated in a configurable and dynamic fashion. With Jenkins, D4Science has now the capabilities to:
- build and release software components from 200+ Git repositories within the same day by launching a single pipeline job;
- reuse the same build jobs across different build stages;
- require zero effort of the development team at release time;
- get reliable, fast Continuous Integration feedback
With minimal effort, D4Science was able to replace the pre-existing build platform that they had relied on for several years. With Jenkins, they gained an easy to maintain, scalable, state-of-the-art CI/CD server that can be upgraded at minimal costs. Most importantly, according to Simi, they solved all the issues that made their previous Continuous Delivery pipeline so expensive.
“With the solutions implemented with Jenkins, we completely reshaped our delivery phase by saving effort, time, and resources,” said Simi. “We now have fast, incremental releases managed by a single person, where previously we had a dedicated team just for the release activities.”
Dozens of research institutions and organizations worldwide are saving time and resources by employing VREs created with gCube. With the capacity to promptly release new software addressing users’ requirements, D4Science continues to receive a stream of projects funded by the European Commission and to attract the interest of diverse communities of practices engaged in research projects, with the promise of more to come.
D4Science, supporting scientific communities and promoting open science practices by serving 11,000+ registered users in 45 countries, introduced a new delivery pipeline replacing their pre-existing build platform.
CHALLENGE:
D4Science needed a cost-effective way to build and release their software framework (gCube) able to support multi-project releases at scale.
SOLUTION:
A Continuous Integration/Continuous Delivery (CI/CD) pipeline, scalable, easy to maintain and upgrade at a minimal cost, that represents an innovative approach to software delivering.
RESULTS
- Automation of all release activities
- Fast, incremental releases to quickly address user requirements
- Build and release software components from 200+ Git repositories within the same day
Jenkins is an excellent CI/CD tool that has excellent functionalities and the best part is that it is free to use. Based on what we discussed in this what is Jenkins blog, we can conclude that Jenkins has features that boost release agility by providing CI services.
Jenkins Pipeline is another feature that enhances the offerings of Jenkins, as it lets you implement a Pipeline as Code using Domain Specific Language (DSL). Though there are other CI/CD tools in the market, Jenkins thrives on its community and powerful set of features (particularly Jenkins Pipeline) that offers faster feedback through CI tools.
Plugins, extensibility, and online community are the core USPs of Jenkins. To improve the usability and functionality of tools like Jenkins, you can use cloud-based cross browser testing platforms. One such platform is LambdaTest that also offers a LambdaTest Jenkins Plugin to accelerate your automated browser testing activities. With LambdaTest, you can perform tests on 2000+ combinations of browsers, operating systems, and devices.
