Jenkins is one of the most popular tools for build automation and it’s pipeline plugin allows us to define the job-configuration as part of our source code. Migrating from manual configuration to the groovy syntax of the pipeline plugin can be very challenging, but it’s definitely worth it. Tracking of changes, dynamic behavior, and complex configurations are just a few areas where the pipeline syntax of Jenkins thrives.
I’ve recently helped a team migrate from manually configured jobs to the pipeline syntax. Looking back on the project, I realize that I had to answer the same questions over and over. Because of this, I want to share a few how-to’s, thoughts and tricks about the “new way” to configure Jenkins.
This “Jenkins 101” is not meant as a tutorial to get started, since there are already a lot of good resources that guide you through the first steps. I’d rather explain some of the solutions to the problems that I struggled with or the things that a lot of people miss out on. If you are just starting out, I recommend the getting started in the Jenkins documentation.
Today’s post is about the glue that holds our CI pipelines together: downstream projects a.k.a. “staring another job”. Creating a chain of multiple executions is an essential part of most CI pipelines. I’ve noticed that people who work with the pipeline syntax are often only aware of one way to connect two jobs. However, there are two ways to link Jobs: triggers and the build command. Both have their own strengths and weaknesses and if you only use one of these, you’re definitely missing out.
Triggered execution
Triggers follow the same logic as the “build after other projects are built” option in the configuration of Jenkins projects.
Most teams work with this model because it’s the default option of Jenkins. When we want to use triggers in the pipeline syntax, we simple define the upstream project that we want to monitor and specify our threshold.
triggers {
upstream(upstreamProjects: "metamorphant/deploy/", threshold: hudson.model.Result.SUCCESS)
}Note that we reference the upstream project by it’s path, not by it’s name. If you’re unsure about the exact path, you can look it up in each project’s overview page:
We can do some customization of the trigger definition, but our options are very limited. For example: you might want to trigger builds on branches with the same name. With the following change to our trigger, Jenkins will do exactly that:
triggers {
upstream(upstreamProjects: "metamorphant/deploy/" + env.BRANCH_NAME.replaceAll("/", "%2F"),
threshold: hudson.model.Result.SUCCESS)
}Since the triggers are asynchronous we do not have to worry about blocking build nodes, timeouts and similar things. However, while this approach has a lot of advantages there’s one thing I often end up missing: a way to share information. Since the upstream project does not take an active part in the execution, no parameters are passed downstream.
There are ways to mitigate this, but it will never get as easy as passing an object downstream. One way to get information about the upstream job is to extract it from the environment metadata of our current job instance:
1
2
3
4
5
6
7
8
9
stage('find-upstream') {
steps {
script {
currentBuild.upstreamBuilds?.each{ b ->
echo "Triggered by upstream project: ${b.getFullDisplayName()}"
}
}
}
}
We’ll wrap up our brief overview of triggered jobs with a short pros and cons list, before moving on to our second method.
pros
- loose coupling
- easy to configure
cons
- data is mostly limited to current build
- loops and repetitions are a headache
- management can get confusing very fast
Actively executing a downstream job
Up next are active calls for the execution of a job. Unlike the triggers these calls are synchronous which makes them ideal for tasks that are linked together very tightly e.g. multiple parts of a deployment.
To do this, we simply tell Jenkins to build a downstream project:
1
2
3
4
5
stage ('Starting downstream job ') {
steps {
build job: 'metamorphant/deploy/master'
}
}
Again, we only need the full path of our downstream project. There are a lot of optional parameters available that let us customize the behavior of this command. Let’s look at an example that’s a bit more advanced:
1
2
3
4
5
stage ('Starting downstream job ') {
steps {
build job: 'metamorphant/deploy/feature%2FPadd-jwt-tokens', parameters: [string(name: 'targetEnvironment', value: 'stage')], propagate: false
}
}
While we’re executing the same job, I’ve specified that I want to build the branch feature/add-jtw-tokens instead of the master . Note that we’re using an URL-encoded string for the branch name. Afterwards, we specify the parameters. They’re passed as an Array of key-value pairs with their respective datatype (e.g. String, Boolean,…). The last argument propagate: false tells Jenkins that I want to treat the current step as a success even if the downstream build fails. This means that my job will continue on errors in the downstream build. There are additional parameters for the build command.
Again, this approach has it’s own ups and downs. I mainly use “build job” for the management of repetitive tasks e.g. when I need to deploy multiple parts of my application. It’s really easy to create a short loop that re-uses the same downstream-job to deploy my application to multiple servers too. Lastly, it makes the evaluation of results very easy since I can just save the return-values of the “build job” function to a variable. But remember: the “root job” will run for the whole time, it’s prone to timeouts.
pros
- easy management of repetitive tasks
- parameterization of downstream projects
cons
- upstream job continues to run and blocks a build node
- can get complex very fast
- it’s confusing to decide when to create a new job and when to use a new step within the current job
All in all it’s safe to say that you shouldn’t use either one exclusively. I use triggered jobs as my default and the “build job” whenever I have to do repetitive tasks. Which approach do you favor and why? Let me know in the comments or send us a tweet @metamorphant.
Post header background image by Ryan McGuire from Pixabay.