Setting Up Jenkins to Build CI/CD Pipelines for MLOps

In a previous series of articles, we explained how to code the scripts to be executed in our group of Docker containers as part of a CI/CD MLOps pipeline. In this series, we’ll set up a Google Kubernetes Engine (GKE) cluster to deploy these containers.

This article series assumes that you are familiar with Deep Learning, DevOps, Jenkins and Kubernetes basics.

In the previous article, we configured GKE and prepared all the components to start our pipeline deployment. In this article, we’ll set up process orchestration with Jenkins.

Jenkins in a Nutshell

Jenkins is an open-source server that enables automation of the building, testing, and deployment stages of the software development cycle. It allows you to trigger actions based on automatically detected code commits in a given repository.

To install Jenkins in your environment, follow the instructions provided in the official documentation. We’ll run Jenkins on a local Ubuntu machine, and will run all of our build, test, and deploy pipelines from there.

Before diving in, a couple of terms need explaining:

• A Jenkins workflow is a set of runnable tasks.
• A Jenkins pipeline is a runnable set of jobs or workflows that follow a certain order of execution. We’ll use scripted workflows coded in the Groovy language.

Installing Jenkins Plugins

The diagram below shows where we are in our project process .

Time to open Jenkins and start setting things up. Let’s install Docker Pipeline, Docker Plugin, docker-build-step, Google Container Registry Auth Plugin, Google Kubernetes Engine Plugin, Google OAuth Credentials plugin, and Git.

To reach the Jenkins interface, enter http://localhost:8080 (or http://127.0.0.1:8000) in your browser, log into the Jenkins dashboard, and select the Manage Jenkins option on the left-hand side.

Select Manage Plugins.

In the Available tab, and search for the required plugin.

Once the plugin is found, select its check box and then click Install without restart.

The installation starts. The installer will notify you when the process ends.

Repeat the above steps for all the required plugins.

Configuring Email Notifications

Let’s configure the email notification plugin to ensure that you receive emails regarding your pipeline operation.

On the Jenkins dashboard, select Manage Jenkins > Configure System > Extended E-mail Notification. Select the Advanced option, fill out the required fields, then click Apply.

Setting Up the GCP Credentials in Jenkins

Now we need to allow Jenkins to use your GCP resources.

First of all, you need a GCP service account. To get it, log into Google Cloud Platform, select your project, and go to IAM & Services > Service Accounts.

Click Create Service Account.

Give a name to your new service account and click Create.

Add the following access roles, then click Done.

Select your service account on the list and click the Keys tab.

Select Add key > Create new key.

Select the JSON format and click Create to download your keys.

On the Jenkins dashboard, select Manage Jenkins > Manage Credentials.

At the bottom, select Jenkins > Global credentials > Add credentials.

From the Kind drop-down list, select Google Service Account from private key, give the project a name (we recommend "AutomaticTrainingCICD"), upload the JSON file you’ve obtained, and click OK.

Next Steps

Now we’re all set to build Jenkins workflows. We’ll build them in the next article. Stay tuned!