Jenkins

Jenkins is the open-source automation server that made continuous integration mainstream: a controller process that schedules and tracks every build, and any number of agents that actually run them, wired together by a Jenkinsfile checked straight into the repository it builds. It predates every SaaS CI product on the market — born as Hudson in 2004, forked into Jenkins in 2011 after a licensing dispute with Oracle, and governed today as a Continuous Delivery Foundation project — and it is still what most organizations reach for when a build needs to run on hardware, a network, or a compliance boundary that a vendor's hosted runner simply cannot touch. This page covers how the controller/agent split actually works, what a real Jenkinsfile looks like in both pipeline syntaxes, why the plugin ecosystem is at once Jenkins's greatest strength and its most-cited maintenance complaint, and the narrow, specific set of situations where choosing Jenkins over a modern SaaS CI product is the right call rather than nostalgia.

What Jenkins is and the problem it solves

☺ Like you're 10: It's a free program that watches your code, builds and tests it every time someone pushes, and runs on machines you own instead of a website you pay per minute.

Jenkins is a free, open-source automation server — a Java process that runs on the JVM, exposes a web UI and a REST API, and executes arbitrary jobs on a schedule or a trigger. "Automation server" is deliberately broader than "CI tool": Jenkins started life as a general-purpose job runner and CI/CD is simply its most common use, not its only one — plugins let the same controller run nightly batch jobs, drive infrastructure automation, or fire off a ChatOps command, all through the same queue and the same audit trail. It was created by Kohsuke Kawaguchi at Sun Microsystems under the name Hudson; after Oracle acquired Sun in 2010 and a dispute broke out over control of the Hudson trademark and its governance, most of the community forked the project in 2011 under a new name, Jenkins, which is now stewarded under the Continuous Delivery Foundation (part of the Linux Foundation).

The one word that defines Jenkins against nearly every competitor covered elsewhere in this course's toolchain is self-hosted. You install the JVM process yourself, on hardware or a VM or a container you control, and everything about it — the data, the network path in and out, the billing model, the uptime — is your organization's responsibility rather than a vendor's. That is a genuine trade, not an automatic downside: see Jenkins vs. SaaS CI below for exactly when that trade is worth making.

Where Jenkins fits in the delivery pipeline

☺ Like you're 10: Jenkins sits right after "someone pushed code" and right before "the finished, tested build goes somewhere" — it doesn't usually own what happens after that.

In the DevOps lifecycle, Jenkins occupies the Build, Test, and Package stages, and very often Deploy too, of the loop covered in CI/CD pipelines. A webhook from GitHub, GitLab, or Bitbucket fires on a push or a pull request; Jenkins checks out the change, runs whatever the Jenkinsfile defines — lint, unit tests, a container build — and pushes the resulting artifact to a registry such as JFrog Artifactory or Sonatype Nexus, or a container image to a registry ahead of a Docker or Kubernetes deployment. From there, one of two things happens: Jenkins itself runs the deploy step (a kubectl apply or a Helm upgrade, gated by an approval), or it hands off to a dedicated continuous-delivery reconciler like Argo CD that watches a Git repo and applies changes independently. Jenkins is not the only orchestrator that can sit in this slot — GitHub Actions, GitLab CI/CD, and CircleCI all compete for exactly this job, and the DevOps toolchain page has the full category breakdown.

Architecture: the controller/agent model

☺ Like you're 10: One brain that plans the work, and a crew of hands that actually do it — the brain almost never lifts a finger itself.

Jenkins renamed its two core roles in 2020 from the old "master/slave" terminology to controller and agent, and the new names describe the split precisely. The controller is the one JVM process that everything else revolves around: it serves the web UI, the REST API, and the CLI; it holds the build queue and scheduler; it hosts every installed plugin; it stores every job definition, every credential, and every build's history and logs under a single directory tree called JENKINS_HOME (by default /var/lib/jenkins on a package install, or /var/jenkins_home in the official container image). What the controller deliberately does not do, in a correctly run production setup, is execute build steps itself.

Agents connect to the controller one of three ways, and the choice mostly comes down to network topology and lifecycle. Static (SSH) agents are long-running machines the controller reaches out to and connects to over SSH — simple, but they're a "pet": the same box, build after build, that can accumulate leftover state, stale dependency caches, or a hand-installed tool nobody documented. Inbound agents connect the other direction — the agent dials the controller over a TCP port (historically 50000, using the JNLP4 protocol) — which is the pattern that works when the agent lives behind a firewall or NAT the controller can't reach first. Ephemeral cloud agents, provisioned by plugins like the Kubernetes plugin, EC2 Fleet, or Azure VM Agents, are created fresh for a single build and destroyed the moment it finishes — the fix for agent drift, and the pattern most new Jenkins deployments default to today. Every agent advertises one or more labels, and every job declares which label it needs (agent { label 'linux && docker' }); the scheduler's entire job is matching the two. Each agent offers some number of executors — parallel build slots — and the controller itself has executors too, by default, which is the single most important setting to change before going to production.

Jenkinsfile as code: declarative vs. scripted

☺ Like you're 10: You write the recipe once, in a file saved with the food itself, and there are two ways to write that recipe: fill in a form with blanks, or write free-form instructions in a real programming language.

A Jenkinsfile is pipeline-as-code for Jenkins specifically: a text file, checked into the repository root (or wherever a multibranch job is told to look), written in a Groovy-based DSL, and version-controlled exactly like the application it builds. Jenkins supports two distinct syntaxes for it, and confusing them is the single most common source of "why doesn't this Jenkinsfile example I copied off the internet work" — they share steps but not structure.

Declarative Pipeline is the newer, structured syntax, wrapped in a top-level pipeline { } block with a fixed set of directives — agent, environment, options, parameters, triggers, stages, post, when — that Jenkins validates against a schema before running anything. That schema is what makes declarative pipelines linter-friendly, easier for a reviewer to read, and safer for a newcomer to extend without breaking the whole file.

// Jenkinsfile — declarative syntax
pipeline {
  agent { label 'linux && docker' }

  options {
    timestamps()
    timeout(time: 30, unit: 'MINUTES')
  }

  parameters {
    booleanParam(name: 'RUN_E2E', defaultValue: false, description: 'Run the e2e suite')
  }

  environment {
    REGISTRY  = 'ghcr.io/acme'
    IMAGE_TAG = "${env.GIT_COMMIT.take(7)}"
  }

  stages {
    stage('Checkout') { steps { checkout scm } }

    stage('Build & unit test') {
      steps {
        sh 'npm ci'
        sh 'npm test -- --ci'
      }
      post { always { junit 'reports/junit/*.xml' } }
    }

    stage('E2E') {
      when { expression { params.RUN_E2E } }
      steps { sh 'npm run test:e2e' }
    }

    stage('Build & push image') {
      steps {
        sh "docker build -t ${REGISTRY}/checkout:${IMAGE_TAG} ."
        withCredentials([usernamePassword(credentialsId: 'ghcr-bot', usernameVariable: 'U', passwordVariable: 'P')]) {
          sh 'echo $P | docker login ghcr.io -u $U --password-stdin'
          sh "docker push ${REGISTRY}/checkout:${IMAGE_TAG}"
        }
      }
    }

    stage('Deploy to prod') {
      when { branch 'main' }
      input { message 'Deploy to production?'; ok 'Ship it' }
      steps { sh "kubectl set image deployment/checkout checkout=${REGISTRY}/checkout:${IMAGE_TAG}" }
    }
  }

  post {
    failure { slackSend channel: '#ci-alerts', message: "Build failed: ${env.BUILD_URL}" }
  }
}

Scripted Pipeline came first and is just Groovy: a node('label') { } block containing arbitrary, general-purpose code — real if/for/try/catch, closures, and helper functions, none of it constrained by a schema. It runs through Jenkins's Groovy CPS (continuation-passing-style) interpreter so a pipeline can be paused and resumed mid-build without losing state — which is also why certain Groovy idioms (some closures, some non-serializable objects) trip the CPS transform and need a method marked @NonCPS to run outside it. Declarative pipelines can still drop into a script { } block to borrow this full power for the one step that genuinely needs it, which is the escape hatch that keeps declarative from being a dead end.

// Jenkinsfile — scripted syntax, doing roughly the same job
node('linux && docker') {
  try {
    stage('Checkout') { checkout scm }
    stage('Build & unit test') {
      sh 'npm ci'
      sh 'npm test -- --ci'
    }
    if (env.BRANCH_NAME == 'main') {
      stage('Deploy to prod') {
        input message: 'Deploy to production?'
        sh 'kubectl set image deployment/checkout checkout=ghcr.io/acme/checkout:latest'
      }
    }
  } catch (err) {
    currentBuild.result = 'FAILURE'
    throw err
  } finally {
    junit 'reports/junit/*.xml'
  }
}

Once a pipeline pattern needs to be shared across many repositories rather than copy-pasted, Jenkins's answer is a shared library — a separate Groovy repository referenced with @Library('name@version'). That mechanism, and how it stacks up against GitHub Actions reusable workflows, GitLab CI/CD components, and CircleCI orbs, is covered in full in Scaling CI/CD Across Teams.

Multibranch pipelines, credentials, and Configuration as Code

☺ Like you're 10: Jenkins can notice every branch and pull request on its own, keep passwords in a locked box instead of pasted in the recipe, and remember its own settings in a file instead of a person's memory.

A plain Pipeline job builds one branch. A Multibranch Pipeline job instead points at a whole repository: Jenkins scans it, finds every branch (and, with the right SCM plugin, every open pull request) that contains a Jenkinsfile, and automatically creates and tears down a sub-job for each one as branches come and go — no one has to hand-register a new job every time a feature branch is opened. An Organization Folder takes that one level higher, scanning an entire GitHub org or GitLab group for repositories that have a Jenkinsfile at all, and provisioning a multibranch job for each match automatically.

Secrets never belong in a Jenkinsfile in plaintext — they belong in the controller's credentials store, referenced by ID and pulled in with withCredentials (as in the declarative example above) so the actual value is masked in the build log and injected only for the steps that need it. Jenkins ships several credential types out of the box — username/password, secret text, SSH private key, X.509 certificate — and for organizations that already run a dedicated secrets manager, the HashiCorp Vault plugin lets Jenkins fetch a secret from HashiCorp Vault at build time instead of storing a long-lived copy on the controller at all; see Secrets & Credential Management for the broader pattern this fits into.

The controller's own configuration — security realm, authorization strategy, credentials, cloud/agent definitions, tool installations — used to live only in whatever an admin clicked through in the UI, which made a Jenkins controller nearly impossible to reproduce or disaster-recover cleanly. Configuration as Code (JCasC) fixes that: a YAML file, checked into version control, applied automatically at controller startup, that a Jenkins admin can diff and review exactly like a Terraform plan. The plugin even ships a built-in schema browser (Manage Jenkins → Configuration as Code → Documentation) so you're not guessing at the YAML shape from memory.

# jenkins.yaml — Configuration as Code, loaded via the JENKINS_HOME/casc_configs mount
jenkins:
  systemMessage: "Managed by JCasC — edit this file, not the UI"
  numExecutors: 0                 # the controller-never-builds rule, enforced by config
  authorizationStrategy:
    roleBased:
      roles:
        global:
          - name: "admin"
            permissions: ["Overall/Administer"]
            assignments: ["platform-team"]

credentials:
  system:
    domainCredentials:
      - credentials:
          - usernamePassword:
              scope: GLOBAL
              id: "ghcr-bot"
              username: "acme-bot"
              password: "${GHCR_BOT_PASSWORD}"   # pulled from an environment variable, never committed plain

# running a controller from the official image, JCasC file mounted alongside it
$ docker run -d --name jenkins-controller \
    -p 8080:8080 -p 50000:50000 \
    -v jenkins_home:/var/jenkins_home \
    -v $(pwd)/jenkins.yaml:/var/jenkins_home/casc_configs/jenkins.yaml \
    -e CASC_JENKINS_CONFIG=/var/jenkins_home/casc_configs/jenkins.yaml \
    jenkins/jenkins:lts-jdk17   # check hub.docker.com/r/jenkins/jenkins for the current LTS tag

# the CLI, once it's up — every command needs an API token, not a password
$ java -jar jenkins-cli.jar -s http://jenkins.internal:8080/ -auth acme-bot:$JENKINS_API_TOKEN list-jobs
$ java -jar jenkins-cli.jar -s http://jenkins.internal:8080/ -auth acme-bot:$JENKINS_API_TOKEN build checkout-pipeline -f
$ java -jar jenkins-cli.jar -s http://jenkins.internal:8080/ -auth acme-bot:$JENKINS_API_TOKEN console checkout-pipeline

# lint a declarative Jenkinsfile without running it
$ curl -X POST -F "jenkinsfile=<Jenkinsfile" http://jenkins.internal:8080/pipeline-model-converter/validate

The plugin ecosystem: superpower and long-term tax

☺ Like you're 10: There's practically an add-on for anything you'd ever want to connect Jenkins to — which is amazing, until you're the one who has to keep all those add-ons working together.

Jenkins's plugin catalog at plugins.jenkins.io lists well over 1,800 plugins (the exact count moves constantly — check the live number there rather than trusting any figure printed here), covering practically every category a real pipeline touches: source control (Git, GitHub, GitLab, Bitbucket), notifications (Slack, Email Extension, Microsoft Teams), build tools (Maven, Gradle, NodeJS), cloud provisioning (Kubernetes, EC2 Fleet, Azure VM Agents), code quality (SonarQube Scanner, Warnings Next Generation), artifact management (Artifactory, Nexus), and generic escape hatches like HTTP Request and Pipeline Utility Steps for the integration nobody's written a dedicated plugin for yet. This is Jenkins's real superpower: whatever the tool, there is very likely already a maintained way to talk to it, without writing custom glue code.

It is also, honestly, Jenkins's most-cited complaint. Every plugin is a separate piece of software with its own release cadence, its own compatibility matrix against Jenkins core, and — for many community plugins — a single volunteer maintainer. Upgrading Jenkins core can break a plugin that hasn't been updated for the new API; upgrading a plugin can break another plugin that depended on its old behavior; and because plugins load into the same JVM as the controller, a bad interaction can take down the whole instance, not just one job. The Plugin Installation Manager Tool and a version-pinned plugins.txt baked into your controller's container image are how mature teams keep this reproducible instead of "whatever happened to be installed the day someone clicked update" — treat plugin versions with the same discipline as application dependencies, because that's exactly what they are.

Gotchas and failure modes

☺ Like you're 10: The filing cabinet is the whole memory — lose it and Jenkins forgets everything — and a few habits that felt fine on day one turn into real problems by month six.

JENKINS_HOME is a single point of failure. Every job definition, every build's history and console log, every stored credential, and the encryption keys that protect those credentials (secrets/master.key and the per-secret files under secrets/) live in that one directory tree. Back it up as a unit — a credentials.xml without its matching master key is undecryptable, not just inconvenient — and treat controller migration as a real, tested runbook, not a "copy the folder and hope" afternoon.

Static agents drift. An SSH agent that's been alive for eight months has whatever tool versions, cached dependencies, and half-remembered manual fixes accumulated on it over that time, and "works on that one agent" is a debugging trap waiting to happen. The fix mirrors the one this course covers generally in Immutable Infrastructure & Golden Images: prefer ephemeral agents (Kubernetes pods, freshly launched cloud instances) that are built from a pinned image and destroyed after one build, so there is nothing left to drift.

Weekly releases vs. LTS. Jenkins core ships a new release every week; roughly every twelve weeks, one of those weekly builds is designated the next Long-Term Support line and receives backported fixes for about the next three months. Run LTS in production. Weekly releases are for testing plugin compatibility against what's coming, not for anything you'd page someone about at 2 a.m. Jenkins also periodically raises the minimum Java version its core requires on new releases — check the current system requirements before planning any controller upgrade, since a Java bump can force an OS or JVM change you didn't budget time for.

Jenkins vs. SaaS CI: when self-hosted control wins

☺ Like you're 10: Most teams should just rent a CI service — but if your build needs to happen on your own turf, owning the whole workshop is worth the extra chores.

For most teams, most of the time, a SaaS CI product is the right default: no controller to patch, no JENKINS_HOME to back up, elastic runners that scale without a Kubernetes plugin to configure, and a UI that doesn't need a plugin to look modern. Jenkins earns its place specifically when the thing you need to control is something a SaaS vendor's hosted runner structurally cannot give you.

The practical rule: choose Jenkins on purpose, for a specific control requirement you can name out loud — not because it's the tool everyone already knows, and not out of inertia from a system nobody's revisited in five years. If the honest answer to "why not just use a hosted runner" is "we've always done it this way," that's usually the moment to re-run the comparison above rather than the moment to justify keeping it. Teams that want Jenkins's control without pure do-it-yourself operations can also look at CloudBees CI, the commercial distribution built on open-source Jenkins with vendor support and managed-controller options — worth knowing the name exists, though check CloudBees's current offering before assuming any specific feature.

None of this is a one-time decision. A team that outgrows a SaaS product's build-minute pricing, or wins a contract requiring on-prem-only build data, can migrate onto Jenkins later; a team drowning in plugin upgrades and controller patching can migrate off it just as validly. The Jenkinsfile concepts on this page — stages, steps, credentials, approval gates — carry over to whichever CI product you land on next, which is exactly why CI/CD pipelines and scaling CI/CD across teams teach the ideas before any tool-specific syntax. Practice the full build stage hands-on in Capstone Part 1 — Pipeline Foundation, or go fix one that's already broken in Drill — Fix a Broken Pipeline.