Chef

Chef is a configuration management tool that describes a machine's desired state as real Ruby code: a recipe is a file full of resources — "this package installed," "this file templated," "this service running" — and cookbooks bundle recipes, templates, and helper logic into a versioned, reusable unit. A persistent agent called chef-client runs on every managed node, pulls its assigned cookbooks and run-list from a central Chef Infra Server, and converges the machine to match, on its own schedule, without anyone pushing anything to it. This page covers that client-server model end to end, what a real cookbook looks like, the resource/provider idempotency model underneath every recipe, the day-to-day CLI, the gotchas that catch newcomers hardest, and a candid, hedge-the-numbers look at where Chef actually stands against Ansible and Puppet as of this writing.

What Chef is and the problem it solves

☺ Like you're 10: Instead of SSHing in and typing commands by hand every time a server needs something installed or fixed, you write the fix once as real code, and a little program on every machine keeps applying it forever.

Before configuration management existed as a category, keeping a fleet of servers consistent meant either hand-running the same shell commands on each one — which drifts the moment anyone forgets a step or types a typo — or writing ad hoc shell scripts that had no concept of "already done," so re-running them could reinstall packages, re-append the same line to a config file twice, or clobber a change someone made by hand. Chef, first released by the company Opscode in 2009 (renamed Chef Software in 2013), answered this with a genuinely programmatic approach: recipes written in a real language — Ruby, extended with a domain-specific set of keywords — that describe what should be true of a machine rather than the literal shell commands to get there. A resource like package 'nginx' doesn't mean "run apt install nginx"; it means "nginx should end up installed," and Chef's own code decides whether that requires doing anything at all.

Chef Software open-sourced its entire product line under the Apache 2.0 license in 2019, and was acquired by Progress Software in 2020; today the product most people mean by "Chef" is formally called Chef Infra, one member of a small family that also includes Chef InSpec (compliance-as-code testing, usable standalone or alongside Infra), Chef Habitat (application packaging and runtime automation), and Chef Automate (a commercial visibility and compliance dashboard that sits on top of Infra Server data). This page is about Chef Infra — cookbooks, recipes, resources, and the server that ties them together — since that is what people mean when they say "we run Chef" on a fleet of nodes.

Where Chef fits in the delivery pipeline

☺ Like you're 10: Something else builds the house and pours the driveway — Chef is the part that keeps unpacking the same boxes and tidying the same rooms, forever, after everyone's moved in.

Chef sits downstream of provisioning. A tool like Terraform or a cloud console creates the virtual machine in the first place — the compute, the network interface, the disk — and Chef takes over once that machine exists, installing packages, laying down config files, managing users, and keeping all of it correct for as long as the machine lives. That's the same category boundary infrastructure as code and configuration management draw generally: provisioning answers "does this resource exist," configuration management answers "is what's running on it correct," and Chef is squarely a configuration management tool, in the same category as Ansible and Puppet rather than a competitor to Terraform.

Chef also shows up one step earlier than that, baking a cookbook's convergence directly into a machine image with Packer so a freshly booted instance already matches the golden image instead of converging from scratch on first boot — the immutable-infrastructure pattern covered in Immutable Infrastructure & Golden Images. That page also names the deeper trend working against Chef's traditional use case: the more a fleet leans on container images and immutable, replace-don't-patch instances, the less work is left for a long-running convergence agent to do on a live machine at all.

Architecture: Workstation, Chef Infra Server, and nodes

☺ Like you're 10: You write the chart on your own laptop, upload it to one shared fridge, and every machine checks that fridge on its own — nobody ever pushes the chart onto a machine directly.

Chef is a strict pull model, and that single fact explains most of its architecture. There are three roles, and confusing them is the first stumbling block for anyone arriving from a push-based tool.

The Workstation is where a human authors cookbooks — a laptop or CI runner with the Chef Workstation bundle installed, which provides the chef and knife CLIs, Ruby, and testing tools. Nothing here is a managed node; it's purely where code is written, tested locally, and uploaded. The Chef Infra Server is the one shared source of truth: it stores every uploaded cookbook version, data bags (global JSON variable stores), node objects (a record of every registered machine, including the facts Ohai collected about it on its last run), roles, environments, and policies, and it exposes a search index over all of it. The server itself is a small constellation of services under the hood — an API layer, PostgreSQL for relational storage, Elasticsearch for the search index, and an object store for cookbook files — packaged for self-hosting; check Progress's current Chef Infra Server and Chef Automate offerings directly before assuming which hosted or managed options exist today. A node is any managed machine running the chef-client agent, plus Ohai, the small tool that inventories the box — OS, IP addresses, CPU, memory, installed packages — and hands those facts to chef-client as automatic attributes at the start of every run.

Two words matter enormously here: compile and converge. Every chef-client run happens in two distinct phases. In the compile phase, Chef loads every recipe in the run-list top to bottom and builds an in-memory resource collection — but any plain Ruby in that recipe (an if, a File.exist? check, a puts) executes immediately, right then, not "in order" alongside the resources around it. Only in the converge phase, after compilation finishes, does Chef walk the finished resource collection and actually call each resource's provider to check and, if needed, fix real state. This two-phase split is the single most common source of "why did that run in the wrong order" confusion for anyone who assumes a recipe executes top to bottom like an ordinary script.

Cookbooks, recipes, and resources — the core abstractions

☺ Like you're 10: A cookbook is the whole recipe book; a recipe is one recipe in it; a resource is one line of that recipe, like "preheat the oven."

A cookbook is a directory with a fixed, convention-driven layout — recipes/, templates/, files/, attributes/, resources/ for custom resources, and a metadata.rb declaring its name, version, and dependencies. A recipe is one Ruby file inside recipes/, and a resource is a single declaration inside that recipe describing one piece of desired state — a package, a file, a service, a user, a cron entry. Chef ships dozens of built-in resources (package, service, template, file, directory, user, group, cron, execute, git, remote_file, cookbook_file) and every one of them follows the same shape: a resource name, a block of properties, and one or more actions.

# metadata.rb — cookbook identity and dependencies
name             'nginx_app'
maintainer       'Acme Platform Team'
maintainer_email 'platform@acme.example'
license          'Apache-2.0'
description      'Installs and configures the Acme checkout frontend'
version          '2.4.0'
chef_version     '>= 17.0'          # minimum chef-client this cookbook requires

depends          'nginx', '~> 12.0' # a SemVer-range dependency, resolved by Policyfile/Berkshelf

# recipes/default.rb — compiled top to bottom, but nothing RUNS until the converge phase
package 'nginx' do
  action :install                    # desired state: installed. Already installed → no-op.
end

template '/etc/nginx/sites-available/checkout.conf' do
  source 'checkout.conf.erb'         # looked up in templates/, rendered with ERB
  owner  'root'
  group  'root'
  mode   '0644'
  variables(
    upstream_port: node['checkout']['port'],       # a node attribute — see below
    server_name:   node['checkout']['server_name']
  )
  notifies :reload, 'service[nginx]', :delayed      # only fires if this template actually changed
end

link '/etc/nginx/sites-enabled/checkout.conf' do
  to 'checkout.conf'
  notifies :reload, 'service[nginx]', :delayed
end

service 'nginx' do
  action [:enable, :start]           # two actions: enabled at boot, and running right now
end

Every resource is backed by a provider — the platform-specific code that knows how to check current state and, only if needed, change it. That check-then-act split is what makes package 'nginx' safe to run a thousand times: the provider inspects the package database first and does nothing at all if nginx is already installed at the right version. notifies (and its inverse, subscribes) is Chef's change-triggered pattern — a resource fires an action on another resource, but only when its own action actually changed something — and :delayed (the default) queues that notification to fire once, at the end of the run, rather than immediately, so five different resources notifying service[nginx] to reload only cause one reload, not five.

Custom resources and the idempotency contract

☺ Like you're 10: When you find yourself writing the same four steps in every recipe, you can name those four steps and give them one new command of their own.

Once a pattern like "install nginx, template a vhost, symlink it in, reload the service" repeats across recipes, Chef's answer is a custom resource: a file in resources/ that declares its own property list and action blocks, callable from any recipe exactly like a built-in resource. This is the composition tool that keeps large cookbook collections from turning into copy-pasted boilerplate.

# resources/checkout_site.rb — a custom resource wrapping the steps above into one call
property :port, Integer, default: 8080
property :server_name, String, required: true, name_property: true

action :create do
  package 'nginx'

  template "/etc/nginx/sites-available/#{new_resource.server_name}.conf" do
    source 'checkout.conf.erb'
    variables(upstream_port: new_resource.port, server_name: new_resource.server_name)
    notifies :reload, 'service[nginx]', :delayed
  end

  link "/etc/nginx/sites-enabled/#{new_resource.server_name}.conf" do
    to "/etc/nginx/sites-available/#{new_resource.server_name}.conf"
    notifies :reload, 'service[nginx]', :delayed
  end

  service('nginx') { action [:enable, :start] }
end

# recipes/default.rb — now a one-line call, readable by someone who's never seen the internals
checkout_site 'checkout.acme.internal' do
  port 8080
end

Run-lists, roles, environments, and the shift to Policyfiles

☺ Like you're 10: The run-list is which chapters of the recipe book this one machine has to follow, and Chef has two different ways over the years to write down "which chapters, with which settings, for which stage of the road to production."

A node's run-list is the ordered list of recipes and roles it should converge — ["recipe[nginx_app::default]", "role[web]"] — stored on its node object on the server. For years, the standard way to manage run-lists and per-stage settings at scale combined roles (named, reusable run-list-plus-default-attributes bundles, like "web"), environments (dev/staging/prod, each pinning acceptable cookbook version constraints and overriding attributes), and the Berkshelf tool for resolving cookbook dependencies — three separate mechanisms an operator had to keep in sync by hand.

Policyfiles are the modern replacement, and most active Chef shops use them today instead of the roles/environments/Berkshelf combination. A single Policyfile.rb declares the run-list and every cookbook source in one file; chef install resolves it into a locked Policyfile.lock.json pinning exact versions (the same idea as a Gemfile.lock or a Terraform lock file); and chef push uploads that locked policy to a named policy group on the server (say, production), which nodes in that group then pull as a single, internally consistent unit instead of resolving version constraints independently at converge time.

# Policyfile.rb — one file replaces roles + environments + a Berksfile for most shops
name 'checkout-web'

default_source :supermarket        # where to resolve public cookbooks from
run_list 'nginx_app::default'

cookbook 'nginx_app', path: '.'    # this cookbook, from the local working directory
cookbook 'nginx', '~> 12.0'        # everything else, resolved and pinned into the lockfile

Day-to-day commands

☺ Like you're 10: Write it, lock the exact versions, push it to the shared fridge, and either wait for a machine to check in on its own or ask it to check right now.

# author and lock a policy from a Workstation
$ chef generate cookbook nginx_app              # scaffold recipes/, templates/, metadata.rb, spec/
$ chef install                                  # resolve Policyfile.rb → Policyfile.lock.json
$ chef push production Policyfile.lock.json     # upload the LOCKED policy to the "production" policy group

# register a brand-new node and run it for the first time
$ knife bootstrap 10.0.4.12 -N web01 -x ubuntu -i ~/.ssh/id_rsa \
    --sudo --node-ssl-verify-mode none \
    --policy-name checkout-web --policy-group production

# on the node itself (or via knife ssh across a fleet)
$ chef-client                                   # a normal converge, reads its policy from the server
$ chef-client --why-run                         # DRY RUN: reports what would change, changes nothing
$ chef-client --local-mode --runlist 'recipe[nginx_app::default]'  # no server at all — aka chef-zero

# inspect what the server knows
$ knife node show web01 -a run_list             # this node's assigned run-list
$ knife node show web01 -a automatic.platform    # an Ohai-collected automatic attribute
$ knife data bag show secrets checkout_db_password
$ knife ssh 'role:web' 'sudo chef-client'       # fan an ad hoc run out across every matching node

# testing, on the Workstation, before anything reaches a real node
$ kitchen list                                  # the cookbook × platform test matrix
$ kitchen converge ubuntu-2204                  # spins up a VM/container, converges the cookbook
$ kitchen verify ubuntu-2204                    # runs InSpec controls against the converged instance
$ kitchen destroy ubuntu-2204

Gotchas and failure modes

☺ Like you're 10: A few habits that feel harmless on your laptop turn into real 2 a.m. mysteries once thirty machines are all quietly redoing the same chore chart on their own schedule.

Compile-phase surprises. The compile/converge split covered above is the single most-reported "Chef did something weird" bug: a Chef::Log.info call, a conditional guard, or a helper method invoked directly in a recipe body runs during compile, before a single resource has actually converged — so code that looks like it runs "between" two resources in the file almost never does. When behavior needs to depend on whether an earlier resource actually changed something, that's exactly what notifies/subscribes and resource guards (only_if, not_if) are for, not inline Ruby conditionals sprinkled through the recipe.

Attribute precedence has a famous asymmetry. Node attributes come from several attribute types (default, normal, override, plus the Ohai-collected automatic) set from several sources (a cookbook's attribute file, a recipe, a role, an environment), and Chef's official precedence table has around fifteen distinct levels. The detail that trips people up specifically: for default-type attributes, an environment outranks a role, but for override-type attributes, a role outranks an environment — the two axes flip relative to each other. Don't try to hold the full table in memory under pressure; keep Chef's own attribute-precedence documentation open the first several times you're debugging "why is this value not what I set," because guessing wrong here wastes far more time than reading the table would.

The Chef Infra Server is a single point of failure for the whole fleet. Every cookbook version, every data bag, and every node's run-list lives there; if it's unreachable, no node can converge and no operator can push a fix — nodes keep running whatever they last pulled, which is a reasonable fail-static default but is not the same as things being fine. Back it up as seriously as the rest of this course's advice on secrets and credential management, since encrypted data bags and Chef Vault items are only as recoverable as the server (and its keys) that stored them.

"chef-solo" is a name that outlived the feature. Older material still refers to running Chef without a server as chef-solo; the current mechanism is chef-client --local-mode (often called chef-zero internally, since it spins up an in-memory, ephemeral Chef server for the duration of one run). Functionally similar, but if a tutorial or a Stack Overflow answer references chef-solo flags directly, check whether it predates the local-mode rename before copying it verbatim.

Chef vs. alternatives: an honest read on adoption

☺ Like you're 10: Chef isn't going away, but fewer new teams are picking it as their first choice these days than they used to — worth knowing honestly instead of pretending every tool in this space is equally popular.

A few structural reasons show up repeatedly in explanations for that trend, and they're worth naming honestly rather than glossing over. Ansible's agentless, SSH-based push model has a materially lower operational floor than Chef's agent-plus-central-server pull model — nothing to install on managed hosts, no Chef Infra Server to run and patch, and YAML playbooks read as more approachable to newcomers than a Ruby DSL with a compile/converge split to internalize. Puppet's own pull-based, agent/master architecture is Chef's closer structural sibling, but it has maintained a steadier enterprise foothold in shops that adopted it earlier and never migrated off. And underneath both comparisons sits the bigger platform shift: containers and immutable, replace-don't-patch images (see Immutable Infrastructure & Golden Images) push a growing share of "configuration" work earlier, into the image build, leaving less work of any kind for a long-running convergence agent on a live host — a trend that erodes demand for Chef, Puppet, and Ansible's ongoing-configuration use case simultaneously, just not evenly.

None of this means an existing Chef estate needs to be ripped out. A cookbook collection that's been hardened over years of real production incidents encodes institutional knowledge that a rewrite in another tool would have to re-earn the hard way, incident by incident. The honest framing for a learner is narrower and more useful than "Chef is dead" or "Chef is dominant": expect to maintain Chef more often than you'll be handed a greenfield choice to adopt it fresh, and weigh a genuinely new configuration-management decision against Ansible's lower operational floor and Puppet's structural similarity rather than assuming Chef is the default. Chef Software historically offered a developer certification track; verify what Progress currently offers, if anything, before pointing a learner toward one — this course does not carry a dedicated Chef certification page for that reason. See the DevOps toolchain for how all of this fits the wider landscape, and configuration management for the push-vs-pull tradeoff underneath the whole comparison.