Drill — Debug a Missing Catalog Entity

A team ships a new service, writes a correct catalog-info.yaml for it, pushes to main, and waits. The portal never shows it. No red banner, no processing error, no orphan tag — nothing to read, because nothing ever tried and failed. That silence is the entire drill: most catalog bugs leave a trail, and this one deliberately doesn't. You'll stand up a real Backstage instance with a GitHub entity provider, ship one repo that ingests cleanly as your control, ship a second that vanishes into nothing, and use the catalog's own REST API to rule out the two failure modes that do leave evidence before landing on the one that doesn't. Fully self-contained — a throwaway local portal and two throwaway GitHub repos, no dependency on the five-part capstone. Budget 25-40 minutes before you open the walkthrough below.

Why this failure mode is worth its own drill

☺ Like you're 10: Most bugs leave a mess behind. This one leaves nothing at all — and "nothing" is a clue you have to learn to read.

The Backstage tools page on this site names three ways an entity can fail to show up cleanly: a malformed or unreachable catalog-info.yaml surfaces a processing error on the entity itself; an orphan is an entity whose parent Location vanished, tagged backstage.io/orphan so it's easy to find; and a stale entity lingers when its source repo is gone but nobody deleted the registration. All three assume an entity exists to be examined. Today's bug is a fourth shape entirely, upstream of all three: an entity provider filter that excludes a repository before the provider ever tries to fetch, let alone parse, its catalog-info.yaml. Nothing is malformed. Nothing is orphaned. Nothing was ever attempted. The catalog has zero opinion about a repo it was never told to look at — and that's precisely why you have to prove the absence of the other two causes before this one is even a reasonable suspect.

Build the scratch portal

☺ Like you're 10: One small Backstage app, one GitHub provider pointed at your own account, configured with a rule nobody updated.

Create a fresh Backstage app — this is the throwaway front desk for today only:

npx @backstage/create-app@latest --path catalog-drill
cd catalog-drill

Configure a GitHub entity provider in app-config.yaml. Two years ago this org put every service repo behind a svc- prefix and wrote that convention straight into the filter — reasonable at the time, and exactly the kind of rule that outlives the reason for it:

# app-config.yaml — add under the existing `catalog:` key
catalog:
  providers:
    github:
      driveOrg:
        organization: ${GITHUB_ORG}
        catalogPath: /catalog-info.yaml
        filters:
          branch: main
          repository: '^svc-.*$'   # written when every service repo had this prefix
        schedule:
          frequency: { seconds: 30 }
          timeout: { minutes: 1 }
          initialDelay: { seconds: 5 }

Give the provider read access with a token — a scoped fine-grained GitHub App is what a production portal should use (see the credential warning on the Backstage tools page), but a personal token is fine for a 30-minute throwaway:

export GITHUB_ORG=$(gh api user -q .login)
export GITHUB_TOKEN=$(gh auth token)

# app-config.yaml — integrations block
integrations:
  github:
    - host: github.com
      token: ${GITHUB_TOKEN}

Ship a control, then ship the one that vanishes

☺ Like you're 10: One repo that follows the old rule, and proves the desk works at all. One repo that doesn't, and disappears.

First, the control — a repo whose name still matches the old prefix, so you have proof the provider works before you go hunting for why something else doesn't:

mkdir svc-notifications && cd svc-notifications && git init -b main
cat > catalog-info.yaml <<'EOF'
apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: notifications
  description: Sends the emails and pushes nobody reads until something breaks.
spec:
  type: service
  lifecycle: production
  owner: group:default/platform-team
EOF
git add -A && git commit -m "chore: catalog-info.yaml"
gh repo create svc-notifications --public --source=. --remote=origin --push
cd ..

Now the one that actually matters — a brand-new service, correctly described, that simply doesn't happen to carry the old prefix, because nobody told whoever named it that the prefix still meant anything:

mkdir payments-ledger && cd payments-ledger && git init -b main
cat > catalog-info.yaml <<'EOF'
apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: payments-ledger
  description: The append-only ledger every payment event gets written to.
spec:
  type: service
  lifecycle: production
  owner: group:default/payments-team
EOF
git add -A && git commit -m "chore: catalog-info.yaml"
gh repo create payments-ledger --public --source=. --remote=origin --push
cd ..

Start the portal and give the provider one full cycle:

cd catalog-drill
yarn install
yarn dev >/tmp/catalog-drill.log 2>&1 &
sleep 45   # initialDelay (5s) + one full frequency tick (30s), with room to spare

Open http://localhost:3000/catalog. notifications is there. payments-ledger is not — not greyed out, not marked with a warning icon, not present in any filtered view. It simply isn't in the list, exactly as if the repo didn't exist.

Rule out the loud failure first

☺ Like you're 10: Ask the front desk directly: do you have any record of this kid at all?

If payments-ledger's catalog-info.yaml were malformed, the entity would still exist as far as the catalog's concerned — just carrying a processing error. Ask directly, by name, and let the HTTP status answer the question before you read a single byte of the body:

curl -s -o /tmp/entity.json -w '%{http_code}\n' \
  -H "Authorization: Bearer $(yarn --silent backstage-cli config:print --format json 2>/dev/null | jq -r '.backend.auth.externalAccess[0].options.token // empty')" \
  "http://localhost:7007/api/catalog/entities/by-name/component/default/payments-ledger"

If your local instance has no auth configured (the default for create-app in dev mode), drop the header entirely — dev mode's catalog API is open on localhost:

curl -s -o /tmp/entity.json -w 'HTTP %{http_code}\n' \
  "http://localhost:7007/api/catalog/entities/by-name/component/default/payments-ledger"
cat /tmp/entity.json | jq .

HTTP 404
{
  "error": {
    "name": "NotFoundError",
    "message": "component:default/payments-ledger not found"
  }
}

404, not 200-with-errors. That's the whole first ruling: the catalog has no record of this entity at all, good or bad. A malformed file that the provider had actually tried to fetch and parse would leave something behind — at minimum a processing error tied to the source location. A flat 404 means the parse step was never reached, which means the bug lives somewhere earlier than the file itself.

Rule out an orphan next

☺ Like you're 10: An orphan is a kid whose enrollment record still exists with "moved away" written on it. This kid was never enrolled — there's no record to have gone stale.

An orphaned entity is, by definition, one that used to ingest successfully and then lost the Location that produced it — Backstage tags the survivor backstage.io/orphan: "true" so it's easy to find every one of them in a single query:

curl -s "http://localhost:7007/api/catalog/entities?filter=metadata.annotations.backstage.io/orphan=true" \
  | jq -r '.[].metadata.name'

# (empty — no output at all)

Empty result, and it has to be: payments-ledger already failed the 404 check above, so there is no prior entity for it to have orphaned from. Orphan status is a property of something that once existed; you've just proven nothing here ever did. Two of the site's three named failure modes are now formally eliminated, by evidence, not by assumption — which is exactly the discipline the CBA blueprint means by "troubleshooting entity ingestion" as its own competency, distinct from just knowing what the catalog is.

Find the sticky note under the desk

☺ Like you're 10: Read the exact rule the desk is using to decide who it even looks at.

With the file and the location both ruled out, the only remaining stage is the one that runs before either — the provider's own filters block. Check the dev server's own log first; the GitHub provider reports what it discovered each cycle:

grep -i "github-provider\|driveOrg" /tmp/catalog-drill.log | tail -5

[catalog] github-provider:driveOrg — discovered 2 repositories in org, 1 matched filters
[catalog] github-provider:driveOrg — applying 1 entity from svc-notifications

2 discovered, 1 matched. The provider saw payments-ledger and correctly, deliberately, did not select it — this isn't a crash or a timeout, it's the filter working exactly as configured. Confirm the regex itself, on the actual repo names, before you touch the config file:

node -e "console.log(/^svc-.*$/.test('svc-notifications'))"   # true
node -e "console.log(/^svc-.*$/.test('payments-ledger'))"     # false

There it is, provably: filters.repository: '^svc-.*$' in app-config.yaml anchors on a naming convention that payments-ledger never had reason to follow. Nobody broke anything — a rule written for a naming scheme that later changed just kept quietly doing its job on repos that no longer matched it.

Fix it, and prove it holds

☺ Like you're 10: Cross out the old rule, save the file, and watch the desk notice the kid it skipped over.

Widen the filter and save — Backstage's dev server watches app-config.yaml and restarts the backend on its own, which re-runs the provider's initialDelay discovery pass immediately:

# app-config.yaml — the fix
      filters:
        branch: main
        repository: '.*'   # was: '^svc-.*$' — the prefix convention is dead

sleep 10   # backend restart + initialDelay
grep -i "driveOrg" /tmp/catalog-drill.log | tail -3

[catalog] github-provider:driveOrg — discovered 2 repositories in org, 2 matched filters
[catalog] github-provider:driveOrg — applying 1 entity from payments-ledger

Confirm the same way you diagnosed — through the API, not just the UI:

curl -s -o /tmp/entity.json -w 'HTTP %{http_code}\n' \
  "http://localhost:7007/api/catalog/entities/by-name/component/default/payments-ledger"
jq '.metadata.annotations."backstage.io/managed-by-location"' /tmp/entity.json

HTTP 200
"url:https://github.com/<you>/payments-ledger/blob/main/catalog-info.yaml"

Done when: the by-name query returns HTTP 200 with a real entity body, backstage.io/managed-by-location points at the GitHub provider's URL (not a manually-created Location you added yourself), and http://localhost:3000/catalog lists payments-ledger right alongside notifications with no page reload needed.

The transferable habit

☺ Like you're 10: When there's nothing to read, that absence is itself the clue — go find the rule that decided not to look.

The instinct most engineers bring to "it's not showing up" is to reread the file that's supposedly wrong — and today's file was never wrong. The move that actually finds this bug is asking the catalog directly, by name, what it knows, and treating a flat 404 as meaningfully different from a 200-with-a-warning. Once you've eliminated "the file failed" and "the location vanished," the only place left to look is whatever decided the file was worth reading in the first place — a provider filter, an allow-list under catalog.rules, a branch requirement, an ownership check on the scaffolder side. That generalizes well past Backstage: a CI job that silently never runs because a path filter didn't match, an alert that never fires because a label selector missed by one character, a webhook that never delivers because an IP allow-list was never updated. All of the same shape — no error, because nothing was ever attempted, and the fix lives in the gate, not in whatever the gate was guarding.

Fixed and holding? Good — that's the whole drill. For the concepts underneath it, see The Backstage Portal Model and Backstage, whose own troubleshooting section is where the three named failure modes this drill built on came from; for the certification this exact competency sits on, see CBA. If your interest is the platform-engineering side of self-service portals rather than Backstage specifically, Self-service & golden paths and the CBA on Platform Engineering's own certifications hub cover the wider ground. Ready for a different single skill? Try Drill — Diagnose a Stuck Argo CD Sync, or go build the full loop in Build Your Cert Tracker — Start Here.