TT Lab
Get started
Learn Learning paths Courses

CBA — Backstage Associate

Services Added to the Catalog Quietly Disappear

Continue in TT Lab

Goal

You bring up a real catalog backend inside a VM, deliberately create four causes of entities not making it into the catalog, and find where traces remain. You check, through real API responses, automatic collection of static locations, locations registered through the API, cleanup of orphan entities, and deletion of a location.

Why it matters

"I uploaded catalog-info.yaml but it does not show in the portal" is the most common question in Backstage operations. Checking one YAML with a validation library is not enough. In a running catalog, there are cases where the failure shows up neither in the API response nor in the default log, and a problem in one entity can even block the whole file. Even if the registration API returns 201, processing can still fail. You can fix it only if you know where to look.

The first VM start takes about 4 minutes, and the installation in step 1 takes about 30 seconds from the internet. Discovery providers that scan external systems such as GitHub need an external account, so they are not covered. Here, "automatic collection" means the catalog periodically re-reads the locations written in the configuration.

Steps

  1. Install the catalog backend packages at pinned versions in /usr/local/cba-ingest.
  2. Confirm that the file location written in the configuration is collected automatically.
  3. Record that an entity missing its owner drops out without a trace.
  4. Attach the log module to bring processing errors out as warn logs.
  5. See that a disallowed kind blocks the whole file, and resolve it with per-location rules.
  6. Register a url location through the API, and fix the read allowlist problem.
  7. See an entity removed from the file become an orphan and then get deleted.
  8. Delete the location and write the ingestion failure report.

Notes

Install the ingredients of a running catalog at pinned versions

In /usr/local/cba-ingest, create an npm project and install four packages at exact versions with no range symbols (in package.json too, without ^). Keep the npm cache at npm_config_cache=/usr/local/cba-ingest/.npmcache. @backstage/backend-defaults@0.17.8, @backstage/plugin-catalog-backend@3.9.1, @backstage/plugin-catalog-backend-module-logs@0.1.25, better-sqlite3@12.4.1.

Use npm install --save-exact. Only install the log module for now and attach it in step 4. better-sqlite3 13.x has no prebuilt binary for this Node 20, so it tries to compile and fails.

A file written in the configuration is collected automatically

Put the following in /usr/local/cba-ingest/app-config.yaml and bring up a backend that has only the catalog plugin (index.js) on port 7007 (output to /usr/local/cba-ingest/backend.log). backend.baseUrl: http://localhost:7007, backend.listen.port: 7007, DB better-sqlite3 and ':memory:', in backend.auth.externalAccess type static, token ingest-lab-token-7f3a9c, and subject ingest-cli, catalog.processingInterval: { seconds: 5 }, catalog.rules: [{allow: [Component, Group, Location]}], and in catalog.locations type: file and target: ./catalog/team.yaml. Put in catalog/team.yaml the Group team-search and the Components search-api and search-indexer owned by that team. It is enough if calling /api/catalog/entities with Authorization: Bearer ingest-lab-token-7f3a9c shows three entities.

The catalog API is behind the default auth policy, so calling it without a token gives 401. In production the static token should be put in as ${환경변수} (the placeholder is an environment variable name), but in this lab you write it in the file to see the flow. Check which file the backstage.io/managed-by-location annotation of an entity that came in points to.

An entity that left out its owner drops out without a trace

At the end of team.yaml, append a Component search-ui (type website, lifecycle production) that has no spec.owner. Without restarting, wait a few seconds, look at the catalog and the log, and write three lines to /root/cba-ingest/silent.txt — search_ui_in_catalog= (yes/no), search_api_in_catalog= (yes/no), and log_lines_mentioning_search_ui= (the number of lines in backend.log at that moment that contain search-ui). Leave search-ui unfixed until the end of this lab (later steps and grading use this state).

You cut the processing interval to 5 seconds, so if you edit the file it is read again soon. See what happens to the rest of the same file when one entity fails validation, and where that failure is recorded (or is not). Count with grep -c search-ui backend.log.

Bring processing errors out into the log

Add @backstage/plugin-catalog-backend-module-logs to index.js and restart the backend. When a warn line about component:default/search-ui is printed to backend.log, save that one line, with the color control characters stripped, to /root/cba-ingest/owner-error.txt. Do not fix team.yaml.

The catalog only emits processing errors as events, and writing them to the log is a separate module. Even after attaching the module, the processing cycle must run once before the line appears. Find it with sed 's/\x1b\[[0-9;]*m//g' backend.log | grep search-ui. The events backend not found warning comes from the absence of the events plugin and is unrelated to this task.

One API keeps the whole file from coming in

In /usr/local/cba-ingest/catalog/billing.yaml, put the Component billing-api (owner team-search, providesApis [billing-openapi]) and the API billing-openapi (type openapi, owner team-search), add type: file and target: ./catalog/billing.yaml to catalog.locations, and restart. Confirm that neither entity comes in, and save one warn line about api:default/billing-openapi to /root/cba-ingest/kind-error.txt. Then leave the global catalog.rules as it is, attach rules: [{allow: [API]}] to only the billing.yaml location, and restart so that both billing-api and billing-openapi come in.

A kind that is not allowed does not just discard that one entity; it fails the entire processing result of that location — compare this with the missing owner. If you put API in the global rules, APIs come in from any file. See the per-location rules in the documentation.

Registration is 201 but nothing comes in

Put the Components etl-runner and etl-scheduler (both with owner team-search) in /usr/local/cba-ingest/incoming/data.yaml, and serve that directory with python3 -m http.server 8088 --bind 127.0.0.1 (in the background). (1) Try to register type: file with target /usr/local/cba-ingest/incoming/data.yaml through the catalog API and look at the response (400), (2) send POST /api/catalog/locations with type: url and target http://localhost:8088/data.yaml, confirm that it is 201 but the entities do not come in, and then save one warn line about that url to /root/cba-ingest/reading-error.txt. (3) Put host: localhost:8088 in backend.reading.allow and restart, and then register the same url again so that the two entities come in. Save the POST response JSON of the last registration to /root/cba-ingest/location.json.

A location registered through the API accepts only the url form. What reads the url is the UrlReader, and a host without an integration must be in the allowlist to be read — this check happens at processing time, not at registration time, so the registration response is a success. The DB is :memory:, so a restart makes the location you registered earlier disappear; check with GET /api/catalog/locations. Also start http.server with setsid nohup ... > 로그 2>&1 < /dev/null & so that it survives after the shell ends (the placeholder is the log file name).

An entity removed from the file becomes an orphan and then gets deleted

Delete the etl-runner document from incoming/data.yaml (keep etl-scheduler). Save to /root/cba-ingest/orphan.json the entity JSON (the response of GET /api/catalog/entities/by-name/component/default/etl-runner) at the moment the annotation backstage.io/orphan: "true" is attached to etl-runner in the catalog. Then wait and confirm that the entity is deleted from the catalog (by-name 404) and the Deleted ... orphaned entities line in the log.

If the location that was emitting an entity no longer emits that entity, it becomes an orphan. Whether to keep or delete orphans is decided by catalog.orphanStrategy, and with the default, the cleanup job (catalog_orphan_cleanup in the log, every 30 seconds) deletes it. The annotation is visible only briefly, so poll at 1-second intervals to catch the moment it is attached.

Delete the location and write the ingestion failure report

Send DELETE /api/catalog/locations/<id> with the id from location.json to unregister, and confirm that etl-scheduler disappears from the catalog. Then in the first five lines of /root/cba-ingest/report.md, write values measured now — registered_location_status= (GET /api/catalog/locations/), etl_scheduler_status= (a by-name lookup), search_ui_status= (a by-name lookup), api_location_count= (the number of results from GET /api/catalog/locations — whether the two static locations appear here), and file_register_status= (the response to the attempt to register type file). Below that, summarize the four causes of ingestion failure in this lab and where traces remained for each.

Every value is one you measured now with curl and the token attached. A static location is managed by the configuration file, so it cannot be deleted through the API and does not appear in the list. The four causes are a missing owner, a disallowed kind, the read allowlist, and an entity dropped from the file (an orphan).