Connecting to a Poorly Documented API

Goal

You attach to a customer API with poor documentation, confirm facts not in the specification through a full survey, and handle retries too.

Why it matters

The first job when attaching to a new API is not writing code but a full survey. You go through every page once and count the total count, the sum, and the number of missing values in each field. This one pass removes weeks of later debugging. If in the first week you ask "of the 60 records in total, 6 have an empty region. How should we handle them?", later no report will come that the regional totals do not match.

The most common bug in pagination is computing the number of pages from total while forgetting the remainder of the division. The last page goes missing entirely, and that fact shows up only as a subtly smaller total, so it is found late. So in step 4, do not trust the computed value; verify by actually looping and counting.

In retries, what you target matters. 5xx and network errors may be transient and are retry targets, but a 4xx gives the same result however many times you send it unless you fix the request, so retrying only raises load.

The API specification (everything written on the wiki)

• GET /health → {"status": "ok"}
• GET /meta → {"version", "page_size", "total"}
• GET /orders?page=N → {"page", "page_size", "total", "has_next", "items": [{id, customer, amount, region}]} — page starts at 1
• GET /flaky → only written that it fails sometimes

Steps

1. Run /opt/app/api.py so that 127.0.0.1:8002/health returns 200.
2. From /meta, write the version value to /root/api/version.txt.
3. Compute the total number of pages and write it to /root/api/pages.txt.
4. Write the total number of records collected by actually looping through all the pages to /root/api/count.txt.
5. Write the sum of the amount of all records to /root/api/sum.txt.
6. Write the number of records whose region is an empty string to /root/api/no_region.txt.
7. Write the status code you finally obtained by retrying /flaky to /root/api/flaky_ok.txt.
8. In /root/api/report.md, summarize the version, the total count, and the amount sum.

Notes

• Start it with python3 /opt/app/api.py &.
• First look at the structure with curl -s http://127.0.0.1:8002/orders?page=1 | python3 -m json.tool.
• Looping: for p in $(seq 1 6); do curl -s "http://127.0.0.1:8002/orders?page=$p"; done
• Common mistake 1: discarding the remainder in step 3 and missing the last page.
• Common mistake 2: copying /meta's total as is in step 4. The purpose of this step is to actually loop and count.

Start the order API

Run /opt/app/api.py so that 127.0.0.1:8002/health returns 200.

When you run /opt/app/api.py, it waits on 127.0.0.1:8002. Check with /health.

Check the API version

From /meta, write the version value to /root/api/version.txt.

The /meta response is JSON. Extract only the value of the version field. jq or python3 makes it easy.

Compute the total number of pages

Compute the total number of pages and write it to /root/api/pages.txt.

Compute it from total and page_size in /meta. If there is a remainder, there is one more page.

Loop through all pages and count

Write the total number of records collected by actually looping through all the pages to /root/api/count.txt.

Actually go through all the pages and count the items. The point is to verify, not to copy the total field as is.

Compute the amount sum

Write the sum of the amount of all records to /root/api/sum.txt.

Add up amount from the items of all pages.

Count the records with gaps

Write the number of records whose region is an empty string to /root/api/no_region.txt.

It is a gap not in the documentation. Count how many records have an empty string for region.

Get through an unstable endpoint

Write the status code you finally obtained by retrying /flaky to /root/api/flaky_ok.txt.

/flaky returns 503 the first few times. Retry until a 200 arrives and write the final status code.

Write the integration result report

In /root/api/report.md, summarize the version, the total count, and the amount sum.

The version, total count, and amount sum must all be in it. Think of it as a document to send to the customer in the first week.