Tag: #reliability

← All posts

• A Sense for Production — What Looks Different After You Have Been Through an Outage

  Two people read the same design doc and one of them reads something else entirely: what logs exist when this breaks at three in the morning, how many minutes a rollback takes, and what happens to the data that already la

  2026-08-15 · 7 min read #career#skills#sre#production#reliability

• Following 657,607 Links, and the Lifespan of a URL — Why Links Die of Connection Failure, Not 404

  A study published its results after following, in 2026, every one of roughly 655,000 shortened links created between 2009 and 2014. More important than the headline number is the composition of the failures: most dead li

  2026-08-14 · 6 min read #web#data-engineering#archival#reliability#url-design

• Running AI Agents in Production — Idempotency, Budgets, and Confidently Wrong Answers

  Moving an agent from prototype to production exposes an operational surface you only discover late: idempotency for retried tool calls, budgets and step limits, observability over non-deterministic control flow, per-tool

  2026-07-31 · 16 min read #ai#agents#observability#reliability#mcp

• Choosing a Rate Limiting Algorithm — What Really Separates Fixed Window, Sliding, and Token Bucket

  Letting 200 requests through a 100-per-minute limit is not a bug. It is the defined behaviour of the fixed window algorithm. This post compares the memory and accuracy tradeoffs of four approaches — fixed window, sliding

  2026-07-26 · 12 min read #web#rate-limiting#api-design#redis#reliability

• Temporal Worker Versioning GA — The Deploy Problem Replay Created, and the Third Answer After Two Discarded Generations

  Durable execution engines like Temporal survive failures by replaying an event history — and it is precisely that replay which makes shipping new code while workflows are still running the hardest problem in this model.

  2026-07-17 · 13 min read #distributed-systems#durable-execution#temporal#workflow-engine#reliability

• How AI Agents Fail in Production — 14 Failure Modes, and Why Retrying Is Not Safe ♪ Listenable

  Putting agents in production hurts in three places. First, failures come from system design, not the model — UC Berkeley's MAST study classified 1642 execution traces into 14 failure modes, and 44.2% of them were system-

  2026-07-17 · 20 min read #ai#agents#observability#reliability#mcp

• Payment Idempotency: Preventing Double Charges

  The network will eventually drop, and when it drops, clients retry. But in payments, a retry can become a double charge. Idempotency keys, dedup windows, unique constraints, the retry-plus-timeout problem, at-least-once

  2026-07-02 · 14 min read #payments#idempotency#reliability

• The Truth About Exactly-Once Semantics

  What the three delivery guarantees (at-most-once, at-least-once, exactly-once) actually mean, why exactly-once delivery is fundamentally impossible while exactly-once processing is achievable through idempotency, dedupli

  2026-06-28 · 13 min read #distributed-systems#messaging#reliability

• Rate Limiting Algorithms, Explained: Fixed Window, Sliding Window, Token Bucket, and Leaky Bucket ♪ Listenable

  A side-by-side tour of the main rate limiting algorithms — fixed window, sliding window log and counter, token bucket, and leaky bucket. How each handles bursts, how to implement rate limiting across a distributed fleet

  2026-06-25 · 11 min read #systems#api#reliability

• Idempotency and Retries: Building Reliable APIs ♪ Listenable

  The network will eventually fail, and when it does you have to retry. The hard case is the request that was "already processed but whose response got lost." This post covers what idempotency is, safe versus unsafe HTTP m

  2026-06-21 · 12 min read #api#reliability#distributed-systems

• Migration Incident Case Studies and Checklists — Learning from Others Failures

  Migrations are among the most common and most painful sources of outages. This article walks through typical failure modes like lock storms, replication lag, data loss, and irreversible changes, derives causes and lesson

  2026-06-16 · 14 min read #database#migration#postmortem#incident#reliability

• The Expand-Contract Pattern — The Right Way to Change Schemas with Zero Downtime

  A thorough walkthrough of the Expand-Contract pattern, the standard technique for changing a live database schema without downtime. We cover per-scenario SQL for adding columns, renaming, dropping, and adding constraints

  2026-06-16 · 15 min read #database#migration#zero-downtime#expand-contract#reliability

• An Introduction to Database Migration Strategy — Schema, Data, and Zero Downtime

  A systematic overview of database migration types and risks, versioned migrations, and zero-downtime deployment principles. It covers transactional DDL, backups and dry runs, environment promotion, team process, and an i

  2026-06-16 · 16 min read #database#migration#schema#devops#reliability

• Code That Fails Well — A Deep Dive into Error Handling and Resilience Design (2026) ♪ Listenable

  Error handling is not decoration you bolt on after the feature is built — it is the design itself. Classify the kinds of failure, decide between exceptions and error values, validate at the edge and trust the core, put a

  2026-05-14 · 25 min read #error-handling#resilience#retry#circuit-breaker#timeout

• Load/Performance Testing Tools 2026 — Deep Dive on k6, Locust, Vegeta, Gatling, Artillery, JMeter (Beyond JMeter)

  Performance testing is still hard in 2026. But the tools have definitely gotten better. JS-scriptable k6, Python-friendly Locust, one-liner Vegeta, enterprise Gatling, YAML-quick Artillery, micro-benchmark wrk/wrk2 and a

  2026-05-14 · 18 min read #load-testing#performance-testing#k6#locust#vegeta

• Durable Execution Engines in 2026 — A Deep Dive Comparison of Temporal, Restate, Inngest, Trigger.dev, DBOS: Escaping Cron and Retry Hell ♪ Listenable

  Stop writing long-running workflows with cron, queues, and if-else. The answer in 2026 is Durable Execution. This deep dive compares Temporal, Restate, Inngest, Trigger.dev, and DBOS — plus AWS Step Functions, Cadence, a

  2026-05-14 · 26 min read #durable-execution#temporal#restate#inngest#trigger-dev

• Agent Evaluation Systems in 2026 — Inspect AI vs Promptfoo vs Phoenix vs LangSmith vs OpenAI Evals (You're Measuring the Agent, Not the Model) ♪ Listenable

  LLM evals measure the model. Agent evals measure whether the model plus the harness plus the tools actually carry a task to completion. They are different problems. This is a map of the 2026 landscape — Inspect AI from U

  2026-05-14 · 20 min read #agent-evaluation#inspect-ai#promptfoo#phoenix#langsmith

• Anatomy of Famous Postmortems — Learning From the Failures of Cloudflare, Fastly, AWS, Knight Capital, and GitLab

  Cloudflare 2022 BGP, Fastly taking down half the internet in 2021, the AWS S3 typo of 2017, Knight Capital losing $440M in 8 minutes, the GitLab DB wipe, and more — the patterns and lessons extracted from famous failures

  2026-04-15 · 12 min read #postmortem#sre#outage#reliability#culture

• Chaos Engineering Deep Dive — Netflix Simian Army, LitmusChaos/Chaos Mesh, AWS FIS, Game Day

  Why Netflix started randomly killing production servers in 2010. From Chaos Monkey philosophy and the 4 principles, to the full Simian Army, LitmusChaos/Chaos Mesh/AWS FIS comparison, Game Day design, and blameless postm

  2026-04-15 · 10 min read #chaos-engineering#sre#reliability#netflix#kubernetes

• AWS Well-Architected Framework Complete Guide 2025: Six Pillars, Practical Adoption, Cost/Security/Performance

  Everything about the AWS Well-Architected Framework: six pillars (Operational Excellence, Security, Reliability, Performance Efficiency, Cost Optimization, Sustainability), practical checklists, Well-Architected Tool usa

  2026-04-15 · 14 min read #aws#well-architected#cloud-architecture#security#reliability