Design for the worst day
Most systems are tested against average traffic and happy paths. Real failures come from peaks, partial outages, and inputs nobody expected. Start every design review by asking what the worst realistic day looks like (month-end, a campaign that takes off, a provider going down) and make sure the design survives it.
Make failure visible early
A system that fails loudly is easier to run than one that fails quietly. Structured logs, a handful of meaningful alerts, and dashboards that show business outcomes (orders placed, payments confirmed) beat hundreds of low-level metrics nobody reads.
- Alert on symptoms customers feel, not every CPU spike
- Track business events alongside technical ones
- Write the runbook before the first incident
Contain the blast radius
Timeouts, retries with backoff, and queues between services stop one slow dependency from dragging everything down with it. When a payment provider is slow, the rest of the checkout should still respond.
Change in small steps
Large releases hide large risks. Small, frequent deployments behind feature flags make every change easy to review, easy to observe, and easy to roll back.
Practise recovery
Backups you have never restored are a hope, not a plan. Schedule restore drills and failover tests, and treat what you learn as part of the system's design.