For the last few months I’ve been keeping a notebook. It started as a handful of markdown files and slowly turned into a small website with a few hundred chapters on system design, debugging, JavaScript and AI.
Every chapter is written the same way. I keep asking “why?” until the answer stops being a definition and starts being a reason. That habit is the only reason any of it stuck.
I’ll forget most of it. Some of it I won’t, because it changed how I write code on an ordinary weekday. These are those ten. Each one links to the full chapter if you want to go deeper.
1. Add the index before you plan the shard
The chapter opens with a story that’s painfully believable. The orders page is slow, p99 is two seconds, and the team starts planning to shard. Two weeks in, somebody finally runs EXPLAIN ANALYZE and finds there’s no index on customer_id. Every page load is reading 40 million rows.
CREATE INDEX CONCURRENTLY idx_orders_customer ON orders (customer_id);p99 drops from 2 seconds to 4 milliseconds. The sharding project quietly goes away.
Database scaling is a ladder. Measure first. Then fix queries and indexes, pool connections, cache, buy a bigger machine, add replicas, partition, and only then shard. Each rung costs more and is harder to undo than the one below it. Sharding feels like serious engineering. Most of the time the serious move is reading the query plan.
Chapter: The database scaling ladder
2. A Postgres connection is a whole process
I thought of a database connection as a cheap handle. It isn’t. In Postgres every connection is its own OS process, around 5 to 10 MB of memory before it runs a single query, and the default limit is 100.
So ten app servers opening a fresh connection per request will hit “too many connections” long before the database runs out of CPU. A pool opens a few connections once and lends them out. Now when I see that error, I go looking for code that isn’t giving its connection back before I even think about raising the limit.
Chapter: Connection pooling
3. The hard part of caching is everything after set
Adding a cache is two lines. A get before the database call and a set after it. That part really is easy.
The hard part is knowing what went stale. Cache user:42 and invalidating it is simple. But user 42 also shows up in a feed, a follower list, a search results page and someone else’s recommendations. Change their job title and all of those are wrong, and nothing tells you which keys to delete.
Two more things I didn’t appreciate until I wrote them down. If the cache restarts empty, the database suddenly gets all the traffic it was sized to see a tenth of. And when one popular key expires, every request misses at the same moment and they all run to the database together. The cheapest defence is almost boring: add a bit of randomness to your TTLs so keys don’t all expire in the same second.
const ttl = 3600 + Math.floor(Math.random() * 600) - 300;Chapters: Why caching is hard and Cache stampedes
4. Retrying is correct, so duplicates are your problem
A client sends “charge this card.” The server charges it. The connection drops before the response gets back. The client retries, which is exactly what it should do, and the card gets charged twice.
The client did nothing wrong. Only the client knows whether a request is new or a retry, so it sends an idempotency key, and the server stores that key along with the result. A second request with the same key gets the same answer back instead of a second charge.
The detail that made it click for me: checking “have I seen this key?” in application code isn’t enough. Two retries can land in the same millisecond and both see “no.” The database has to enforce it with a unique constraint, so exactly one insert wins and the other one reads the winner’s result.
Chapter: Idempotency for write APIs
5. “Exactly once” usually means “at least once, and duplicates are harmless”
A queue can promise one of three things. At most once, so you might lose messages. At least once, so you might process some twice. Exactly once, which needs the producer, the broker and the consumer to coordinate through every kind of failure.
What almost everyone actually runs is at-least-once delivery with consumers that don’t mind seeing a message twice. Once I accepted that, I stopped hunting for the magic setting and started writing handlers that are safe to run twice. Which is just idea 4 again, one layer down.
Chapter: Delivery guarantees
6. You can’t write to a database and a queue at the same time
You save an order, then publish “OrderPlaced” to a queue. If the process dies between those two lines, the order exists and nothing downstream ever hears about it. Swap the order and you can announce an order that never got saved.
The outbox pattern is a small, lovely trick. Write the event into an outbox table in the same transaction as the order. Now both happen or neither does. A separate worker reads the outbox and publishes. If it crashes halfway, it publishes again, which is fine because of idea 5.
This one stopped being theory for me. The generation pipeline in Prajna is built on it, because a lost job there means a family is waiting on a story that never gets made.
Chapter: The outbox pattern
7. Timeouts are per hop. Users wait end to end.
Service A calls B with a 2 second timeout. B calls C with its own 2 seconds. C calls D with another 2. If D hangs, the user can wait 6 seconds even though A promised 2.
A deadline fixes the maths. You pass along the time that’s left, not a fresh timeout at every hop. If only 100 ms remain, the next service shouldn’t start work it can’t possibly finish in time.
Retries multiply too. If two layers each try three times, one failure at the bottom turns into 9 requests. With four layers it’s 81. A retry budget caps retries as a share of total traffic, so a struggling service doesn’t get buried by its own callers trying to help.
Chapter: Timeouts, deadlines and retry budgets
8. A queue doesn’t fix a slow consumer. It hides it.
Requests come in at 5,000 a second and the database can handle 1,000. “Just add a queue” only changes how you fail. An unbounded queue eats memory until the process dies. A bounded one fills up, and long before it’s full, every request is waiting behind thousands of others.
That second failure is the sneaky one. Errors turn into slowness, and slowness rarely pages anybody.
Backpressure is the slow side telling the fast side to slow down, and the fast side actually listening. A bounded queue that blocks, a 429 the caller respects, TCP shrinking its window. The buffer is there to absorb a spike, not to hide a permanent gap.
Chapter: Backpressure
9. Read replicas can show you the past
A replica is usually a few milliseconds behind the primary. Under load, that can grow to seconds. So a user updates their bio, the write goes to the primary, the page reloads from a replica, and they see the old bio. As far as they’re concerned, the save didn’t work.
The fix is unglamorous. For a short while after someone writes, read their data from the primary. But you only think of it if you remember that a replica is always a little bit in the past.
Chapter: Replication and replication lag
10. CAP is a choice you make during a failure
I’d always heard databases described as “CP” or “AP,” like a personality type. What untangled it for me was going back to what the theorem actually says. When the network splits, you choose between answering with possibly stale data or refusing to answer. When the network is healthy, a well-built system gives you both.
So the useful question is a lot more concrete. When the link between my two regions drops, do I want users to see errors or stale data? For a payment, errors. For a like count, stale is fine.
Chapter: The CAP theorem
The one sentence I’d keep
If I could keep only one line from the whole notebook, it would be this: something will fail between two lines of your code, so decide now what should happen when it does.
Almost every idea above is an answer to that. Retry safely. Record the intent before acting on it. Give up on time. Slow down instead of piling up.
The full notebook lives at trilo-upskill.vercel.app. I wrote it for myself first, so it reads like notes from someone figuring things out. Which is exactly what it is.