Make a Slow App Measurably Faster

This is not the guide for a prototype. This is the guide for a system that works: it ships, it has customers, it makes money, and every quarter it is a little slower than the last. The p95 on the main endpoint has crept from 200ms to 900ms over eighteen months and no single commit did it. The database bill grows faster than the user count. A feature that would have taken two days last year takes a week now, because the code that used to be one thing has become four things wearing a trench coat. Nobody can point at the cause, because there isn’t one — there are two hundred, and most of them were reasonable at the time.

The reflex response is a rewrite, and the second reflex is a week of “performance work” where everybody optimizes the part of the code they personally find annoying. Both fail the same way: they spend real effort on code that was never the problem. Donald Knuth’s line about premature optimization is quoted to death and usually misquoted — the point was never that optimization is bad. It is that you cannot know which 3% of the code matters until you measure, and unmeasured effort goes into the other 97% by default.

So this guide runs on two rules that gate everything else: measure before optimizing, pin before restructuring. The profiler decides what you work on. The safety net decides what you are allowed to touch. Each phase is driven by a skill: Working with Legacy Code builds the net and the baseline, Clean Architecture and A Philosophy of Software Design fix the structure that made it slow to change, Refactoring reshapes the hot paths safely, System Design and Data-Intensive Apps find and fix the real bottleneck, Release It! keeps it stable under stress, and The Pragmatic Programmer turns the result into budgets that hold.

A change that does not move the number it targeted is not an improvement you keep. It is a diff you revert.

Phase 1 — Pin the behavior, then measure the baseline

Two things have to exist before anything changes, and neither is a code change. The first is a safety net: characterization tests over the code you intend to touch. The Working with Legacy Code skill defines legacy code as code without tests, which makes a hot path with no coverage legacy code no matter how recently it was written. Its recipe is mechanical — call the code, assert something you know is wrong, run it, read the real output from the failure, pin that value. You are not judging the behavior, you are photographing it, so that when you restructure, an unintended change announces itself as a red test instead of as a 3am page.

The obstacle is always the same: the hot path constructs its own dependencies, so you cannot substitute anything. The skill’s answer is seams — the least invasive being Parameterize Constructor with a production default, so every existing caller compiles untouched while a test can pass in a fake.

The second thing is a baseline with real numbers. Not “it feels slow” — p50, p95 and p99 per endpoint under production-like load, the slow query log, an actual CPU or wall-clock profile of the worst offender, and the current infrastructure cost. Write them down with the date. Everything from here cites this document, and the single most common failure of performance work is that nobody recorded the before, so every later claim is a memory.

Phase one ends with two artifacts: a map of what is pinned and what is not, and a numbers document. Both gate everything after. If a later phase wants to touch code outside the safety net, it pins it first or it does not touch it.

Phase 2 — Find where the boundary drifted

With the net in place, look at structure before you look at speed — because in a system that grew, the structural drift and the slowness are usually the same fact seen from two angles. The Clean Architecture skill’s single rule is the Dependency Rule: source-code dependencies point inward, from frameworks and drivers toward use cases and entities, never outward. A system that once obeyed it and now does not will have business rules that import the ORM, entities that know about HTTP, and use cases that cannot be exercised without booting the framework.

That drift has a direct performance cost, and it is not abstract. When business logic lives inside an ORM model, the code that decides what to fetch is tangled with the code that decides how, so nobody can see that a rule fires one query per item in a loop. When a use case can only run inside a request, you cannot profile it in isolation, so you profile the whole request and learn nothing specific. Restoring the boundary is what makes the next four phases possible.

The skill is also the right tool for deciding what not to do. A boundary is a cost as well as a benefit, and a system that grew slow does not need every module inverted behind an interface. Draw the boundary where the volatility is — around the things you have replaced or expect to replace — and leave the stable parts alone. Over-boundarying a system in the name of cleanliness is its own performance problem, paid in indirection.

Phase 3 — Check whether the structure itself is the complexity

Structure has a second axis, and A Philosophy of Software Design supplies it. John Ousterhout’s central idea is deep modules: a module’s value is its interface area divided by the functionality it hides. A deep module presents a small interface over substantial work. A shallow one presents an interface nearly as complicated as the thing it wraps, which means it adds cost without removing any.

Codebases that grew under time pressure accumulate shallow modules — the pass-through service that calls one repository method, the “helper” that takes eight parameters and switches on three of them, the layer that exists because the previous architect said layers were good. Each one is a place where a caller must understand two things instead of one, and collectively they are why a two-day feature now takes a week. The skill’s related idea, information leakage, catches the other half: when the same design decision appears in three modules, changing it means changing all three, and eventually someone changes two.

Ousterhout’s rule of tactical versus strategic programming is the one to internalize for the rest of this guide. Tactical programming optimizes for getting this change done; strategic programming spends a small, deliberate extra effort each time to keep the design good. Every slow, tangled codebase is a record of tactical decisions that were each individually correct. The point of this phase is not to atone for that but to decide, explicitly, which parts you are switching to strategic mode for — usually the hot paths, and usually only those.

Phase 4 — Reshape the hot paths in named steps

Now you may change the shape of the code the profiler pointed at, and the Refactoring skill governs how. Martin Fowler’s definition matters here more than usual: a refactoring is a behavior-preserving transformation applied in small named steps with the tests green between each one. Not “cleaning up”. Not “while I’m in here”. A named move, applied, tested, committed.

For hot paths the useful catalogue is small. Extract Function to name a step so you can measure it separately. Replace Loop with Pipeline, or the reverse, when one allocates and the other does not. Introduce Parameter Object when a signature has grown to seven arguments and half the callers pass defaults. Replace Conditional with Polymorphism when a switch on type is executed a million times a day. Move Function when logic sits in the wrong module and forces a data round trip to get what it needs.

The discipline that makes this safe is the separate commits rule: a commit either changes structure or changes behavior, never both. When a commit is purely structural and a test goes red, the test is telling you unambiguously that you broke something, and you can revert one small thing. When the two are mixed, every failure is a debate. Optimization commits are a third category — they change behavior only in timing, and they cite the baseline they target.

Phase 5 — Ask the measured load what the bottleneck is

Now go back to the numbers. The System Design skill’s contribution is back-of-the-envelope honesty: estimate before you architect, and compare the estimate to what you measured. Requests per second at peak, payload sizes, working-set size against available memory, the latency budget of a single request split across its components. Most “we need to scale” conversations end quietly when someone works out that the system handles 40 requests per second and a single modern machine handles thousands.

The output of this phase is a ranked list of bottlenecks with the cheapest viable fix beside each. The order that usually falls out is unglamorous and correct: fix the query before you add a cache, add a cache before you add a queue, add a queue before you add a service. Each step up that ladder buys latency at the cost of a new failure mode and a new thing to operate — a cache introduces invalidation and staleness, a queue introduces ordering and retry semantics, a service introduces a network call that can hang.

Caching deserves a specific warning because it is the most over-applied fix in this phase. A cache in front of a slow query hides the slow query; it does not remove it, and the first cold cache after a deploy or an eviction gives you the original latency at the worst possible moment, often with a thundering herd on top. Cache what is genuinely expensive and genuinely reusable, after the query itself is as fast as it reasonably gets — not instead of fixing it.

Phase 6 — Fix the data layer, where the latency actually lives

In most applications that got slow, the answer is here, and Data-Intensive Apps is the skill for it. Martin Kleppmann’s material covers storage engines, indexing, replication, partitioning and transaction isolation — and the practical top of the list is short and boring.

N+1 queries are the single most common cause of an endpoint that got slower as the product grew: one query for a list, then one per item, so latency scales with content and nobody notices until a customer has 500 of something. Missing or wrong indexes are the second: a query that scans a table which had 10,000 rows at launch and has 10 million now. Unbounded result sets are the third — an endpoint with no pagination that was fine when the biggest account had 40 records. Over-fetching is the fourth: selecting every column, including the large text one, to render a list that shows two fields.

Then the correctness half, which matters the moment you start making things concurrent to make them fast. Isolation levels are not a detail: read-committed, the default in most databases, permits lost updates on read-modify-write cycles, which is exactly what a naive counter, a balance, or an inventory decrement is. If your fast new path lets two requests do that at once, you have traded latency for a data bug that appears under exactly the load you were optimizing for.

Phase 7 — Stay fast when something else is slow

A system tuned to be fast in the happy path can still fall over the first time a dependency degrades, and it usually fails worse than a slow system would. Release It! is the skill for this, and Michael Nygard’s central observation is that the worst failures come from things that are slow rather than down. A dead dependency fails fast and you handle it. A dependency answering in 30 seconds holds every thread that called it, and the queue behind them, until your service is unavailable for reasons that have nothing to do with your code.

The patterns are specific and cheap. Every outbound call gets a timeout — an unbounded call is a resource leak with a slow fuse. Every critical dependency gets a circuit breaker, so repeated failures stop being retried into the ground and start failing fast with a degraded response. Bulkheads keep one slow integration from consuming the whole thread or connection pool. And back-pressure with a bounded queue is what stops a burst from turning into an out-of-memory crash.

This phase is not judged on latency. Adding a circuit breaker will not make the p95 lower on a good day; it is what stops the p95 becoming a timeout on a bad one. Judge it by its own done-when: no unbounded call, no unbounded queue, no shared pool without a bulkhead, and a documented degraded mode for each critical dependency.

Phase 8 — Turn the gains into budgets that hold

Performance work that is not defended decays, because the next feature is written by someone who does not know what any of the numbers cost to earn. The Pragmatic Programmer supplies the habits that make the gains durable, and the mechanism that matters most here is turning a measurement into a gate.

Write the budgets down as numbers — p95 per critical endpoint, maximum queries per request, maximum payload size — and enforce them in CI, so a regression fails a build instead of arriving as a support ticket in six weeks. Add the query-count assertion to the tests for hot endpoints, because that single check catches the N+1 that a future ORM relationship will reintroduce. Keep the optimization ledger: every change, its baseline, its after, and whether it was kept.

The skill’s broader themes — DRY as being about knowledge rather than code, tracer bullets over big-bang delivery, orthogonality, and knowing what you actually own — are what stop the next eighteen months from recreating the situation you just spent a quarter fixing.

Your checklist

Work down in order; each item assumes the ones above it.

• Behavior pinned. Every hot path you intend to change has characterization tests, and what is not pinned is written down as such.
• Baseline recorded. Latency percentiles, query counts, a real profile, and current cost — dated, under production-like load.
• Boundary restored. Business rules on the hot paths no longer depend on the framework, and you can exercise them without booting it.
• Shallowness removed. Pass-through modules collapsed, leaked design decisions consolidated, hot paths readable enough to optimize.
• Hot paths reshaped. Named refactorings only, tests green between steps, structural and behavioral changes in separate commits.
• Bottleneck identified. A ranked list with the cheapest viable fix beside each, arrived at from measured load rather than intuition.
• Data layer fixed. No N+1 on hot paths, indexes match the real queries, everything paginated, read-modify-write cycles safe under concurrency.
• Degradation handled. Timeouts everywhere, circuit breakers on critical dependencies, bulkheads on shared pools, a documented degraded mode.
• Budgets enforced. Numbers in CI, query-count assertions on hot endpoints, and a ledger where every optimization shows its before and after.

Common mistakes

Optimizing without a profile. The intuition about which code is slow is wrong far more often than it is right, and the code that looks worst is rarely the code that costs most.

Caching instead of fixing. A cache over a bad query is a bad query that now also has an invalidation problem and a cold-start cliff.

Rewriting instead of measuring. A rewrite reproduces the old system’s behavior imperfectly at enormous cost, and the new one is slow in different places you have not measured yet.

Restructuring unpinned code. “Behavior-preserving” is a claim, and without tests it is an unverified one. This is how a performance sprint produces an incident.

Mixing structure, behavior, and optimization in one commit. Every subsequent failure becomes ambiguous and every revert becomes surgery.

Keeping a change that missed its number. If it did not beat the baseline it cited, it is complexity you are now maintaining for nothing. Revert it and write down what you learned.

Scaling out before scaling the query. Adding machines to hide a full table scan is the most expensive way to not fix a problem.

Frequently asked questions

How is this different from the technical-debt guide?

Aim. Refactor a Codebase Buried in Technical Debt is for a codebase that people are afraid to change: the goal is making it safe to touch again, module by module, while the team keeps shipping. This guide assumes the code broadly works and asks a narrower question — where is the time actually going, and what is the cheapest thing that removes it. Structure work appears here as a means to speed, not as the goal. If your complaint is “it is slow”, start here. If it is “we cannot change it”, start there.

Do I need all eight phases, or can I go straight to the database?

You can go straight there, and sometimes it is right — if the profile says one unindexed query is 80% of your p95, fix it today. What phase one is non-negotiable about is having that profile and having the path pinned before you change it. The structural phases earn their place when the same slow pattern keeps coming back: an N+1 is usually a boundary problem, and a hot path nobody can read is a hot path nobody can safely change. Defer them if time is short; record that you deferred them and why.

What does “measure before optimizing” mean in practice for a small team?

Less than you think. It means production-like load rather than your laptop, latency percentiles rather than an average, a query count per request, and one real profile of the worst endpoint — captured once, written down with the date. That is an afternoon, not a project. The average is the part worth insisting on: a mean latency of 300ms can hide a p99 of nine seconds, and the p99 is what your loudest customers experience every day.

Why is the resilience work judged differently from the rest?

Because it is not supposed to make anything faster. Timeouts, circuit breakers and bulkheads change what happens when something else fails, so measuring them against a latency delta on a healthy day would score them as useless. They are judged by their own condition: no unbounded call, no unbounded queue, no shared pool without a bulkhead, and a stated degraded mode for each critical dependency. The same applies to CI budgets, which exist to protect gains rather than produce them.

Our slowness is in the front end, not the server. Does this apply?

Partly. The measure-first and pin-first rules transfer directly, and High Performance Browser Networking is the skill for the transport side — connection setup, protocol overhead, what actually blocks a first render. But if the complaint is a marketing site’s page-load score rather than an application’s endpoint latency, Improve an Existing Website covers Core Web Vitals as part of a broader pass and is the better starting point.

Start with the baseline

The quarter of performance work that produces nothing always starts the same way: with a change. Somebody had a theory, the theory was plausible, and by the time anyone asked what it was supposed to move there was no before to compare against. An afternoon spent capturing percentiles, query counts and one honest profile is what separates that quarter from the one where every change has a number beside it.

Install the library with npx skills add wondelai/skills --all --global and work the phases in order — or hand the whole sequence to the Architecture Optimization journey, which asks you the decision questions at each phase and keeps the baseline, the ledger and the safety-net map in your project’s docs/ folder so the work survives between sessions.