Reliability

The Other Production

Players only see production. Engineers live in the systems behind it. When those systems fail, the product fails later. The two worlds are closer than they look.

Players only ever see one world. They judge the product by what reaches their hands. Everything upstream is invisible to them, and that is entirely reasonable. Engineers work in a different world. Their day can depend on shared systems. Source control. Build pipelines. Shared services. Test and preview environments. When any of these falter, progress stops. Not later. On the spot.

This is the part organisations often miss. These internal systems are production for the people who build the product. If source control is down, it is a production outage. If CI will not run, it is a production outage. If a shared service drifts from its contract because it is being bent into a shape it was never meant to hold, it is a production outage. Players may never touch these systems, but their experience depends on them all the same.

The temptation is to treat internal platforms as second class because they do not appear in customer monitoring. It seems harmless. It rarely is. When the basics cannot be trusted, engineers resort to workarounds. Parallel copies. Manual steps. Guesswork. These patch the day but leak into the product. The debt accumulates quietly, then arrives loudly.

The teams who run these systems feel this tension most. They are asked to deliver production-grade stability, yet the environments they manage are described as non production. The disconnect weakens priorities and excuses unreliability. The honest framing is simpler. Anything that blocks engineers blocks the company. The label on the environment does not reduce the impact.

None of this is easy. Keeping internal systems healthy is hard with twenty five engineers. It is hard with two hundred and fifty. It is hard with two thousand five hundred. Scale changes the shape of the work but not the difficulty. The dependence is universal.

Reliable software begins with reliable internal systems. Treating them as best endeavours only shifts the cost downstream where it spreads into the product itself. A stable development chain is not accessory work. It is the start of everything players feel.

If we want consistency in the world players see, we need consistency in the world engineers inhabit. They are two sides of the same production line. The only difference is which side the audience stands on.

Continue from here

Follow a related thread.

These essays continue the same argument.

Latest in this thread

Keys to the Kingdom

Finding the Keys That Matter

4 min read · Platform Engineering

Local Without Losing Shape

A player in São Paulo should not feel like a guest in a London lobby. Same skeleton, different clothes. The craft is to let the outfit change without replacing the bones every

5 min read · Developer Experience

Continue from here

Follow a related thread.

Selected automatically from shared topics, newest first.

Latest related essay

The Coordination Tax in Your Software Estate

Repository fragmentation turns shared work into internal transactions. Pricing those transactions reveals opportunities to move faster, release capacity and simplify control.

5 min read · Monorepo

Faux InnerSource

A contribution process is not InnerSource merely because another team can submit code. The test is whether repeated contributions build reusable engineering capability or repeatedly enter the same queue.

7 min read · InnerSource

Nx Should Make Code Graphs Cacheable

Nx already owns the Project Graph and distributed cache. Should it make finer Code Graph indexes reusable across the organisation?

6 min read · Monorepo

Browse by question

What are you trying to decide?

Find an argument

Search the shelf.

Search all published essays by title and standfirst.

New arguments, occasionally

Worth sending.

No cadence theatre. No filler. Just new essays when the argument is ready.