How to make your APIs more reliable in production

An order gets paid for but never reaches the warehouse system. A customer sees an error because the payment provider is taking a little longer than usual. Or an integration books the same order three times, because the network dropped for a second and the sending system politely tried again. These are not small technical hiccups. They cost revenue, they cost support hours, and they cost you trust with customers who did nothing wrong.

So the question of how to make your APIs more reliable is rarely just a question about code. The answer sits in the whole chain: your application, the infrastructure underneath it, the systems of the partners you connect to, and the way all of that is managed day to day.

And no, a reliable API is not an API that never fails. That does not exist, certainly not when four parties sit in the same request path. Reliable means something else: failures follow a route you thought about in advance, they are easy to trace, and they do as little damage as possible to the process behind them.

Start with the process, not with the endpoint

Plenty of organisations measure exactly one thing: is the endpoint up? If it is, the dashboard is green. But an API can be perfectly reachable and still be useless to the business. An order endpoint that returns an error within 200 milliseconds counts as available, and your customer is still stuck.

Three questions get you further. Which process depends on this integration? What happens if that process stalls for an hour? And how quickly does it need to run again before it genuinely hurts? For a product feed, an hour of delay is usually survivable. For a payment, a stock update or access control, a few minutes is already too long.

Then turn those answers into numbers you can hold each other to: the share of requests that has to succeed, the response time you accept, and the window within which stuck messages still need to be processed. It sounds formal, but on a bad day it saves you an hour of arguing. You know not just that something is failing, but whether it falls inside or outside what was agreed, and who is up next.

Measure signals that actually mean something

A green tick on a status page tells you very little. Measure successful requests, errors broken down by type, response times and volume. Measure what happens after the API as well. An order can be accepted cleanly while the background job that should push it onwards dies an hour later on a missing field.

Give every request its own ID and put it in all your log lines. Support can then follow an order from the webshop to the API, from the API to the database and from the database to the carrier's system. Without that thread running through it, an incident turns into a lot of searching and guessing. We go into setting up that foundation in more detail in our piece on uptime and server monitoring.

Clear contracts prevent most surprises

In practice, a lot of integration failures have nothing to do with a broken server. They come from expectations that were never written down anywhere. One system wants an amount as a number, the other sends "12.50" as text. A supplier makes a field mandatory and forgets to mention it. Or someone starts sending dates in a different format.

So treat your specification as a contract. Which endpoints exist, which fields are required, which error codes can a consumer expect, and what happens when things are slow? Versioning is part of that. If you change something that can break existing integrations, run the new version alongside the old one for a while. Consumers then migrate at a moment that suits them, rather than on a Tuesday morning when nothing works any more.

Validate input at the edge of your system, and be specific in the error: which field, which value, and what it should have been. A bare 500 for a malformed email address helps nobody. It also pollutes your monitoring, because a customer's typo ends up in the same statistic as a real outage.

Then there is idempotency, which is the third example from the top of this article. A call should be safe to send again without an order, a payment or a stock change landing twice. An idempotency key per action, with the result stored, solves that: if the same key shows up again, you return the earlier response instead of creating a second order. That is exactly the situation where a client got no answer, has no idea whether it worked, and tries once more.

Absorb temporary failures on purpose

Networks stutter, DNS breaks occasionally, and external services run maintenance at the worst possible moment. The question is not whether a dependency goes down, but what your system does while it is down. Wait indefinitely and your own workers fill up, which means you have adopted somebody else's outage. Give up at the first hiccup and you throw away work that would have succeeded half a second later.

Timeouts are your first line of defence. Give every outbound call a maximum wait that fits the process it serves. A customer waiting on a page needs a far tighter limit than a sync that runs at three in the morning. One default value for everything is almost always wrong: too generous for the front end, too tight for the batch job.

Retries come next, but with a policy. Retry temporary problems such as a timeout, a 429 or a 503. Do not retry a validation error or a rejected token, because repeating the call will not change the outcome. Increase the wait between attempts and add some variation, so you do not finish off a struggling service with a wave of simultaneous retries.

If a process does not have to finish immediately, a queue is often the better answer. The API accepts the request, stores it durably and processes it asynchronously afterwards. Messages that do not make it after a handful of attempts move to a separate error queue where someone can inspect them and resubmit. Nothing disappears, and that is the whole point.

There is a condition attached to that, though. Asynchronous processing means the user does not see the final result right away. So return a status that is honest, something like "request received, we will let you know once it has been processed", and make sure that notification actually arrives. Answering fast is only worth something if it is clear what the answer means.

Keep spikes and chain problems out

Almost any system copes with a normal working day. Things break on the outliers: a mailing that goes out at ten in the morning, a bulk import from a new customer, an integration that sends thousands of requests a minute because of a bug, or a partner that happens to be four times slower than usual today.

Rate limiting stops a single consumer from claiming all your capacity. Set limits per API key, per customer or per endpoint. It is not a punishment, it is protection for everyone else on the platform. Do tell the consumer in the response when retrying makes sense, so they are not left guessing.

Also limit the damage when an external service falls over. A circuit breaker stops calling for a while as soon as it is clear that nothing sensible is coming back. Instead of letting hundreds of requests pile up, you return a controlled error or switch to a limited alternative route. After a cooling-off period, the system carefully checks whether the service is back.

Caching helps for data that does not need to be correct to the second, like product categories, country lists and configuration. For prices, stock levels and permissions it is risky, because stale data there does real damage. So always weigh the cost of a slow response against the cost of a wrong one. Sometimes waiting is the smaller evil.

See it before your customer does

Monitoring is only useful once it answers one question: does someone need to act right now? Alerting on every individual error produces alert fatigue within a week, and after that nobody looks. Alerting when the share of failed payments stays above your threshold for ten minutes straight is genuinely useful. Your logs can contain everything; your alerts cannot.

Combine technical checks with functional ones. An endpoint can cheerfully return a 200 with an empty or half-filled body. So run a real action periodically: a test order, a login, a stock lookup. That way you check the chain the way a user experiences it, instead of only the outside of it.

Make sure an alert leads somewhere useful: recent errors, the systems involved, response times and volumes. Add tracing where you can, so a single correlation ID follows a request through every service. That is the difference between naming the cause in ten minutes and three teams each building their own theory in parallel.

Write down the process around it too. Who assesses an alert outside office hours? Who calls the customer or the supplier? When do you escalate? And who checks, after recovery, whether the messages that got stuck have actually been processed? That last step is the one most often skipped, and it is exactly where quiet data damage happens. Technology without ownership stays fragile.

Change in small steps, even when it is urgent

Old code is rarely the culprit. A good share of incidents start with a change: a quick hotfix, a dependency update, an edited configuration, a certificate that was renewed but not yet rolled out everywhere. That is often how those unwritten expectations bite as well, because the change breaks an agreement nobody ever made explicit. So keep deployments small, repeatable and reversible.

Test more than your own API. Test the agreements with the systems hanging off it. Contract tests show you when a change breaks an existing consumer, before that consumer discovers it themselves. Roll out to a slice of traffic before you switch everything over. And keep the rollback plan current, so nobody has to improvise during an incident with an angry customer on the phone.

Finally, write down what currently lives in somebody's head: which external systems are critical, where keys and certificates live and when they expire, which queue you check first during an outage, and who has access to what. This is not bureaucracy. It stops recovery from depending on the one colleague who happens to be on a plane right now.

One party that sees the whole stack

As soon as development, hosting, monitoring and support are spread across different parties, grey areas appear. The host sees no server problem, the developer points at the external API, and the supplier of that API only starts looking once a complete ticket has been filed. Meanwhile your process is stalled and nobody connects those three observations.

A party that knows both the application and the infrastructure underneath it can make that connection. At LJPc we deliberately keep those layers together, whether the issue is a bug in an integration, a capacity shortage, or a background job that has been stuck since Friday evening.

If you want to improve something, start small. Pick one integration where real money or real customers are involved, map out what happens when it fails, and arrange things so that during the next outage you know within minutes what is broken, who is acting, and how the stuck messages get put right. That gives you a better API, and more importantly a calmer operation.