A deployment should not turn an ordinary request into a mystery halfway through a database write. Yet a service can pass its health checks, survive load testing, and still drop useful work every time someone replaces a process.
Graceful shutdown is the agreement between the application and the machinery that stops it. That agreement needs an admission boundary, a deadline, and a recovery story. “Handle SIGTERM” is a starting point. It does not tell you whether the last checkout completed, whether a background job will run twice, or whether a WebSocket client knows how to reconnect.
The goal is bounded completion: give accepted work a reasonable opportunity to finish, stop accumulating new work, and leave anything unfinished in a state another process can understand.
Separate planned termination from dependency failure
During a dependency outage, the service is still expected to accept some useful work and enforce its runtime failure policy. During planned termination, this particular instance is leaving service. Keeping it busy forever defeats the point.
The dependency-failure guide covers timeouts, retries, fallbacks, and ambiguous writes. Those controls remain relevant while draining, but shutdown adds a new constraint: the process has a finite remaining lifetime.
Start with three questions:
- What counts as accepted work: an admitted HTTP request, an acknowledged job, or a durable operation record?
- Which work can finish inside the shutdown budget?
- What evidence lets another instance recover after the deadline?
Different answers may be appropriate for a read-only lookup, a payment request, and an hour-long export. Pretending they share one lifecycle is how a tidy shutdown handler becomes a production bug.
Close admission without forgetting routing propagation
Removing an instance from a routing pool is a distributed change. The application, service discovery, proxies, and clients do not all observe it simultaneously. Existing connections can also carry additional requests.
Define an explicit transition from serving to draining. Make it safe to trigger more than once, and use it to stop new business work at the application boundary as well as through the router. Keep enough of the process alive to finish already accepted requests and expose useful shutdown state.
There is a tradeoff during routing convergence. Immediate refusal can produce visible errors for callers still sent to this instance. Briefly continuing to serve can reduce those errors, but extends the set of work that must drain. Choose and test a bounded convergence policy rather than hiding an arbitrary sleep in a shell script.
For example, a service might mark itself unavailable to new routing, serve during a short measured convergence window, then close admission and drain. Another might reject immediately and rely on a verified upstream failover policy. Neither policy gives permission to replay an unsafe write blindly.
In Kubernetes, terminating endpoints are marked not ready for regular traffic. That endpoint state is part of the shutdown mechanism, not proof that every routing component has already stopped sending traffic. Test your ingress, service mesh, and client connection behavior as deployed.
Spend one termination budget
The deadline outside the application wins. If a supervisor kills the process after 30 seconds, a 45-second application drain timeout cannot create another 15 seconds.
The Kubernetes container-hook documentation makes an easily missed detail explicit: the termination grace period includes both the preStop hook and the container's normal stopping time. A hook that spends most of the allowance waiting leaves little time for application cleanup.
Here is an illustrative 40-second shutdown budget. These are design allocations to validate, not recommended defaults:
| Time (seconds) | Intended behavior |
|---|---|
| 0–5 | Begin withdrawal; stop job intake; allow bounded routing convergence. |
| 5–30 | Close request admission; let accepted requests finish. |
| 30–35 | Cancel remaining work and record recoverable outcomes. |
| 35–40 | Close resources, flush bounded telemetry, retain a safety margin. |
If a hook consumes the first five seconds, the application cannot restart this clock when it receives its stop signal. Coordinate the budgets across layers. Use monotonic elapsed-time measurement inside the process, and avoid restarting the allowance when a second shutdown signal arrives.
Measure actual shutdown duration under load. Include slow dependency calls and resource contention; a shutdown that works only when idle tells you very little.
Drain requests before closing their dependencies
An accepted request still needs its database pool, outbound client, and authentication material. Closing shared resources before handlers finish turns graceful shutdown into a carefully scheduled outage.
A useful application sequence is:
- Enter draining state and stop admitting new work at the chosen boundary.
- Wait for tracked active work within the remaining budget.
- Request cooperative cancellation for work that exceeds its allowance.
- Wait briefly for cancellation and durable cleanup.
- Close shared resources after their users have finished, or record forced termination if they have not.
Track work at the level that matters. Counting TCP connections is insufficient when a connection carries multiple requests. Detached tasks started by handlers also need an owner; otherwise the request counter reaches zero while important work is still running.
Go's http.Server.Shutdown illustrates the distinction. It closes listeners and idle connections, then waits for active connections to become idle. Give it a bounded context and keep the program alive until the shutdown coordination completes. A context timeout returns an error; it does not by itself implement the application's policy for abandoned work.
Keep the shutdown coordinator separate from the signal-canceled context used to tell ordinary workers to stop. Otherwise the graceful wait may begin with an already canceled deadline and return immediately.
Give long-lived connections their own exit
A streaming response or WebSocket may be healthy precisely because it never becomes idle. Waiting for every connection to end naturally can therefore mean waiting forever.
Specify protocol-aware behavior:
- Stop creating new sessions when draining begins.
- Notify existing clients when the protocol supports it.
- Allow a bounded interval for reconnecting or finishing a unit of work.
- Define how clients resume without losing or duplicating application events.
Go's shutdown API does not close or wait for hijacked connections such as WebSockets. Applications must coordinate those separately. That is a useful reminder to inspect framework promises instead of assuming “graceful” covers every transport.
For an export stream, decide whether a reconnect starts over, resumes from a durable cursor, or retrieves an already completed artifact. A process-local byte count is not a recovery mechanism once the process is gone. Test a client disconnect during the transition, not just the server's happy-path notification.
Treat background jobs as owned work
Stop fetching jobs early. Prefetching a large backlog while the worker is leaving makes the shutdown problem larger and delays other workers from doing useful work.
Separate jobs that can finish quickly from jobs that need checkpoints, leases, or durable rescheduling. A detached thread is still process-owned work unless responsibility has actually moved elsewhere. Returning an HTTP success response does not make that thread durable.
For AMQP 0-9-1 consumers, RabbitMQ documents automatic requeueing of unacknowledged deliveries when a channel or connection closes. Its consumer guide distinguishes canceling a subscription from dealing with deliveries already in flight. Stopping intake and completing ownership transfer are separate steps.
Consider a worker that writes a result, then dies before its acknowledgment reaches the broker. Redelivery may repeat the job. Make the side effect idempotent, or retain a durable operation identity that lets the next worker recognize completion. Shutdown cannot manufacture exactly-once execution across independent systems.
Do not acknowledge unfinished work just to make the queue look clean. If a job must be abandoned, use the queue's documented recovery behavior and verify it in a representative environment. A graceful process exit is not a successful business outcome if it silently loses the task.
Preserve evidence when the deadline wins
Some accepted work will exceed the budget. Plan for that outcome without labeling every interrupted operation a safe failure.
For a write, the database may have committed even if the response never reached the caller. Keep an operation identifier and a way to check the durable result. Retrying should follow the operation's semantics, not the assumption that a closed socket means nothing happened.
Emit a compact shutdown record with the trigger, phase durations, accepted work remaining, deadline outcome, and recoverable operation references. Avoid sensitive payloads. Record forced exits in deployment monitoring so repeated deadline exhaustion becomes visible instead of disappearing into routine rollout noise.
If shutdown regularly consumes the whole allowance, investigate the work model. Increasing the grace period may be appropriate, but it can also conceal an unbounded handler, an enormous prefetch window, or a missing recovery mechanism.
Verify the contract through the real traffic path
A local signal test is useful, but it does not exercise the router or supervisor. Run a bounded test in a representative environment and capture both client outcomes and server-side state.
Include these cases:
- A short request finishes after draining starts.
- New work arriving during routing convergence follows the documented policy.
- A slow request reaches the deadline and leaves an inspectable outcome.
- A write commits immediately before its client connection disappears.
- A worker stops intake and an unfinished delivery becomes available again.
- A stream reconnects or resumes according to its documented behavior.
- Cleanup encounters a stalled dependency without exceeding the outer deadline.
The incident-rehearsal guide can help clarify who observes and stops a test; discussion alone does not prove the runtime behavior.
A service is ready to stop when its accepted work has either completed or has a credible recovery path. Make that condition observable, keep the wait finite, and make routine deployments boring for callers. More practical engineering guidance lives at Slaptijack.