How To Debug Bazel Remote Cache Misses Without Guessing

Remote cache misses are where build-system optimism goes to get humbled.

The sales pitch for remote caching is simple: someone already built the thing, so you should not have to build it again. In a healthy Bazel setup, that can be beautiful. CI writes reusable outputs. Developers pull the same branch and get work back from the cache. Expensive generated code, compilation, packaging, and test setup stop burning the same minutes over and over.

Then the cache misses.

At that point, too many teams start debugging from vibes. Maybe it is the toolchain. Maybe it is the Docker image. Maybe somebody changed a flag. Maybe Bazel is weird. Maybe the cache server is haunted. That last one is emotionally satisfying, but not an engineering method.

Bazel remote cache misses are usually not mysterious. They are usually evidence. The trick is to compare the right evidence instead of staring at a low hit rate and guessing.

This article is the practical follow-up to When Remote Build Caching Is Worth It. That piece covered whether the investment makes sense. This one is about what to do when the investment exists, the cache is configured, and the results are not what you expected. It also pairs naturally with Making Local CI Commands Boring Enough for Humans and AI Agents, because cache debugging gets much easier when the repository already has a stable local validation interface.

Start With The Actual Symptom

"The cache is bad" is not a symptom.

Start with a smaller statement:

A specific target misses when it should hit.
CI writes outputs but developer machines do not reuse them.
Two developer machines miss each other's work.
A target hits on Linux but misses on macOS.
Cache hits disappeared after a toolchain change.
A clean build hits, but an incremental workflow does not.
Tests hit but compile actions do not, or the reverse.

That level of precision matters because Bazel caching is action-based. Bazel does not cache "the build" as one blob. Its remote cache stores action result metadata and output artifacts. The official remote caching docs describe the basic model: actions have inputs, output names, command lines, and environment variables, and the cache uses action results plus a content-addressable store for the outputs.

If an action misses, something about the cache lookup did not match, the result was not available, the cache could not be reached, or the action was not eligible for reuse. Your job is to narrow which one.

The first pass should answer three questions:

Is Bazel talking to the remote cache?
Is the writer actually uploading the outputs you expect?
Is the reader building the same action with the same relevant inputs?

Do not skip the first two because they feel too basic. Plenty of cache "debugging" sessions eventually turn into "the local command was missing the same .bazelrc config as CI."

Read The Status Line, But Do Not Worship It

Bazel tells you some useful information at the end of the run. The remote cache debugging docs show output like:

INFO: 7 processes: 3 remote cache hit, 4 linux-sandbox.

That is a good first signal. It tells you remote cache hits happened, and it tells you which actions ran locally. It is not enough to diagnose the miss.

There are a few traps here:

Local cache hits are not the same as remote cache hits.
A high hit rate on cheap actions may not save much time.
A low hit rate after a source change may be perfectly reasonable.
Different target sets can make hit rates impossible to compare.
The status line does not explain which input changed.

Use the status line to notice a problem, not to finish the investigation.

I like to capture a small before-and-after note whenever a cache miss matters:

Target: //services/payments:payments_test
Writer: CI on main at abc123
Reader: developer laptop at abc123
Expected: remote hit for Java compile actions
Actual: 0 remote hits, actions executed with linux-sandbox

That note prevents the conversation from drifting. You are no longer debating whether "Bazel caching works." You are debugging a target, a commit, an execution environment, and an expected action class.

Confirm The Reader And Writer Are Actually Compatible

Remote caching is a two-party system. One Bazel invocation writes outputs. A later invocation reads them. If those two invocations do not agree on the important parts, misses are the correct behavior.

Check the boring configuration first:

Are both invocations using the same --remote_cache endpoint?
Is the writer allowed to upload results?
Is the reader accidentally running with --remote_upload_local_results=false only in a place where you expected it to write?
Is the target tagged with no-remote-cache?
Are credentials valid for both read and write?
Are warnings about remote cache reads or writes present in the output?

Bazel's remote caching guide calls out --remote_upload_local_results=false as the flag for read-only remote cache usage. That is a perfectly sensible configuration for developer machines when CI is the trusted writer. It is a terrible configuration if you are trying to prove that one local machine can populate the cache for another.

Also check .bazelrc layering. A repo-level .bazelrc, user-level config, CI flags, wrapper scripts, and environment-specific startup options can produce subtly different commands while everyone insists they ran "the same build."

For serious debugging, write down the exact command line, including config expansions. If your team hides Bazel behind make, just, or a script, make sure the wrapper can print the Bazel invocation it ran. The wrapper should make the normal path easier, not make the debugging path opaque.

Compare Actions, Not Feelings

When a miss is not explained by connectivity or obvious configuration, move to action comparison.

Bazel can write execution logs. The command-line reference documents --execution_log_json_file, which emits executed spawns as newline-delimited JSON, and also points to compact and binary execution log formats. For human-driven debugging, JSON is often the easiest starting point even if the compact format is cheaper at scale.

A simple workflow looks like this:

bazel clean
bazel test //path/to:target \
  --config=remote-cache \
  --execution_log_json_file=/tmp/writer.json

bazel clean
bazel test //path/to:target \
  --config=remote-cache-readonly \
  --execution_log_json_file=/tmp/reader.json

Then compare the actions that should have matched.

You are looking for differences in:

Arguments.
Environment.
Input digests.
Tool paths.
Execution platform.
Working directory assumptions.
Output paths and names.
Param files.
Test-related settings.
Configuration transitions.

Do not start by comparing the entire log by eye. Pick one high-value missed action. A Java compile action, TypeScript transpilation step, code generation action, container packaging step, or expensive test setup action is a better debugging subject than a tiny copy action.

The question is not "Why was my build slow?" The question is "Why did this one action have a different cache key or no reusable result?"

Once you can answer that, the broader pattern usually appears.

The Usual Miss Causes

Most remote cache miss investigations eventually land in one of a few buckets.

Different Targets Or Different Configs

This is the embarrassing one, so check it early.

CI may build //..., while the developer builds //service:all. The repo may have a --config=ci path that uses a different platform, different test flags, or different incompatible flags than local development. Someone may have added --define, --features, or a Starlark build setting that changes the action inputs.

The cure is not cleverness. Capture the target pattern and effective flags for both runs. Make the intended read/write configurations explicit.

Environment Leaks

Bazel is good at making builds deterministic when rules declare their inputs. It is not magic around every ambient machine detail.

The remote caching docs warn about environment variables leaking into action definitions and call out --action_env as one mechanism that affects what environment is included. If $PATH, locale settings, home-directory paths, temporary directories, credentials, or host-specific values get into actions, cache reuse across machines will suffer.

The practical test is straightforward: compare the action environment from the execution logs. If the action key changes because one machine's environment is different, decide whether that environment really belongs in the action.

Sometimes it does. Often it does not.

Toolchains Outside The Workspace

A remote cache assumes that identical action inputs produce identical outputs. If an action calls /usr/bin/clang, /usr/bin/python, or some other host tool that Bazel is not really tracking, two machines can appear compatible while actually using different tools.

This can create both misses and scarier correctness risks. The fix is usually to move toward declared, pinned toolchains and repository-managed dependencies. That work may feel slower than flipping another cache flag, but it pays back in trust.

Generated Files And Non-Hermetic Inputs

Generated files are wonderful until they smuggle in timestamps, absolute paths, random IDs, machine names, network results, or unordered output.

If a code generator produces different output on each run, remote caching will either miss constantly or make you nervous for good reasons. Compare generated outputs from two supposedly identical builds. If they differ, fix the generator or isolate the action from remote caching until it deserves trust.

Platform And Execution Strategy Drift

Remote cache reuse depends on the action being compatible with the platform that produced the output. Linux CI and a macOS laptop should not blindly share everything. x86 and ARM may differ. Containerized and non-containerized execution may differ. Sandbox strategy changes can reveal undeclared inputs.

Platform drift is not a cache failure. It is the cache telling you the world is not as uniform as you hoped.

Debug High-Value Misses First

Not every miss deserves investigation.

Some actions are cheap. Some are expected to change frequently. Some are configuration-specific enough that reuse is not worth chasing. The trap is trying to make the hit rate beautiful instead of making the build faster and more trustworthy.

Start with actions that are:

Expensive in wall-clock time.
Frequent across CI and developer workflows.
Stable enough that a hit is realistic.
Important enough that wrong reuse would hurt.
Representative of a broader class of work.

For example, a 30-second compile action that runs hundreds of times per day is worth attention. A 200-millisecond stamping action that changes on every commit is not your first problem.

The best cache debugging sessions end with one of three outcomes:

The action now hits because configuration or inputs were fixed.
The action is intentionally excluded because reuse is unsafe or low-value.
The team learned that a deeper hermeticity or toolchain issue needs a real project, not a quick flag change.

All three are useful.

Make The Investigation Repeatable

Once a team has debugged cache misses twice, the third time should not start from scratch.

Document a small runbook:

Confirm the target and commit.
Capture writer and reader commands.
Check remote cache warnings.
Confirm read/write policy.
Run clean builds with execution logs.
Compare one missed high-value action.
Classify the miss.
Decide whether to fix, exclude, or ignore.

You can put this in a build README, an internal developer productivity guide, or a tools/cache-debug wrapper that produces logs in known locations. I am fond of wrappers for this kind of work because they keep the sharp flags out of tribal memory.

But keep the wrapper honest. It should print what it is doing. It should avoid uploading local results unless that is the point of the test. It should make it hard to accidentally poison a shared cache from an experiment.

Watch For Security And Trust Boundaries

Remote cache debugging is not only about performance.

The cache stores build outputs. Sometimes those outputs are binaries, generated source, packaged artifacts, logs, or test outputs. Bazel's docs explicitly remind teams to take care with who can write to the remote cache. CI-as-writer and developers-as-readers is a common starting point because it gives the team a clearer trust boundary.

If a miss investigation tempts you to let every laptop write every result to a shared cache, slow down. That may be fine in some teams. It may be reckless in others.

Ask:

Who can write?
Who can read?
Are credentials scoped?
Can a bad result be purged?
Are release artifacts built with stricter rules?
Do logs or outputs contain sensitive material?

Performance work that quietly weakens your supply chain is not developer productivity. It is debt with a faster progress bar.

The Practical Checklist

When a Bazel remote cache miss matters, use this checklist:

Name the exact target, commit, writer, and reader.
Confirm --remote_cache is set in both places.
Check for remote read/write warnings.
Confirm read-only versus read/write intent.
Compare the effective Bazel flags and configs.
Start from a clean build when measuring remote hits.
Capture execution logs for both invocations.
Compare one expensive missed action.
Look at arguments, environment, inputs, tool paths, and platform.
Check for no-remote-cache tags or non-cacheable behavior.
Decide whether the action should hit, should be fixed, or should be excluded.

That list is intentionally boring. Boring is how you turn cache debugging from a campfire story into engineering work.

The Bottom Line

Bazel remote cache misses are not solved by staring at the hit rate and hoping the next run looks better.

Start with the exact symptom. Confirm the cache is reachable. Make the writer and reader configuration explicit. Compare actions. Treat environment, toolchains, generated outputs, and platform differences as evidence.

Remote caching rewards teams that make builds honest. Debugging cache misses is one of the fastest ways to find where the build is still relying on folklore.

For more engineering craft notes like this, visit Slaptijack.