How To Run a Focused Terminal Investigation Without Losing the Evidence

Published · Programming

The terminal is a superb place to investigate an unfamiliar system. It is also a terrible place to keep a durable record by accident. A useful command disappears into scrollback, the output that changed your mind is buried under a second search, and an hour later you have a conclusion but not the evidence that would let another engineer challenge it.

That is a problem in incident response, debugging, and ordinary maintenance. The goal is not to create a transcript of every keystroke. It is to make the small chain of evidence recoverable: what question you started with, what you looked at, what the system actually said, what you infer from it, and what still needs a decision.

Start with one question and a stopping condition

Before opening six panes, write a question in a scratch file or issue comment. Good questions are falsifiable: “Which configuration selects this backend?” or “Did this deploy change the request path?” “Find the bug” is not a question; it is an invitation to collect attractive facts forever.

Give the investigation a stopping condition too. For example: identify the owner and affected call path, reproduce the failure locally, or collect enough evidence to choose between two safe fixes. A stopping condition protects the team from an investigation that quietly turns into redesign work.

Make an evidence directory before commands get interesting

For a repository investigation, create a small, ignored working area outside the change itself:

mkdir -p /tmp/session-loader-investigation
cd /tmp/session-loader-investigation
git -C /path/to/repo status --short --branch > repo-state.txt
date -u +'%Y-%m-%dT%H:%M:%SZ' > started-at.txt

Use files with descriptive names for results worth preserving. Redirect output when the exact version matters, and record the command above it when it does not fit naturally in shell history:

{
  echo '$ rg -n "SessionLoader|negative cache" src tests'
  rg -n "SessionLoader|negative cache" /path/to/repo/src /path/to/repo/tests
} > call-sites.txt

This is deliberately boring. It avoids a common failure mode of screenshots and terminal logs: reviewers can see the result, but cannot tell what generated it. It also separates reusable evidence from credentials, customer data, access tokens, and noisy full logs that should never enter a ticket.

Keep raw output, then add a short observation

Raw output is not the same as a conclusion. Next to each saved command, add a plain-language observation in notes.md:

## 14:10 — cache write path

Evidence: `call-sites.txt` shows the loader writes a cache entry after a null
lookup. The existing regression test asserts that behavior.

Inference: changing the loader may affect intentional negative caching, not
just the reported miss.

Question: what TTL and consumer behavior make that entry safe?

The distinction matters. “The loader causes the bug” often joins an observed line of code, a causal theory, and a proposed fix into one unreviewable sentence. Labeling evidence, inference, and questions lets a teammate correct the right part without relitigating everything you found.

Capture the environment that can change the answer

Most investigations are more environment-dependent than they first appear. Save the commit, relevant config, commands, and test invocation. If a service is involved, capture safe identifiers such as a release version, feature-flag state, and time window—not a production dump.

The point is reproducibility, not ceremony. A command that passed against your local working tree has limited value if the report does not say whether local changes were present. git diff --stat, git status --short, and the exact test command are often enough context for a focused patch.

For longer work, let a terminal multiplexer preserve the live session, but do not mistake persistence for evidence. Ghostty, tmux, and SSH: A Terminal Workflow That Survives Context Switching explains why the emulator, session, and durable record solve different problems.

Use a narrow command sequence

An investigation usually benefits from moving outward in a predictable order:

  1. Locate the symbol, configuration key, error text, or test name with rg.
  2. Read the smallest owning file and its immediate callers.
  3. Run the narrowest test or reproduction that can disprove the current theory.
  4. Inspect history only when it can answer a specific “why did this exist?” question.
  5. Broaden the search only after recording what the narrow result ruled in or out.

This is not a ban on exploration. It stops broad searches from producing a pile of related-looking material with no decision value. If you inspect a log, record the time range and filter used. If you compare branches, record both revisions. Small boundaries make a result reviewable.

Know when to stop collecting

Stop when the next command would merely make the report feel more confident without changing a decision. This is where senior engineers save real time. You do not need to read every call site to prove a configuration is loaded from one file; you need enough direct evidence to name the boundary and a test that would catch a wrong change. If a new fact could choose a different owner, risk, or validation, collect it. If it only decorates the narrative, leave it out.

That restraint also makes escalation clearer. A compact evidence bundle lets a domain owner answer the one question you cannot resolve, instead of asking them to reconstruct an afternoon of terminal exploration from fragments in chat.

End with a handoff, not a terminal archaeology project

Close the investigation with five headings: question, evidence, current model, decision needed, and validation. The handoff can live in an issue, a pull request description, or a compact plan. It should say what changed in your understanding and what would change it again.

This is also the right shape for agent-assisted work. An agent can gather paths and summarize output quickly, but its fluent summary is still a claim until a reviewer can follow the evidence. How To Turn Agent Investigation Notes Into an Engineering Plan covers turning reconnaissance into bounded, accountable work.

The best terminal investigation is not the one with the most panes or longest history. It leaves a future engineer able to reproduce the observations, understand the uncertainty, and make the next decision without repeating the whole search.

For more practical engineering essays and tools, visit Slaptijack.

Slaptijack's Koding Kraken