research-observability

Research observability

Experiments you cannot replay are experiments you have to rerun. This skill installs a small, file-based observability layer over any LLM/agent experiment and the tools to read it: every generation, every tool call and result, every reasoning trace, every reward — on disk as plain JSONL, greppable, diffable, and re-gradable months later.

The design bet: the recording format is fixed and platform-independent, so adding a platform is writing one ~15-line normalizer, not re-plumbing observability. LiteLLM, OpenAI, Gemini, Anthropic, Tinker and a bespoke inference server all produce identical files, and every viewer, report and diff downstream works with no further changes.

Start here

Adding observability to a project. Follow references/integration.md end to end — it is a seven-step procedure, starting with a survey of the repo (which platform, where the generations happen, where results go today) and ending with the existing runs imported so old and new sit in the same viewer. Do the survey before writing code; the answers determine every later choice.

A platform not yet wired. references/platforms.md has working recipes for OpenAI, Gemini/Vertex, Anthropic, LiteLLM, Tinker, and vLLM/self-hosted, plus the 15-line template for anything else and the framework-level patterns (subclass the model class, or use the callback API).

Understanding or extending the format. references/data-model.md is the contract: the directory layout, every field, both sequence counters and why there are two, and the concurrency rules.

Reading and comparing runs. references/viewers.md covers the three tools, the diffing recipe, plotting, and how to add domain-specific panels without changing the trace format.

Eval scripts across many models. references/runner-scripts.md — one shared runner holds the protocol, each model gets a ten-line script. Template in assets/_common_eval.sh.

Seeing it at full scale. references/reference-implementation.md maps the tau2-bench system this was generalized from, with the details worth stealing verbatim.

What gets installed

Copy from scripts/ into the project (vendor them — research code always wants project-specific events, and a file you can edit beats a dependency):

file	role
obs.py	the core. Run + the logging surface. No provider imports, no domain knowledge
obs_adapters.py	per-platform normalizers and Observed* client wrappers
view_trace.py	one-shot dump: pipe it, grep it, diff two arms, --follow a live run
view_trace_tui.py	interactive curses viewer: run list → session list → conversation, with notes
obs_report.py	cross-run comparison table with ± standard error
obs_import.py	backfill existing results into the format so old runs show up too

The shape of an instrumented run

from obs import Run
from obs_adapters import ObservedLiteLLM, log_tool_results

run = Run("eval", config=run_config, root="runs",
          run_name=f"eval-{stamp}-{model_slug}")
llm = ObservedLiteLLM(run, role="agent")      # or ObservedOpenAI/Gemini/Anthropic

for task in tasks:
    with run.session(task=task.id, trial=k):
        run.system_prompt("agent", policy)
        while not done:
            resp = llm.completion(model=AGENT_MODEL, messages=msgs, tools=tools)
            results = execute_tools(resp)
            log_tool_results(run, results, requestor="agent")
        run.episode_done(reward=reward)
        run.artifact(episode, reward=reward)
run.close()

Then:

python view_trace.py --task task_007      # what happened
python view_trace_tui.py                  # read it properly
python obs_report.py --run runs/arm-a runs/arm-b   # did the arm actually win

Principles that make this worth doing

Log every speaker, not just the agent. The user simulator, the LLM judge, the summarizer, the retrieval model — each under its own role. An episode whose grade came from an unlogged judge cannot be debugged, only re-run. This is the single most common reason a trace stops explaining an outcome.

Tool results are part of the conversation. Tools run between generations, so results never pass through a sample call. Emit them as they arrive, so the trace reads turn → result → turn. A trace where the agent calls tools into a void cannot explain what it did next.

One config dict, printed and written. Build it once, print it to the log, hand it to Run. Then the log and the run folder can never disagree about what was run — and a field you did not record is a comparison you cannot make.

Retries and recoveries are events. Truncation retries, duplicate-call resamples, context overflows, cache misses. Each changes the sampled distribution; unlogged, they make a run irreproducible in a way nobody notices.

Keep token ids when the platform gives them. They make a training run auditable — the tokens the policy was updated on are the tokens in the file — and let you recompute logprobs offline. Hosted APIs cannot give you this.

Report variance, not just means. obs_report.py prints ± standard error and whether a two-arm gap clears 2·sem. On a 26-task benchmark a 0.04 reward difference is almost always noise, and the table should say so before you write it up.

Default it on. Export the echo and threshold env vars from the runner scripts. Observability that requires remembering to enable it is observability you do not have.

Environment knobs

Prefix defaults to OBS; pass Run(..., env_prefix="MYPROJ") to adopt a project's existing convention.

var	default	effect
<P>_ECHO	full	live per-turn stdout trace: full / preview / off
<P>_PREVIEW_CHARS	240	truncation width for previews
<P>_TOOL_REPEAT_WARN	3	emit tool_call_repeat at N identical calls
<P>_TOOL_TOTAL_WARN	15	emit tool_call_flood past N calls in a session
OBS_NOTES_DIR	.obs_notes	where the TUI's persistent notes live
OBS_EXPORTS_DIR	exports	where E writes session exports

Related

For Tinker specifically — training, inference, hyperparameters, LoRA, checkpointing, pricing — use the tinker-training-inference skill alongside this one. That skill runs the loop; this one records it.