ts-idea2story

ts-idea2story — idea → grounded research story (the upstream of ts-paper)

Distilled from the product's idea→story link. The idea is the PROTAGONIST; a recalled/searched pattern is the TOOL it wields. Claude is the packager, the recaller, the searcher, the storyteller, and the critic — all in one context (so coherence and anti-stacking come for free). Two scripts are irreducible: kg_recall.py (vector/graph retrieval) and story_lint.py (the gate); embeddings (optional) reuse ../ts-kg-build/scripts/embed.py's TS_EMBED_* config.

Inputs / Outputs

In: a raw idea (text); optional kg/ dir (from ts-kg-build, or the bundled kg_ts); optional TS_EMBED_* endpoint; a retrieval_focus dial in {trust_kg, balanced, go_search_web}. Out (in the workdir):

• story.json — the 8 fields: title, abstract, problem_framing, gap_pattern, solution, method_skeleton, innovation_claims[], experiments_plan.
• story_proposal.md — the Markdown projection of the 8 fields (this is what ts-paper-plan reads — a story IS a structured proposal; story2paper == proposal2paper).
• retrieved_papers.json — the real papers found, the citation seed for ts-paper-cite (§ Reuse).
• logs/*.io.md, optional novelty_report.json.

Procedure

1. Package the idea (Claude)

Write idea_brief.json: motivation, problem, assumptions_explicit[], assumptions_inferred[], constraints, retrieval_query. Faithfulness guard: keep assumptions you inferred separate from what the user stated — never fabricate user intent. The retrieval_query is an English, search-friendly reformulation. Never raise on a thin idea; normalize and proceed.

2. Recall candidate patterns (script + Claude)

If a kg/ exists: python3 scripts/kg_recall.py --idea "<retrieval_query>" --kg <kg_dir> --out recall.json --topk 8. It returns Top-K candidate patterns (semantic+graph if embeddings configured, else lexical-only, labeled — treat lexical hits as weaker). Then YOU reason over them comparatively in one pass, scoring each on three independent axes (no per-pattern round-trips needed):

• stability — reliable skeleton, well-trodden; novelty — differentiating, rare;
• domain_distance — how far the pattern's home domain is from the idea (sorted ASCENDING; a near-domain pattern transfers more safely, a far one is a riskier "storyteller" move). Size ≠ stability (a big cluster can be incoherent — check the pattern's coherence). If no KG: skip to search (or, with neither, generate the story from the idea alone — weakest, allowed).

3. Agentic external search (Claude WebSearch/WebFetch) — also seeds citations

Plan intent-tagged query packs (with forced minimums) and run them via native search: core_method, task_setting, contrast (what you depart from — don't skip this), evaluation (datasets/metrics/baselines). Force-keep the literal user idea as one query. For each kept paper, read the abstract and extract a story-first pattern (as in ts-kg-build). Record every real paper into retrieved_papers.json (schema in § Reuse) — this is the citation seed. Never fabricate a paper or an abstract.

4. Select the seed pattern (Claude)

Pick the pattern that best fits the idea as a stable skeleton (prefer a near-domain, coherent KG pattern for the first draft); hold novelty/far-domain patterns in reserve for the refine loop.

5. Generate the Story (Claude — the craft)

Idea = protagonist, pattern = tool. Produce the 8 fields. The non-negotiable craft:

• Reframe, don't combine. The gap_pattern/solution must say "reframe X from Y to Z" — recast the problem, don't staple two methods together. Banned verbs: combine, integrate, merge, stack, plug in. Required verbs: reframe, recast, transform, unify, exploit. Include a one-line rationale why_not_straightforward_combination inside solution.
• Title: names the idea/insight, not the architecture. Good: "Labels for free: dense prediction as retrieval". Bad: "A CNN-Transformer framework for X".
• solution vs method_skeleton split: solution is the narrative (what changes and why it works); method_skeleton is the concrete step list (≥ a few real steps). Keep them distinct.
• Honesty: experiments_plan and innovation_claims are forward-looking ("we evaluate… we expect…") — no fabricated numbers / "beats SOTA by X%".

6. Critique → refine → fuse loop (Claude, in-context; bounded ≤3 rounds)

• Critic (blind, comparative): judge the story against 1–2 exemplar stories with titles/authors STRIPPED (so prestige can't anchor you); prefer comparative better/worse over absolute scores; anti-length bias (longer ≠ better); tie on thin evidence. Diagnose the single WEAKEST of: faithfulness-to-idea, novelty, narrative organic-ness.
• Defect → axis routing: novelty weak → pull a novel/rare reserved pattern; stability/soundness weak → pull a robust near-domain pattern; storytelling flat → pull a far domain_distance pattern.
• Fuse conceptually (not A+B): weave the new pattern into the story so the two co-evolve (restructure the framing), then run a StoryReflector organic-ness check — reject a story that reads as two bolted-on ideas; do not let it self-approve over a stacking warning.
• Calibrated pass bar (accept condition): accept only when the story is judged better-or-equal to ≥2 of the N exemplar anchors on ≥2 of the 3 axes (faithfulness, novelty, narrative). If <2 anchors are available, fall back to a fixed bar — each axis judged at least good (≥7/10 equivalent). (Claude-native analog of the source's two-of-three-q75 + pass-score-7.0 gate.)
• Loop control: keep the global best, not the last (rollback if a round regresses); don't retry a pattern that already failed the same axis; stop early when the pass bar is met. Cap at 3 refine rounds; if the bar is still unmet after 3 rounds, ship the global best and record the unmet axis in the log.

7. Novelty / anti-collision check (script when an embedding endpoint is configured)

When TS_EMBED_* is configured, run python3 scripts/novelty_check.py <workdir>: it embeds the story text (reusing the embed_one() pattern), cosine-compares it against an available reference set (the retrieved_papers.json abstracts embedded on the fly, else the pattern index kg/pattern_emb.npy), and writes novelty_report.json {max_similarity, risk_level, threshold_high, threshold_medium, top_similar, verdict, basis}. Act on the bands: ≥0.88 high → pivot to a more differentiating reserved pattern and re-tell; ≥0.82 medium → warn; cap pivots at 2. Graceful degrade: with no embedding endpoint the script exits 0 with a "lexical/judgement only — not a semantic guarantee" note; in that case do the lexical/judgement check against the recalled exemplars + retrieved_papers.json yourself. Either way novelty_report.json must exist (top similar works + verdict + basis) so the check is never silently skipped.

8. Emit + gate

Write story.json, render story_proposal.md (the 8 fields as a proposal: Problem / Gap / Proposed method / Evaluation plan / Contributions), and finalize retrieved_papers.json. Then: python3 scripts/story_lint.py <workdir> — must be ok:true (8 fields non-empty, no schema-echo noise, method_skeleton concrete, no fabricated numbers). Soft requirement: confirm novelty_report.json exists (from step 7, semantic or degraded) so the novelty check was not silently skipped. Hand story_proposal.md to ts-paper-plan.

Reuse — retrieved_papers.json (cuts the cite stage's search)

Write the real papers found in steps 2–3 here; ts-paper-cite ingests it as a verified seed.

{"source":"ts-idea2story", "papers":[
  {"paper_id":"openalex:W..","title":"..","authors":[".."],"year":2024,"venue":"..",
   "volume":"","issue":"","pages":"","doi":"10..","url":"https://..","abstract":"(real)",
   "abstract_source":"openalex|arxiv|openreview|missing",
   "relevance":"core|context|baseline|dataset_metric|contrast",
   "supports_claim":"the exact story claim this paper backs"}]}

Rules: dedup by paper_id / normalized title; on merge keep the longest non-empty abstract/title; never fabricate an abstract (abstract_source:"missing"); pool = the exemplar/agentic papers the story was derived from + the novelty candidates. The cite stage still verifies each via doi2bib, places it by abstract, gap-fills missing angles, and enforces its coverage floor — the seed just removes the cold-start sweep.

Integrity & trace

This is a proposal — no real results anywhere. Keep explicit-vs-inferred assumptions honest. Write logs/idea2story.io.md — INPUT (idea, retrieval_focus, kg on/off, embeddings on/off), DECISIONS (packaged brief, recalled patterns + 3-axis scores, search packs + kept/rejected, seed pattern, each critique round: weakest axis → pattern pulled → fuse result, novelty verdict), OUTPUT (story.json summary, story_lint result, retrieved_papers count).