search

Search the awesome-japanese-nlp-resources database for the user's query.

Claude Code and Codex

This skill is shared by the Claude Code and Codex versions of the plugin. The steps are the same in both tools; only these details differ:

• Query — Claude Code: the arguments of /awesome-japanese-nlp-resources:search, appended at the end of this skill as ARGUMENTS: …. Codex: the user's message that invoked $awesome-japanese-nlp-resources:search, minus that $… mention. If the skill was picked automatically rather than invoked by name, use the user's request as the query.
• Plugin root — Claude Code: ${CLAUDE_PLUGIN_ROOT}. Codex: the directory two levels above this SKILL.md (use its absolute path).
• Shell — run the commands below with Claude Code's Bash tool or Codex's shell tool. Copy each Python script in full and run it as written, changing only its placeholders (RESOURCES_PATH, the keyword lists) — don't shorten it, drop passes, or alter its scores and thresholds.
• Commands — write any command you show the user in the current tool's form: /awesome-japanese-nlp-resources:<skill> in Claude Code, $awesome-japanese-nlp-resources:<skill> in Codex.

Results must come from the bundled data. If the data file can't be read (for example, shell commands are blocked or fail to start), say so and link https://github.com/taishi-i/awesome-japanese-nlp-resources instead of answering from memory or web search.

Instructions

Step 0 — Validate input

If the query is empty or blank, stop immediately and output (in Codex, write the commands with $ instead of /):

Usage: /awesome-japanese-nlp-resources:search <query>

Examples:
  /awesome-japanese-nlp-resources:search morphological analysis
  /awesome-japanese-nlp-resources:search BERT
  /awesome-japanese-nlp-resources:search named entity recognition
  /awesome-japanese-nlp-resources:search text classification dataset
  /awesome-japanese-nlp-resources:search sentence embedding

Please pass the keyword(s) you want to search for as the argument.

---

使い方: /awesome-japanese-nlp-resources:search <query>

クエリ例:
  /awesome-japanese-nlp-resources:search 形態素解析
  /awesome-japanese-nlp-resources:search BERT
  /awesome-japanese-nlp-resources:search 固有表現認識
  /awesome-japanese-nlp-resources:search テキスト分類 データセット
  /awesome-japanese-nlp-resources:search 文埋め込み

検索したいキーワードを引数に指定してください。

Do not proceed to Step 1 if the query is empty.

Step 1 — Interpret the query

The data descriptions are in English, so always convert the query intent to English keywords before searching.

Keyword rules — read before choosing keywords:

1. Use stems, not full words. Substring match is used, so morpholog catches "morphology", "morphological", "morphological analyzer". Other examples: embed → embedding/embeddings, classif → classification/classifier, translat → translation/translate, generat → generation/generative, segment → segmentation/segmenter, recogni → recognition/recognizer, extract → extraction/extractor, retriev → retrieval/retrieve.
2. Add domain-specific tool names. When the query maps to a known NLP domain, include the well-known tool names present in the database:

Domain (Japanese query hint)	Stem keywords	Tool names to add
形態素解析 / morphological analysis	morpholog, segment	mecab, janome, sudachi, kytea, kuromoji, jumanpp, nagisa
固有表現認識 / NER	named entit, NER, recogni	ginza, spacy, knp
係り受け解析 / dependency parsing	depend, parse, syntax	cabocha, knp, ginza, spacy
文章分類 / text classification	classif, sentiment, categor	bert, fasttext
感情分析 / sentiment analysis	sentiment, emotion, opinion	oseti, wrime
埋め込み / word vectors / embeddings	embed, vector, represent	word2vec, fasttext, bert, sbert
事前学習モデル / pretrained model	pretrain, language model, bert, gpt	bert, gpt, llama, rinna, elyza, calm, swallow
テキスト生成 / text generation	generat, language model	gpt, llm, llama, rinna, elyza
機械翻訳 / machine translation	translat, machine translation	opus, marian, fairseq
音声認識 / speech recognition	speech, recogni, audio, asr	whisper, julius, espnet
音声合成 / text-to-speech	speech, synthesis, tts	voicevox, espnet
質問応答 / QA	question, answer, qa	bert, t5
要約 / summarization	summari, abstract	bart, t5, pegasus
辞書・IME / dictionary	dict, lexicon, ime	mecab, sudachi, mozc
コーパス・データセット / corpus	corpus, dataset, annot	(rely on stems)
チュートリアル / learning	tutorial, introduc, learn	(rely on stems)
OCR / 光学文字認識	ocr, optical character, recogni	manga-ocr, donut, tesseract
RAG / 検索拡張生成	retriev, rag, embed	ruri, glucose, faiss
ファインチューニング / fine-tuning	fine-tun, finetun, lora, peft	lora, peft, qlora
ベンチマーク・評価 / benchmark	benchmark, evaluat, jglue	llm-jp-eval, jglue, nejumi

3. When the query contains Japanese text, also keep 2–4 raw Japanese terms/phrases lifted directly from the query (not translated) as a separate ja_keywords list. Aliases and some descriptions (al, d_ja — see Step 3) are Japanese-only, so a literal Japanese substring catches entries an English-only translation would miss entirely — nicknames like ボイボ (VOICEVOX), めかぶ (mecab), or a Japanese technical term that never got glossed into the English description. Leave ja_keywords empty for English queries.
4. Aim for 4–6 keywords. Fewer miss items; more than 6 inflates low-quality partial matches.
5. If none of the above domains fit, translate the query intent literally to English stems.

Step 2 — Locate the data file

The data file ships with the plugin at data/resources.json under the plugin root (see "Claude Code and Codex" above). Resolve its absolute path, falling back to a scoped search only if the install is unusual:

PLUGIN_ROOT="${CLAUDE_PLUGIN_ROOT}"  # Codex: replace with the plugin root, two levels above this SKILL.md
RESOURCES_PATH="$PLUGIN_ROOT/data/resources.json"
[ -f "$RESOURCES_PATH" ] || RESOURCES_PATH="$(find "${CODEX_HOME:-$HOME/.codex}/plugins" "${HOME}/.claude/plugins" -type f -name resources.json 2>/dev/null | grep "awesome-japanese-nlp-resources/" | head -1)"
echo "RESOURCES_PATH=$RESOURCES_PATH"

Use the resulting absolute RESOURCES_PATH wherever Step 3 opens the data file — write the path itself into the script, since shell variables may not persist between commands.

The plugin also ships data/multilingual_resources.json (same item format) listing multilingual libraries, models, and datasets (GitHub repositories) that also support Japanese, from docs/multilingual.md. The scripts below load it automatically when it exists; its items have categories like Multilingual (Speech recognition).

Step 3 — Search and score with Python

Do not read the data file directly (no Read tool, cat, head, or similar) — it is about 660 KB and would flood the context. Instead, run the scoring in a single shell command using Python.

Each item in the JSON array has:

• u: GitHub or Hugging Face URL
• n: repository/model name
• d: English description
• d_ja: Japanese description (GitHub-origin items only; match your ja_keywords against this)
• al: curated alternate names / kana nicknames, e.g. ["VOICEVOX", "ボイスボックス", "ボイボ"] (array of strings, only ~40 items have this — treat a hit here as strong as a name match)
• c: category (e.g. Python library, HuggingFace Model (Text Generation), Corpus, Tutorial, Multilingual (Speech recognition), ...)
• s: subcategory / semantic labels (array of strings)
• st: GitHub star count (GitHub items only; absent or 0 otherwise)
• ns: normalized star score 0–10 (log-scaled, GitHub items only)
• dl: Hugging Face download count (HF items only; absent or 0 otherwise)
• nd: normalized download score 0–10 (log-scaled, HF items only)
• sc: pre-computed quality score (higher = more popular/active)
• status: "ok" or "not_found" — items whose repo 404s (~8 of ~1200) are filtered out below; never recommend one

Run the following, substituting RESOURCES_PATH with the absolute path from Step 2, keywords with your English keywords and ja_keywords with your raw Japanese terms, both from Step 1 (ja_keywords may be []):

python3 << 'EOF'
import json, os

with open("RESOURCES_PATH") as f:    # absolute path from Step 2
    data = json.load(f)
multilingual_path = os.path.join(os.path.dirname("RESOURCES_PATH"), "multilingual_resources.json")
if os.path.exists(multilingual_path):
    with open(multilingual_path) as f:
        data += json.load(f)

keywords = ["keyword1", "keyword2", "keyword3"]  # English stems, from Step 1
ja_keywords = []  # raw Japanese terms from Step 1 -- [] for English queries

results = []
for item in data:
    if item.get("status") == "not_found":
        continue  # dead repo -- never recommend it

    n = item.get("n", "").lower()
    d = item.get("d", "").lower()
    d_ja = item.get("d_ja") or ""
    s = " ".join(item.get("s") or []).lower()
    c = item.get("c", "").lower()
    al = " ".join(item.get("al") or []).lower()

    text_score = 0
    for kw in keywords:
        kw = kw.lower()
        if n == kw:       text_score += 20
        elif kw in n:     text_score += 10
        if kw in d:       text_score += 5
        if kw in s:       text_score += 3
        if kw in c:       text_score += 2
        if kw in al:      text_score += 10  # alias hit is name-equivalent

    for kw in ja_keywords:
        if kw in n:       text_score += 10
        if kw in d_ja:    text_score += 5
        if kw in al:      text_score += 10

    if text_score < 8:
        continue

    ns = item.get("ns") or 0
    nd = item.get("nd") or 0
    sc = item.get("sc") or 0
    pop = (ns if ns else nd) * 2.5
    qual = min(5, sc * 5 / 21)
    combined = text_score + pop + qual

    results.append((combined, text_score, item))

results.sort(key=lambda x: -x[0])
seen = {item['n'] for _, _, item in results}

# Supplemental pass: surface high-popularity items from matching categories
# that may have been missed because their descriptions are in Japanese.
# Keys are stems to match against user keywords; values are category prefixes
# (prefix match covers "HuggingFace Model (Text Generation)" etc.).
CATEGORY_KEYWORDS = {
    "tutorial": "Tutorial", "introduc": "Tutorial", "learn": "Tutorial",
    "morpholog": "Python library", "segment": "Python library",
    "mecab": "Python library", "janome": "Python library", "sudachi": "Python library",
    "spacy": "Python library", "ginza": "Python library",
    "corpus": "Corpus", "dataset": "Corpus",
    "bert": "HuggingFace Model", "gpt": "HuggingFace Model",
    "llm": "HuggingFace Model", "llama": "HuggingFace Model",
    "pretrain": "HuggingFace Model", "embed": "HuggingFace Model",
    "model": "Pretrained model",
}
supplement_cats = set()
for kw in keywords:
    for ck, cat in CATEGORY_KEYWORDS.items():
        if ck in kw.lower():
            supplement_cats.add(cat)

if supplement_cats:
    def cat_match(c):
        return any(c == cat or c.startswith(cat + " ") for cat in supplement_cats)
    extras = [
        item for item in data
        if cat_match(item.get("c", ""))
        and (item.get("st", 0) or item.get("dl", 0))
        and item["n"] not in seen
        and item.get("status") != "not_found"
    ]
    extras.sort(key=lambda x: -max(x.get("ns") or 0, x.get("nd") or 0))
    for item in extras[:5]:
        ns = item.get("ns") or 0
        nd = item.get("nd") or 0
        sc = item.get("sc") or 0
        # base 8 = category-match credit (same as the text_score threshold)
        combined = 8 + max(ns, nd) * 2.5 + min(5, sc * 5 / 21)
        results.append((combined, 0, item))
        seen.add(item["n"])

results.sort(key=lambda x: -x[0])
for combined, text_score, item in results[:20]:
    st = item.get("st", 0) or 0
    dl = item.get("dl", 0) or 0
    flag = " [supplemental]" if text_score == 0 else ""
    print(f"score={combined:.1f} text={text_score} st={st} dl={dl}{flag}")
    print(f"  n={item['n']}")
    print(f"  u={item['u']}")
    print(f"  c={item['c']}")
    print(f"  s={item.get('s','')}")
    if item.get('al'):
        print(f"  al={item['al']}")
    print(f"  d={item.get('d','')[:120]}")
    if item.get('d_ja'):
        print(f"  d_ja={item['d_ja'][:120]}")
    print()
EOF

This returns up to 20 candidates. Items marked [supplemental] were added by the category-based pass to recover high-star resources whose descriptions are in Japanese. In Step 4, evaluate supplemental items on semantic fit before including them in the final list.

Step 4 — Re-rank with your judgment

You now have up to 20 candidates. Apply your semantic judgment to produce the final ordered list of up to 10 results.

Re-rank by evaluating each candidate on:

1. Semantic centrality — how directly does this resource address the query's core intent? A BERT model is more central to "BERT fine-tuning" than a generic transformer library.
2. Popularity as a proxy for quality — high stars/downloads generally signal battle-tested, well-documented tools. Prefer them when candidates are otherwise equivalent.
3. Category fit — match the resource type to the implied need:
   • "how to learn / 勉強" → prefer Tutorial, Research summary
   • "I need a model" → prefer Pretrained model, HuggingFace Model
   • "find a dataset / コーパス" → prefer Corpus, HuggingFace Dataset
   • "build an app / ライブラリ" → prefer Python library, language-specific libs
   • "multilingual / 多言語 / other languages too" → include Multilingual (...) items; otherwise prefer Japanese-specific resources when they fit equally well, and use Multilingual (...) items to fill gaps such as speech, OCR, language detection or search engines
4. Specificity — a resource specialized for the exact task beats a general one.
5. Recency signal — when sc is significantly higher among otherwise-similar items, it usually reflects more recent activity; prefer those.

Do not mechanically follow the combined score from Step 3 — use it as a starting point, then move items up or down based on the criteria above.

Step 5 — Format the output

Language detection rule (apply before writing any output):

• The query contains Japanese characters (hiragana / katakana / kanji) → Japanese
• Otherwise → English (default)

Apply the detected language to all headings and prose.

Present the final re-ranked results:

## Search results for "<query>"

*(Searched for: keyword1, keyword2, ...)*

Found N result(s).

### 1. [repository-name](url)
**Category:** category > subcategory
**Popularity:** ⭐ {st} stars  (or  📥 {dl} downloads for HF)
Description text here.

### 2. ...

If no results, suggest alternate keywords and link to: https://github.com/taishi-i/awesome-japanese-nlp-resources

Step 6 — Output use-case selection guide table

After the search results list, append a guide table that helps the user pick the right resource for their specific situation.

Match the section heading and table language to the query language — translate the heading and column headers into the query language (e.g. Japanese query → Japanese heading and headers).

## Use-case Selection Guide

| Use case | Recommended | Popularity | Why |
|---|---|---|---|
| ... | [name](url) | ⭐N or 📥N | short reason |

Rules:

• List 3–6 distinct use cases derived from the top 10 results. Each row should represent a meaningfully different scenario (e.g., "fine-tune an LLM" vs "evaluate an LLM"), not just a restatement of the search query.
• For each row, select the single best resource from the top 10 results.
• Popularity column: use ⭐{st} for GitHub stars, 📥{dl} for HuggingFace downloads. If both are 0, omit.
• Why: write a 10–15 word reason in the query language explaining why this resource is the best fit for that use case. Do not copy the description verbatim. Focus on the practical benefit.
• If two use cases would map to the same resource, merge them into one row or drop the weaker one.
• If there are fewer than 3 meaningfully distinct use cases in the results, output as many rows as make sense (minimum 1).