model-compute-simulation

Model Compute Simulation

Overview

Use this when the question is about operator order, tensor dimensions, FLOPs, MFU, or parallelism checks. The simulator loads a model config, builds the representative operator sequence, prints tensor shapes and FLOPs, and can estimate MFU from measured latency.

Inputs

Before running a simulation, collect or verify these inputs:

Item	Why it matters	How to obtain	Default if user skips
Model name	Resolves to config in model-config-index.json; determines entire architecture	Ask user or infer from trace context	— (required)
Config accuracy	Indexed values may differ from actual serving config (e.g. routed_expert_intermediate_size, compress_ratios)	Ask user to provide config.json or verify key params against HuggingFace	Use indexed values with a caveat
GPU type	Determines peak FLOPS for MFU denominator	Ask user	— (required for MFU)
dtype (bf16 / fp8)	Affects peak FLOPS selection; fp8 doubles peak	Ask user	bf16
Batch size & seq len	Directly affects FLOPs and tensor shapes	Ask user	B=1, S=1 (decode)
TP / DP / EP	TP splits GEMM FLOPs across GPUs; EP splits expert FLOPs	Ask user	Read the run manifest; otherwise report assumptions explicitly
Measured latency (ms)	Required for MFU numerator; must be per-GPU forward-pass wall-clock	Ask user or extract from a profiler trace	— (optional, no MFU without it)

If the model is not in model-config-index.json, ask the user for a config.json path or add an indexed config before running estimates. The 2026-10-05 index pins public HF config revisions for V4 Flash/Pro/V4.1, K3, GLM-5.2/5.3/5.3-Flash, Qwen3.8-Flash-Next, Hy3/Hy4, Inkling, dots3, Step-3.7, Nemotron-3/3.5, LongCat and Gemma4, and corrects older model dimensions. Qwen/Qwen3.8-A95B returned HTTP 401 at this audit and is not indexed. Use --config /path/to/config.json for a raw nested HF config; the local normalizer expands text wrappers and expert/layer aliases.

Shape and timing limits for speculative MoE

For DSV4.1, obtain actual target/draft row counts from the trace: request BS=1 can issue six target-verify rows with DSPARK block size 5. Do not substitute acceptance length for the number of rows computed. Account for TP-sharded shared-expert shapes separately from routed expert assignments and EP. Do not divide an unverified model template by TP/EP and call it measured FLOPs. The simulator applies uniform TP division outside routed experts and uniform EP division inside routed experts. This is a what-if approximation: replicated routers, norms, latent projections and backend-dependent shared-expert ownership require a run-specific adjustment before interpreting MFU. The pinned V4.1 entry is a static target-model template; DSPARK draft, vision, Engram/PLE memory traffic and backend dispatch require separate evidence.

Kernel-duration sums include overlap and possibly PDL dependency waits. Use wall-clock time for a whole-pass MFU denominator, and measured GPU SKU/dtype peaks rather than assuming every FP8 operation doubles useful FLOPs. Follow the dispatch and timing principles when assigning kernel names to attention, shared experts, routing or mHC.

Workflow

Step 1: Load model config

Resolve the model name and load its configuration parameters:

python3 skills/model-compute-simulation/scripts/model_compute_simulator.py "<model name>" --list-models

The script resolves the model name against references/model-config-index.json, which stores public HuggingFace config parameters (hidden_size, num_experts, MLA ranks, etc.).

If the model is not indexed, tell the user to provide a config.json path or request an index update.

Step 2: Generate execution flow and tensor dimensions

Run the simulator with batch size, sequence length, and parallelism configuration:

python3 skills/model-compute-simulation/scripts/model_compute_simulator.py "Qwen3-235B-A22B" \
  --batch-size 1 --seq-len 1 \
  --tp 8 --dp 1 --ep 8 \
  --gpu b200 --dtype bf16

The simulator prints:

• Per-layer operator sequence with FLOPs and tensor shapes (shape_in → shape_out)
• Attention vs MoE/FFN FLOPs proportion per layer
• Total model FLOPs for a single forward pass

For decode: use --seq-len 1 --context-len TOTAL_KV_LENGTH. For extend, --context-len is the prefix length, with causal work for the new --seq-len tokens. Full/window attention includes KV context in QK and AV FLOPs. The LM head computes last-token logits unless --all-token-logits is given. Every run prints batch/shape/parallelism/GPU/dtype assumptions to stderr. Sparse selection, KDA/Mamba scalar work, norm/Sinkhorn and zero-expert top-k are architecture estimates, not measured instruction counts. LongCat routed FLOPs assume all top-k are nonzero (an upper bound until routing is measured). For prefill: use --seq-len <prompt_length>.

Step 3: Estimate MFU with measured latency

Provide the measured forward-pass latency to compute MFU:

python3 skills/model-compute-simulation/scripts/model_compute_simulator.py "Qwen3-235B-A22B" \
  --batch-size 1 --seq-len 1 \
  --tp 8 --dp 1 --ep 8 \
  --gpu b200 --dtype bf16 \
  --measured-ms 15.0

MFU = theoretical_min_time / measured_time × 100%

The simulator prints:

• Overall MFU
• Per-layer MFU (uniform layer-time assumption)
• Per-operator FLOPs proportion (for identifying which ops dominate)

GPU peak FLOPS are loaded from references/gpu-specs.json. The bundled hardware table includes H100/H200/B200, B300, GB300, MI355X, RTX PRO 6000 and DGX Spark, each with official URLs. H20 is retained as an unverified historical estimate. Unknown dense peaks are null and MFU for that dtype is refused; RTX Server and Spark sparsity is unconfirmed. B300/GB300 memory capacity is physical maximum, not usable allocation. --dtype accepts bf16, fp8/mxfp8 and fp4/mxfp4/nvfp4; INT8 must use its separately published peak. Use aliases such as --gpu h100, --gpu b2000, or --gpu b200 when running on those local boxes.

Step 4: Per-operator MFU with kernel-level latency

When you have per-kernel measured latency, compute per-operator MFU by mapping kernel durations to the compute flow.

Method A: --kernel-flow (kernel-level MFU, recommended)

Provide per-kernel detail as JSON, then feed it to the simulator for kernel-level MFU analysis. This preserves every kernel row from the compute flow and adds FLOPs/MFU columns.

python3 skills/model-compute-simulation/scripts/model_compute_simulator.py "Qwen3-235B-A22B" \
  --batch-size 1 --seq-len 8192 \
  --tp 8 --dp 1 --ep 8 \
  --gpu b200 --dtype bf16 \
  --measured-ms 100 --kernel-flow @/tmp/layer3_detail.json

The --kernel-flow parameter accepts a JSON string or @file path. It produces a kernel-level MFU table that preserves all kernel rows from the compute flow and adds:

• Mapped Op: which operator this kernel maps to
• FLOPs: operator's total FLOPs
• Theo(us): theoretical minimum time
• MFU%: measured FLOPs utilization
• shape_in→shape_out: operator tensor dimensions

When --kernel-flow is provided, the static per-operator template is omitted because the kernel-level MFU table already carries per-kernel shape and FLOPs information. The output keeps the model summary, serving configuration, total FLOPs, overall MFU calculated from the measured layer duration, and the kernel-level MFU table. In --format json, the raw evidence is stored under kernel_flow and no text table is appended.

Mapping rules:

• Direct-match kernels (mla, moe, mhc, rmsnorm, hadamard, rope, quant, topk, etc.): time is assigned directly to the corresponding operators
• Generic GEMM kernels (gemm_fp8, gemm_bf16): time is distributed to remaining unassigned projection GEMM operators by FLOPs share
• Overhead kernels (allreduce, moe_align, moe_sort, other): rows preserved, FLOPs/MFU marked as N/A

FP8 kernel MFU correction: Kernels in categories moe (fused_moe_kernel) and gemm_fp8 use fp8 math internally even when --dtype bf16 is specified. For these kernels, the MFU denominator uses the GPU's fp8 peak FLOPS (2x bf16 peak) instead of bf16 peak. The resulting MFU is marked with a superscript ⁸ (for example, 63.7%⁸) to show that the fp8 denominator was used. gemm_bf16 kernels still use the bf16 peak FLOPS denominator.

Method B: --kernel-detail (operator-level MFU, legacy)

Same input as --kernel-flow but outputs an operator-level summary table (aggregated by operator, not per-kernel). Use when you want a compact view.

python3 skills/model-compute-simulation/scripts/model_compute_simulator.py "Qwen3-235B-A22B" \
  --batch-size 1 --seq-len 8192 \
  --tp 8 --dp 1 --ep 8 \
  --gpu b200 --dtype bf16 \
  --measured-ms 100 --kernel-detail @/tmp/layer3_detail.json

Method C: --kernel-ms (category-level approximation)

Use a hand-aggregated category map only when per-kernel detail is unavailable:

python3 skills/model-compute-simulation/scripts/model_compute_simulator.py "Qwen3-235B-A22B" \
  --batch-size 1 --seq-len 8192 \
  --tp 8 --dp 1 --ep 8 \
  --gpu b200 --dtype bf16 \
  --measured-ms 100 --kernel-ms '{
    "mla": 4.922, "moe": 1.644, "allreduce": 0.769,
    "hadamard": 0.348, "mhc": 1.388, "gemm_fp8": 1.692,
    "gemm_bf16": 0.125, "rmsnorm": 0.227, "quant": 0.311,
    "rope": 0.209, "topk": 0.122, "activation": 0.071,
    "other": 0.437
  }'

The --kernel-ms parameter accepts a JSON object mapping kernel category names to their measured durations in milliseconds. It uses FLOPs-proportional distribution across entire categories, which is less precise than --kernel-detail because generic GEMM categories (gemm_fp8, gemm_bf16) span multiple operator categories.

Choose only one of --per-layer-ms, --kernel-ms, --kernel-detail, or --kernel-flow. All measured durations must be positive. The three kernel inputs also require --measured-ms from whole-pass wall-clock timing; the 100 ms in these examples is a placeholder to replace with the measurement. Kernel-duration sums cannot establish elapsed time when streams overlap.

Output includes:

• Model architecture summary (layers, hidden_size, attention_type, MoE config)
• Per-layer compute flow: operator sequence with tensor dimensions, FLOPs, shape_in→shape_out
• Per-operator MFU table: each operator's FLOPs, theoretical time, measured time (from trace), MFU%
• Kernel → operator mapping explanation (direct-match vs FLOPs-proportional vs overhead)
• Overall and per-layer MFU

When To Use It

• when you need compute-level detail for a known model or config
• when the user asks about execution flow, tensor dimensions, or FLOPs for a specific serving shape
• when the user asks about MFU and can provide measured forward-pass latency
• when comparing compute profiles across different parallelism configurations

Useful Commands

List known model IDs:

python3 skills/model-compute-simulation/scripts/model_compute_simulator.py --list-models

List known GPU types:

python3 skills/model-compute-simulation/scripts/model_compute_simulator.py --list-gpus

Emit JSON for automation:

python3 skills/model-compute-simulation/scripts/model_compute_simulator.py "GLM-5" --format json

Reporting Checklist

Include:

1. Model architecture summary: model name, config source, num_layers, hidden_size, attention_type, MoE config (num_experts, topk, shared_experts), MHC, head_dim
2. Serving configuration: batch_size, seq_len, TP, DP, EP, GPU, dtype
3. Per-layer compute flow (showing first representative layer in detail):
   • Operator sequence table: name, category, FLOPs, shape_in → shape_out
   • Attention vs MoE/FFN FLOPs proportion
4. Total model FLOPs for a single forward pass
5. Kernel-level MFU table (when --kernel-flow provided):
   • Preserves ALL kernel rows from the compute flow (never deleted)
   • Per-kernel columns: # | Half | Category | Simplified Name | dur(us) | % | Mapped Op | FLOPs | Theo(us) | MFU% | shape_in→shape_out
   • Direct-match kernels: show mapped operator FLOPs/MFU
   • Overhead kernels: show N/A for FLOPs/MFU, row preserved
6. Operator-level MFU table (when --kernel-detail or --measured-ms provided):
   • Each operator: name, category, total FLOPs, per-GPU FLOPs, theoretical time, measured time (from trace), MFU%
   • Kernel category → operator mapping explained
7. Overall MFU and per-layer MFU
8. One-line summary: dominant compute category, MFU status, key bottleneck

Trace-Based Validation (extract_compute_flow_from_trace.py)

Use scripts/extract_compute_flow_from_trace.py to extract the real operator sequence and tensor dimensions from a torch profiler trace, then compare against the static template as ground truth validation.

# Extract compute flow from a trace
python3 skills/model-compute-simulation/scripts/extract_compute_flow_from_trace.py \
  --input /path/to/trace.json.gz --format text

# Compare trace against static template
python3 skills/model-compute-simulation/scripts/extract_compute_flow_from_trace.py \
  --input /path/to/trace.json.gz \
  --compare qwen3-235b-a22b \
  --batch-size 1 --seq-len 1 --tp 8 --ep 8

Trace/template operation presence is compared through semantic families such as matmul, attention, norm, and router; raw aten::* names are not compared directly with semantic template names. A positive --min-flops threshold removes operations whose FLOPs could not be determined.

Compute Flow Confirmation Hierarchy

When the static template or trace extraction cannot fully confirm the compute process (e.g. ambiguous scope, missing shapes, new model architecture), follow this escalation hierarchy:

1. Static template (model_compute_simulator.py + model-config-index.json) — fast, covers known models

2. Trace extraction (extract_compute_flow_from_trace.py) — validates template against real execution

3. Inference framework source code — when trace is insufficient (missing Input Dims, CUDA Graph replay, compiled kernels without scope), read the model's forward flow directly from the serving framework source:

   • SGLang: python/sglang/srt/models/<model_name>.py — contains the forward() method and declare_attn/declare_ffn stage boundaries in python/sglang/srt/layers/layer_boundary/factories.py
   • vLLM: vllm/models/<family>/{common,nvidia,amd}/ for migrated 2026 families; older models remain under model_executor/models
   • TensorRT-LLM: tensorrt_llm/_torch/models/modeling_<family>.py + model configs

   When consulting framework source, focus on:

   • The forward() method: operator call order and residual connections
   • QKV / O projection: whether LoRA-style down/up projections are used (q_lora_rank, o_lora_rank)
   • MoE routing: top-k selection, shared vs routed expert split
   • TP/EP slicing: which dimensions are split and how FLOPs divide across GPUs
   • Any model-specific ops not in the static template (e.g. MHC, Hadamard, indexer)

   Action: If the framework source reveals discrepancies with the static template, update model-config-index.json and/or build_layer_ops() accordingly.

Limitations of Trace Extraction

Limitation	Detail	Workaround
record_shapes=True required	Trace must be captured with shape recording enabled; without it, Input Dims fields are absent and FLOPs cannot be computed	SGLang live capture and vLLM torch_profiler_with_stack=true already enable this; TensorRT-LLM requires a py_executor.py override adding record_shapes=True
CUDA Graph mode	During graph replay, cpu_op events may only appear once (at capture time); shape information for replayed iterations is not re-recorded	The script detects graph capture phases and annotates affected ops; use eager-mode traces for full coverage
TP-sliced dimensions	Trace shows post-TP-split dimensions (e.g. H/TP), not the full-model view	Use --tp in --compare mode to scale trace FLOPs back to full-model equivalents
Scope attribution quality	Python scope depends on with_stack=True; some frameworks or compiled paths may produce shallow or missing scope chains	Graceful degradation: ops with unresolved scope are categorized as "other"
Not a replacement for static templates	Trace extraction is a validation and discovery tool; static templates remain the primary fast-analysis path	Use trace extraction to verify templates for new models, then update model-config-index.json if discrepancies are found

References

• references/model-config-index.json: model configuration parameters (hidden_size, expert counts, MLA ranks, etc.).
• references/gpu-specs.json: GPU peak FLOPS specifications for MFU calculation.
• scripts/extract_compute_flow_from_trace.py: trace-based compute flow extraction and template validation tool.

Source audit: SGLang V4, K3 latent experts, NVIDIA HGX dense/sparse specs, AMD MI355X specs. HF field provenance is hf_config_source at hf_config_revision; construct https://huggingface.co/<repo>/blob/<revision>/config.json for the exact source. No GPU run is implied by a config refresh.