vss-benchmark-video-summarization

Instructions

Follow the routing table and step-by-step workflow below. Execute each step in order on a first run. For repeat runs, go directly to the Repeat Runs section. Detailed reference material lives in references/ and benchmark scripts live in scripts/.

⚠️ Shared GPU systems: On a multi-user host, multiple LVS instances may be running on different ports. Always benchmark against the instance YOU deployed. Never assume port 38111 — always use the exact endpoint URL returned by vss-build-vision-ai when you deployed LVS. If you did not deploy an LVS instance in this session, deploy one first before running benchmarks.

Purpose

Measure the latency and throughput limits of a deployed LVS (Long Video Summarization) instance, identify GPU and pipeline bottlenecks, and suggest configuration changes that improve performance. Produces XLSX reports and JSON result files per scenario, plus a natural-language analysis with improvement recommendations.

Do NOT use this skill for:

• Deploying LVS — use the vss-build-vision-ai skill with the lvs profile.
• Production monitoring or alerting — this is an offline benchmarking tool.
• Non-LVS VSS profiles (RTVI, search-only, etc.) — the /files API and /summarize endpoint are LVS-specific.

When to Use

• Measure single-file latency or burst throughput of a deployed LVS instance
• Identify whether the VLM or LLM stage is the pipeline bottleneck
• Get concrete config recommendations (chunk size, frame count, resolution, concurrency) after a benchmark run
• Compare performance across hardware profiles or config changes on the same LVS deployment
• Re-run a benchmark scenario after setup is already complete (see Repeat Runs)

Routing

Situation	Action
User has not deployed LVS in this session	Deploy LVS with vss-build-vision-ai (lvs profile) using a unique COMPOSE_PROJECT_NAME — note the exact endpoint URL it returns, then return here
LVS_BACKEND is not set	Ask the user for the endpoint URL of the LVS instance they deployed — do not guess or default
Results directory already exists with data and user asks to analyze only	Skip to Step 6: Analyze Results
First run or new hardware configuration	Run the full workflow Steps 0–7
Repeat run, setup already complete	Use run_benchmark.sh (see Repeat Runs section)

---

Prerequisites

Requirement	How to Check
LVS deployed by you via vss-build-vision-ai (lvs profile) with a unique COMPOSE_PROJECT_NAME	curl -sf ${LVS_BACKEND}/v1/ready returns 200
LVS_BACKEND set to your deployment's endpoint	echo $LVS_BACKEND (e.g. http://localhost:38111)
LVS_CONTAINER_NAME set to your LVS container name	echo $LVS_CONTAINER_NAME (e.g. vss-lvs)
VIA_DEV_API=true on YOUR LVS container	docker inspect ${LVS_CONTAINER_NAME} --format '{{range .Config.Env}}{{println .}}{{end}}' | grep VIA_DEV_API
NGC_API_KEY set in shell	echo $NGC_API_KEY (non-empty)
ngc CLI installed	ngc --version
Python 3.10+	python3 --version
ffprobe and ffmpeg installed	ffprobe -version && ffmpeg -version
Docker with Compose plugin	docker compose version

---

Deploy LVS for Benchmarking

Before benchmarking, deploy a fresh LVS instance using vss-build-vision-ai. On shared systems, always use a unique COMPOSE_PROJECT_NAME so your containers and volumes are isolated from other users.

# Choose a unique project name (e.g. your username)
export COMPOSE_PROJECT_NAME="lvs-bench-$(whoami)"

cd <repo>/deploy/docker

# REQUIRED FOR BENCHMARKING: the benchmark uploads videos via the LVS dev /files
# route, which is gated by VIA_DEV_API (default false; POST /files returns 404 when
# off). The lvs-server container loads its environment from
# services/video-summarization/.env (the compose `env_file:`), so VIA_DEV_API must
# be set THERE — putting it in any other env file will NOT reach the container.
# Set it before deploying:
grep -q '^VIA_DEV_API=' services/video-summarization/.env \
  && sed -i 's/^VIA_DEV_API=.*/VIA_DEV_API=true/' services/video-summarization/.env \
  || echo 'VIA_DEV_API=true' >> services/video-summarization/.env

# Deploy LVS on your designated GPUs
./scripts/dev-profile.sh up \
  --profile lvs \
  --hardware-profile RTXPRO6000BW \
  --llm-device-id <LLM_GPU> \
  --vlm-device-id <VLM_GPU>

# If LVS was already running, re-run the deploy command above so compose
# recreates the container with the new env (a plain `docker restart` does
# NOT re-read env_file changes).

Note the LVS endpoint (http://<HOST_IP>:38111) and container name: the compose file sets it statically to vss-lvs (verify with docker ps --filter name=lvs). Set these before continuing:

export LVS_BACKEND=http://localhost:38111
export LVS_CONTAINER_NAME=vss-lvs
export VLM_GPUS=<VLM_GPU>
export LLM_GPUS=<LLM_GPU>

Model downloads: First deployment downloads LLM (~20 GB) and VLM (~17 GB) model weights. This takes 20–40 minutes depending on network speed. Subsequent deployments reuse the volumes created under your COMPOSE_PROJECT_NAME and start in minutes.

---

Step 0: Pre-flight Check

On shared systems another tenant may hold port 38111 — so the benchmark must verify it is hitting YOUR instance and YOUR GPUs, not someone else's. preflight.sh does this (and fails fast if not).

Set your deployment's values, then run the pre-flight check:

export LVS_BACKEND=http://localhost:38111                          # YOUR LVS /summarize endpoint
export LVS_CONTAINER_NAME=vss-lvs                                   # YOUR LVS container (see deploy step)
export VLM_GPUS=<VLM_GPU>                                           # GPU(s) your VLM uses
export LLM_GPUS=<LLM_GPU>                                           # GPU(s) your LLM uses
export VSS_PUBLIC_URL=http://localhost:8000                         # origin serving the VSS agent

./scripts/preflight.sh

preflight.sh exits non-zero unless all of the following hold:

• the config parses and vlm_gpus/llm_gpus are valid GPU ids within the host's GPU range;
• LVS_BACKEND is reachable (/v1/ready → 200) and the dev /files route is enabled (not 404);
• your LVS_CONTAINER_NAME actually owns the backend port — its server bound successfully (no address already in use in its logs) and no other container publishes that port;
• the configured VLM_GPUS/LLM_GPUS are reserved by your LVS's VLM/LLM containers — so you can't silently benchmark idle GPUs or another tenant's instance (the exact failure this guards against);
• the deployed VSS version satisfies the requires-vss range in this skill's front matter (>=3.2.0,<4.0.0) — an incompatible deployment produces numbers that cannot be compared against anything, so the run stops rather than measuring it.

run_benchmark.sh runs preflight.sh automatically before every run; you can also run it standalone (above) any time.

Version compatibility

preflight.sh checks the deployment against the requires-vss range declared in this skill's own front matter, using services/agent/scripts/check_vss_version.py (standard library only). The range is read from SKILL.md, not passed in, so what a run enforces cannot drift from what the skill publishes.

GET /api/v1/version is served by the VSS agent, not by LVS. The check uses VSS_PUBLIC_URL — the deployment origin vss-build-vision-ai publishes, whose ingress routes /api to the agent — and falls back to LVS_BACKEND when that is unset. On a Compose deployment VSS_PUBLIC_URL is unset by design and LVS_BACKEND is the LVS server, which returns 404 for this route. Because compatibility cannot be determined from that origin, preflight stops. Point it at the agent:

export VSS_PUBLIC_URL=http://localhost:8000        # YOUR VSS agent / deployment origin
./scripts/preflight.sh

Outcomes, which preflight maps onto its own ok/FAIL lines:

Checker exit	Meaning	Preflight
0	deployed version is inside requires-vss	ok
3	deployed version is outside requires-vss	FAIL — benchmark stops
1	version or range could not be determined (404, 503, non-SemVer, unreachable, absent/malformed requires-vss)	FAIL — benchmark stops

A prerelease counts as its release (3.3.0-65576357eb80 satisfies exactly what 3.3.0 does), so the check behaves identically on Helm and Compose defaults. The requires-vss range is owned by this skill's author — bump it here when a VSS release ships a new minor; a new deployment version never silently widens it.

If /files returns 404, the dev route is off — enable VIA_DEV_API=true on your LVS (see the deploy step) before continuing.

---

Step 1: Download Test Videos

The benchmark uses warehouse surveillance videos from the VSS sample dataset, hosted in NGC. scripts/fetch-videos.sh downloads the package and places a curated set — warehouse_4min.mp4, warehouse_5min.mp4, warehouse_10min.mp4 — into <VSS_BENCHMARK_DATA_DIR>/videos/, the exact filenames the default scripts/config.yaml references. It requires the ngc CLI and an NGC API key, and prints install/auth instructions if either is missing.

export VSS_BENCHMARK_DATA_DIR=${VSS_BENCHMARK_DATA_DIR:-$HOME/vss-benchmark-data}

# Idempotent; add FORCE=1 to re-fetch, or pass a package version (default 3.2.0)
./scripts/fetch-videos.sh

# Probe video durations to verify what was downloaded
find "${VSS_BENCHMARK_DATA_DIR}/videos" -name "*.mp4" | while read f; do
  dur=$(ffprobe -v quiet -show_entries format=duration -of csv=p=0 "$f" 2>/dev/null | cut -d. -f1)
  echo "$(basename $f): ${dur}s"
done

---

Step 2: Generate Test Clips (if needed)

Normally Step 1 provides real 5- and 10-minute clips and this step can be skipped. Use it only if a curated clip is missing (e.g. the NGC package layout changed) or you need custom durations: loop an available source video into the filenames the default scripts/config.yaml references — warehouse_5min.mp4 (300s) and warehouse_10min.mp4 (600s).

# Find a source video to loop from
SOURCE=$(find "${VSS_BENCHMARK_DATA_DIR}" -name "*.mp4" | head -1)
echo "Using source: ${SOURCE}"

# Create a videos/ subdirectory where the media server expects files
mkdir -p "${VSS_BENCHMARK_DATA_DIR}/videos"

for spec in warehouse_5min:300 warehouse_10min:600; do
  NAME="${spec%%:*}"; DURATION="${spec##*:}"
  OUT="${VSS_BENCHMARK_DATA_DIR}/videos/${NAME}.mp4"
  [ -f "$OUT" ] && echo "Already exists: ${OUT}" && continue
  ffmpeg -stream_loop -1 -i "${SOURCE}" -t ${DURATION} -c copy "${OUT}" -y -loglevel error
  echo "Created: ${OUT}"
done

# Verify clips
find "${VSS_BENCHMARK_DATA_DIR}/videos" -name "*.mp4" | while read f; do
  dur=$(ffprobe -v quiet -show_entries format=duration -of csv=p=0 "$f" 2>/dev/null | cut -d. -f1)
  echo "$(basename $f): ${dur}s"
done

---

Step 3: Start Media Server

The media server serves test videos over HTTP so the LVS /files endpoint can download them by URL.

SKILL_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"

# Start the media server (nginx:alpine)
cd "${SKILL_DIR}"
docker compose -f scripts/media-server.yaml up -d
sleep 2

# Verify health
curl -sf "http://localhost:8888/health" && echo "Media server ready" || echo "Media server not ready"

# List available videos (JSON directory listing)
curl -s "http://localhost:8888/videos/" | python3 -m json.tool 2>/dev/null || \
  find "${VSS_BENCHMARK_DATA_DIR}/videos" -name "*.mp4" -exec basename {} \; | sed 's|^|http://localhost:8888/videos/|'

Note the filenames returned — you will need them in the next step to update config.yaml.

---

Step 4: Configure

Edit scripts/config.yaml with the actual video filenames discovered in Step 3:

1. Update video URLs — In both single_file_test and file_burst_test sections, replace HOST_IP with your host's LAN IP (hostname -I | awk '{print $1}' — NOT localhost; LVS downloads videos from inside its container) and make sure the filenames match those served by the media server (e.g., http://<HOST_IP>:8888/videos/warehouse_5min.mp4).

2. Set GPU assignments — Update vlm_gpus and llm_gpus to match your LVS deployment. Check YOUR container:

docker inspect "${LVS_CONTAINER_NAME}" \
  --format '{{range .Config.Env}}{{println .}}{{end}}' | grep -E "VLM_GPUS|LLM_GPUS|CUDA_VISIBLE"

3. Adjust chunk sizes — The default chunk_sizes: [10, 30] tests both 10-second and 30-second chunking. Larger chunks reduce API call overhead but increase per-chunk latency.

4. Adjust concurrency levels — The default concurrency_levels: [1, 2, 4, 8] for file_burst. Remove levels that exceed your hardware's memory capacity.

---

Step 5: Run Benchmark

SKILL_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
cd "${SKILL_DIR}/scripts"

# Create Python virtual environment if it doesn't exist
[ ! -d "vss-bench-env" ] && python3 -m venv vss-bench-env

# Activate venv and install requirements
source vss-bench-env/bin/activate
pip install -r requirements.txt -q

# LVS_BACKEND must already be set to YOUR deployment's endpoint
if [ -z "${LVS_BACKEND}" ]; then
  echo "ERROR: LVS_BACKEND is not set. Set it to your own LVS endpoint before running."
  exit 1
fi
export VIA_BACKEND="${LVS_BACKEND}"
export VIA_VLM_GPUS="${VLM_GPUS:?ERROR: VLM_GPUS must be set (e.g. export VLM_GPUS=6)}"
export VIA_LLM_GPUS="${LLM_GPUS:?ERROR: LLM_GPUS must be set (e.g. export LLM_GPUS=7)}"

# Run single_file scenario. The wrapper runs preflight first.
./run_benchmark.sh --scenario single_file_test

# Run file_burst scenario (can be run separately or together)
./run_benchmark.sh --scenario file_burst_test

The benchmark creates an output directory (default: vss-perf-report/) with per-scenario subdirectories. Each scenario run generates an XLSX report and execution_summary.json.

Note: The output directory must not exist or must be empty before each run. Move or rename previous results before re-running:

mv vss-perf-report vss-perf-report-$(date +%Y%m%d-%H%M%S)

---

Step 6: Analyze Results

SKILL_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
OUTPUT_DIR="${SKILL_DIR}/vss-perf-report"

# Show top-level execution summary for single_file
echo "=== single_file_test execution summary ==="
cat "${OUTPUT_DIR}/single_file_test/execution_summary.json" | python3 -m json.tool

# Show per-test-case summaries
find "${OUTPUT_DIR}" -name "test_case_summary.json" | while read f; do
  echo ""
  echo "=== $(dirname $f | xargs basename) ==="
  cat "$f" | python3 -m json.tool
done

# List generated XLSX reports
find "${OUTPUT_DIR}" -name "*.xlsx" | while read f; do
  echo "Report: $f"
done

Read the JSON result files and extract the following key metrics for analysis:

• E2E latency (e2e_latency in seconds): total time from request to response
• VLM pipeline latency (vlm_pipeline_latency): time spent in the vision model pipeline
• VLM pipeline %: vlm_pipeline_latency / e2e_latency * 100
• CA-RAG latency (ca_rag_latency): context-aware RAG inference time
• VLM GPU utilization mean (vlm_gpu_usage_mean): GPU compute utilization % for VLM
• LLM GPU utilization mean (llm_gpu_usage_mean): GPU compute utilization % for LLM
• GPU memory mean (vlm_gpu_memory_mean, llm_gpu_memory_mean): memory pressure %
• File burst throughput (throughput_files_per_second): concurrent files processed per second
• Optimal concurrency (optimal_target_concurrency.estimated_concurrency): the concurrency level that meets target_latency_seconds

---

Step 7: Suggest Improvements

After analyzing the results, present a findings table and recommendations based on the following rules:

Observation	Likely Cause	Recommended Action
VLM GPU utilization mean < 70%	VLM is under-utilized — chunks too small or few frames	Increase chunk_duration (e.g., 10 → 30) or num_frames_per_chunk (e.g., 20 → 40)
VLM pipeline % > 70% of E2E latency	VLM processing dominates — resolution too high	Reduce vlm_input_width/vlm_input_height (e.g., 1312x736 → 896x504 for ~4k tokens)
GPU memory mean > 85%	Near out-of-memory — risk of OOM at higher load	Reduce batch size or lower num_frames_per_chunk; avoid higher concurrency levels
File burst throughput plateaus between concurrency levels	Hardware is saturated — optimal concurrency found	The concurrency level just before plateau is the recommended operating point
High iteration variance (std% > 20%)	Thermal throttling or memory pressure between runs	Allow longer cooldown between iterations; check GPU temperatures
CA-RAG latency > 20% of E2E latency	Elasticsearch indexing or retrieval bottleneck	Check Elasticsearch container health; consider increasing heap size
LLM GPU utilization mean < 50%	LLM is waiting on VLM output	VLM is the bottleneck; optimize VLM settings first

---

Repeat Runs

Once setup is complete (Steps 1–4), use run_benchmark.sh for subsequent runs:

SKILL_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
cd "${SKILL_DIR}"

# Run single_file scenario
./scripts/run_benchmark.sh --scenario single_file_test

# Run file_burst scenario
./scripts/run_benchmark.sh --scenario file_burst_test

# Run with a versioned output directory
./scripts/run_benchmark.sh --scenario single_file_test --output-dir vss-perf-report-v2

# Run all scenarios in config.yaml
./scripts/run_benchmark.sh

# Debug mode (verbose API payloads)
./scripts/run_benchmark.sh --scenario single_file_test --debug

run_benchmark.sh checks LVS readiness, starts the media server if needed, creates/reuses the Python venv, and passes all extra arguments directly to vss_perf_benchmark.py.

---

Troubleshooting

Symptom	Cause	Fix
POST /files returns 404	VIA_DEV_API not set or set to false	Set VIA_DEV_API=true in services/video-summarization/.env (the lvs-server env_file) and recreate the LVS service
GPU metrics all zeros	GPU monitoring requires local execution with NVML access	Run the benchmark on the GPU host directly (not via SSH without GPU passthrough)
Videos not found on media server	VSS_BENCHMARK_DATA_DIR not set correctly or videos/ subdirectory missing	Check path with ls ${VSS_BENCHMARK_DATA_DIR}/videos/; ensure Step 2 clips were created
OOM on LVS container during file_burst	Too many concurrent requests consuming GPU memory	Reduce concurrency_levels in config.yaml (remove the highest levels)
ngc download fails	NGC_API_KEY not set or expired	Run ngc config set and verify the key at https://ngc.nvidia.com/setup/api-key
Output directory not empty error	Previous run's results exist	Move previous results: mv vss-perf-report vss-perf-report-backup
ModuleNotFoundError in benchmark	Python venv not activated or requirements not installed	Run source scripts/vss-bench-env/bin/activate && pip install -r scripts/requirements.txt

---

Cross-reference

• vss-build-vision-ai — deploy LVS with the lvs profile before benchmarking
• vss-summarize-video — operational use of LVS for video summarization (not benchmarking)
• Benchmark modes reference — references/benchmark-modes.md
• Analyzing results reference — references/analyzing-results.md

bump:3