xiaomi-ocr

Xiaomi-OCR-0

Use this Skill when a task needs OCR, document parsing, KIE, or OCR-VQA. Inference runs on the user's machine with the Hugging Face checkpoint SeerRay-Lab/Xiaomi-OCR-0 and SGLang, vLLM, or the experimental Ollama backend. The MCP service owns prompts, image/PDF preparation, optional layout segmentation, crop processing, concurrency, and output formatting. Call the task tool; do not recreate these steps with separate tools or direct model requests.

Set up local inference

Choose the inference backend

Respect an explicit backend/model preference and the user's accuracy requirements. Otherwise:

• Existing local service: reuse a working Xiaomi-OCR-0 endpoint and its configured backend/model. Inspect MCP configuration, the running service, and its model list; an OpenAI-compatible URL alone does not identify the runtime. Keep a working backend unless it conflicts with the user's requirements.
• Supported GPU environment: prefer SGLang or vLLM with the original SeerRay-Lab/Xiaomi-OCR-0 checkpoint, especially for accuracy-sensitive work. Reuse a compatible installed runtime and cached weights. For a fresh setup, check current official runtime support for the host and Qwen3_5ForConditionalGeneration; choose vLLM when supported, or SGLang when it is the compatible option.
• Desktop/local setup where SGLang/vLLM is unavailable or unsuitable: use a Qwen3.5-compatible Ollama if the host has enough memory for the BF16 model. This is the convenient local route, including compatible macOS setups. Read the Ollama setup notes from the complete repository. Use longwayxu/xiaomi-ocr-0:bf16 by default; reuse it if present, otherwise pull it under the user's setup/download authorization. Do not silently substitute a quantized tag.

Ollama remains experimental: explain the documented formula differences when selecting it. BF16 weights do not establish HF output parity. If the user requires HF-equivalent results, prefer the original checkpoint through SGLang/vLLM; if that route is unavailable, explain the constraint and resolve the accuracy tradeoff with the user before choosing Ollama. Do not promise numerical equivalence from any backend name alone.

Set all three variables before running the MCP installer so values from a previous backend are not accidentally retained:

Service	XIAOMI_OCR_BACKEND	XIAOMI_OCR_LOCAL_URL (default)	XIAOMI_OCR_MODEL (default)
SGLang / vLLM	openai	http://127.0.0.1:8000/v1	SeerRay-Lab/Xiaomi-OCR-0
Ollama	ollama	http://127.0.0.1:11434/v1	longwayxu/xiaomi-ocr-0:bf16

For an existing service, use its actual loopback port and served model name. The installer persists these values in MCP configuration. Keep image preparation and task prompts inside this MCP service, including for Ollama.

Install and connect

1. Inspect existing services, runtime installations, cached weights, and available memory; select the backend using the rules above. Explain the selected route and any needed environment creation or downloads. Use existing setup authorization when it covers these changes; ask only for missing authorization or an unresolved compatibility/accuracy choice. The user does not need to select a runtime just to start installation.

2. If the selected runtime is missing, check the OS, GPU, driver and relevant CUDA/PyTorch environment against its current official installation guide. For SGLang/vLLM, require support for Qwen3_5ForConditionalGeneration. For Ollama, follow the linked setup notes. Reuse compatible installations instead of replacing another runtime by default.

3. Start the selected service bound to loopback. For Ollama, follow the linked notes and use the model tag above. For SGLang/vLLM, examples are below (adjust tensor parallelism and memory limits to the hardware):

   python -m sglang.launch_server \
     --model-path SeerRay-Lab/Xiaomi-OCR-0 \
     --host 127.0.0.1 --port 8000 --context-length 16384
   

   vllm serve SeerRay-Lab/Xiaomi-OCR-0 \
     --host 127.0.0.1 --port 8000 \
     --trust-remote-code --max-model-len 16384
   

   If the engine cannot load the model or accept images, consult its current compatibility notes. Obtain authorization for upgrades if existing setup authorization does not cover them. Do not switch to a hosted endpoint. After connecting MCP, verify a real image request before reporting inference ready.

4. Reuse or clone the complete GitHub repository. The MCP imports its shared evaluation_pipeline/ and postprocess/ modules. Run the installer in skills/xiaomi-ocr/ in that checkout (bash install.sh --agent=codex for Codex, bash install.sh for other agents). If using a detached Skill, set XIAOMI_OCR_REPO to the complete checkout. Before running bash install.sh, explain that it creates a private .venv, downloads the packages in requirements.txt, and registers one MCP entry in Codex config when applicable. Reuse an existing compatible MCP environment when available. Get the user's authorization before creating the environment or downloading dependencies if that authorization has not already been given. For other agents, merge the generated mcp-config.json entry into the current agent's MCP settings without replacing unrelated entries, then restart/rescan it. The MCP service connects only to the model's loopback listener.

5. Include printed-document region mode when completing a full agent-assisted installation, unless the user requests page-only setup. Explain the PaddlePaddle/PaddleX packages and possible PP-DocLayoutV3 weight download as part of the setup plan. Use existing setup authorization when it covers these downloads; ask only if it does not.

   Check the OS, Python version, CPU/GPU support and available GPU memory against the official PaddlePaddle installation guide and PaddleX installation guide. Install the compatible PaddlePaddle wheel in the MCP environment, then use that environment's Python to run -m pip install -r <repo>/requirements-paddlex.txt. This includes paddlex[cv] and shapely; plain paddlex alone is insufficient for PP-DocLayoutV3. Reuse compatible packages and cached weights. If GPU memory is limited, use a supported CPU build and set XIAOMI_OCR_LAYOUT_DEVICE=cpu in the MCP environment. If the Demo uses a separate Python environment, install its region dependencies there too when setting up the Demo.

   Load PP-DocLayoutV3 locally (the first use may download its weights), then verify one real ocr_image call with mode="region" before reporting region mode ready. Region detection, cropping and OCR stay inside the MCP server. If the platform is unsupported or setup fails, report the failing step and retain the verified page OCR, KIE and VQA capabilities; do not silently present page mode as region mode.

Official runtime references: SGLang Qwen3.5 guide · vLLM Qwen3.5 model implementation · vLLM supported models.

Choose the right task and document mode

• Printed, regular documents: use ocr_image or ocr_pdf with mode="region". The MCP runs PP-DocLayoutV3 locally, crops regions, sends independent requests concurrently, then restores reading order and combines output. Model prompts request OTSL; MCP converts tables to HTML inside the returned Markdown and formats LaTeX formulas.
• Scene text, handwritten notes, calligraphy, historical books, or irregular layouts: use mode="page". Region detection may split a line, column, or connected handwriting incorrectly; that can lose context and compound recognition errors. Whole-page mode preserves those relationships.
• Key information extraction: call kie_image. Pass fields when the output keys are known; the MCP builds the schema prompt and returns parsed JSON. Leave fields empty for general key-value extraction.
• Question answering about a document image: call vqa_image with the image and question. The MCP applies the concise answer format.
• PDF: call ocr_pdf, choosing page/region by the same document guidance. Prefer local file paths over base64 to avoid payload truncation.

Available MCP tools are ocr_image, ocr_pdf, kie_image, and vqa_image. Page/region preprocessing and postprocessing are internal to these calls.

Output handling

Keep the model's reading order and Markdown. Do not fill in unreadable content; retain the model's uncertainty. Table conversion and document assembly happen inside MCP. Do not ask the agent to invoke a separate postprocessor. For KIE, preserve JSON keys and values; for VQA, return the concise answer without adding unsupported detail.

If MCP reports that the local model is unavailable, inspect the configured service first, then follow the backend selection and setup authorization rules above. If region mode reports missing PaddleX/PaddlePaddle, explain the extra packages and possible layout-weight download, then use existing authorization or ask if it does not cover the installation; whole-page mode remains available in the meantime.