Ollama vision and image description

Vision and image description

The bundled Ollama plugin registers Ollama as an image-capable media-understanding provider, so OpenClaw can route explicit image-description requests and configured image-model defaults through local or hosted Ollama vision models.

ollama pull qwen2.5vl:7b
export OLLAMA_API_KEY="ollama-local"
openclaw infer image describe --file ./photo.jpg --model ollama/qwen2.5vl:7b --json

--model must be a full <provider/model> ref; when set, infer image describe tries that model first instead of skipping description for models that already support native vision. If the call fails, OpenClaw can continue through agents.defaults.imageModel.fallbacks; file/URL preparation errors fail before fallback is attempted. Use infer image describe for OpenClaw's image-understanding flow and configured imageModel; use infer model run --file for a raw multimodal check with a custom prompt.

To make Ollama the default image-understanding provider for inbound media:

{
  agents: {
    defaults: {
      imageModel: {
        primary: "ollama/qwen2.5vl:7b",
      },
    },
  },
}

Prefer the full ollama/<model> ref. A bare imageModel ref such as qwen2.5vl:7b normalizes to ollama/qwen2.5vl:7b only when that exact model is listed under models.providers.ollama.models with input: ["text", "image"] and no other configured image provider exposes the same bare id; otherwise use the provider prefix explicitly.

Slow local vision models can need a longer image-understanding timeout than cloud models, and can crash on constrained hardware if Ollama tries to allocate the model's full advertised vision context. Set a capability timeout and cap num_ctx:

{
  models: {
    providers: {
      ollama: {
        models: [
          {
            id: "qwen2.5vl:7b",
            name: "qwen2.5vl:7b",
            input: ["text", "image"],
            params: { num_ctx: 2048, keep_alive: "1m" },
          },
        ],
      },
    },
  },
  tools: {
    media: {
      models: [
        {
          provider: "ollama",
          model: "qwen2.5vl:7b",
          timeoutSeconds: 300,
          capabilities: ["image"],
        },
      ],
      image: {
        timeoutSeconds: 180,
      },
    },
  },
}

This timeout applies to inbound image understanding and to the explicit view_image tool. models.providers.ollama.timeoutSeconds still controls the underlying Ollama HTTP request guard for normal model calls.

Live verification:

OPENCLAW_LIVE_TEST=1 OPENCLAW_LIVE_OLLAMA_IMAGE=1 \
  pnpm test:live -- src/agents/tools/image-tool.ollama.live.test.ts

If you define models.providers.ollama.models manually, mark vision models explicitly:

{
  id: "qwen2.5vl:7b",
  name: "qwen2.5vl:7b",
  input: ["text", "image"],
  contextWindow: 128000,
  maxTokens: 8192,
}

OpenClaw rejects image-description requests for models not marked image-capable. With implicit discovery, this comes from /api/show's vision capability.