pixi-environment-builder

Pixi Environment Builder

Overview

Use this skill to design, migrate, and debug pixi-managed environments. The core principle is to clarify environment intent before editing pixi.toml: version constraints, project scope, package source priority, mirror/network policy, special packages, cache location, and validation tasks.

Pixi can solve dependencies automatically, but mixed conda + PyPI environments need deliberate package ownership. Most hard failures come from unclear ownership, unconstrained top-level packages, inaccessible mirrors, or non-registry packages.

Preflight Questions

Before creating or changing a pixi environment, ask these questions unless the answer is already known from repo files, user context, or error logs:

1. Required versions

   • Are any package versions fixed by previous results, notebooks, papers, models, CUDA drivers, or collaborators?
   • Examples: python, cuda, pytorch, domain packages, model libraries, analysis frameworks.

2. Environment scope

   • Is this project-level, user/global-level, or temporary?
   • Project-level: create or edit repo pixi.toml.
   • User/global-level: prefer pixi global for reusable CLI tools, not complex project workflows.

3. Package source priority

   • Should conda or PyPI own the main dependency graph?
   • Which packages should be installed from conda, PyPI, Git, local path, or system modules?
   • Avoid specifying the same package unconstrained in both conda and PyPI.

4. Multiple environments or kernels

   • Does the user need separate named environments, solve groups, or Jupyter/VS Code kernels?
   • Clarify whether they need identical packages in separate prefixes or different feature sets.

5. Mirror and network policy

   • Which conda channels and mirrors are reachable?
   • Which PyPI index is reachable?
   • Can the machine access GitHub, pypi.org, files.pythonhosted.org, prefix.dev, or internal mirrors?

6. Non-registry packages

   • Are any packages installed from local source, private Git repos, wheels, editable paths, or unpublished projects?

7. Cache and storage

   • Use pixi defaults unless there is a permissions, quota, or sharing requirement.
   • If custom cache is needed, ask where writable shared cache should live.

8. Validation

   • What imports, version checks, CLI commands, GPU checks, or kernel registration prove the environment works?

Design Rules

Prefer project-level manifests for project workflows

For analysis projects, put environment definition in the repo:

[workspace]
name = "project-name"
channels = ["conda-forge"]
platforms = ["linux-64"]

Use user/global environments mostly for standalone tools.

Assign package ownership

Choose one owner for each important package family.

Prefer conda for:

• Python interpreter
• compiled libraries and hard-to-build scientific packages
• CUDA/PyTorch stacks when conda binaries are desired
• R, rpy2, system libraries, CLI bioinformatics tools
• packages requiring consistent native ABI

Prefer PyPI for:

• packages whose canonical release is PyPI
• fast-moving Python-only libraries
• packages unavailable or stale on conda
• top-level frameworks that expect pip-style dependency resolution
• headless/server variants such as opencv-python-headless

Avoid this pattern:

[dependencies]
scanpy = "*"
anndata = "*"
scipy = "*"

[pypi-dependencies]
some-framework-that-also-depends-on-scanpy = "*"

This can make conda pin versions before PyPI solves, causing conflicts.

Start minimal, then add constraints only when evidence requires them

Do not mechanically copy an entire old conda environment. Start from:

• interpreter/runtime
• top-level packages the user directly uses
• hardware/runtime packages
• Jupyter/kernel tooling if needed
• non-registry packages

Add transitive pins only when solver output or runtime validation proves they are needed.

Migrating From Conda

When migrating an existing conda/mamba environment:

1. Inspect by path if env-name lookup is unreliable:

conda list -p /path/to/env
conda env export -p /path/to/env --no-builds

2. Identify direct imports and notebook evidence:

rg -n "^(import|from) " scripts src notebooks tests --glob '*.py' --glob '*.ipynb'
rg -n "Version:|__version__|import " notebooks scripts --glob '*.ipynb'

3. Classify packages:

   • direct user dependencies
   • transitive dependencies
   • runtime/system dependencies
   • local/Git/private packages
   • packages only needed for old experiments

4. Preserve known compatibility anchors:

   • versions printed in notebook outputs
   • versions required by published workflow
   • CUDA/PyTorch compatibility
   • package versions known to affect results

5. Leave unrelated transitive packages out of pixi.toml.

Multiple Environments And Kernels

Use multiple named environments when the user needs isolation or parallel notebooks.

Use one solve group when environments should have identical package versions:

[environments]
worker-1 = { solve-group = "analysis" }
worker-2 = { solve-group = "analysis" }
worker-3 = { solve-group = "analysis" }

Use separate features when environments differ:

[feature.gpu.dependencies]
pytorch-cuda = "*"

[feature.r.dependencies]
r-base = "*"

[environments]
cpu = []
gpu = ["gpu"]
r-analysis = ["r"]

For VS Code/Jupyter kernels, add explicit kernel tasks:

[tasks]
kernel-1 = "python -m ipykernel install --user --name worker-1 --display-name 'Python (worker-1)'"
kernel-2 = "python -m ipykernel install --user --name worker-2 --display-name 'Python (worker-2)'"
kernels = "pixi run -e worker-1 kernel-1 && pixi run -e worker-2 kernel-2"

Tell VS Code users: after registration, select the kernel in VS Code; they do not need to launch notebooks through pixi run.

Mirrors And Network

Use mirrors deliberately. Do not assume a mirror works for all package types.

Recommended checks:

pixi config list
sed -n '1,120p' ~/.config/uv/uv.toml 2>/dev/null
sed -n '1,120p' ~/.config/pip/pip.conf 2>/dev/null
env | rg "PIP|UV|PIXI|RATTLER|HTTP|HTTPS|PROXY"

For PyPI, prefer setting only the index URL in pixi.toml:

[pypi-options]
index-url = "https://example-mirror/simple"

Avoid unnecessary files.pythonhosted.org mirror rewrites unless verified. Some mirrors serve simple index pages but fail wheel metadata URLs.

If official PyPI times out, switch to a reachable mirror. If a mirror gives 404 for metadata, try another mirror or the official file server directly.

For conda, use .pixi/config.toml mirrors when needed:

[mirrors]
"https://conda.anaconda.org/conda-forge" = [
  "https://your-conda-mirror/anaconda/cloud/conda-forge",
  "https://conda.anaconda.org/conda-forge"
]

Conda + PyPI Mapping

Pixi needs conda-to-PyPI name mapping when conda and PyPI dependencies are mixed. If fetching mapping from prefix.dev fails, use a local mapping file.

In pixi.toml:

[workspace]
conda-pypi-map = { "conda-forge" = "config/conda-pypi-map.json" }

Example config/conda-pypi-map.json:

{
  "scikit-learn": "scikit-learn",
  "matplotlib-base": "matplotlib",
  "pytorch": "torch",
  "torchvision": "torchvision",
  "torchaudio": "torchaudio"
}

Validate it:

python -m json.tool config/conda-pypi-map.json

Keep this mapping small and project-specific. Add entries only for packages relevant to mixed solving.

Non-Registry Packages

If a package is not found on PyPI or conda, inspect how it was installed before guessing.

Check old environment metadata:

find /path/to/env/lib/python*/site-packages -maxdepth 3 \
  \( -iname '*dist-info' -o -path '*dist-info/direct_url.json' \)

Use local path dependency when reproducible on this machine:

[pypi-dependencies]
my-package = { path = "/absolute/path/to/source" }

Use Git dependency when portability matters:

[pypi-dependencies]
my-package = { git = "https://github.com/org/repo.git", rev = "commit-sha" }

Prefer a fixed commit/tag for reproducibility.

Common Failure Patterns

Package not found in registry

Root cause: package is unpublished, private, named differently, or only installed from source.

Actions:

• Inspect old direct_url.json.
• Search project docs for install command.
• Use path or Git dependency.
• Do not keep retrying PyPI.

Version conflict after conda solve

Root cause: conda pinned a transitive package version that conflicts with PyPI requirements.

Actions:

• Read the solver message for pinned packages.
• Decide whether top-level package should own those dependencies.
• Remove conda-side transitive packages, or bound them to a compatible range.
• Pin only compatibility anchors, not every transitive package.

Network timeout fetching PyPI package

Root cause: inaccessible PyPI index, file host, proxy, or mirror.

Actions:

• Check current [pypi-options] index-url.
• Check user uv/pip config for reachable mirrors.
• Change only PyPI index first.
• Clean PyPI cache and retry.

pixi clean cache --pypi -y
pixi install --all

Mirror metadata 404

Root cause: simple index mirror works but wheel metadata/file mirror is incomplete.

Actions:

• Remove files.pythonhosted.org mirror rewrites.
• Use a different PyPI index.
• Avoid mixing multiple PyPI mirror layers unless verified.

OpenCV solve conflict

Root cause: conda opencv pulls GUI/Qt/Python ABI-specific builds.

Server/headless fix:

[pypi-dependencies]
opencv-python-headless = ">=4.10,<5"

Use conda opencv only when GUI functionality is required.

CUDA/PyTorch mismatch

Root cause: CUDA runtime, driver, PyTorch build, and channel priorities disagree.

Actions:

• Ask for nvidia-smi.
• Pin CUDA runtime intentionally.
• Use a single coherent PyTorch source.
• Validate with torch.cuda.is_available().

Cache Guidance

Pixi has default cache. Do not create custom cache directories unless the user requests shared cache, has permission errors, or needs a specific storage path.

Useful commands:

pixi clean cache --pypi -y
pixi clean cache --repodata -y
pixi clean cache --mapping -y

If custom cache is needed:

PIXI_CACHE_DIR=/path/to/pixi-cache \
RATTLER_CACHE_DIR=/path/to/rattler-cache \
pixi install --all

Verification Tasks

Add a check task for complex environments. It should verify the actual success criteria, not just installation.

Examples:

[tasks]
check = "python -c \"import sys; print(sys.version)\""

For GPU Python environments:

check = "python -c \"import torch; print(torch.__version__); print(torch.version.cuda); print(torch.cuda.is_available()); print(torch.cuda.get_device_name(0) if torch.cuda.is_available() else 'NO GPU')\""

Run:

pixi run -e <environment> check

For Jupyter/VS Code workflows, verify kernel registration separately:

pixi run kernels
jupyter kernelspec list

OpenBLAS Thread Tuning For R Environments

conda-forge r-base ships with OpenBLAS, but when OPENBLAS_NUM_THREADS is unset on a high-core server, the default thread scheduling is extremely poor — SVD on 96 cores without explicit thread count is slower than single-threaded. This directly impacts Seurat RunPCA() (backed by irlba() randomized SVD).

When To Apply

• User reports RunPCA / SVD / matrix operations are slow in a pixi R environment
• OPENBLAS_NUM_THREADS and OMP_NUM_THREADS are both unset
• Multi-core Linux server (>16 cores)

Diagnostic Flow

Step 1 - Confirm BLAS Implementation

PREFIX=$(pixi info --manifest-path pixi-workspaces/<env>/pixi.toml 2>/dev/null | grep "Prefix location" | awk '{print $NF}')
readlink -f "$PREFIX/lib/libblas.so.3"

• Points to libopenblasp-*.so → OpenBLAS, this section applies
• Points to libflexiblas.so → different approach needed (FlexiBLAS backend switching)

Step 2 - Baseline Benchmark

pixi run --manifest-path pixi-workspaces/<env>/pixi.toml \
  Rscript -e '
  cat("OPENBLAS_NUM_THREADS =", Sys.getenv("OPENBLAS_NUM_THREADS"), "\n")
  set.seed(42); n <- 3000; X <- matrix(rnorm(n*n), n, n)
  t <- system.time({ svd(X, nu=10, nv=0) })
  cat("SVD(3000) time:", round(t["elapsed"], 3), "sec\n")
  cat("Detected cores:", parallel::detectCores(), "\n")
  '

• SVD(3000) > 15 sec on 96 cores → confirmed thread scheduling problem
• SVD(3000) < 5 sec → already optimized, no action needed

Step 3 - Thread Count Sweep

for threads in 1 8 16 32 64; do
    echo "=== OPENBLAS_NUM_THREADS=$threads ==="
    OPENBLAS_NUM_THREADS=$threads OMP_NUM_THREADS=$threads \
    pixi run --manifest-path pixi-workspaces/<env>/pixi.toml \
    Rscript -e '
    set.seed(42); n <- 3000; X <- matrix(rnorm(n*n), n, n)
    t <- system.time({ svd(X, nu=10, nv=0) })
    cat("SVD(3000) time:", round(t["elapsed"],3), "sec\n")
    ' 2>/dev/null
done

Step 4 - Pick Optimal Thread Count

Benchmark results on AMD EPYC 7K62 (96 cores, Zen2):

Threads	SVD(3000) Time	Speedup
Default (unset)	~20 sec	1x (baseline)
1	~12 sec	1.6x
8	~3.8 sec	5.2x
16	~3.2 sec	6.2x
32	~2.9 sec	6.8x (optimal)
64	~3.3 sec	6.0x (overhead degrades)

Rule of thumb: optimal thread count ≈ 1/3 of total cores (96→32, 64→16-24, 32→8-16). Beyond the sweet spot, thread synchronization overhead degrades performance.

Fix - pixi.toml activation.env (Recommended)

Add [activation.env] to the workspace pixi.toml:

[activation.env]
OPENBLAS_NUM_THREADS = "32"
OMP_NUM_THREADS = "32"

Takes effect on every pixi run or environment activation. Jupyter kernels (IRkernel) registered via pixi run kernel also inherit these variables.

Gotchas

• Do NOT use all cores: 96 cores fully loaded is ~15% slower than 32 threads
• Set OMP_NUM_THREADS too: some R packages (data.table, RcppParallel) use OpenMP
• conda-forge r-base already links OpenBLAS: unlike system R (/opt/R), no need to manually replace libRblas.so symlinks
• Optimal thread count varies by CPU: AMD EPYC (Zen2) vs Intel Xeon may differ

Debugging Discipline

• Treat each error type separately: network timeout, package not found, version conflict, mapping failure, runtime import failure.
• Change one thing per solver error where possible.
• Do not rewrite the manifest blindly after every failure.
• Do not run install commands if the user asked only for diagnosis or commands.
• Explain whether warnings are harmless or actionable.