ambient-healthcare-agent-with-nemotron-voice-agent

Ambient Healthcare Agent with Nemotron Voice Agent

Purpose

Customize src/examples/generic/ in a user-selected NVIDIA Nemotron Voice Agent (NVA) checkout. Use this skill for appointment making, five-field patient intake, or a developer-defined healthcare workflow that fits one Pipecat pipeline with an LLM prompt, OpenAI-style tool schemas, and Python handlers.

This skill is owned by Healthcare TME. It is not maintained or endorsed by the NVA team, and it does not live in the NVA repository. For ordinary NVA deployment or non-healthcare voice work, follow the public NVA documentation instead.

Read the user-experience flowchart when a visual overview of the gated workflow or bundled scenario state machines would help.

Requirements

• A compatible checkout of https://github.com/NVIDIA-AI-Blueprints/nemotron-voice-agent
• Python 3.10+ with PyYAML, Git, and Docker Compose
• Network access and credentials for the user-selected LLM, ASR, and TTS services
• Permission to write only to the NVA checkout path the user explicitly supplies
• Human review before sending fictional healthcare-style histories to any model endpoint

Treat the skill loader's installed directory as SKILL_DIR. Resolve bundled scripts/ and references/ from that directory, not from the current working directory.

Required Welcome

Begin a new positive workflow with this phrase verbatim:

Welcome to the NVIDIA Nemotron Voice Agent (NVA for short). We will customize NVA for creating ambient healthcare agents.
Now I will make a fresh clone of the Nemotron Voice Agent repository, and this will be the directory we work out of. Where would you like me to clone the repo to? Please provide a path.
If you already have a clone of the repository somewhere, please point me to the path.

Then stop. Do not inspect files, search for clones, reuse a path from earlier context, run tools, or choose a default. Set NVA_ROOT only from a path the user provides or confirms after this welcome.

Instructions

1. Resolve and validate the NVA checkout

If the user requests a fresh clone, clone only to their exact destination and only with network and write permission:

git clone https://github.com/NVIDIA-AI-Blueprints/nemotron-voice-agent.git "$NVA_ROOT"

If cloning fails, report the observed reason and ask the user to either grant the session the required access via /permissions and request a retry, or manually clone the NVA repository and provide its path. Do not retry until the user grants access or supplies a checkout path. For either a fresh or existing checkout, require these markers:

test -f "$NVA_ROOT/docker-compose.yml" && \
test -f "$NVA_ROOT/examples_registry.yaml" && \
test -d "$NVA_ROOT/src/examples/generic"

An invalid path is a hard stop. Do not modify any repository before this check passes.

After the markers pass, create $NVA_ROOT/.env by copying $NVA_ROOT/.env.example when the target does not already exist. Preserve an existing .env and never print its contents. If .env is absent and the template is missing, report that setup failure and stop. Do this immediately after validating a fresh clone or an existing checkout, before asking the user to choose hosted services.

2. Inspect compatibility and service defaults

Run:

python3 "$SKILL_DIR/scripts/inspect_nva_generic_defaults.py" \
  --nva-root "$NVA_ROOT" --check-compatibility

Stop on compatibility errors. Report the inspected prompt plus the LLM, ASR, and TTS key, display name, model/server, base URL when present, and catalog source. Never substitute release-specific defaults from memory.

The preflight checks Python structure and required capabilities rather than an NVA version number or one exact source string. It also discovers deployment skills from both skills/*/SKILL.md and .agents/skills/*/SKILL.md. A passing check is not permission to guess through an unknown layout: stop when syntax or a required semantic hook is ambiguous.

3. Select the runtime and verify setup access

Explain that the default is the compatible public NVIDIA AI Endpoint entries in the NVA cloud catalog. Before any inference, let the user choose public NVIDIA endpoints, NVA-managed local NIMs, existing NIM endpoints, or a mixed layout. Never silently fall back to public endpoints after opt-out. In the same message as these choices, give the user the actual absolute path to $NVA_ROOT/.env and tell them to fill in its NVIDIA_API_KEY= entry if they choose public NVIDIA AI Endpoints. Explain that the key authenticates access to those endpoints. Direct the user to the public NVA deployment documentation for credential setup; never ask them to paste a secret into chat or display the file contents. State:

The NVIDIA_API_KEY is required to utilize public NVIDIA AI Endpoints. With this key configured, I will be running live tests while customizing and standing up a Nemotron Voice Agent application.

Before applying a healthcare overlay:

1. Run python3 "$SKILL_DIR/scripts/verify_docker_compose_access.py" and require all checks to pass. If Docker Compose access is blocked by session permissions, report the observed failure, ask the user to grant the required access via /permissions, and stop until the user requests a retry.
2. Use the public NVA deployment instructions to identify one recipe and the selected runtime's credential and endpoint requirements. Do not start the unmodified Generic recipe or run inference at this stage.

If repository compatibility or Docker Compose access fails, report the exact failed gate and stop before applying an overlay. Missing credentials or unavailable endpoints do not prevent overlay and static validation, but they block the later authenticated service checks, live validation, and handoff. Do not change runtime modes silently or claim the app is ready.

4. Obtain informed scenario selection

After checkout compatibility and Docker Compose access pass, restate the resolved runtime and services, then ask exactly:

What type of voice agent application would you like to create? We have two example default use cases, appointment making and patient intake, or you could tell me your own use case.

Offer appointment-making example, patient-intake example, and customize your own use case. In the same message, replace the destination placeholder below with the actual resolved LLM display name and base URL or host:

If you choose either example, I will apply its customization and send its bundled fictional conversation histories to <resolved destination> for live validation during setup and before handoff. The appointment example includes the fictional patient Jordan Patel, date of birth 1979-09-24, and appointment details. The patient-intake example includes the fictional patient Maya Chen, date of birth 1988-04-12, symptoms, current medications, and pharmacy details.

Wait for selection. Selecting a preset after this disclosure authorizes only the disclosed fixture and destination. Ask again if either changes.

5. Apply a preset

For a selected preset, apply the bundled overlay without additional design questions:

python3 "$SKILL_DIR/scripts/apply_generic_agent_template.py" \
  --nva-root "$NVA_ROOT" \
  --scenario-dir "$SKILL_DIR/references/appointment-making"

Use references/patient-intake for patient intake. The applier must preserve current LLM/ASR/TTS defaults, set the scenario prompt as the Generic default, patch only supported insertion points, and remain idempotent.

The applier reruns compatibility with the selected scenario before writing. To inspect that gate separately, pass --scenario appointment-making, --scenario patient-intake, or --scenario custom together with --check-compatibility. Scenario checks must cover every scenario-specific hook, including deterministic session startup for both examples.

Appointment making installs SQLite support, initializes data/appointment-making/appointment_schedule.sqlite, and creates or merges docker-compose.override.yml; it must not edit the base Compose file. It also queues the exact fixed opening greeting once at session start and suppresses the model-generated intro. Patient intake collects name, date of birth, symptoms, current medications, and preferred pharmacy, and installs deterministic turn/speech guards plus generated unit tests. Preserve its shared conversation state, earliest-missing-field question, exactly-once direct tool responses, one-time welcome, silent empty tool transitions, interruption forwarding, and interruptible welcome. These are code-backed safety invariants, not prompt-only suggestions. Read references/example-design-choices.md when implementation detail is needed.

6. Design a custom workflow

For customize your own use case, first ask what the conversation should accomplish; which fields are required, optional, or sensitive; what must be known and confirmed before each tool call; what each tool should read, write, and return; and which representative histories demonstrate message content and tool timing.

Use references/custom/guide.md and its templates. Present the proposed expected-conversation artifact, resolved destination, and data fields to the user. Do not implement or transmit it until the user approves that exact artifact and destination.

7. Validate, start, and test voice

Run static and scenario tests after applying the overlay. Check the selected runtime's credentials, then start the customized NVA recipe using its public deployment instructions; build when the source changes require it. Require the app and selected LLM/ASR/TTS health and authentication checks to pass. If a credential, startup, or service check fails, report the exact failed gate and stop before live validation. Never start the unmodified Generic recipe.

For an approved preset or custom fixture, export the selected endpoint credential in the process environment without displaying it, then run:

python3 "$SKILL_DIR/scripts/run_expected_conversation.py" \
  --nva-root "$NVA_ROOT" --live-test-approved

The runner checks tool timing, arguments, status, and result identifiers. Review the actual assistant message for semantic alignment with expected_next_message_content; do not claim success if a deterministic or semantic check fails.

After any validation-driven change, rebuild or restart the selected recipe as directed by the public NVA documentation and recheck its services. Complete one real microphone-to-ASR-to-LLM/tool-to-TTS round trip. Preset handoff is blocked until static tests, approved live histories, service health, and voice validation pass.

8. Handoff

Report the NVA path and commit; compatibility/default inspection and catalog sources; runtime mode, recipe, and repository/Docker/authentication health gates; modified files, prompt key, tool names, and data paths; static, live-history, service-health, and voice results; and the verified UI URL. If anything is incomplete, name it as pending or failed rather than saying the application is ready.

Available Scripts

Script	Purpose	Main arguments
inspect_nva_generic_defaults.py	Resolve defaults and validate supported patch capabilities	--nva-root, --check-compatibility, optional --scenario
apply_generic_agent_template.py	Apply a preset or custom overlay idempotently	--nva-root, --scenario-dir or explicit artifact paths
run_expected_conversation.py	Run an authorized live LLM conversation contract	--nva-root, --live-test-approved, optional endpoint overrides
verify_docker_compose_access.py	Test Docker daemon, Compose, and disposable startup	optional image and timeout flags
nva_generic_defaults.py	Shared inspection library imported by other scripts	library module; do not invoke directly

Invoke scripts with python3 as shown. Agent runtimes that expose a run_script facility may use it with the same argument vector.

Examples

• “Customize NVA Generic for appointment scheduling” → use this skill and start with the exact welcome.
• “Build patient intake directly in the Pipecat Generic example” → use this skill.
• “Deploy ordinary NVA Generic” → do not use this skill; follow NVA deployment documentation.
• “Create a FastAPI/LangGraph healthcare backend” → use a backend-oriented skill instead.

Limitations

• The bundled examples are demonstrations, not clinical decision support or production records systems.
• The skill does not diagnose, triage, recommend treatment, or replace privacy/security review.
• Upstream NVA changes can invalidate required capabilities; syntax-aware compatibility and the selected scenario check must both pass.
• Live calls can transmit approved fictional fixture content and incur endpoint charges.
• Voice validation requires interactive audio hardware and cannot be inferred from text-only tests.

Troubleshooting

Failure	Action
Path is not an NVA checkout	Ask for a valid explicit path; do not search the workspace
Compatibility preflight fails	Stop and report the missing layout, default, catalog, or insertion point
Authentication/service health fails	Ask the user to correct the selected endpoint configuration; do not change modes silently
Docker access/startup fails	Report daemon, permission, network, or image-pull failure and stop
Overlay application fails	Preserve the checkout, report the exact patch point, and do not hand-edit around the guard
Live history differs from expectation	Correct prompt/tool behavior, rerun static tests, then rerun the approved history

See references/deployment-modes.md, references/example-design-choices.md, and the scenario directories for deeper implementation detail.