### Install a single example package Source: https://docs.nvidia.com/nemo/agent-toolkit/latest/resources/contributing/index.html Install a single example package using `uv pip install -e ./examples/`. Example shown installs the Simple Calculator example. ```bash uv pip install -e ./examples/getting_started/simple_web_query ``` -------------------------------- ### Install nat_simple_web_query workflow Source: https://docs.nvidia.com/nemo/agent-toolkit/latest/get-started/quick-start.html Installs the simple web query workflow example using uv. ```bash uv pip install -e examples/getting_started/simple_web_query ``` -------------------------------- ### Install simple_calculator_observability workflow Source: https://docs.nvidia.com/nemo/agent-toolkit/latest/run-workflows/observe/observe.html Installs the simple_calculator example workflow used in the Weave guide. ```bash uv pip install -e examples/observability/simple_calculator_observability ``` -------------------------------- ### Install the simple calculator example Source: https://docs.nvidia.com/nemo/agent-toolkit/latest/improve-workflows/sizing-calc.html Install the simple calculator example for the sizing calculator. ```bash uv pip install -e examples/evaluation_and_profiling/simple_calculator_eval ``` -------------------------------- ### Run simple calculator observability example with Arize AX Source: https://docs.nvidia.com/nemo/agent-toolkit/latest/run-workflows/observe/observe.html Installs the example package and runs the workflow with the Arize AX config. Expect a log line 'Started exporter 'arize_ax'' and traces in the Arize project. ```bash uv pip install -e examples/observability/simple_calculator_observability/ nat run --config_file examples/observability/simple_calculator_observability/configs/config-arize-ax.yml --input "What is 2 * 4?" ``` -------------------------------- ### Install NeMo Agent Toolkit and simple web query example Source: https://docs.nvidia.com/nemo/agent-toolkit/latest/build-workflows/llms/using-local-llms.html Run from the repository root to install the toolkit and the simple web query example package. ```bash uv pip install -e . uv pip install -e examples/getting_started/simple_web_query ``` -------------------------------- ### Install and Serve Simple Calculator Workflow Source: https://docs.nvidia.com/nemo/agent-toolkit/latest/reference/rest-api/api-server-endpoints.html Installs the simple calculator example and starts the NeMo Agent Toolkit server on http://localhost:8000. Run these commands before using the API examples. ```bash uv pip install -e examples/getting_started/simple_calculator ``` ```bash nat serve --config_file examples/getting_started/simple_calculator/configs/config.yml ``` -------------------------------- ### Install nvidia-nat with uv Source: https://docs.nvidia.com/nemo/agent-toolkit/latest/index.html Install the NeMo Agent Toolkit package using the uv package manager. ```bash uv pip install nvidia-nat ``` -------------------------------- ### Install and Run simple_web_query Workflow with OTel+Dynatrace Source: https://docs.nvidia.com/nemo/agent-toolkit/latest/run-workflows/observe/observe.html Commands to install the simple_web_query example workflow and run it with OTel+Dynatrace telemetry settings. Telemetry data will appear in Dynatrace as the workflow runs. ```bash # Install the workflow and plugins uv pip install -e examples/getting_started/simple_web_query # Run the workflow with OTel+Dynatrace telemetry settings nat run --config_file examples/getting_started/simple_web_query/configs/config.yml --input "What is LangSmith?" ``` -------------------------------- ### Install from source with vLLM backend Source: https://docs.nvidia.com/nemo/export-deploy/latest/index.html Installs the package with the vLLM backend after installing prerequisites. ```bash pip install --no-build-isolation .[vllm] ``` -------------------------------- ### Install pre-commit hooks Source: https://docs.nvidia.com/nemo/agent-toolkit/latest/resources/contributing/index.html Install and configure pre-commit hooks. First run takes longer than normal. ```bash pre-commit install ``` -------------------------------- ### nat start fastapi example command Source: https://docs.nvidia.com/nemo/agent-toolkit/latest/reference/cli.html Launches a FastAPI server for a workflow using the specified configuration file, binding to host 0.0.0.0 on port 8000. After starting, Swagger API documentation is available at http://localhost:8000/docs. ```bash nat start fastapi --config_file=path/to/config --host 0.0.0.0 --port 8000 ``` -------------------------------- ### Install and Run simple_calculator_observability Workflow with Galileo Source: https://docs.nvidia.com/nemo/agent-toolkit/latest/run-workflows/observe/observe.html Commands to install the simple_calculator_observability example and run it with Galileo telemetry settings. Configuration may need updating based on your Galileo account. ```bash # Install the workflow and plugins uv pip install -e examples/observability/simple_calculator_observability/ # Run the workflow with Galileo telemetry settings # Note, you may have to update configuration settings based on your Galileo account nat run --config_file examples/observability/simple_calculator_observability/configs/config-galileo.yml --input "What is 1*2?" ``` -------------------------------- ### Set OpenAI API key environment variable Source: https://docs.nvidia.com/nemo/agent-toolkit/latest/get-started/quick-start.html Optional setup for workflows that require OpenAI. Replace with your actual key. ```bash export OPENAI_API_KEY="" ``` -------------------------------- ### Install from source with TRT-ONNX backend Source: https://docs.nvidia.com/nemo/export-deploy/latest/index.html Installs the package with the TRT-ONNX backend after installing prerequisites. ```bash pip install --no-build-isolation .[trt-onnx] ``` -------------------------------- ### Start server with --host and --port flags Source: https://docs.nvidia.com/nemo/agent-toolkit/latest/components/auth/mcp-auth/index.html Start the server with host and port matching the redirect_uri. The first command is for a remote server example; the second is for production behind a reverse proxy. ```bash # For the remote server example above nat serve --host 192.168.1.100 --port 8080 # Or for production with a reverse proxy nat serve --host 0.0.0.0 --port 8000 ``` -------------------------------- ### Set NVIDIA API key environment variable Source: https://docs.nvidia.com/nemo/agent-toolkit/latest/get-started/quick-start.html Required before running any NeMo Agent Toolkit examples. Replace with your actual NVIDIA API key. ```bash export NVIDIA_API_KEY= ``` -------------------------------- ### Example nat optimize command with placeholder config path Source: https://docs.nvidia.com/nemo/agent-toolkit/latest/improve-workflows/optimizer.html Example invocation of the optimizer command. Replace the placeholder with the actual path to your configuration file. The command starts the optimization process and logs progress, parameters tested, and scores for each trial. ```bash nat optimize --config_file ``` -------------------------------- ### Install and launch NeMo AutoModel with uv pip and automodel CLI Source: https://docs.nvidia.com/nemo/automodel/latest Install the nemo-automodel package and run a training job using the automodel CLI with a YAML recipe. The example uses two processes per node and the llama3_2_1b_squad.yaml recipe. ```bash uv pip install nemo-automodel automodel --nproc-per-node=2 llama3_2_1b_squad.yaml ``` -------------------------------- ### setup() Source: https://docs.nvidia.com/nemo/rl/latest/index.html Sets up the GRPO training environment. ```APIDOC ## setup() ### Description Sets up the GRPO training environment. ``` -------------------------------- ### setup() (Distillation) Source: https://docs.nvidia.com/nemo/rl/latest/index.html Sets up the distillation training environment. ```APIDOC ## setup() (Distillation) ### Description Sets up the distillation training environment. ``` -------------------------------- ### Install nvidia-nat-eval[full] for full eval support Source: https://docs.nvidia.com/nemo/agent-toolkit/latest/resources/migration-guide.html Install the full extra for nvidia-nat-eval to get complete eval functionality. ```bash pip install "nvidia-nat-eval[full]" ``` -------------------------------- ### Run nat_simple_web_query workflow Source: https://docs.nvidia.com/nemo/agent-toolkit/latest/get-started/quick-start.html Executes the simple web query workflow with a sample input. ```bash nat run --config_file=examples/getting_started/simple_web_query/configs/config.yml --input "What is LangSmith" ``` -------------------------------- ### Run Hello World example with nat CLI Source: https://docs.nvidia.com/nemo/agent-toolkit/latest/index.html Runs the Hello World example using the nat CLI and the workflow.yml file. The command outputs the workflow result to the console. ```bash nat run --config_file workflow.yml --input "List five subspecies of Aardvarks" ``` -------------------------------- ### Tavily Internet Search Configuration (Breaking) Source: https://docs.nvidia.com/nemo/agent-toolkit/latest/resources/migration-guide.html Configuration for the old Tavily internet search function. This is a breaking change in v1.8.0; workflows using this configuration no longer get Tavily search by installing only nvidia-nat[langchain]. ```yaml functions: internet_search: _type: tavily_internet_search ``` -------------------------------- ### Basic finetuning command Source: https://docs.nvidia.com/nemo/agent-toolkit/latest/improve-workflows/finetuning/rl_with_openpipe.html Starts a finetuning run using the OpenPipe ART plugin. Requires a running ART server, an LLM with logprobs enabled, a training dataset, and a reward function. ```bash nat finetune --config_file=configs/finetune.yml ``` -------------------------------- ### Install nvidia-nat-security for nat red-team Source: https://docs.nvidia.com/nemo/agent-toolkit/latest/resources/migration-guide.html Install the package providing the `nat red-team` command. ```bash pip install nvidia-nat-security ``` -------------------------------- ### Access configuration in create_mcp_server() Source: https://docs.nvidia.com/nemo/agent-toolkit/latest/extend/custom-components/mcp-server.html Example showing how to access configuration via self.front_end_config (name, host, port, debug) and customize server creation based on debug mode. ```python async def create_mcp_server(self) -> FastMCP: # Access server name from config server_name = self.front_end_config.name # Customize based on debug mode if self.front_end_config.debug: logger.info(f"Creating debug server: {server_name}") return FastMCP( name=server_name, host=self.front_end_config.host, port=self.front_end_config.port, debug=self.front_end_config.debug, ) ``` -------------------------------- ### Install nvidia-nat-eval CLI package Source: https://docs.nvidia.com/nemo/agent-toolkit/latest/resources/migration-guide.html Install the package providing the `nat eval` command. ```bash pip install "nvidia-nat[eval]" ``` -------------------------------- ### Install QEMU for multi-architecture CI Source: https://docs.nvidia.com/nemo/agent-toolkit/latest/resources/contributing/testing/running-ci-locally.html Commands to install QEMU on an apt-based system for emulating a different architecture (e.g., arm64 on amd64). ```bash sudo apt install qemu-utils qemu-system-arm qemu-user-static ``` -------------------------------- ### Install nvidia-nat-ragas for RAGAS evaluator Source: https://docs.nvidia.com/nemo/agent-toolkit/latest/resources/migration-guide.html Install the RAGAS evaluator package when using `_type: ragas`. ```bash pip install nvidia-nat-ragas ``` -------------------------------- ### Curriculum Learning Training Schedule Source: https://docs.nvidia.com/nemo/agent-toolkit/latest/improve-workflows/finetuning/concepts.html Shows the training schedule for curriculum learning, where the model starts with the easiest 30% of examples and gradually expands to all examples over epochs. ```text Epoch 1-5: Train on easiest 30% of examples Epoch 6-10: Train on easiest 50% of examples Epoch 11-15: Train on easiest 70% of examples Epoch 16+: Train on all examples ``` -------------------------------- ### Hello World example output Source: https://docs.nvidia.com/nemo/agent-toolkit/latest/index.html Example console output from running the nat CLI command, showing the workflow result listing five subspecies of Aardvarks. ```text Workflow Result: ['Here are five subspecies of Aardvarks:\n\n1. Orycteropus afer afer (Southern aardvark)\n2. O. a. adametzi Grote, 1921 (Western aardvark)\n3. O. a. aethiopicus Sundevall, 1843\n4. O. a. angolensis Zukowsky & Haltenorth, 1957\n5. O. a. erikssoni Lönnberg, 1906'] ``` -------------------------------- ### Install profiling support with nvidia-nat[profiler] Source: https://docs.nvidia.com/nemo/agent-toolkit/latest/resources/migration-guide.html Install the profiling package when needed. Use the extra or combine with eval. ```bash pip install "nvidia-nat[profiler]" ``` ```bash pip install "nvidia-nat[eval, profiler]" ``` -------------------------------- ### Create workflow.yml configuration file Source: https://docs.nvidia.com/nemo/agent-toolkit/latest/index.html Defines the agents, tools, and workflows for the Hello World example. Includes a wikipedia_search tool, a nim_llm LLM, and a react_agent workflow with verbose output and retry settings. ```yaml functions: # Add a tool to search wikipedia wikipedia_search: _type: wiki_search max_results: 2 llms: # Tell NeMo Agent Toolkit which LLM to use for the agent nim_llm: _type: nim model_name: meta/llama-3.1-70b-instruct temperature: 0.0 workflow: # Use an agent that 'reasons' and 'acts' _type: react_agent # Give it access to our wikipedia search tool tool_names: [wikipedia_search] # Tell it which LLM to use llm_name: nim_llm # Make it verbose verbose: true # Retry up to 3 times parse_agent_response_max_retries: 3 ``` -------------------------------- ### Install nvidia-nat-config-optimizer package Source: https://docs.nvidia.com/nemo/agent-toolkit/latest/resources/migration-guide.html Install the optional package that owns the `nat optimize` command and optimizer implementation modules. ```bash pip install nvidia-nat-config-optimizer ``` -------------------------------- ### Install nvidia-nat[config-optimizer] extra Source: https://docs.nvidia.com/nemo/agent-toolkit/latest/resources/migration-guide.html Install config optimizer support via the renamed extra (previously `nvidia-nat[optimizer]`). ```bash pip install "nvidia-nat[config-optimizer]" ``` -------------------------------- ### Example nat finetune commands with options Source: https://docs.nvidia.com/nemo/agent-toolkit/latest/improve-workflows/finetuning/concepts.html Example commands showing basic finetuning, overriding config values with -o, adding validation with --validation_dataset and --validation_interval, and using a remote endpoint with --endpoint and --endpoint_timeout. The --endpoint_timeout default is 300 seconds; --validation_interval default is 5 epochs. ```bash # Basic finetuning nat finetune --config_file=configs/finetune.yml # Override number of epochs nat finetune --config_file=configs/finetune.yml -o finetuning.num_epochs 20 # With validation nat finetune --config_file=configs/finetune.yml \ --validation_dataset=data/val.json \ --validation_interval=3 # Using remote endpoint nat finetune --config_file=configs/finetune.yml \ --endpoint=http://localhost:8000/generate \ --endpoint_timeout=600 ``` -------------------------------- ### A365 MCP Tooling Configuration Example Source: https://docs.nvidia.com/nemo/agent-toolkit/latest/api/nat/plugins/a365/tooling/tooling_config/index.html YAML configuration for discovering and registering MCP servers from Agent 365. Requires the nvidia-nat-mcp package (install with `uv pip install nvidia-nat-mcp` or `uv pip install 'nvidia-nat[mcp]'`, or from source with `uv pip install 'nvidia-nat-a365[mcp]'`). ```yaml function_groups: - type: a365_mcp_tooling agentic_app_id: "your-agent-id" auth_token: "your-auth-token" ``` -------------------------------- ### Install from source with InFramework backend Source: https://docs.nvidia.com/nemo/export-deploy/latest/index.html Clones the Export-Deploy repository and installs the package with the InFramework backend. ```bash git clone https://github.com/NVIDIA-NeMo/Export-Deploy cd Export-Deploy/ pip install --no-build-isolation . ``` -------------------------------- ### Install and run simple_calculator_observability with Phoenix Source: https://docs.nvidia.com/nemo/agent-toolkit/latest/run-workflows/observe/observe.html Installs the workflow and plugins, then runs the workflow with Phoenix telemetry settings. Telemetry data will start appearing in Phoenix as the workflow runs. ```bash # Install the workflow and plugins uv pip install -e examples/observability/simple_calculator_observability/ # Run the workflow with Phoenix telemetry settings nat run --config_file examples/observability/simple_calculator_observability/configs/config-phoenix.yml --input "What is 1*2?" ``` -------------------------------- ### nat start fastapi --help output Source: https://docs.nvidia.com/nemo/agent-toolkit/latest/reference/cli.html Displays the help for the `nat start fastapi` command, listing all options for serving a workflow as a FastAPI microservice. The `--config_file` option is required and accepts a JSON/YAML file. Additional options include `--override` for dot-notation config overrides, `--host`, `--port`, `--reload`, `--workers`, `--step_adaptor`, `--workflow`, `--endpoints`, `--use_gunicorn`, and `--runner_class`. ```bash $ nat start fastapi --help Usage: nat start fastapi [OPTIONS] Options: --config_file FILE A JSON/YAML file that sets the parameters for the workflow. [required] --override ... Override config values using dot notation (e.g., --override llms.nim_llm.temperature 0.7) --root_path TEXT The root path for the API --host TEXT Host to bind the server to --port INTEGER Port to bind the server to --reload BOOLEAN Enable auto-reload for development --workers INTEGER Number of workers to run --step_adaptor STEPADAPTORCONFIG --workflow ENDPOINTBASE Endpoint for the default workflow. --endpoints ENDPOINT Additional endpoints to add to the FastAPI app which run functions within the NAT configuration. Each endpoint must have a unique path. --use_gunicorn BOOLEAN Use Gunicorn to run the FastAPI app --runner_class TEXT The NAT runner class to use when launching the FastAPI app from multiple processes. Each runner is responsible for loading and running the NAT workflow. Note: This is different from the worker class used by Gunicorn. --help Show this message and exit. ``` -------------------------------- ### Complete interactive workflow example Source: https://docs.nvidia.com/nemo/agent-toolkit/latest/build-workflows/advanced/interactive-workflows.html Full example of an async function that acquires the manager, creates a prompt, prompts for input, and processes the response with error handling. ```python async def _inner(prompt: str) -> str: try: context = Context.get() user_input_manager = context.user_interaction_manager human_prompt_text = HumanPromptText(text="Hello, how are you today?", required=True, placeholder="default") response = await user_input_manager.prompt_user_input(human_prompt_text) assert (isinstance(response.content, HumanResponseText)) return response.content.text except Exception as e: logger.error("An error occurred when getting interaction content: %s", e) raise ``` -------------------------------- ### Install plugin with pip Source: https://docs.nvidia.com/nemo/agent-toolkit/latest/extend/third-party-plugins.html Alternative installation method using pip. ```bash pip install nemo-agent-toolkit-tavily ``` -------------------------------- ### nat start console --help output Source: https://docs.nvidia.com/nemo/agent-toolkit/latest/reference/cli.html Displays the help for the `nat start console` command, which runs a workflow from a configuration file with inputs from the command line or a file. The `--config_file` option is required. Options include `--override` for dot-notation overrides, `--input` for a single input, `--input_file` for a plain text file (batch evaluation should use `nat eval`), `--user_id` (defaults to 'nat_run_user_id'), and `--conversation_id` for isolating memory-backed runs. ```bash $ nat start console --help Usage: nat start console [OPTIONS] Options: --config_file FILE A JSON/YAML file that sets the parameters for the workflow. [required] --override ... Override config values using dot notation (e.g., --override llms.nim_llm.temperature 0.7) --input TEXT A single input to submit the the workflow. --input_file FILE Path to a plain text file containing a single input to submit to the workflow. For batch evaluation of multiple inputs, use 'nat eval' instead. --user_id TEXT User ID to use for the workflow session. Defaults to 'nat_run_user_id' for single-user CLI execution. --conversation_id TEXT Conversation ID to use for the workflow session. Set this to isolate memory-backed CLI runs. --help Show this message and exit. ``` -------------------------------- ### Install NeMo Agent Toolkit with all groups and most extras Source: https://docs.nvidia.com/nemo/agent-toolkit/latest/get-started/installation.html Installs the library with developer tools (`--all-groups`) and most dependencies for profiling and plugins (`--extra most`). Use this for a full source installation. ```bash uv sync --all-groups --extra most ``` -------------------------------- ### Training Quickstart with Pre-configured Recipe Source: https://docs.nvidia.com/nemo/megatron-bridge/latest/ Launch random-init pretraining using a pre-configured Llama 3.2 1B recipe on a single H100 GPU. Override cfg.model to use a different Hugging Face model ID or local directory, and optionally set cfg.checkpoint.pretrained_checkpoint for SFT/PEFT workflows. ```python from megatron.bridge import AutoBridge from megatron.bridge.recipes.llama.h100 import llama32_1b_pretrain_1gpu_h100_bf16_config from megatron.bridge.training.gpt_step import forward_step from megatron.bridge.training.pretrain import pretrain if __name__ == "__main__": # The recipe uses the Llama 3.2 1B architecture from Hugging Face. # This is random-init pretraining and does not require a converted Megatron checkpoint. cfg = llama32_1b_pretrain_1gpu_h100_bf16_config() # The recipe already sets cfg.model internally using this pattern. # Override cfg.model to choose a different Hugging Face model ID as the architecture source. # cfg.model = AutoBridge.from_hf_pretrained("meta-llama/Llama-3.2-1B").to_megatron_provider(load_weights=False) # Optional: use a local Hugging Face model/config directory instead. # cfg.model = AutoBridge.from_hf_pretrained("/path/to/local/hf_model").to_megatron_provider(load_weights=False) # Optional: initialize weights from a converted Megatron checkpoint for SFT/PEFT # or other pretrained-weight workflows. # cfg.checkpoint.pretrained_checkpoint = "/path/to/megatron/checkpoint" # Override training parameters cfg.train.train_iters = 10 cfg.scheduler.lr_decay_iters = 10000 cfg.model.vocab_size = 8192 cfg.tokenizer.vocab_size = cfg.model.vocab_size pretrain(cfg, forward_step) ``` -------------------------------- ### ToolTestRunner usage example Source: https://docs.nvidia.com/nemo/agent-toolkit/latest/api/nat/test/tool_test_runner/index.html Shows how to instantiate ToolTestRunner and test a tool with minimal setup or with mocked dependencies using the with_mocks() context manager. The example is from the class docstring and demonstrates the expected usage pattern. ```python >>> runner = ToolTestRunner() >>> # Test a tool with minimal setup >>> result = await runner.test_tool( ... config_type=MyToolConfig, ... config_params={"param1": "value1"}, ... input_data="test input" ... ) >>> # Test a tool with mocked dependencies >>> async with runner.with_mocks() as mock_builder: >>> mock_builder.mock_llm("my_llm", "mocked response") >>> result = await runner.test_tool( ... config_type=MyToolConfig, ... config_params={"llm_name": "my_llm"}, ... input_data="test input" ... ) ``` -------------------------------- ### Start built container Source: https://docs.nvidia.com/nemo/export-deploy/latest/index.html Runs the container built with the Dockerfile, mounting the current directory and enabling GPU access. ```bash docker run --rm -it -w /workdir -v $(pwd):/workdir \ --entrypoint bash \ --gpus all \ nemo-export-deploy ``` -------------------------------- ### Install nvidia-nat-profiler for performance evaluators Source: https://docs.nvidia.com/nemo/agent-toolkit/latest/resources/migration-guide.html Install the profiler package when using performance evaluators or profiling workflows. Note: the previous dependency group name `nvidia-nat-profiling` is now `nvidia-nat-profiler`. ```bash pip install nvidia-nat-profiler ``` -------------------------------- ### get_outstanding_step_count() Source: https://docs.nvidia.com/nemo/agent-toolkit/latest/api/nat/builder/intermediate_step_manager/index.html Get the number of outstanding (started but not ended) steps. ```APIDOC ## get_outstanding_step_count() ### Description Get the number of outstanding (started but not ended) steps. ### Method get_outstanding_step_count ### Returns int: Number of steps that have been started but not yet ended ``` -------------------------------- ### Show nat run help and options Source: https://docs.nvidia.com/nemo/agent-toolkit/latest/reference/cli.html Displays the help output for the `nat run` command, which is an alias for `nat start console`. The help lists all available options including `--config_file`, `--override`, `--input`, and `--input_file`. The `--config_file` option is required. ```bash $ nat run --help Usage: nat run [OPTIONS] Options: --config_file FILE A JSON/YAML file that sets the parameters for the workflow. [required] --override ... Override config values using dot notation (e.g., --override llms.nim_llm.temperature 0.7) --input TEXT A single input to submit the the workflow. --input_file FILE Path to a json file of inputs to submit to the workflow. --help Show this message and exit. ``` -------------------------------- ### Reinstall workflow example Source: https://docs.nvidia.com/nemo/agent-toolkit/latest/reference/cli.html Example command to reinstall a workflow named 'my_workflow' after updating dependencies in pyproject.toml. The reinstall process rebuilds the package, uninstalls the existing version, installs the updated version, and verifies registered components. ```bash nat workflow reinstall my_workflow ``` -------------------------------- ### Install plugin with uv Source: https://docs.nvidia.com/nemo/agent-toolkit/latest/extend/third-party-plugins.html Install a third-party plugin package into the same Python environment as nvidia-nat-core using uv. ```bash uv add nemo-agent-toolkit-tavily ```