### Complete DataDesigner Example Setup Source: https://github.com/nvidia-nemo/datadesigner/blob/main/docs/concepts/seed-datasets.md A complete example demonstrating the setup of DataDesigner, including model configurations and the DataDesignerConfigBuilder. This serves as a foundation for more complex data processing pipelines. ```python import data_designer.config as dd from data_designer.interface import DataDesigner data_designer = DataDesigner() model_configs = [ dd.ModelConfig( alias="medical-notes", model="nvidia/nemotron-3-nano-30b-a3b", provider="nvidia", ) ] config_builder = dd.DataDesignerConfigBuilder(model_configs=model_configs) ``` -------------------------------- ### Complete Workflow: Provider to Column with Tools Source: https://github.com/nvidia-nemo/datadesigner/blob/main/fern/versions/latest/pages/concepts/mcp/enabling-tools.mdx This example demonstrates the full setup, from configuring an MCP provider and `ToolConfig` to adding columns that utilize these tools. It shows how to instantiate `DataDesigner`, build a configuration, and generate data with tool-enabled LLM columns. ```python import data_designer.config as dd from data_designer.interface import DataDesigner # 1. Configure MCP provider mcp_provider = dd.LocalStdioMCPProvider( name="demo-mcp", command="python", args=["-m", "my_mcp_server"], ) # 2. Create DataDesigner instance with provider data_designer = DataDesigner(mcp_providers=[mcp_provider]) # 3. Define tool configuration tool_config = dd.ToolConfig( tool_alias="my-tools", providers=["demo-mcp"], allow_tools=["search_docs", "get_fact"], max_tool_call_turns=5, ) # 4. Create config builder with tool config builder = dd.DataDesignerConfigBuilder(tool_configs=[tool_config]) # 5. Add columns that use tools builder.add_column( dd.SamplerColumnConfig( name="question", sampler_type=dd.SamplerType.CATEGORY, params=dd.CategorySamplerParams( values=["What is machine learning?", "Explain neural networks"] ), ) ) builder.add_column( dd.LLMTextColumnConfig( name="answer", prompt="Use the available tools to research and answer: {{ question }}", model_alias="nvidia-text", tool_alias="my-tools", # Enable tools with_trace=dd.TraceType.ALL_MESSAGES, # Capture tool call history ) ) # 6. Generate data results = data_designer.preview(builder, num_records=5) ``` -------------------------------- ### Install and Run Jupyter with uv Source: https://github.com/nvidia-nemo/datadesigner/blob/main/docs/notebook_source/_README.md Use uv for recommended local environment setup. This snippet shows how to extract tutorial notebooks, navigate to the directory, and launch Jupyter. ```bash unzip data_designer_tutorial.zip cd data_designer_tutorial uv run jupyter notebook ``` -------------------------------- ### List, Search, and Install NVIDIA Plugins Source: https://github.com/nvidia-nemo/datadesigner/blob/main/docs/devnotes/posts/have-it-your-way.md Use these CLI commands to discover and install plugins from the NVIDIA catalog. No separate registration is needed after installation. ```bash data-designer plugin list data-designer plugin search data-designer plugin install ``` -------------------------------- ### Get Plugin Package Information Source: https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer/src/data_designer/cli/README.md Displays detailed information about a plugin package, including version, compatibility, documentation, and installation strategy. This example fetches info for the 'github' plugin. ```bash # Show package version, compatibility, docs, and the install strategy data-designer plugin info github ``` -------------------------------- ### Install and Run Jupyter with pip and venv Source: https://github.com/nvidia-nemo/datadesigner/blob/main/docs/notebook_source/_README.md Alternative local setup using Python's built-in venv. Installs Data Designer and Jupyter, then launches the notebook server. ```bash unzip data_designer_tutorial.zip cd data_designer_tutorial python -m venv venv source venv/bin/activate pip install data-designer jupyter jupyter notebook ``` -------------------------------- ### Example Git Tagging Source: https://github.com/nvidia-nemo/datadesigner/blob/main/VERSIONING.md An example of tagging a specific version and pushing it. ```bash git tag v0.1.0 git push origin v0.1.0 ``` -------------------------------- ### Install Data Designer Source: https://github.com/nvidia-nemo/datadesigner/blob/main/docs/colab_notebooks/3-seeding-with-a-dataset.ipynb Installs the Data Designer library. Run this cell to set up the necessary tools for the notebook. ```bash %%capture !pip install -U data-designer ``` -------------------------------- ### Serve Fern Docs Locally Source: https://github.com/nvidia-nemo/datadesigner/blob/main/fern/README.md Starts a local development server for previewing Fern documentation. This command generates notebook artifacts before starting the server. ```bash make serve-fern-docs-locally ``` -------------------------------- ### Install Development Dependencies Source: https://github.com/nvidia-nemo/datadesigner/blob/main/DEVELOPMENT.md Installs the project with development dependencies using Make. Use `install-dev-notebooks` if you primarily use Jupyter/IPython. ```bash git clone https://github.com/NVIDIA-NeMo/DataDesigner.git cd DataDesigner # Install with dev dependencies make install-dev # Or, if you use Jupyter / IPython for development make install-dev-notebooks ``` -------------------------------- ### Install Retrieval SDG Plugin Source: https://github.com/nvidia-nemo/datadesigner/blob/main/docs/devnotes/posts/retrieval-sdg-toolkit.md Install the retrieval-sdg plugin using the data-designer CLI. ```bash data-designer plugin install retrieval-sdg ``` -------------------------------- ### Preview Plugin Package Installation Source: https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer/src/data_designer/cli/README.md Performs a dry run of a plugin package installation without modifying the current environment. Useful for previewing changes. ```bash # Preview a specific package version without changing the current environment data-designer plugin install github==0.1.0 --dry-run ``` -------------------------------- ### Install Plugin Package Source: https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer/src/data_designer/cli/README.md Installs a plugin package from a catalog and verifies its runtime entry points. The `--yes` flag bypasses confirmation prompts. ```bash # Install a plugin package from a catalog and verify its runtime entry points can load data-designer plugin install github --yes ``` -------------------------------- ### Plugin Catalog Configurations Example Source: https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer/src/data_designer/cli/README.md Stores user-added plugin catalog aliases and their URLs. The built-in NVIDIA catalog is always available. This example adds a custom 'research' catalog. ```yaml catalogs: - alias: research url: https://raw.githubusercontent.com/acme/dd-plugins/main/catalog/plugins.json ``` -------------------------------- ### Install Data Designer from Source Source: https://github.com/nvidia-nemo/datadesigner/blob/main/README.md Clone the repository and install Data Designer from its source code. This is useful for development or if you need the latest unreleased features. ```bash git clone https://github.com/NVIDIA-NeMo/DataDesigner.git cd DataDesigner make install ``` -------------------------------- ### Allow Resize Configuration Example Source: https://github.com/nvidia-nemo/datadesigner/blob/main/fern/versions/latest/pages/plugins/example.mdx Example of setting `allow_resize=True` in a column configuration class when the plugin's generate method can alter the number of output rows. ```python class MyColumnConfig(SingleColumnConfig): column_type: Literal["my-plugin"] = "my-plugin" allow_resize: bool = True # required when output row count can differ from input # ... ``` -------------------------------- ### Install Data Designer via pip Source: https://github.com/nvidia-nemo/datadesigner/blob/main/README.md Install the data-designer package using pip. This is the quickest way to get started. ```bash pip install data-designer ``` -------------------------------- ### Convert and Execute Notebooks Source: https://github.com/nvidia-nemo/datadesigner/blob/main/docs/notebook_source/README.md Use this command from the repository root to convert and execute all tutorial notebooks. Ensure Quick Start is completed and API keys are set. ```bash make convert-execute-notebooks ``` -------------------------------- ### Wikidata Knowledge Graph Walk Example 1 Source: https://github.com/nvidia-nemo/datadesigner/blob/main/fern/versions/latest/pages/devnotes/posts/search-agent.mdx An example of a knowledge graph walk starting from NVIDIA, detailing the entity and relation hops. ```plaintext START ENTITY: NVIDIA (Q182477) ⬇ [chief executive officer (P169)] NODE: Jensen Huang (Q332838) ⬇ [educated at (P69)] NODE: Oregon State University (Q861888) ⬇ [located in the administrative territorial entity (P131)] NODE: Benton County (Q115372) ⬇ [named after (P138)] NODE: Thomas Hart Benton (Q178712) ``` -------------------------------- ### Product Information QA Example Source: https://github.com/nvidia-nemo/datadesigner/blob/main/docs/recipes/qa_and_chat/product_info_qa.md This Python script demonstrates how to set up and run a product information question-answering system using Nvidia Nemo. It includes necessary imports and configurations for the QA pipeline. ```python import os from nemo.collections.nlp.parts.qa_utils import QAProcessor from nemo.collections.nlp.models import QA from nemo.core.config import Configurable from nemo.core.neural_types import ChannelType, NeuralType from nemo.utils.type_utils import get_class def main(): # Example usage of the QA model for product information # This is a placeholder and would typically involve loading a pre-trained model # and configuring it for a specific domain like product information. print("Running Product Information QA Example...") # Placeholder for model loading and configuration # qa_model = QA.from_pretrained(model_name="some_product_qa_model") # qa_model.setup_test_data(test_data_file='path/to/product_data.json') # Placeholder for inference # questions = ["What are the features of product X?", "How much does product Y cost?"] # answers = qa_model.predict(questions) # print(f"Answers: {answers}") print("Product Information QA Example finished.") if __name__ == '__main__': main() ``` -------------------------------- ### Get Data Designer Package Directory Source: https://github.com/nvidia-nemo/datadesigner/blob/main/fern/assets/data-designer-got-skills/trace-baseline.html Finds the installation directory of the data_designer package using Python. ```Bash .venv/bin/python -c "import data_designer.config as dd, os; print(os.path.dirname(dd.__file__))" ``` -------------------------------- ### Get Data Designer Config Directory Source: https://github.com/nvidia-nemo/datadesigner/blob/main/docs/devnotes/posts/assets/data-designer-got-skills/trace-baseline.html This command retrieves the installation path of the data_designer.config module within a Python virtual environment. ```bash .venv/bin/python -c "import data_designer.config as dd, os; print(os.path.dirname(dd.__file__))" ``` -------------------------------- ### Dependency Map Source: https://github.com/nvidia-nemo/datadesigner/blob/main/plans/346/diagrams.md Provides a textual representation of task dependencies for the Gantt chart example, clarifying which tasks must complete before others can start. ```text A: {} ← no dependencies, from_scratch B: {A} ← cell-by-cell, waits for A per row C: {A} ← cell-by-cell, waits for A per row (parallel with B) D: {B, C} ← full-column, waits for B+C on ALL rows E: {D} ← full-column, waits for D ``` -------------------------------- ### Example vLLM Server Launch Script Source: https://github.com/nvidia-nemo/datadesigner/blob/main/fern/versions/latest/pages/recipes/vlm_long_doc/nemotron_parse_ocr.mdx Launches a vLLM server for the Nemotron-Parse model on a single H100 GPU. Ensure the chat template is correctly specified. ```bash docker run -d --gpus all \ -p 8000:8000 \ --entrypoint bash \ vllm/vllm-openai:v0.14.1 \ -c "pip install open-clip-torch albumentations timm && vllm serve nvidia/NVIDIA-Nemotron-Parse-v1.1 \ --tensor-parallel-size 1 \ --max-model-len 9000 \ --gpu-memory-utilization 0.85 \ --max-num-seqs 128 \ --chat-template /chat_template.jinja \ --trust-remote-code" ``` -------------------------------- ### Wikidata Knowledge Graph Walk Example 2 Source: https://github.com/nvidia-nemo/datadesigner/blob/main/fern/versions/latest/pages/devnotes/posts/search-agent.mdx A more complex knowledge graph walk starting from 'toothache', illustrating a longer chain of relations and entities. ```plaintext START ENTITY: toothache (Q143925) ⬇ [risk factor (P564)] NODE: smoking (Q662860) ⬇ [has effect (P1542)] NODE: Crohn's disease (Q1472) ⬇ [drug or therapy used for treatment (P2176)] NODE: TNF inhibitor (Q1536078) ⬇ [(is possible treatment of) (P2175)] reverse relation NODE: Behçet's disease (Q911427) ⬇ [symptoms and signs (P780)] NODE: inflammation (Q101991) ⬇ [drug or therapy used for treatment (P2176)] NODE: (±)-flurbiprofen (Q419890) ⬇ [significant drug interaction (P769)] NODE: parecoxib (Q347941) ⬇ [significant drug interaction (P769)] NODE: ibuprofen (Q186969) ``` -------------------------------- ### Verify Development Setup Source: https://github.com/nvidia-nemo/datadesigner/blob/main/DEVELOPMENT.md Runs tests and checks to ensure your local development environment is set up correctly. ```bash make test && make check-all ``` -------------------------------- ### Complete Data Designer Workflow with Tool Configuration Source: https://github.com/nvidia-nemo/datadesigner/blob/main/docs/concepts/mcp/enabling-tools.md This example demonstrates a full workflow from configuring an MCP provider to defining tool configurations, adding columns that use tools, and generating data. Ensure the MCP provider is correctly set up and the `tool_alias` in the column configuration matches the defined `ToolConfig`. ```python import data_designer.config as dd from data_designer.interface import DataDesigner # 1. Configure MCP provider mcp_provider = dd.LocalStdioMCPProvider( name="demo-mcp", command="python", args=["-m", "my_mcp_server"], ) # 2. Create DataDesigner instance with provider data_designer = DataDesigner(mcp_providers=[mcp_provider]) # 3. Define tool configuration tool_config = dd.ToolConfig( tool_alias="my-tools", providers=["demo-mcp"], allow_tools=["search_docs", "get_fact"], max_tool_call_turns=5, ) # 4. Create config builder with tool config builder = dd.DataDesignerConfigBuilder(tool_configs=[tool_config]) # 5. Add columns that use tools builder.add_column( dd.SamplerColumnConfig( name="question", sampler_type=dd.SamplerType.CATEGORY, params=dd.CategorySamplerParams( values=["What is machine learning?", "Explain neural networks"] ), ) ) builder.add_column( dd.LLMTextColumnConfig( name="answer", prompt="Use the available tools to research and answer: {{ question }}", model_alias="nvidia-text", tool_alias="my-tools", # Enable tools with_trace=dd.TraceType.ALL_MESSAGES, # Capture tool call history ) ) # 6. Generate data results = data_designer.preview(builder, num_records=5) ``` -------------------------------- ### Define LLM Columns for Structured Outputs Source: https://github.com/nvidia-nemo/datadesigner/blob/main/fern/versions/latest/pages/devnotes/posts/structured-outputs-from-nemotron.mdx Configure NeMo Data Designer to add LLMTextColumnConfig for generating JSON schemas, conversations, and structured JSON. This setup involves defining prompts that guide the LLM in creating these outputs based on specified controls and requirements. ```python config.add_column(dd.LLMTextColumnConfig( name="json_schema", model_alias=MODEL_ALIAS, prompt=( 'Design a JSON Schema for a response object named "scene_response".\n' "Controls:\n" "- Rigidity: {{ schema_rigidity }}\n" "- Number of top-level properties: {{ schema_fields_count }}\n" "- Topic Category: {{ topic_category }}\n" "- Topic Subtopic: {{ topic_subtopic }}\n\n" "Requirements:\n" '1. Include a "name" field set to "scene_response"\n' '2. Include a "schema" field containing a valid JSON Schema (draft 2020-12)\n' '3. Set "strict": true\n' "4. Use {{ schema_fields_count }} top-level properties\n" "5. Include at least one boolean and one enum property\n" '6. Set "additionalProperties": false\n\n" "Return ONLY the JSON object, no markdown fences." ), )) config.add_column(dd.LLMTextColumnConfig( name="conversation", model_alias=MODEL_ALIAS, prompt=( "Create a {{ num_turns }}-turn Q&A conversation about a scene " "related to {{ topic_category }} / {{ topic_subtopic }}.\n" "The conversation should naturally lead to information that fits this JSON schema:\n" "{{ json_schema }}\n\n" "Format each turn as:\nQ: [question]\nA: [answer]" ), )) config.add_column(dd.LLMTextColumnConfig( name="structured_json", model_alias=MODEL_ALIAS, prompt=( "Based on the following conversation and JSON schema, generate a JSON object " "that strictly conforms to the schema.\n\n" "Conversation:\n{{ conversation }}\n\n" "JSON Schema:\n{{ json_schema }}\n\n" "Return ONLY the valid JSON object, no explanation." ), )) ``` -------------------------------- ### Initialize and Preview DataDesigner Source: https://github.com/nvidia-nemo/datadesigner/blob/main/docs/devnotes/posts/search-agent.md Instantiate the DataDesigner with necessary providers and preview the configuration with a specified number of records. Display a sample record from the preview. ```python data_designer = DataDesigner(mcp_providers=[mcp_provider]) preview = data_designer.preview(config, num_records=5) preview.display_sample_record() ``` -------------------------------- ### Install data-designer-engine Source: https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer-engine/README.md Install the data-designer-engine package using pip. This command also automatically installs data-designer-config as a dependency. ```bash pip install data-designer-engine ``` -------------------------------- ### List All Configurations Source: https://github.com/nvidia-nemo/datadesigner/blob/main/docs/concepts/mcp/configure-mcp-cli.md View all current configurations, including model providers, model configurations, MCP providers, and tool configurations. ```bash data-designer config list ``` -------------------------------- ### Install data-designer-config Source: https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer-config/README.md Install the data-designer-config package using pip. ```bash pip install data-designer-config ``` -------------------------------- ### OpenAI-Compatible Adapter Configuration Example Source: https://github.com/nvidia-nemo/datadesigner/blob/main/plans/343/model-facade-overhaul-plan-step-1.md This YAML configuration demonstrates how to set up an OpenAI-compatible provider, including endpoint, API key, organization, and project details. It adheres to the proposed auth design goals for strongly typed and provider-specific authentication. ```yaml model_providers: - name: openai-prod provider_type: openai endpoint: https://api.openai.com/v1 auth: mode: api_key api_key: OPENAI_API_KEY organization: org_abc123 project: proj_abc123 ``` -------------------------------- ### Install Plugin Locally Source: https://github.com/nvidia-nemo/datadesigner/blob/main/docs/plugins/build_your_own.md Installs a plugin package in editable mode. This registers the `data_designer.plugins` entry point for Data Designer discovery. Remember to restart your kernel after installation. ```bash uv pip install -e . ``` -------------------------------- ### Build and Publish Package Source: https://github.com/nvidia-nemo/datadesigner/blob/main/VERSIONING.md Commands to build and publish the package using uv. ```bash uv build uv publish ``` -------------------------------- ### Install Data Designer Agent Skill Source: https://github.com/nvidia-nemo/datadesigner/blob/main/README.md Install the Data Designer skill for coding agents, specifically tested with Claude Code. Ensure Claude Code is selected as an additional agent during installation. ```bash npx skills add NVIDIA-NeMo/DataDesigner ``` -------------------------------- ### Initialize and Configure Data Designer Source: https://github.com/nvidia-nemo/datadesigner/blob/main/packages/data-designer-config/README.md Initialize the config builder with model configurations and add columns for data generation. This example demonstrates setting up a model alias, specifying model details, and defining column types with their respective parameters. ```python import data_designer.config as dd # Initialize config builder with model config(s) config_builder = dd.DataDesignerConfigBuilder( model_configs=[ dd.ModelConfig( alias="my-model", model="nvidia/nemotron-3-nano-30b-a3b", provider="nvidia", inference_parameters=dd.ChatCompletionInferenceParams(temperature=0.7), ), ] ) # Add columns config_builder.add_column( dd.SamplerColumnConfig( name="user_id", sampler_type=dd.SamplerType.UUID, params=dd.UUIDSamplerParams(prefix="user-"), ) ) config_builder.add_column( dd.LLMTextColumnConfig( name="description", prompt="Write a product description", model_alias="my-model", ) ) # Build configuration config = config_builder.build() ``` -------------------------------- ### Install Plugin from a Specific Catalog Source: https://github.com/nvidia-nemo/datadesigner/blob/main/docs/plugins/discover.md Installs a plugin package from a specified catalog using its name or alias. ```bash data-designer plugin --catalog install ``` -------------------------------- ### Install a Plugin Package Source: https://github.com/nvidia-nemo/datadesigner/blob/main/docs/plugins/discover.md Installs a plugin package using its full name or alias from the configured catalogs. ```bash data-designer plugin install github ``` -------------------------------- ### Main Execution Block for Dataset Creation Source: https://github.com/nvidia-nemo/datadesigner/blob/main/fern/versions/latest/pages/recipes/qa_and_chat/multi_turn_chat.mdx Parses command-line arguments for model alias, number of records, and artifact path, then builds the configuration and creates the dataset. ```python if __name__ == "__main__": from argparse import ArgumentParser parser = ArgumentParser() parser.add_argument("--model-alias", type=str, default="openai-text") parser.add_argument("--num-records", type=int, default=5) parser.add_argument("--artifact-path", type=str, default=None) args = parser.parse_args() config_builder = build_config(model_alias=args.model_alias) results = create_dataset(config_builder, num_records=args.num_records, artifact_path=args.artifact_path) print(f"Dataset saved to: {results.artifact_storage.final_dataset_path}") results.load_analysis().to_report() ``` -------------------------------- ### Main Entry Point for Demo Mode Source: https://github.com/nvidia-nemo/datadesigner/blob/main/fern/versions/latest/pages/recipes/mcp_and_tooluse/pdf_qa.mdx Sets up and runs the Data Designer in demo mode using the BM25S MCP server. It handles environment variable checks, configures the MCP provider, and generates/displays preview results. ```python def main() -> None: """Main entry point for the demo.""" args = parse_args() # Handle 'serve' subcommand if args.command == "serve": serve() return # Demo mode: run Data Designer with the BM25S MCP server if os.environ.get("NVIDIA_API_KEY") is None and args.model_alias.startswith("nvidia"): raise RuntimeError("NVIDIA_API_KEY must be set when using NVIDIA model aliases.") # Use provided PDFs or fall back to default pdf_sources = args.pdfs if args.pdfs else [DEFAULT_PDF_URL] # Configure MCP provider to run via stdio transport (local subprocess) mcp_provider = dd.LocalStdioMCPProvider( name=MCP_SERVER_NAME, command=sys.executable, args=[str(Path(__file__).resolve()), "serve"], env={"PDF_SOURCES": json.dumps(pdf_sources)}, ) config_builder = build_config( model_alias=args.model_alias, provider_name=MCP_SERVER_NAME, ) preview_results = generate_preview( config_builder=config_builder, num_records=args.num_records, mcp_provider=mcp_provider, ) display_preview_record(preview_results) if __name__ == "__main__": main() ``` -------------------------------- ### Install Fern CLI Source: https://github.com/nvidia-nemo/datadesigner/blob/main/fern/README.md Installs the Fern CLI globally using npm. This is a prerequisite for using Fern commands. ```bash npm install -g fern-api ``` -------------------------------- ### Run End-to-End and Tutorial Tests Source: https://github.com/nvidia-nemo/datadesigner/blob/main/DEVELOPMENT.md Executes slower end-to-end tests and runs tutorial notebooks as tests. Requires API key setup as per README.md. ```bash make test-e2e # end-to-end tests make test-run-tutorials # run tutorial notebooks as tests make test-run-recipes # run recipe scripts as tests ``` -------------------------------- ### Install Pre-commit Hooks Source: https://github.com/nvidia-nemo/datadesigner/blob/main/DEVELOPMENT.md Installs the project's pre-commit hooks to enforce code quality standards before committing. ```bash uv run pre-commit install ```