### Install Data Designer Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1 Install the data-designer package using pip. This is the initial setup step. ```bash pip install data-designer ``` -------------------------------- ### Usage Example Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/code_reference/run_config Example of how to set the run configuration using the RunConfig class. ```APIDOC ```python import data_designer.config as dd from data_designer.interface import DataDesigner data_designer = DataDesigner() data_designer.set_run_config(dd.RunConfig( buffer_size=500, max_conversation_restarts=3, )) ``` ``` -------------------------------- ### Install Local or Published Plugin Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/plugins/overview Use uv pip to install a local plugin for development or pip to install a published plugin from PyPI. ```bash # Install a local plugin (for development and testing) uv pip install -e /path/to/your/plugin ``` ```bash # Or install a published plugin from PyPI pip install data-designer-{plugin-name} ``` -------------------------------- ### Verify Development Setup Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/CONTRIBUTING Run tests and checks to ensure the environment is configured correctly. ```bash make test && make check-all ``` -------------------------------- ### Install Plugin Locally for Testing Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/plugins/overview Install your plugin in editable mode using uv pip for local testing and iteration. ```bash # Install your plugin locally with `uv pip install -e .` (editable mode) ``` -------------------------------- ### Install Plugin Locally Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/plugins/example Install the plugin in editable mode using uv pip for immediate use and development without PyPI publishing. ```bash # From the plugin directory uv pip install -e . ``` -------------------------------- ### Install Project Dependencies Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/CONTRIBUTING Commands to install development dependencies, including options for Jupyter/IPython environments. ```bash # Install project with dev dependencies make install-dev # Or, if you use Jupyter / IPython for development make install-dev-notebooks ``` -------------------------------- ### Validate Configuration Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/notebooks/5-generating-images Confirms the configuration setup is valid. ```text [16:32:12] [INFO] ✅ Validation passed ``` -------------------------------- ### Preview Dataset with Custom Configuration Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/concepts/models/custom-model-settings Generate a preview of the dataset based on the configured builder. The `display_sample_record()` method shows an example of the generated data. ```python preview_result = data_designer.preview(config_builder=config_builder) preview_result.display_sample_record() ``` -------------------------------- ### Validate Configuration Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/notebooks/2-structured-outputs-and-jinja-expressions Verify the configuration setup using the validate method. ```text [16:29:54] [INFO] ✅ Validation passed ``` -------------------------------- ### Generate First Dataset with Data Designer Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1 Initialize Data Designer, configure columns using a builder, and generate a preview of the synthetic dataset. This example demonstrates creating multilingual greetings. ```python import data_designer.config as dd from data_designer.interface import DataDesigner # Initialize with default model providers data_designer = DataDesigner() config_builder = dd.DataDesignerConfigBuilder() # Add a sampler column to randomly select a language config_builder.add_column( dd.SamplerColumnConfig( name="language", sampler_type=dd.SamplerType.CATEGORY, params=dd.CategorySamplerParams( values=["English", "Spanish", "French", "German", "Italian"], ), ) ) # Add an LLM text generation column config_builder.add_column( dd.LLMTextColumnConfig( name="greeting", model_alias="nvidia-text", prompt="Write a casual and formal greeting in {{ language }}.", ) ) # Generate a preview results = data_designer.preview(config_builder) results.display_sample_record() ``` -------------------------------- ### Complete Data Designer Workflow with Tool Use Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/concepts/mcp/enabling-tools This example demonstrates a full workflow from configuring an MCP provider to generating data with LLM columns that utilize tools. Ensure the `tool_alias` in the `LLMTextColumnConfig` matches the `tool_alias` defined in the `ToolConfig`. ```python import data_designer.config as dd from data_designer.interface import DataDesigner # 1. Configure MCP provider mcp_provider = dd.LocalStdioMCPProvider( name="demo-mcp", command="python", args=["-m", "my_mcp_server"], ) # 2. Create DataDesigner instance with provider data_designer = DataDesigner(mcp_providers=[mcp_provider]) # 3. Define tool configuration tool_config = dd.ToolConfig( tool_alias="my-tools", providers=["demo-mcp"], allow_tools=["search_docs", "get_fact"], max_tool_call_turns=5, ) # 4. Create config builder with tool config builder = dd.DataDesignerConfigBuilder(tool_configs=[tool_config]) # 5. Add columns that use tools builder.add_column( dd.SamplerColumnConfig( name="question", sampler_type=dd.SamplerType.CATEGORY, params=dd.CategorySamplerParams( values=["What is machine learning?", "Explain neural networks"] ), ) ) builder.add_column( dd.LLMTextColumnConfig( name="answer", prompt="Use the available tools to research and answer: {{ question }}", model_alias="nvidia-text", tool_alias="my-tools", # Enable tools with_trace=dd.TraceType.ALL_MESSAGES, # Capture tool call history ) ) # 6. Generate data results = data_designer.preview(builder, num_records=5) ``` -------------------------------- ### Verify Data Designer Configuration Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1 Use the CLI command to list pre-configured model providers and models. This verifies your API key setup and available services. ```bash data-designer config list ``` -------------------------------- ### Get CLI Help Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/concepts/models/configure-model-settings-with-the-cli Access help information for the Data Designer CLI. Use `--help` to see available commands and sub-commands. ```bash data-designer --help ``` ```bash data-designer config --help ``` -------------------------------- ### Example Output of Custom Plugin Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/plugins/example This is the expected output when using the IndexMultiplierColumnGenerator plugin to create a 'scaled_index' column. ```text category scaled_index 0 B 0 1 A 5 2 C 10 3 A 15 4 B 20 ... ``` -------------------------------- ### Expand Topics with Full Column Strategy and Resizing Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/concepts/custom_columns Demonstrates the `FULL_COLUMN` generation strategy with `allow_resize=True` to expand input rows into multiple output rows. This example generates three variations for each input topic. ```python @dd.custom_column_generator( required_columns=["topic"], side_effect_columns=["variation_id"], ) def expand_topics(df: pd.DataFrame, params: None, models: dict) -> pd.DataFrame: rows = [] for _, row in df.iterrows(): for i in range(3): # Generate 3 variations per input rows.append({ "topic": row["topic"], "question": f"Question {i+1} about {row['topic']}", "variation_id": i, }) return pd.DataFrame(rows) dd.CustomColumnConfig( name="question", generator_function=expand_topics, generation_strategy=dd.GenerationStrategy.FULL_COLUMN, allow_resize=True, ) ``` -------------------------------- ### Example Trace with Tool Use Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/concepts/traces A JSON representation of a full conversation trace including system messages, user prompts, assistant tool calls, and tool responses. ```json [ # System message { "role": "system", "content": [{"type": "text", "text": "You must call tools before answering. Only use tool results."}] }, # User message (the rendered prompt) { "role": "user", "content": [{"type": "text", "text": "What documents are in the knowledge base about machine learning?"}] }, # Assistant requests tool calls { "role": "assistant", "content": [{"type": "text", "text": ""}], "tool_calls": [ { "id": "call_abc123", "type": "function", "function": { "name": "list_docs", "arguments": "{\"query\": \"machine learning\"}" } } ] }, # Tool response (linked by tool_call_id) { "role": "tool", "content": [{"type": "text", "text": "Found 3 documents: intro_ml.pdf, neural_networks.pdf, transformers.pdf"}], "tool_call_id": "call_abc123" }, # Final assistant response { "role": "assistant", "content": [{"type": "text", "text": "The knowledge base contains three documents about machine learning: ..."}] } ] ``` -------------------------------- ### Run Product Info Q&A Recipe Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/recipes/qa_and_chat/product_info_qa Commands to execute the recipe or view help options. ```bash uv run product_info_qa.py ``` ```bash uv run product_info_qa.py --help ``` -------------------------------- ### GET /api/config/seed Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/code_reference/config_builder Retrieves the seed configuration for the current Data Designer setup. ```APIDOC ## GET /api/config/seed ### Description Get the seed config for the current Data Designer configuration. Returns the seed config if configured, otherwise None. ### Method GET ### Endpoint /api/config/seed ### Response #### Success Response (200) - **seed_config** (SeedConfig | None) - The seed config if configured, None otherwise. #### Response Example ```json { "seed_config": { "seed_value": 12345 } } ``` ```json { "seed_config": null } ``` ``` -------------------------------- ### Display CLI Help Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/concepts/mcp/configure-mcp-cli View available commands for the configuration interface. ```bash data-designer config --help ``` -------------------------------- ### Set Up Environment for Data Designer Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/notebooks Commands to extract tutorial files and launch Jupyter using either uv or standard pip/venv workflows. ```bash # Extract tutorial notebooks unzip data_designer_tutorial.zip cd data_designer_tutorial # Launch Jupyter uv run jupyter notebook ``` ```bash # Extract tutorial notebooks unzip data_designer_tutorial.zip cd data_designer_tutorial # Create Python virtual environment and install required packages python -m venv venv source venv/bin/activate pip install data-designer jupyter # Launch Jupyter jupyter notebook ``` -------------------------------- ### List All Configurations Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/concepts/mcp/configure-mcp-cli Display all currently configured model providers, models, MCP providers, and tool configurations. ```bash data-designer config list ``` -------------------------------- ### Python Code Validator Output Example Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/concepts/validators Example of a validation result from the Python code validator. It includes validity status, a quality score, the highest severity level found, and a list of linter messages. ```json { "is_valid": False, "python_linter_score": 0, "python_linter_severity": "error", "python_linter_messages": [ { "type": "error", "symbol": "F821", "line": 1, "column": 7, "message": "Undefined name `it`" } ] } ``` -------------------------------- ### Configure Tool Settings Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/concepts/mcp/configure-mcp-cli Launch the interactive wizard to manage tool configurations. ```bash data-designer config tools ``` -------------------------------- ### SQL Code Validator Output Examples Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/concepts/validators Examples of validation results from the SQL code validator. The first shows a valid SQL query with an empty error message, and the second shows an invalid query with a parsing error. ```json # Valid SQL { "is_valid": True, "error_messages": "" } ``` ```json # Invalid SQL { "is_valid": False, "error_messages": "PRS: Line 1, Position 1: Found unparsable section: 'NOT SQL'" } ``` -------------------------------- ### Main Execution Block Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/recipes/qa_and_chat/multi_turn_chat Parses command-line arguments for model alias, number of records, and artifact path, then builds the configuration and creates the dataset. ```python if __name__ == "__main__": from argparse import ArgumentParser parser = ArgumentParser() parser.add_argument("--model-alias", type=str, default="openai-text") parser.add_argument("--num-records", type=int, default=5) parser.add_argument("--artifact-path", type=str, default=None) args = parser.parse_args() config_builder = build_config(model_alias=args.model_alias) results = create_dataset(config_builder, num_records=args.num_records, artifact_path=args.artifact_path) print(f"Dataset saved to: {results.artifact_storage.final_dataset_path}") results.load_analysis().to_report() ``` -------------------------------- ### Register Plugin Entry Point Format Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/plugins/example Understand the format for registering plugins as entry points, specifying the entry point name and the module path to the plugin instance. ```text [project.entry-points."data_designer.plugins"] = ":" ``` -------------------------------- ### GET /api/config/profilers Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/code_reference/config_builder Retrieves a list of all configured profiler objects. ```APIDOC ## GET /api/config/profilers ### Description Get all profilers. Returns a list of profiler configuration objects. ### Method GET ### Endpoint /api/config/profilers ### Response #### Success Response (200) - **profilers** (list[ColumnProfilerConfigT]) - A list of profiler configuration objects. #### Response Example ```json { "profilers": [ { "name": "profiler_name", "type": "column", "params": { ... } } ] } ``` ``` -------------------------------- ### Get all profilers Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/code_reference/config_builder Retrieves a list of all configured profiler objects. ```python def get_profilers(self) -> list[ColumnProfilerConfigT]: """Get all profilers. Returns: A list of profiler configuration objects. """ return self._profilers ``` -------------------------------- ### Commit Changes Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/CONTRIBUTING Example of a descriptive commit message referencing an issue. ```bash git commit -m "Add XYZ generator for synthetic data" -m "Fixes #123" ``` -------------------------------- ### Manage MCP and Tool Configurations via CLI Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/concepts/mcp/configure-mcp-cli Commands for configuring providers, tools, and listing current settings. ```bash # Configure MCP providers data-designer config mcp # Configure tool configs data-designer config tools # List all configurations (including MCP) data-designer config list ``` -------------------------------- ### Get All Column Configurations Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/code_reference/config_builder Retrieves a list of all available column configurations. ```APIDOC ## GET /api/columns ### Description Get all column configurations. ### Method GET ### Endpoint `/api/columns` ### Response #### Success Response (200) - **list[ColumnConfigT]** (array) - A list of all column configuration objects. #### Response Example ```json [ { "name": "column1", "type": "integer" }, { "name": "column2", "type": "string" } ] ``` ``` -------------------------------- ### Get Required Columns Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/code_reference/column_configs Retrieves the list of columns that are mandatory for data validation. ```APIDOC ## GET /websites/nvidia-nemo_github_io_datadesigner_0_5_1/required_columns ### Description Returns the columns that need to be validated. ### Method GET ### Endpoint /websites/nvidia-nemo_github_io_datadesigner_0_5_1/required_columns ### Parameters #### Query Parameters - **None** ### Request Example ```json { "example": "No request body needed for this GET request." } ``` ### Response #### Success Response (200) - **required_columns** (list of strings) - A list of column names that are required for validation. #### Response Example ```json { "required_columns": [ "column1", "column2", "column3" ] } ``` ``` -------------------------------- ### Run Text-to-SQL Generation Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/recipes/code_generation/text_to_sql Commands to execute the generation script and access help documentation. ```bash # Basic usage (generates 5 records by default) uv run text_to_sql.py # For help message and available options uv run text_to_sql.py --help ``` -------------------------------- ### GET /api/config/columns/count/{column_type} Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/code_reference/config_builder Counts the number of columns that match a specified data type. ```APIDOC ## GET /api/config/columns/count/{column_type} ### Description Get the count of columns of the specified type. Returns the number of columns matching the specified type. ### Method GET ### Endpoint /api/config/columns/count/{column_type} ### Parameters #### Path Parameters - **column_type** (DataDesignerColumnType) - Required - The type of columns to count. ### Response #### Success Response (200) - **count** (int) - The number of columns matching the specified type. #### Response Example ```json { "count": 15 } ``` ``` -------------------------------- ### GET /api/config/tools/{alias} Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/code_reference/config_builder Retrieves a specific tool configuration object using its alias. ```APIDOC ## GET /api/config/tools/{alias} ### Description Get a tool configuration by alias. Returns the tool configuration object. ### Method GET ### Endpoint /api/config/tools/{alias} ### Parameters #### Path Parameters - **alias** (str) - Required - The alias of the tool configuration to retrieve. ### Response #### Success Response (200) - **tool_config** (ToolConfig) - The tool configuration object. #### Response Example ```json { "tool_config": { "tool_alias": "my_tool", "type": "some_type", "params": { ... } } } ``` ### Error Handling #### Error Response (404) - **error** (str) - If no tool configuration with the given alias exists. #### Error Example ```json { "error": "No tool configuration with alias 'non_existent_alias' found" } ``` ``` -------------------------------- ### Manage Model Configurations Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/concepts/models/configure-model-settings-with-the-cli Initiate the interactive command to manage model configurations. Ensure at least one provider is configured before adding models. ```bash data-designer config models ``` -------------------------------- ### Get seed configuration Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/code_reference/config_builder Retrieves the seed configuration if it has been set, otherwise returns None. ```python def get_seed_config(self) -> SeedConfig | None: """Get the seed config for the current Data Designer configuration. Returns: The seed config if configured, None otherwise. """ return self._seed_config ``` -------------------------------- ### Run MCP Server Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/recipes/mcp_and_tooluse/pdf_qa Initializes the search index and runs the MCP server. Reads PDF sources from the PDF_SOURCES environment variable or defaults to a sample PDF. ```python def serve() -> None: """Run the MCP server (called when launched as subprocess by Data Designer).""" pdf_sources_json = os.environ.get("PDF_SOURCES", "[]") pdf_sources = json.loads(pdf_sources_json) if not pdf_sources: pdf_sources = [DEFAULT_PDF_URL] initialize_search_index(pdf_sources) mcp_server.run() ``` -------------------------------- ### Get Constraints for Target Column Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/code_reference/config_builder Retrieves all constraints associated with a specific target column. ```APIDOC ## GET /api/constraints/{target_column} ### Description Get all constraints for the given target column. ### Method GET ### Endpoint `/api/constraints/{target_column}` ### Parameters #### Path Parameters - **target_column** (str) - Required - Name of the column to get constraints for. ### Response #### Success Response (200) - **list[ColumnConstraintT]** (array) - A list of constraint objects targeting the specified column. #### Response Example ```json [ { "target_column": "example_column", "constraint_type": "unique" } ] ``` ``` -------------------------------- ### Get Column Configurations of Type Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/code_reference/config_builder Retrieves column configurations that match a specified type. ```APIDOC ## GET /api/columns/type/{column_type} ### Description Get all column configurations of the specified type. ### Method GET ### Endpoint `/api/columns/type/{column_type}` ### Parameters #### Path Parameters - **column_type** (DataDesignerColumnType) - Required - The type of columns to filter by. ### Response #### Success Response (200) - **list[ColumnConfigT]** (array) - A list of column configurations matching the specified type. #### Response Example ```json [ { "name": "column2", "type": "string" } ] ``` ``` -------------------------------- ### Display Sample Record Preview Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/notebooks/1-the-basics Run this code multiple times to cycle through preview records. It displays generated columns and their values. ```text Generated Columns ┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Name ┃ Value ┃ ┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ product_category │ Clothing │ ├─────────────────────┼──────────────────────────────────────────────────────────────────────────────────────┤ │ product_subcategory │ Women's Clothing │ ├─────────────────────┼──────────────────────────────────────────────────────────────────────────────────────┤ │ target_age_range │ 18-25 │ ├─────────────────────┼──────────────────────────────────────────────────────────────────────────────────────┤ │ customer │ { │ │ │ 'uuid': 'f37b4c69-76a8-4e08-863e-aaf29b4fc656', │ │ │ 'locale': 'en_US', │ │ │ 'first_name': 'David', │ │ │ 'last_name': 'Henry', │ │ │ 'middle_name': None, │ │ │ 'sex': 'Male', │ │ │ 'street_number': '95130', │ │ │ 'street_name': 'Miranda Land', │ │ │ 'city': 'West Timothy', │ │ │ 'state': 'New Jersey', │ │ │ 'postcode': '18917', │ │ │ 'age': 51, │ │ │ 'birth_date': '1974-05-18', │ │ │ 'country': 'Poland', │ │ │ 'marital_status': 'married_present', │ │ │ 'education_level': 'graduate', │ │ │ 'unit': '', │ │ │ 'occupation': 'Architect', │ │ │ 'phone_number': '769.369.4438x0153', │ │ │ 'bachelors_field': 'education' │ │ │ } │ ├─────────────────────┼──────────────────────────────────────────────────────────────────────────────────────┤ │ number_of_stars │ 5 │ ├─────────────────────┼──────────────────────────────────────────────────────────────────────────────────────┤ │ review_style │ detailed │ ├─────────────────────┼──────────────────────────────────────────────────────────────────────────────────────┤ │ product_name │ Neon Pulse Skirt │ │ │ Pulse │ │ │ Electric Tapestry Tee │ │ │ Tapestry │ │ │ Circuit Skirt │ │ │ Skirt │ ``` -------------------------------- ### GET /config_builder/info/samplers Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/notebooks/1-the-basics Retrieves a list of available samplers and their configuration requirements for synthetic data generation. ```APIDOC ## GET /config_builder/info/samplers ### Description Displays the available samplers, their parameters, data types, and constraints for the NeMo Data Designer. ### Method GET ### Endpoint config_builder.info.display("samplers") ### Response #### Success Response (200) - **Type** (string) - The name of the sampler type (e.g., bernoulli, gaussian, person). - **Parameter** (string) - The specific configuration parameter required for the sampler. - **Data Type** (string) - The expected data type for the parameter. - **Required** (boolean) - Indicates if the parameter is mandatory. - **Constraints** (string) - Any specific validation rules or ranges for the parameter. ``` -------------------------------- ### Configure MCP Providers Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/concepts/mcp/configure-mcp-cli Launch the interactive wizard to add, update, or delete MCP providers. ```bash data-designer config mcp ``` -------------------------------- ### GET /api/config/processors Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/code_reference/config_builder Retrieves all available processor configuration objects, categorized by dataset builder stage. ```APIDOC ## GET /api/config/processors ### Description Get processor configuration objects. Returns a dictionary of processor configuration objects by dataset builder stage. ### Method GET ### Endpoint /api/config/processors ### Response #### Success Response (200) - **processors** (list[ProcessorConfigT]) - A dictionary of processor configuration objects by dataset builder stage. #### Response Example ```json { "processors": [ { "stage": "stage_name", "config": { ... } } ] } ``` ``` -------------------------------- ### Clone and Configure Repository Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/CONTRIBUTING Initial commands to clone a fork and set the upstream remote. ```bash git clone https://github.com/YOUR_GITHUB_USERNAME/DataDesigner.git cd DataDesigner git remote add upstream https://github.com/NVIDIA-NeMo/DataDesigner.git ``` -------------------------------- ### Get Column Configurations Excluding Type Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/code_reference/config_builder Retrieves column configurations, excluding those of a specified type. ```APIDOC ## GET /api/columns/exclude-type/{column_type} ### Description Get all column configurations excluding the specified type. ### Method GET ### Endpoint `/api/columns/exclude-type/{column_type}` ### Parameters #### Path Parameters - **column_type** (DataDesignerColumnType) - Required - The type of columns to exclude. ### Response #### Success Response (200) - **list[ColumnConfigT]** (array) - A list of column configurations that do not match the specified type. #### Response Example ```json [ { "name": "column1", "type": "integer" } ] ``` ``` -------------------------------- ### Prepare and add a seed dataset Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/notebooks/3-seeding-with-a-dataset Downloads an external CSV file and registers it as a seed source within the configuration builder. ```python # Download sample dataset from Github import urllib.request url = "https://raw.githubusercontent.com/NVIDIA/GenerativeAIExamples/refs/heads/main/nemo/NeMo-Data-Designer/data/gretelai_symptom_to_diagnosis.csv" local_filename, _ = urllib.request.urlretrieve(url, "gretelai_symptom_to_diagnosis.csv") # Seed datasets are passed as reference objects to the config builder. seed_source = dd.LocalFileSeedSource(path=local_filename) config_builder.with_seed_dataset(seed_source) ``` -------------------------------- ### Get Column Configuration by Name Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/code_reference/config_builder Retrieves a specific column configuration using its unique name. ```APIDOC ## GET /api/columns/{name} ### Description Get a column configuration by name. ### Method GET ### Endpoint `/api/columns/{name}` ### Parameters #### Path Parameters - **name** (str) - Required - Name of the column to retrieve the config for. ### Response #### Success Response (200) - **ColumnConfigT** (object) - The column configuration object. #### Response Example ```json { "name": "example_column", "type": "string", "description": "An example column" } ``` ### Errors - **KeyError**: If no column with the given name exists. ``` -------------------------------- ### Configure and Run Data Designer Pipeline Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/devnotes/graduate-level-science-reasoning-data-with-nemo-data-designer Demonstrates the full workflow for configuring a reasoning model, defining column generation strategies, and executing the dataset creation process. ```python import data_designer.config as dd from data_designer.interface import DataDesigner # Configure your model model_configs = [ dd.ModelConfig( alias="reasoning-model", model="qwen/qwen3-235b-a22b", provider="nvidia", inference_parameters=dd.ChatCompletionInferenceParams( max_tokens=8192, timeout=300, # 5 minute timeout for long reasoning chains ), ), ] # Build the workflow config = dd.DataDesignerConfigBuilder(model_configs=model_configs) config.with_seed_dataset( dd.LocalFileSeedSource(path="path/to/your_seed_data.parquet"), sampling_strategy=dd.SamplingStrategy.SHUFFLE, ) # Generate questions config.add_column( dd.LLMTextColumnConfig( name="question", prompt=QUESTION_PROMPT, model_alias="reasoning-model", ) ) # Generate answers with reasoning trace config.add_column( dd.LLMTextColumnConfig( name="answer", prompt="{{ question }}", model_alias="reasoning-model", extract_reasoning_content=True, # Extract reasoning into separate column ) ) # Combine into final sample config.add_column( dd.ExpressionColumnConfig( name="rqa_sample", expr="{{ question }}\n\n{{ answer__reasoning_content }}\n\n{{ answer }}", ) ) # Run generation and save to disk data_designer = DataDesigner() result = data_designer.create( config_builder=config, num_records=N_RECORDS, dataset_name="rqa_dataset", ) ``` -------------------------------- ### Get processor configurations Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/code_reference/config_builder Retrieves a dictionary of processor configuration objects organized by dataset builder stage. ```python def get_processor_configs(self) -> list[ProcessorConfigT]: """Get processor configuration objects. Returns: A dictionary of processor configuration objects by dataset builder stage. """ return self._processor_configs ``` -------------------------------- ### Configure ModelProvider with API Keys Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/concepts/models/model-providers Demonstrates the two methods for setting the API key: referencing an environment variable or providing a direct string. ```python # Method 1: Environment variable (recommended) provider = ModelProvider( name="nvidia", endpoint="https://integrate.api.nvidia.com/v1", api_key="NVIDIA_API_KEY", # Will be resolved from environment ) # Method 2: Direct value (not recommended) provider = ModelProvider( name="nvidia", endpoint="https://integrate.api.nvidia.com/v1", api_key="nvapi-abc123...", # Direct API key ) ``` -------------------------------- ### Configure DataDesigner with Sampler Columns Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/notebooks/5-generating-images Initializes a configuration builder and adds multiple category-based sampler columns for image generation attributes. ```python config_builder = dd.DataDesignerConfigBuilder(model_configs=model_configs) config_builder.add_column( dd.SamplerColumnConfig( name="style", sampler_type=dd.SamplerType.CATEGORY, params=dd.CategorySamplerParams( values=[ "photorealistic", "oil painting", "watercolor", "digital art", "sketch", "anime", ], ), ) ) config_builder.add_column( dd.SamplerColumnConfig( name="dog_breed", sampler_type=dd.SamplerType.CATEGORY, params=dd.CategorySamplerParams( values=[ "a Golden Retriever", "a German Shepherd", "a Labrador Retriever", "a Bulldog", "a Beagle", "a Poodle", "a Corgi", "a Siberian Husky", "a Dalmatian", "a Yorkshire Terrier", "a Boxer", "a Dachshund", "a Doberman Pinscher", "a Shih Tzu", "a Chihuahua", "a Border Collie", "an Australian Shepherd", "a Cocker Spaniel", "a Maltese", "a Pomeranian", "a Saint Bernard", "a Great Dane", "an Akita", "a Samoyed", "a Boston Terrier", ], ), ) ) config_builder.add_column( dd.SamplerColumnConfig( name="cat_breed", sampler_type=dd.SamplerType.CATEGORY, params=dd.CategorySamplerParams( values=[ "a Persian", "a Maine Coon", "a Siamese", "a Ragdoll", "a Bengal", "an Abyssinian", "a British Shorthair", "a Sphynx", "a Scottish Fold", "a Russian Blue", "a Birman", "an Oriental Shorthair", "a Norwegian Forest Cat", "a Devon Rex", "a Burmese", "an Egyptian Mau", "a Tonkinese", "a Himalayan", "a Savannah", "a Chartreux", "a Somali", "a Manx", "a Turkish Angora", "a Balinese", "an American Shorthair", ], ), ) ) config_builder.add_column( dd.SamplerColumnConfig( name="dog_age", sampler_type=dd.SamplerType.CATEGORY, params=dd.CategorySamplerParams( values=["1-3", "3-6", "6-9", "9-12", "12-15"], ), ) ) config_builder.add_column( dd.SamplerColumnConfig( name="cat_age", sampler_type=dd.SamplerType.CATEGORY, params=dd.CategorySamplerParams( values=["1-3", "3-6", "6-9", "9-12", "12-18"], ), ) ) config_builder.add_column( dd.SamplerColumnConfig( name="dog_look_direction", sampler_type=dd.SamplerType.CATEGORY, params=dd.CategorySamplerParams( values=["left", "right", "front", "up", "down"], ), ) ) config_builder.add_column( dd.SamplerColumnConfig( name="cat_look_direction", sampler_type=dd.SamplerType.CATEGORY, params=dd.CategorySamplerParams( values=["left", "right", "front", "up", "down"], ), ) ) config_builder.add_column( dd.SamplerColumnConfig( name="dog_emotion", sampler_type=dd.SamplerType.CATEGORY, params=dd.CategorySamplerParams( values=["happy", "curious", "serious", "sleepy", "excited"], ), ) ) config_builder.add_column( dd.SamplerColumnConfig( name="cat_emotion", sampler_type=dd.SamplerType.CATEGORY, params=dd.CategorySamplerParams( values=["aloof", "curious", "content", "sleepy", "playful"], ), ) ) config_builder.add_column( dd.ImageColumnConfig( name="generated_image", prompt=( """ A {{ style }} family pet portrait of a {{ dog_breed }} dog of {{ dog_age }} years old looking {{dog_look_direction}} with an {{ dog_emotion }} expression and {{ cat_breed }} cat of {{ cat_age }} years old looking {{ cat_look_direction }} with an {{ cat_emotion }} expression in the background. Both subjects should be in focus. """ ), model_alias=MODEL_ALIAS, ) ) data_designer.validate(config_builder) ``` -------------------------------- ### GET /dataset/statistics/filter Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/code_reference/analysis Filters column statistics within the DatasetProfilerResults object based on the specified column type. ```APIDOC ## GET /dataset/statistics/filter ### Description Filters the column statistics of a dataset profile to return only those matching a specific column type. ### Method GET ### Endpoint /dataset/statistics/filter ### Parameters #### Query Parameters - **column_type** (DataDesignerColumnType) - Required - The type of column to filter by. ### Response #### Success Response (200) - **statistics** (list[ColumnStatisticsT]) - A list of statistics objects matching the requested column type. ``` -------------------------------- ### Configure Resizable Column Generator Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/plugins/example Example configuration for a plugin that modifies row counts, requiring allow_resize to be set to True. ```python class MyColumnConfig(SingleColumnConfig): column_type: Literal["my-plugin"] = "my-plugin" allow_resize: bool = True # required when output row count can differ from input # ... ``` -------------------------------- ### Build Data Designer Config with Sampler and Image Columns Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/notebooks/5-generating-images Initializes a `DataDesignerConfigBuilder` and adds multiple `SamplerColumnConfig` for dog and cat attributes (breed, age, look direction, emotion), followed by an `ImageColumnConfig` that uses a Jinja2 prompt to generate images based on these attributes. This setup is useful for creating diverse datasets of pet portraits. ```python config_builder = dd.DataDesignerConfigBuilder(model_configs=model_configs) config_builder.add_column( dd.SamplerColumnConfig( name="style", sampler_type=dd.SamplerType.CATEGORY, params=dd.CategorySamplerParams( values=[ "photorealistic", "oil painting", "watercolor", "digital art", "sketch", "anime", ], ), ) ) config_builder.add_column( dd.SamplerColumnConfig( name="dog_breed", sampler_type=dd.SamplerType.CATEGORY, params=dd.CategorySamplerParams( values=[ "a Golden Retriever", "a German Shepherd", "a Labrador Retriever", "a Bulldog", "a Beagle", "a Poodle", "a Corgi", "a Siberian Husky", "a Dalmatian", "a Yorkshire Terrier", "a Boxer", "a Dachshund", "a Doberman Pinscher", "a Shih Tzu", "a Chihuahua", "a Border Collie", "an Australian Shepherd", "a Cocker Spaniel", "a Maltese", "a Pomeranian", "a Saint Bernard", "a Great Dane", "an Akita", "a Samoyed", "a Boston Terrier", ], ), ) ) config_builder.add_column( dd.SamplerColumnConfig( name="cat_breed", sampler_type=dd.SamplerType.CATEGORY, params=dd.CategorySamplerParams( values=[ "a Persian", "a Maine Coon", "a Siamese", "a Ragdoll", "a Bengal", "an Abyssinian", "a British Shorthair", "a Sphynx", "a Scottish Fold", "a Russian Blue", "a Birman", "an Oriental Shorthair", "a Norwegian Forest Cat", "a Devon Rex", "a Burmese", "an Egyptian Mau", "a Tonkinese", "a Himalayan", "a Savannah", "a Chartreux", "a Somali", "a Manx", "a Turkish Angora", "a Balinese", "an American Shorthair", ], ), ) ) config_builder.add_column( dd.SamplerColumnConfig( name="dog_age", sampler_type=dd.SamplerType.CATEGORY, params=dd.CategorySamplerParams( values=["1-3", "3-6", "6-9", "9-12", "12-15"], ), ) ) config_builder.add_column( dd.SamplerColumnConfig( name="cat_age", sampler_type=dd.SamplerType.CATEGORY, params=dd.CategorySamplerParams( values=["1-3", "3-6", "6-9", "9-12", "12-18"], ), ) ) config_builder.add_column( dd.SamplerColumnConfig( name="dog_look_direction", sampler_type=dd.SamplerType.CATEGORY, params=dd.CategorySamplerParams( values=["left", "right", "front", "up", "down"], ), ) ) config_builder.add_column( dd.SamplerColumnConfig( name="cat_look_direction", sampler_type=dd.SamplerType.CATEGORY, params=dd.CategorySamplerParams( values=["left", "right", "front", "up", "down"], ), ) ) config_builder.add_column( dd.SamplerColumnConfig( name="dog_emotion", sampler_type=dd.SamplerType.CATEGORY, params=dd.CategorySamplerParams( values=["happy", "curious", "serious", "sleepy", "excited"], ), ) ) config_builder.add_column( dd.SamplerColumnConfig( name="cat_emotion", sampler_type=dd.SamplerType.CATEGORY, params=dd.CategorySamplerParams( values=["aloof", "curious", "content", "sleepy", "playful"], ), ) ) config_builder.add_column( dd.ImageColumnConfig( name="generated_image", prompt=( """ A {{ style }} family pet portrait of a {{ dog_breed }} dog of {{ dog_age }} years old looking {{dog_look_direction}} with an {{ dog_emotion }} expression and {{ cat_breed }} cat of {{ cat_age }} years old looking {{ cat_look_direction }} with an {{ cat_emotion }} expression in the background. Both subjects should be in focus. """ ), model_alias=MODEL_ALIAS, ) ) data_designer.validate(config_builder) ``` -------------------------------- ### Get tool configuration by alias Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/code_reference/config_builder Retrieves a specific tool configuration using its alias. Raises a KeyError if the alias is not found. ```python def get_tool_config(self, alias: str) -> ToolConfig: """Get a tool configuration by alias. Args: alias: The alias of the tool configuration to retrieve. Returns: The tool configuration object. Raises: KeyError: If no tool configuration with the given alias exists. """ for tc in self._tool_configs: if tc.tool_alias == alias: return tc raise KeyError(f"No tool configuration with alias {alias!r} found") ``` -------------------------------- ### Resize Image Utility Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/notebooks/4-providing-images-as-context Resizes a PIL Image object to a target height while preserving the aspect ratio. Ensure PIL is installed. ```python def resize_image(image, height: int): """ Resize image while maintaining aspect ratio. Args: image: PIL Image object height: Target height in pixels Returns: Resized PIL Image object """ original_width, original_height = image.size width = int(original_width * (height / original_height)) return image.resize((width, height)) ``` -------------------------------- ### Initialize and Run MCP Server Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/devnotes/deep-research-trajectories-with-nemo-data-designer-and-mcp-tool-use Initializes the corpus and runs the MCP server. The corpus path is read from the CORPUS_PATH environment variable, defaulting to 'corpus.jsonl'. Ensure the environment variable is set or the default file exists. ```python def serve() -> None: """Run as MCP server subprocess (called by Data Designer).""" corpus_path = os.environ.get("CORPUS_PATH", "corpus.jsonl") initialize(corpus_path) mcp_server.run() ```