NeMo Data Designer

https://nvidia-nemo.github.io/DataDesigner/0.5.1
Admin
NeMo Data Designer is an orchestration framework for generating high-quality synthetic data with LLM...
Tokens:113,574
Snippets:976
Trust Score:8.3
Update:6 months ago
Show doc for...
Tokens:
Raw
### Install Data Designer

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1

Install the data-designer package using pip. This is the initial setup step.

```bash
pip install data-designer

```

--------------------------------

### Usage Example

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/code_reference/run_config

Example of how to set the run configuration using the RunConfig class.

```APIDOC
```python
import data_designer.config as dd
from data_designer.interface import DataDesigner

data_designer = DataDesigner()
data_designer.set_run_config(dd.RunConfig(
    buffer_size=500,
    max_conversation_restarts=3,
))
```
```

--------------------------------

### Install Local or Published Plugin

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/plugins/overview

Use uv pip to install a local plugin for development or pip to install a published plugin from PyPI.

```bash
# Install a local plugin (for development and testing)
uv pip install -e /path/to/your/plugin
```

```bash
# Or install a published plugin from PyPI
pip install data-designer-{plugin-name}
```

--------------------------------

### Verify Development Setup

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/CONTRIBUTING

Run tests and checks to ensure the environment is configured correctly.

```bash
make test && make check-all

```

--------------------------------

### Install Plugin Locally for Testing

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/plugins/overview

Install your plugin in editable mode using uv pip for local testing and iteration.

```bash
# Install your plugin locally with `uv pip install -e .` (editable mode)
```

--------------------------------

### Install Plugin Locally

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/plugins/example

Install the plugin in editable mode using uv pip for immediate use and development without PyPI publishing.

```bash
# From the plugin directory
uv pip install -e .

```

--------------------------------

### Install Project Dependencies

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/CONTRIBUTING

Commands to install development dependencies, including options for Jupyter/IPython environments.

```bash
# Install project with dev dependencies
make install-dev

# Or, if you use Jupyter / IPython for development
make install-dev-notebooks

```

--------------------------------

### Validate Configuration

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/notebooks/5-generating-images

Confirms the configuration setup is valid.

```text
[16:32:12] [INFO] ✅ Validation passed
```

--------------------------------

### Preview Dataset with Custom Configuration

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/concepts/models/custom-model-settings

Generate a preview of the dataset based on the configured builder. The `display_sample_record()` method shows an example of the generated data.

```python
preview_result = data_designer.preview(config_builder=config_builder)
preview_result.display_sample_record()
```

--------------------------------

### Validate Configuration

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/notebooks/2-structured-outputs-and-jinja-expressions

Verify the configuration setup using the validate method.

```text
[16:29:54] [INFO] ✅ Validation passed
```

--------------------------------

### Generate First Dataset with Data Designer

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1

Initialize Data Designer, configure columns using a builder, and generate a preview of the synthetic dataset. This example demonstrates creating multilingual greetings.

```python
import data_designer.config as dd
from data_designer.interface import DataDesigner

# Initialize with default model providers
data_designer = DataDesigner()
config_builder = dd.DataDesignerConfigBuilder()

# Add a sampler column to randomly select a language
config_builder.add_column(
    dd.SamplerColumnConfig(
        name="language",
        sampler_type=dd.SamplerType.CATEGORY,
        params=dd.CategorySamplerParams(
            values=["English", "Spanish", "French", "German", "Italian"],
        ),
    )
)

# Add an LLM text generation column
config_builder.add_column(
    dd.LLMTextColumnConfig(
        name="greeting",
        model_alias="nvidia-text",
        prompt="Write a casual and formal greeting in {{ language }}.",
    )
)

# Generate a preview
results = data_designer.preview(config_builder)
results.display_sample_record()

```

--------------------------------

### Complete Data Designer Workflow with Tool Use

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/concepts/mcp/enabling-tools

This example demonstrates a full workflow from configuring an MCP provider to generating data with LLM columns that utilize tools. Ensure the `tool_alias` in the `LLMTextColumnConfig` matches the `tool_alias` defined in the `ToolConfig`.

```python
import data_designer.config as dd
from data_designer.interface import DataDesigner

# 1. Configure MCP provider
mcp_provider = dd.LocalStdioMCPProvider(
    name="demo-mcp",
    command="python",
    args=["-m", "my_mcp_server"],
)

# 2. Create DataDesigner instance with provider
data_designer = DataDesigner(mcp_providers=[mcp_provider])

# 3. Define tool configuration
tool_config = dd.ToolConfig(
    tool_alias="my-tools",
    providers=["demo-mcp"],
    allow_tools=["search_docs", "get_fact"],
    max_tool_call_turns=5,
)

# 4. Create config builder with tool config
builder = dd.DataDesignerConfigBuilder(tool_configs=[tool_config])

# 5. Add columns that use tools
builder.add_column(
    dd.SamplerColumnConfig(
        name="question",
        sampler_type=dd.SamplerType.CATEGORY,
        params=dd.CategorySamplerParams(
            values=["What is machine learning?", "Explain neural networks"]
        ),
    )
)

builder.add_column(
    dd.LLMTextColumnConfig(
        name="answer",
        prompt="Use the available tools to research and answer: {{ question }}",
        model_alias="nvidia-text",
        tool_alias="my-tools",  # Enable tools
        with_trace=dd.TraceType.ALL_MESSAGES,  # Capture tool call history
    )
)

# 6. Generate data
results = data_designer.preview(builder, num_records=5)


```

--------------------------------

### Verify Data Designer Configuration

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1

Use the CLI command to list pre-configured model providers and models. This verifies your API key setup and available services.

```bash
data-designer config list

```

--------------------------------

### Get CLI Help

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/concepts/models/configure-model-settings-with-the-cli

Access help information for the Data Designer CLI. Use `--help` to see available commands and sub-commands.

```bash
data-designer --help
```

```bash
data-designer config --help
```

--------------------------------

### Example Output of Custom Plugin

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/plugins/example

This is the expected output when using the IndexMultiplierColumnGenerator plugin to create a 'scaled_index' column.

```text
  category  scaled_index
0        B             0
1        A             5
2        C            10
3        A            15
4        B            20
...

```

--------------------------------

### Expand Topics with Full Column Strategy and Resizing

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/concepts/custom_columns

Demonstrates the `FULL_COLUMN` generation strategy with `allow_resize=True` to expand input rows into multiple output rows. This example generates three variations for each input topic.

```python
@dd.custom_column_generator(
    required_columns=["topic"],
    side_effect_columns=["variation_id"],
)
def expand_topics(df: pd.DataFrame, params: None, models: dict) -> pd.DataFrame:
    rows = []
    for _, row in df.iterrows():
        for i in range(3):  # Generate 3 variations per input
            rows.append({
                "topic": row["topic"],
                "question": f"Question {i+1} about {row['topic']}",
                "variation_id": i,
            })
    return pd.DataFrame(rows)

dd.CustomColumnConfig(
    name="question",
    generator_function=expand_topics,
    generation_strategy=dd.GenerationStrategy.FULL_COLUMN,
    allow_resize=True,
)
```

--------------------------------

### Example Trace with Tool Use

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/concepts/traces

A JSON representation of a full conversation trace including system messages, user prompts, assistant tool calls, and tool responses.

```json
[
    # System message
    {
        "role": "system",
        "content": [{"type": "text", "text": "You must call tools before answering. Only use tool results."}]
    },
    # User message (the rendered prompt)
    {
        "role": "user",
        "content": [{"type": "text", "text": "What documents are in the knowledge base about machine learning?"}]
    },
    # Assistant requests tool calls
    {
        "role": "assistant",
        "content": [{"type": "text", "text": ""}],
        "tool_calls": [
            {
                "id": "call_abc123",
                "type": "function",
                "function": {
                    "name": "list_docs",
                    "arguments": "{\"query\": \"machine learning\"}"
                }
            }
        ]
    },
    # Tool response (linked by tool_call_id)
    {
        "role": "tool",
        "content": [{"type": "text", "text": "Found 3 documents: intro_ml.pdf, neural_networks.pdf, transformers.pdf"}],
        "tool_call_id": "call_abc123"
    },
    # Final assistant response
    {
        "role": "assistant",
        "content": [{"type": "text", "text": "The knowledge base contains three documents about machine learning: ..."}]
    }
]
```

--------------------------------

### Run Product Info Q&A Recipe

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/recipes/qa_and_chat/product_info_qa

Commands to execute the recipe or view help options.

```bash
uv run product_info_qa.py
```

```bash
uv run product_info_qa.py --help
```

--------------------------------

### GET /api/config/seed

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/code_reference/config_builder

Retrieves the seed configuration for the current Data Designer setup.

```APIDOC
## GET /api/config/seed

### Description
Get the seed config for the current Data Designer configuration. Returns the seed config if configured, otherwise None.

### Method
GET

### Endpoint
/api/config/seed

### Response
#### Success Response (200)
- **seed_config** (SeedConfig | None) - The seed config if configured, None otherwise.

#### Response Example
```json
{
  "seed_config": {
    "seed_value": 12345
  }
}
```
```json
{
  "seed_config": null
}
```
```

--------------------------------

### Display CLI Help

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/concepts/mcp/configure-mcp-cli

View available commands for the configuration interface.

```bash
data-designer config --help
```

--------------------------------

### Set Up Environment for Data Designer

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/notebooks

Commands to extract tutorial files and launch Jupyter using either uv or standard pip/venv workflows.

```bash
# Extract tutorial notebooks
unzip data_designer_tutorial.zip
cd data_designer_tutorial

# Launch Jupyter
uv run jupyter notebook
```

```bash
# Extract tutorial notebooks
unzip data_designer_tutorial.zip
cd data_designer_tutorial

# Create Python virtual environment and install required packages
python -m venv venv
source venv/bin/activate
pip install data-designer jupyter

# Launch Jupyter
jupyter notebook
```

--------------------------------

### List All Configurations

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/concepts/mcp/configure-mcp-cli

Display all currently configured model providers, models, MCP providers, and tool configurations.

```bash
data-designer config list
```

--------------------------------

### Python Code Validator Output Example

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/concepts/validators

Example of a validation result from the Python code validator. It includes validity status, a quality score, the highest severity level found, and a list of linter messages.

```json
{
    "is_valid": False,
    "python_linter_score": 0,
    "python_linter_severity": "error",
    "python_linter_messages": [
        {
            "type": "error",
            "symbol": "F821",
            "line": 1,
            "column": 7,
            "message": "Undefined name `it`"
        }
    ]
}
```

--------------------------------

### Configure Tool Settings

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/concepts/mcp/configure-mcp-cli

Launch the interactive wizard to manage tool configurations.

```bash
data-designer config tools
```

--------------------------------

### SQL Code Validator Output Examples

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/concepts/validators

Examples of validation results from the SQL code validator. The first shows a valid SQL query with an empty error message, and the second shows an invalid query with a parsing error.

```json
# Valid SQL
{
    "is_valid": True,
    "error_messages": ""
}
```

```json
# Invalid SQL
{
    "is_valid": False,
    "error_messages": "PRS: Line 1, Position 1: Found unparsable section: 'NOT SQL'"
}
```

--------------------------------

### Main Execution Block

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/recipes/qa_and_chat/multi_turn_chat

Parses command-line arguments for model alias, number of records, and artifact path, then builds the configuration and creates the dataset.

```python
if __name__ == "__main__":
    from argparse import ArgumentParser

    parser = ArgumentParser()
    parser.add_argument("--model-alias", type=str, default="openai-text")
    parser.add_argument("--num-records", type=int, default=5)
    parser.add_argument("--artifact-path", type=str, default=None)
    args = parser.parse_args()

    config_builder = build_config(model_alias=args.model_alias)
    results = create_dataset(config_builder, num_records=args.num_records, artifact_path=args.artifact_path)

    print(f"Dataset saved to: {results.artifact_storage.final_dataset_path}")

    results.load_analysis().to_report()
```

--------------------------------

### Register Plugin Entry Point Format

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/plugins/example

Understand the format for registering plugins as entry points, specifying the entry point name and the module path to the plugin instance.

```text
[project.entry-points."data_designer.plugins"]
<entry-point-name> = "<module.path>:<plugin-instance-name>"

```

--------------------------------

### GET /api/config/profilers

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/code_reference/config_builder

Retrieves a list of all configured profiler objects.

```APIDOC
## GET /api/config/profilers

### Description
Get all profilers. Returns a list of profiler configuration objects.

### Method
GET

### Endpoint
/api/config/profilers

### Response
#### Success Response (200)
- **profilers** (list[ColumnProfilerConfigT]) - A list of profiler configuration objects.

#### Response Example
```json
{
  "profilers": [
    {
      "name": "profiler_name",
      "type": "column",
      "params": { ... }
    }
  ]
}
```
```

--------------------------------

### Get all profilers

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/code_reference/config_builder

Retrieves a list of all configured profiler objects.

```python
def get_profilers(self) -> list[ColumnProfilerConfigT]:
    """Get all profilers.

    Returns:
        A list of profiler configuration objects.
    """
    return self._profilers
```

--------------------------------

### Commit Changes

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/CONTRIBUTING

Example of a descriptive commit message referencing an issue.

```bash
git commit -m "Add XYZ generator for synthetic data" -m "Fixes #123"

```

--------------------------------

### Manage MCP and Tool Configurations via CLI

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/concepts/mcp/configure-mcp-cli

Commands for configuring providers, tools, and listing current settings.

```bash
# Configure MCP providers
data-designer config mcp

# Configure tool configs
data-designer config tools

# List all configurations (including MCP)
data-designer config list
```

--------------------------------

### Get All Column Configurations

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/code_reference/config_builder

Retrieves a list of all available column configurations.

```APIDOC
## GET /api/columns

### Description
Get all column configurations.

### Method
GET

### Endpoint
`/api/columns`

### Response
#### Success Response (200)
- **list[ColumnConfigT]** (array) - A list of all column configuration objects.

#### Response Example
```json
[
  {
    "name": "column1",
    "type": "integer"
  },
  {
    "name": "column2",
    "type": "string"
  }
]
```
```

--------------------------------

### Get Required Columns

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/code_reference/column_configs

Retrieves the list of columns that are mandatory for data validation.

```APIDOC
## GET /websites/nvidia-nemo_github_io_datadesigner_0_5_1/required_columns

### Description
Returns the columns that need to be validated.

### Method
GET

### Endpoint
/websites/nvidia-nemo_github_io_datadesigner_0_5_1/required_columns

### Parameters
#### Query Parameters
- **None**

### Request Example
```json
{
  "example": "No request body needed for this GET request."
}
```

### Response
#### Success Response (200)
- **required_columns** (list of strings) - A list of column names that are required for validation.

#### Response Example
```json
{
  "required_columns": [
    "column1",
    "column2",
    "column3"
  ]
}
```
```

--------------------------------

### Run Text-to-SQL Generation

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/recipes/code_generation/text_to_sql

Commands to execute the generation script and access help documentation.

```bash
# Basic usage (generates 5 records by default)
uv run text_to_sql.py

# For help message and available options
uv run text_to_sql.py --help
```

--------------------------------

### GET /api/config/columns/count/{column_type}

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/code_reference/config_builder

Counts the number of columns that match a specified data type.

```APIDOC
## GET /api/config/columns/count/{column_type}

### Description
Get the count of columns of the specified type. Returns the number of columns matching the specified type.

### Method
GET

### Endpoint
/api/config/columns/count/{column_type}

### Parameters
#### Path Parameters
- **column_type** (DataDesignerColumnType) - Required - The type of columns to count.

### Response
#### Success Response (200)
- **count** (int) - The number of columns matching the specified type.

#### Response Example
```json
{
  "count": 15
}
```
```

--------------------------------

### GET /api/config/tools/{alias}

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/code_reference/config_builder

Retrieves a specific tool configuration object using its alias.

```APIDOC
## GET /api/config/tools/{alias}

### Description
Get a tool configuration by alias. Returns the tool configuration object.

### Method
GET

### Endpoint
/api/config/tools/{alias}

### Parameters
#### Path Parameters
- **alias** (str) - Required - The alias of the tool configuration to retrieve.

### Response
#### Success Response (200)
- **tool_config** (ToolConfig) - The tool configuration object.

#### Response Example
```json
{
  "tool_config": {
    "tool_alias": "my_tool",
    "type": "some_type",
    "params": { ... }
  }
}
```

### Error Handling
#### Error Response (404)
- **error** (str) - If no tool configuration with the given alias exists.

#### Error Example
```json
{
  "error": "No tool configuration with alias 'non_existent_alias' found"
}
```
```

--------------------------------

### Manage Model Configurations

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/concepts/models/configure-model-settings-with-the-cli

Initiate the interactive command to manage model configurations. Ensure at least one provider is configured before adding models.

```bash
data-designer config models
```

--------------------------------

### Get seed configuration

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/code_reference/config_builder

Retrieves the seed configuration if it has been set, otherwise returns None.

```python
def get_seed_config(self) -> SeedConfig | None:
    """Get the seed config for the current Data Designer configuration.

    Returns:
        The seed config if configured, None otherwise.
    """
    return self._seed_config
```

--------------------------------

### Run MCP Server

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/recipes/mcp_and_tooluse/pdf_qa

Initializes the search index and runs the MCP server. Reads PDF sources from the PDF_SOURCES environment variable or defaults to a sample PDF.

```python
def serve() -> None:
    """Run the MCP server (called when launched as subprocess by Data Designer)."""
    pdf_sources_json = os.environ.get("PDF_SOURCES", "[]")
    pdf_sources = json.loads(pdf_sources_json)
    if not pdf_sources:
        pdf_sources = [DEFAULT_PDF_URL]
    initialize_search_index(pdf_sources)
    mcp_server.run()
```

--------------------------------

### Get Constraints for Target Column

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/code_reference/config_builder

Retrieves all constraints associated with a specific target column.

```APIDOC
## GET /api/constraints/{target_column}

### Description
Get all constraints for the given target column.

### Method
GET

### Endpoint
`/api/constraints/{target_column}`

### Parameters
#### Path Parameters
- **target_column** (str) - Required - Name of the column to get constraints for.

### Response
#### Success Response (200)
- **list[ColumnConstraintT]** (array) - A list of constraint objects targeting the specified column.

#### Response Example
```json
[
  {
    "target_column": "example_column",
    "constraint_type": "unique"
  }
]
```
```

--------------------------------

### Get Column Configurations of Type

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/code_reference/config_builder

Retrieves column configurations that match a specified type.

```APIDOC
## GET /api/columns/type/{column_type}

### Description
Get all column configurations of the specified type.

### Method
GET

### Endpoint
`/api/columns/type/{column_type}`

### Parameters
#### Path Parameters
- **column_type** (DataDesignerColumnType) - Required - The type of columns to filter by.

### Response
#### Success Response (200)
- **list[ColumnConfigT]** (array) - A list of column configurations matching the specified type.

#### Response Example
```json
[
  {
    "name": "column2",
    "type": "string"
  }
]
```
```

--------------------------------

### Display Sample Record Preview

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/notebooks/1-the-basics

Run this code multiple times to cycle through preview records. It displays generated columns and their values.

```text
                                                                                                              
                                              Generated Columns                                               
┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Name                ┃ Value                                                                                ┃
┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ product_category    │ Clothing                                                                             │
├─────────────────────┼──────────────────────────────────────────────────────────────────────────────────────┤
│ product_subcategory │ Women's Clothing                                                                     │
├─────────────────────┼──────────────────────────────────────────────────────────────────────────────────────┤
│ target_age_range    │ 18-25                                                                                │
├─────────────────────┼──────────────────────────────────────────────────────────────────────────────────────┤
│ customer            │ {                                                                                    │
│                     │     'uuid': 'f37b4c69-76a8-4e08-863e-aaf29b4fc656',                                  │
│                     │     'locale': 'en_US',                                                               │
│                     │     'first_name': 'David',                                                           │
│                     │     'last_name': 'Henry',                                                            │
│                     │     'middle_name': None,                                                             │
│                     │     'sex': 'Male',                                                                   │
│                     │     'street_number': '95130',                                                        │
│                     │     'street_name': 'Miranda Land',                                                   │
│                     │     'city': 'West Timothy',                                                          │
│                     │     'state': 'New Jersey',                                                           │
│                     │     'postcode': '18917',                                                             │
│                     │     'age': 51,                                                                       │
│                     │     'birth_date': '1974-05-18',                                                      │
│                     │     'country': 'Poland',                                                             │
│                     │     'marital_status': 'married_present',                                             │
│                     │     'education_level': 'graduate',                                                   │
│                     │     'unit': '',                                                                      │
│                     │     'occupation': 'Architect',                                                       │
│                     │     'phone_number': '769.369.4438x0153',                                             │
│                     │     'bachelors_field': 'education'                                                   │
│                     │ }                                                                                    │
├─────────────────────┼──────────────────────────────────────────────────────────────────────────────────────┤
│ number_of_stars     │ 5                                                                                    │
├─────────────────────┼──────────────────────────────────────────────────────────────────────────────────────┤
│ review_style        │ detailed                                                                             │
├─────────────────────┼──────────────────────────────────────────────────────────────────────────────────────┤
│ product_name        │ Neon Pulse Skirt                                                                     │
│                     │ Pulse                                                                                │
│                     │ Electric Tapestry Tee                                                                │
│                     │ Tapestry                                                                             │
│                     │ Circuit Skirt                                                                        │
│                     │ Skirt                                                                                │

```

--------------------------------

### GET /config_builder/info/samplers

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/notebooks/1-the-basics

Retrieves a list of available samplers and their configuration requirements for synthetic data generation.

```APIDOC
## GET /config_builder/info/samplers

### Description
Displays the available samplers, their parameters, data types, and constraints for the NeMo Data Designer.

### Method
GET

### Endpoint
config_builder.info.display("samplers")

### Response
#### Success Response (200)
- **Type** (string) - The name of the sampler type (e.g., bernoulli, gaussian, person).
- **Parameter** (string) - The specific configuration parameter required for the sampler.
- **Data Type** (string) - The expected data type for the parameter.
- **Required** (boolean) - Indicates if the parameter is mandatory.
- **Constraints** (string) - Any specific validation rules or ranges for the parameter.
```

--------------------------------

### Configure MCP Providers

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/concepts/mcp/configure-mcp-cli

Launch the interactive wizard to add, update, or delete MCP providers.

```bash
data-designer config mcp
```

--------------------------------

### GET /api/config/processors

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/code_reference/config_builder

Retrieves all available processor configuration objects, categorized by dataset builder stage.

```APIDOC
## GET /api/config/processors

### Description
Get processor configuration objects. Returns a dictionary of processor configuration objects by dataset builder stage.

### Method
GET

### Endpoint
/api/config/processors

### Response
#### Success Response (200)
- **processors** (list[ProcessorConfigT]) - A dictionary of processor configuration objects by dataset builder stage.

#### Response Example
```json
{
  "processors": [
    {
      "stage": "stage_name",
      "config": { ... } 
    }
  ]
}
```
```

--------------------------------

### Clone and Configure Repository

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/CONTRIBUTING

Initial commands to clone a fork and set the upstream remote.

```bash
git clone https://github.com/YOUR_GITHUB_USERNAME/DataDesigner.git

cd DataDesigner

git remote add upstream https://github.com/NVIDIA-NeMo/DataDesigner.git

```

--------------------------------

### Get Column Configurations Excluding Type

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/code_reference/config_builder

Retrieves column configurations, excluding those of a specified type.

```APIDOC
## GET /api/columns/exclude-type/{column_type}

### Description
Get all column configurations excluding the specified type.

### Method
GET

### Endpoint
`/api/columns/exclude-type/{column_type}`

### Parameters
#### Path Parameters
- **column_type** (DataDesignerColumnType) - Required - The type of columns to exclude.

### Response
#### Success Response (200)
- **list[ColumnConfigT]** (array) - A list of column configurations that do not match the specified type.

#### Response Example
```json
[
  {
    "name": "column1",
    "type": "integer"
  }
]
```
```

--------------------------------

### Prepare and add a seed dataset

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/notebooks/3-seeding-with-a-dataset

Downloads an external CSV file and registers it as a seed source within the configuration builder.

```python
# Download sample dataset from Github
import urllib.request

url = "https://raw.githubusercontent.com/NVIDIA/GenerativeAIExamples/refs/heads/main/nemo/NeMo-Data-Designer/data/gretelai_symptom_to_diagnosis.csv"
local_filename, _ = urllib.request.urlretrieve(url, "gretelai_symptom_to_diagnosis.csv")

# Seed datasets are passed as reference objects to the config builder.
seed_source = dd.LocalFileSeedSource(path=local_filename)

config_builder.with_seed_dataset(seed_source)
```

--------------------------------

### Get Column Configuration by Name

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/code_reference/config_builder

Retrieves a specific column configuration using its unique name.

```APIDOC
## GET /api/columns/{name}

### Description
Get a column configuration by name.

### Method
GET

### Endpoint
`/api/columns/{name}`

### Parameters
#### Path Parameters
- **name** (str) - Required - Name of the column to retrieve the config for.

### Response
#### Success Response (200)
- **ColumnConfigT** (object) - The column configuration object.

#### Response Example
```json
{
  "name": "example_column",
  "type": "string",
  "description": "An example column"
}
```

### Errors
- **KeyError**: If no column with the given name exists.
```

--------------------------------

### Configure and Run Data Designer Pipeline

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/devnotes/graduate-level-science-reasoning-data-with-nemo-data-designer

Demonstrates the full workflow for configuring a reasoning model, defining column generation strategies, and executing the dataset creation process.

```python
import data_designer.config as dd
from data_designer.interface import DataDesigner

# Configure your model
model_configs = [
    dd.ModelConfig(
        alias="reasoning-model",
        model="qwen/qwen3-235b-a22b",
        provider="nvidia",
        inference_parameters=dd.ChatCompletionInferenceParams(
            max_tokens=8192,
            timeout=300,  # 5 minute timeout for long reasoning chains
        ),
    ),
]

# Build the workflow
config = dd.DataDesignerConfigBuilder(model_configs=model_configs)
config.with_seed_dataset(
    dd.LocalFileSeedSource(path="path/to/your_seed_data.parquet"),
    sampling_strategy=dd.SamplingStrategy.SHUFFLE,
)

# Generate questions
config.add_column(
    dd.LLMTextColumnConfig(
        name="question",
        prompt=QUESTION_PROMPT,
        model_alias="reasoning-model",
    )
)

# Generate answers with reasoning trace
config.add_column(
    dd.LLMTextColumnConfig(
        name="answer",
        prompt="{{ question }}",
        model_alias="reasoning-model",
        extract_reasoning_content=True,  # Extract reasoning into separate column
    )
)

# Combine into final sample
config.add_column(
    dd.ExpressionColumnConfig(
        name="rqa_sample",
        expr="{{ question }}\n\n{{ answer__reasoning_content }}\n\n{{ answer }}",
    )
)

# Run generation and save to disk
data_designer = DataDesigner()
result = data_designer.create(
    config_builder=config,
    num_records=N_RECORDS,
    dataset_name="rqa_dataset",
)
```

--------------------------------

### Get processor configurations

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/code_reference/config_builder

Retrieves a dictionary of processor configuration objects organized by dataset builder stage.

```python
def get_processor_configs(self) -> list[ProcessorConfigT]:
    """Get processor configuration objects.

    Returns:
        A dictionary of processor configuration objects by dataset builder stage.
    """
    return self._processor_configs
```

--------------------------------

### Configure ModelProvider with API Keys

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/concepts/models/model-providers

Demonstrates the two methods for setting the API key: referencing an environment variable or providing a direct string.

```python
# Method 1: Environment variable (recommended)
provider = ModelProvider(
    name="nvidia",
    endpoint="https://integrate.api.nvidia.com/v1",
    api_key="NVIDIA_API_KEY",  # Will be resolved from environment
)

# Method 2: Direct value (not recommended)
provider = ModelProvider(
    name="nvidia",
    endpoint="https://integrate.api.nvidia.com/v1",
    api_key="nvapi-abc123...",  # Direct API key
)
```

--------------------------------

### Configure DataDesigner with Sampler Columns

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/notebooks/5-generating-images

Initializes a configuration builder and adds multiple category-based sampler columns for image generation attributes.

```python
config_builder = dd.DataDesignerConfigBuilder(model_configs=model_configs) config_builder.add_column( dd.SamplerColumnConfig( name="style", sampler_type=dd.SamplerType.CATEGORY, params=dd.CategorySamplerParams( values=[ "photorealistic", "oil painting", "watercolor", "digital art", "sketch", "anime", ], ), ) ) config_builder.add_column( dd.SamplerColumnConfig( name="dog_breed", sampler_type=dd.SamplerType.CATEGORY, params=dd.CategorySamplerParams( values=[ "a Golden Retriever", "a German Shepherd", "a Labrador Retriever", "a Bulldog", "a Beagle", "a Poodle", "a Corgi", "a Siberian Husky", "a Dalmatian", "a Yorkshire Terrier", "a Boxer", "a Dachshund", "a Doberman Pinscher", "a Shih Tzu", "a Chihuahua", "a Border Collie", "an Australian Shepherd", "a Cocker Spaniel", "a Maltese", "a Pomeranian", "a Saint Bernard", "a Great Dane", "an Akita", "a Samoyed", "a Boston Terrier", ], ), ) ) config_builder.add_column( dd.SamplerColumnConfig( name="cat_breed", sampler_type=dd.SamplerType.CATEGORY, params=dd.CategorySamplerParams( values=[ "a Persian", "a Maine Coon", "a Siamese", "a Ragdoll", "a Bengal", "an Abyssinian", "a British Shorthair", "a Sphynx", "a Scottish Fold", "a Russian Blue", "a Birman", "an Oriental Shorthair", "a Norwegian Forest Cat", "a Devon Rex", "a Burmese", "an Egyptian Mau", "a Tonkinese", "a Himalayan", "a Savannah", "a Chartreux", "a Somali", "a Manx", "a Turkish Angora", "a Balinese", "an American Shorthair", ], ), ) ) config_builder.add_column( dd.SamplerColumnConfig( name="dog_age", sampler_type=dd.SamplerType.CATEGORY, params=dd.CategorySamplerParams( values=["1-3", "3-6", "6-9", "9-12", "12-15"], ), ) ) config_builder.add_column( dd.SamplerColumnConfig( name="cat_age", sampler_type=dd.SamplerType.CATEGORY, params=dd.CategorySamplerParams( values=["1-3", "3-6", "6-9", "9-12", "12-18"], ), ) ) config_builder.add_column( dd.SamplerColumnConfig( name="dog_look_direction", sampler_type=dd.SamplerType.CATEGORY, params=dd.CategorySamplerParams( values=["left", "right", "front", "up", "down"], ), ) ) config_builder.add_column( dd.SamplerColumnConfig( name="cat_look_direction", sampler_type=dd.SamplerType.CATEGORY, params=dd.CategorySamplerParams( values=["left", "right", "front", "up", "down"], ), ) ) config_builder.add_column( dd.SamplerColumnConfig( name="dog_emotion", sampler_type=dd.SamplerType.CATEGORY, params=dd.CategorySamplerParams( values=["happy", "curious", "serious", "sleepy", "excited"], ), ) ) config_builder.add_column( dd.SamplerColumnConfig( name="cat_emotion", sampler_type=dd.SamplerType.CATEGORY, params=dd.CategorySamplerParams( values=["aloof", "curious", "content", "sleepy", "playful"], ), ) ) config_builder.add_column( dd.ImageColumnConfig( name="generated_image", prompt=( """ A {{ style }} family pet portrait of a {{ dog_breed }} dog of {{ dog_age }} years old looking {{dog_look_direction}} with an {{ dog_emotion }} expression and {{ cat_breed }} cat of {{ cat_age }} years old looking {{ cat_look_direction }} with an {{ cat_emotion }} expression in the background. Both subjects should be in focus. """ ), model_alias=MODEL_ALIAS, ) ) data_designer.validate(config_builder)
```

--------------------------------

### GET /dataset/statistics/filter

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/code_reference/analysis

Filters column statistics within the DatasetProfilerResults object based on the specified column type.

```APIDOC
## GET /dataset/statistics/filter

### Description
Filters the column statistics of a dataset profile to return only those matching a specific column type.

### Method
GET

### Endpoint
/dataset/statistics/filter

### Parameters
#### Query Parameters
- **column_type** (DataDesignerColumnType) - Required - The type of column to filter by.

### Response
#### Success Response (200)
- **statistics** (list[ColumnStatisticsT]) - A list of statistics objects matching the requested column type.
```

--------------------------------

### Configure Resizable Column Generator

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/plugins/example

Example configuration for a plugin that modifies row counts, requiring allow_resize to be set to True.

```python
class MyColumnConfig(SingleColumnConfig):
    column_type: Literal["my-plugin"] = "my-plugin"
    allow_resize: bool = True  # required when output row count can differ from input
    # ...
```

--------------------------------

### Build Data Designer Config with Sampler and Image Columns

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/notebooks/5-generating-images

Initializes a `DataDesignerConfigBuilder` and adds multiple `SamplerColumnConfig` for dog and cat attributes (breed, age, look direction, emotion), followed by an `ImageColumnConfig` that uses a Jinja2 prompt to generate images based on these attributes. This setup is useful for creating diverse datasets of pet portraits.

```python
config_builder = dd.DataDesignerConfigBuilder(model_configs=model_configs)

config_builder.add_column(
    dd.SamplerColumnConfig(
        name="style",
        sampler_type=dd.SamplerType.CATEGORY,
        params=dd.CategorySamplerParams(
            values=[
                "photorealistic",
                "oil painting",
                "watercolor",
                "digital art",
                "sketch",
                "anime",
            ],
        ),
    )
)

config_builder.add_column(
    dd.SamplerColumnConfig(
        name="dog_breed",
        sampler_type=dd.SamplerType.CATEGORY,
        params=dd.CategorySamplerParams(
            values=[
                "a Golden Retriever",
                "a German Shepherd",
                "a Labrador Retriever",
                "a Bulldog",
                "a Beagle",
                "a Poodle",
                "a Corgi",
                "a Siberian Husky",
                "a Dalmatian",
                "a Yorkshire Terrier",
                "a Boxer",
                "a Dachshund",
                "a Doberman Pinscher",
                "a Shih Tzu",
                "a Chihuahua",
                "a Border Collie",
                "an Australian Shepherd",
                "a Cocker Spaniel",
                "a Maltese",
                "a Pomeranian",
                "a Saint Bernard",
                "a Great Dane",
                "an Akita",
                "a Samoyed",
                "a Boston Terrier",
            ],
        ),
    )
)

config_builder.add_column(
    dd.SamplerColumnConfig(
        name="cat_breed",
        sampler_type=dd.SamplerType.CATEGORY,
        params=dd.CategorySamplerParams(
            values=[
                "a Persian",
                "a Maine Coon",
                "a Siamese",
                "a Ragdoll",
                "a Bengal",
                "an Abyssinian",
                "a British Shorthair",
                "a Sphynx",
                "a Scottish Fold",
                "a Russian Blue",
                "a Birman",
                "an Oriental Shorthair",
                "a Norwegian Forest Cat",
                "a Devon Rex",
                "a Burmese",
                "an Egyptian Mau",
                "a Tonkinese",
                "a Himalayan",
                "a Savannah",
                "a Chartreux",
                "a Somali",
                "a Manx",
                "a Turkish Angora",
                "a Balinese",
                "an American Shorthair",
            ],
        ),
    )
)

config_builder.add_column(
    dd.SamplerColumnConfig(
        name="dog_age",
        sampler_type=dd.SamplerType.CATEGORY,
        params=dd.CategorySamplerParams(
            values=["1-3", "3-6", "6-9", "9-12", "12-15"],
        ),
    )
)

config_builder.add_column(
    dd.SamplerColumnConfig(
        name="cat_age",
        sampler_type=dd.SamplerType.CATEGORY,
        params=dd.CategorySamplerParams(
            values=["1-3", "3-6", "6-9", "9-12", "12-18"],
        ),
    )
)

config_builder.add_column(
    dd.SamplerColumnConfig(
        name="dog_look_direction",
        sampler_type=dd.SamplerType.CATEGORY,
        params=dd.CategorySamplerParams(
            values=["left", "right", "front", "up", "down"],
        ),
    )
)

config_builder.add_column(
    dd.SamplerColumnConfig(
        name="cat_look_direction",
        sampler_type=dd.SamplerType.CATEGORY,
        params=dd.CategorySamplerParams(
            values=["left", "right", "front", "up", "down"],
        ),
    )
)

config_builder.add_column(
    dd.SamplerColumnConfig(
        name="dog_emotion",
        sampler_type=dd.SamplerType.CATEGORY,
        params=dd.CategorySamplerParams(
            values=["happy", "curious", "serious", "sleepy", "excited"],
        ),
    )
)

config_builder.add_column(
    dd.SamplerColumnConfig(
        name="cat_emotion",
        sampler_type=dd.SamplerType.CATEGORY,
        params=dd.CategorySamplerParams(
            values=["aloof", "curious", "content", "sleepy", "playful"],
        ),
    )
)

config_builder.add_column(
    dd.ImageColumnConfig(
        name="generated_image",
        prompt=(
            """
A {{ style }} family pet portrait of a {{ dog_breed }} dog of {{ dog_age }} years old looking {{dog_look_direction}} with an {{ dog_emotion }} expression and
{{ cat_breed }} cat of {{ cat_age }} years old looking {{ cat_look_direction }} with an {{ cat_emotion }} expression in the background. Both subjects should be in focus.
        """
        ),
        model_alias=MODEL_ALIAS,
    )
)

data_designer.validate(config_builder)

```

--------------------------------

### Get tool configuration by alias

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/code_reference/config_builder

Retrieves a specific tool configuration using its alias. Raises a KeyError if the alias is not found.

```python
def get_tool_config(self, alias: str) -> ToolConfig:
    """Get a tool configuration by alias.

    Args:
        alias: The alias of the tool configuration to retrieve.

    Returns:
        The tool configuration object.

    Raises:
        KeyError: If no tool configuration with the given alias exists.
    """
    for tc in self._tool_configs:
        if tc.tool_alias == alias:
            return tc
    raise KeyError(f"No tool configuration with alias {alias!r} found")
```

--------------------------------

### Resize Image Utility

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/notebooks/4-providing-images-as-context

Resizes a PIL Image object to a target height while preserving the aspect ratio. Ensure PIL is installed.

```python
def resize_image(image, height: int):
    """
    Resize image while maintaining aspect ratio.

    Args:
        image: PIL Image object
        height: Target height in pixels

    Returns:
        Resized PIL Image object
    """
    original_width, original_height = image.size
    width = int(original_width * (height / original_height))
    return image.resize((width, height))
```

--------------------------------

### Initialize and Run MCP Server

Source: https://nvidia-nemo.github.io/DataDesigner/0.5.1/devnotes/deep-research-trajectories-with-nemo-data-designer-and-mcp-tool-use

Initializes the corpus and runs the MCP server. The corpus path is read from the CORPUS_PATH environment variable, defaulting to 'corpus.jsonl'. Ensure the environment variable is set or the default file exists.

```python
def serve() -> None:
    """Run as MCP server subprocess (called by Data Designer)."""
    corpus_path = os.environ.get("CORPUS_PATH", "corpus.jsonl")
    initialize(corpus_path)
    mcp_server.run()
```
Tokens:
Raw