### Development Installation of DeepFabric

Source: https://github.com/lukehinds/deepfabric/blob/main/docs/getting-started/installation.md

Installs the `uv` package manager, clones the DeepFabric repository, and installs all dependencies including development tools using `uv sync`. This setup is for contributing to the project.

```bash
# Install uv package manager
curl -LsSf https://astral.sh/uv/install.sh | sh

# Clone the repository
git clone https://github.com/lukehinds/deepfabric.git
cd deepfabric

# Install all dependencies including development tools
uv sync --all-extras
```

--------------------------------

### Example of Multi-Tool Workflow in JSON

Source: https://github.com/lukehinds/deepfabric/blob/main/docs/guide/instruction-formats/agent-tool-calling/tutorials/getting-started.md

This JSON illustrates a multi-tool workflow for an AI agent. It outlines a user's complex request involving web search and calculation, detailing the step-by-step planning and the reasoning behind selecting each tool.

```json
{
  "question": "Search for Italian restaurants in NYC and calculate tip for $75",
  "tool_planning": [
    {
      "step_number": 1,
      "reasoning": "Search for Italian restaurants in NYC",
      "selected_tool": "search_web"
    },
    {
      "step_number": 2,
      "reasoning": "Calculate tip amount for $75 bill",
      "selected_tool": "calculator"
    }
  ]
}
```

--------------------------------

### Example of Single Tool Usage in JSON

Source: https://github.com/lukehinds/deepfabric/blob/main/docs/guide/instruction-formats/agent-tool-calling/tutorials/getting-started.md

This JSON snippet demonstrates a simplified agent response where only one tool is required. It shows the user's question, the planning phase with reasoning for tool selection, and the tool itself without execution details.

```json
{
  "question": "What's the weather in Paris?",
  "tool_planning": [
    {
      "reasoning": "User wants current weather for Paris - need weather tool",
      "selected_tool": "get_weather"
    }
  ]
}
```

--------------------------------

### Quick Start DeepFabric CLI and Python

Source: https://github.com/lukehinds/deepfabric/blob/main/examples/README.md

Demonstrates the fastest way to start using DeepFabric. It shows how to initiate a process using a YAML configuration file via the CLI and programmatically using a Python script.

```bash
# Using a YAML config file
deepfabric start examples/configs/01-quickstart.yaml
```

```python
# Or programmatically with Python
python examples/code/01-quickstart.py
```

--------------------------------

### Verify DeepFabric Installation

Source: https://github.com/lukehinds/deepfabric/blob/main/docs/getting-started/installation.md

Confirms that DeepFabric has been successfully installed and is accessible from the command line. This is a crucial step after installation to ensure the environment is set up correctly.

```bash
deepfabric --version
```

--------------------------------

### Display DeepFabric Installation Information

Source: https://github.com/lukehinds/deepfabric/blob/main/docs/getting-started/installation.md

Retrieves and displays comprehensive information about the DeepFabric installation, including its version, available commands, and detected environment variables. This command serves as a final verification step.

```bash
deepfabric info
```

--------------------------------

### Configure Topic Prompts for Tool Usage (YAML)

Source: https://github.com/lukehinds/deepfabric/blob/main/docs/guide/instruction-formats/agent-tool-calling/tutorials/getting-started.md

This YAML snippet illustrates how to set topic prompts to encourage tool usage in agents. By defining prompts that naturally require external data or actions, the agent is guided towards utilizing available tools.

```yaml
topic_tree:
  topic_prompt: "Tasks requiring weather data, calculations, or web searches"

```

--------------------------------

### Bash Commands for DeepFabric Setup Verification

Source: https://github.com/lukehinds/deepfabric/blob/main/docs/guide/instruction-formats/chain-of-thought/tutorials/math-reasoning.md

These bash commands verify the DeepFabric installation and OpenAI API key setup. 'echo $OPENAI_API_KEY' checks if the API key environment variable is set, and 'deepfabric --version' confirms the tool's installation and provides its version number.

```bash
# Verify setup
echo $OPENAI_API_KEY
deepfabric --version
```

--------------------------------

### Set up Ollama for Local Models

Source: https://github.com/lukehinds/deepfabric/blob/main/docs/getting-started/installation.md

Installs Ollama, pulls a model (e.g., mistral), and lists available models for local execution. This is ideal for privacy-focused or offline model usage.

```bash
# Install Ollama (macOS/Linux)
curl -fsSL https://ollama.com/install.sh | sh

# Pull a model
ollama pull mistral

# Verify the model is available
ollama list
```

--------------------------------

### Run DeepFabric Development Test Suite

Source: https://github.com/lukehinds/deepfabric/blob/main/docs/getting-started/installation.md

Executes the project's test suite using a `make` command. This verifies that the development environment is correctly configured and all dependencies are functioning as expected.

```bash
make test
```

--------------------------------

### Generate Dataset with CLI (Bash)

Source: https://github.com/lukehinds/deepfabric/blob/main/docs/guide/instruction-formats/agent-tool-calling/tutorials/getting-started.md

Executes the DeepFabric command-line interface to start the dataset generation process based on the specified YAML configuration file.

```bash
deepfabric start agent-getting-started.yaml

```

--------------------------------

### Run DeepFabric Dataset Generation

Source: https://github.com/lukehinds/deepfabric/blob/main/examples/mcp/README.md

Command to start the dataset generation process using a specified DeepFabric configuration file. It utilizes the 'deepfabric start' command with a path to a YAML configuration.

```bash
deepfabric start examples/mcp/firecrawl_config.yaml
```

--------------------------------

### Manage API Keys with .env File

Source: https://github.com/lukehinds/deepfabric/blob/main/docs/getting-started/installation.md

Creates a `.env` file to store API keys and other sensitive credentials for various services. This promotes organized management of environment-specific configurations.

```bash
# .env file
OPENAI_API_KEY=your-openai-key
ANTHROPIC_API_KEY=your-anthropic-key
GEMINI_API_KEY=your-gemini-key
HF_TOKEN=your-huggingface-token
```

--------------------------------

### YAML Examples for Dataset System Prompt

Source: https://github.com/lukehinds/deepfabric/blob/main/docs/guide/instruction-formats/chain-of-thought/configuration/yaml-config.md

Provides examples of system prompts for the dataset generation process. These prompts define the assistant's role and behavior, emphasizing different reasoning styles (free-text, structured, hybrid) to guide the LLM.

```yaml
# Examples by format
dataset_system_prompt: "You are a helpful teacher who shows step-by-step reasoning."  # Free-text
dataset_system_prompt: "You are an instructor who guides students systematically."    # Structured
dataset_system_prompt: "You are an expert who combines intuition with analysis."     # Hybrid
```

--------------------------------

### Deepfabric Development Setup

Source: https://github.com/lukehinds/deepfabric/blob/main/README.md

Shell commands for setting up the Deepfabric project for development. This includes cloning the repository, installing dependencies with development extras, running tests, and formatting the code.

```bash
git clone https://github.com/lukehinds/deepfabric
cd deepfabric
uv sync --all-extras  # Install with dev dependencies
make test            # Run tests
make format          # Format code
```

--------------------------------

### Login to Hugging Face Hub via CLI

Source: https://github.com/lukehinds/deepfabric/blob/main/docs/getting-started/installation.md

Installs the `huggingface_hub` library and logs into the Hugging Face Hub using the command-line interface. This is an alternative method to setting the `HF_TOKEN` environment variable for authentication.

```bash
pip install huggingface_hub
huggingface-cli login
```

--------------------------------

### Verify DeepFabric Installation and Basic Functionality

Source: https://github.com/lukehinds/deepfabric/blob/main/docs/guide/instruction-formats/chain-of-thought/reference/troubleshooting.md

Commands to check DeepFabric installation, API key presence, and test basic validation functionality. These are essential first steps in diagnosing issues.

```bash
# 1. Verify installation
deepfabric --version

# 2. Check API keys
echo $OPENAI_API_KEY | head -c 10
echo $ANTHROPIC_API_KEY | head -c 10

# 3. Test basic functionality
deepfabric validate examples/quickstart.yaml
```

--------------------------------

### Generate Dataset with Python (Python)

Source: https://github.com/lukehinds/deepfabric/blob/main/docs/guide/instruction-formats/agent-tool-calling/tutorials/getting-started.md

Uses the DeepFabric Python library to programmatically load configuration, generate topics, create agent tool-calling samples, and save the dataset to a JSONL file.

```python
import asyncio
from deepfabric.config import DeepFabricConfig
from deepfabric.tree import Tree
from deepfabric.generator import DataSetGenerator
from deepfabric.dataset import Dataset

async def main():
    # Load configuration
    config = DeepFabricConfig.from_yaml("agent-getting-started.yaml")

    # Generate topics
    tree_params = config.get_topic_tree_params()
    tree = Tree(tree_params)
    topics = await tree.generate()
    print(f"Generated {len(topics)} topics")

    # Generate agent dataset
    engine_params = config.get_engine_params()
    engine_params["topics"] = topics

    generator = DataSetGenerator(engine_params)
    samples = await generator.generate()
    print(f"Generated {len(samples)} agent samples")

    # Save dataset
    dataset = Dataset.from_list(samples)
    dataset.save("my_first_agent_dataset.jsonl")
    print("Dataset saved!")

# Run generation
asyncio.run(main())

```

--------------------------------

### Install DeepFabric using Pip

Source: https://github.com/lukehinds/deepfabric/blob/main/docs/getting-started/installation.md

Installs the latest stable release of DeepFabric via pip, suitable for most use cases. This command fetches the package and its core dependencies from the Python Package Index.

```bash
pip install deepfabric
```

--------------------------------

### Example JSON Structure for Tool Output

Source: https://github.com/lukehinds/deepfabric/blob/main/docs/guide/instruction-formats/agent-tool-calling/tutorials/getting-started.md

This JSON object represents a detailed breakdown of a user query, including the question, available tools, analysis, tool planning, execution steps, and the final synthesized answer. This structure is often used in AI agent frameworks to document decision-making processes.

```json
{
  "question": "What's the weather like in Seattle and what's 15% of the temperature in Fahrenheit?",
  "available_tools": [
    {
      "name": "get_weather",
      "description": "Get current weather conditions for a location",
      "parameters": [...],
      "returns": "Weather data including temperature and conditions",
      "category": "information"
    },
    {
      "name": "calculator",
      "description": "Perform mathematical calculations",
      "parameters": [...],
      "returns": "Calculation result",
      "category": "computation"
    }
  ],
  "initial_analysis": "The user wants two things: current weather information for Seattle, and a calculation of 15% of the temperature value. I'll need to use the weather tool first to get the temperature, then use the calculator to compute 15% of that value.",
  "tool_planning": [
    {
      "step_number": 1,
      "reasoning": "Need to get current weather for Seattle to obtain temperature data",
      "selected_tool": {...},
      "parameter_reasoning": {
        "location": "Seattle is explicitly specified by the user"
      },
      "expected_result": "Current weather conditions including temperature in Fahrenheit"
    },
    {
      "step_number": 2,
      "reasoning": "Need to calculate 15% of the temperature value obtained from weather data",
      "selected_tool": {...},
      "parameter_reasoning": {
        "expression": "Will use the temperature from step 1 multiplied by 0.15"
      },
      "expected_result": "15% of the temperature value"
    }
  ],
  "tool_executions": [
    {
      "function": "get_weather",
      "arguments": {"location": "Seattle"},
      "reasoning": "Getting weather data for Seattle as planned",
      "result": "Seattle: 65°F, partly cloudy, 60% humidity, light wind from northwest"
    },
    {
      "function": "calculator",
      "arguments": {"expression": "65 * 0.15"},
      "reasoning": "Calculating 15% of 65°F temperature",
      "result": "9.75"
    }
  ],
  "result_synthesis": "I obtained the current weather for Seattle (65°F, partly cloudy) and calculated that 15% of the temperature (65°F) equals 9.75. This provides both pieces of information the user requested.",
  "final_answer": "The current weather in Seattle is 65°F and partly cloudy with 60% humidity and light winds from the northwest. 15% of the current temperature (65°F) is 9.75."
}
```

--------------------------------

### Configure OpenAI API Key

Source: https://github.com/lukehinds/deepfabric/blob/main/docs/getting-started/installation.md

Sets the OpenAI API key as an environment variable, enabling DeepFabric to authenticate with OpenAI services for models like GPT-4 and GPT-3.5. Replace 'your-api-key-here' with your actual key.

```bash
export OPENAI_API_KEY="your-api-key-here"
```

--------------------------------

### YAML Examples for Topic Prompt Specification

Source: https://github.com/lukehinds/deepfabric/blob/main/docs/guide/instruction-formats/chain-of-thought/configuration/yaml-config.md

Illustrates different uses of the `topic_prompt` parameter, showing how to specify the domain, complexity, and target audience for topic generation. Examples range from elementary math to advanced physics problems.

```yaml
# Domain-specific examples
topic_prompt: "Elementary mathematics word problems suitable for grades 3-6"
topic_prompt: "Computer science algorithms and data structures for undergraduate level"
topic_prompt: "Advanced physics problems requiring multi-step problem solving"
```

--------------------------------

### Agent Tool-Calling Configuration (YAML)

Source: https://github.com/lukehinds/deepfabric/blob/main/docs/guide/instruction-formats/agent-tool-calling/tutorials/getting-started.md

Defines the parameters for generating an agent tool-calling dataset. It specifies prompts, model configurations, available tools, and dataset generation settings.

```yaml
# Basic Agent Tool-Calling Configuration
dataset_system_prompt: "You are an AI assistant with access to various tools. Always explain your reasoning when selecting and using tools."

topic_tree:
  topic_prompt: "Daily tasks requiring tool usage: weather checks, simple calculations, web searches"
  provider: "openai"
  model: "gpt-4o-mini"
  temperature: 0.7
  depth: 2
  degree: 3

data_engine:
  generation_system_prompt: "You excel at reasoning about tool selection and parameter construction. Create realistic scenarios showing step-by-step tool usage."
  provider: "openai"
  model: "gpt-4o-mini"
  temperature: 0.8
  conversation_type: "agent_cot_tools"
  available_tools:
    - "get_weather"
    - "calculator"
    - "search_web"
  max_tools_per_query: 2
  max_retries: 3

dataset:
  creation:
    num_steps: 5  # Start small for testing
    batch_size: 1
    sys_msg: false
  save_as: "my_first_agent_dataset.jsonl"

```

--------------------------------

### Improve Agent Reasoning Prompts (YAML)

Source: https://github.com/lukehinds/deepfabric/blob/main/docs/guide/instruction-formats/agent-tool-calling/tutorials/getting-started.md

This YAML configuration demonstrates how to enhance the system prompt for agent generation to improve reasoning quality. It instructs the model to focus on the 'WHY' and 'HOW' of tool selection and parameter determination.

```yaml
data_engine:
  generation_system_prompt: "Focus on WHY tools are selected and HOW parameters are determined. Provide detailed reasoning at each step."

```

--------------------------------

### Configure Google Gemini API Key

Source: https://github.com/lukehinds/deepfabric/blob/main/docs/getting-started/installation.md

Sets the Google Gemini API key as an environment variable, enabling DeepFabric to interact with Google's Gemini models. Substitute 'your-api-key-here' with your actual Gemini API key.

```bash
export GEMINI_API_KEY="your-api-key-here"
```

--------------------------------

### Define Custom Tools for Agent (YAML)

Source: https://github.com/lukehinds/deepfabric/blob/main/docs/guide/instruction-formats/agent-tool-calling/tutorials/getting-started.md

This YAML configuration defines a custom tool named 'book_restaurant' with its parameters and return type. This allows agents to interact with custom functionalities.

```yaml
tools:
  - name: "book_restaurant"
    description: "Book a restaurant reservation"
    parameters:
      - name: "restaurant"
        type: "str"
        description: "Restaurant name"
        required: true
      - name: "party_size"
        type: "int"
        description: "Number of people"
        required: true
    returns: "Reservation confirmation"
    category: "booking"

```

--------------------------------

### View First Dataset Entry with jq

Source: https://github.com/lukehinds/deepfabric/blob/main/docs/guide/instruction-formats/agent-tool-calling/tutorials/getting-started.md

This command displays the first line of the 'my_first_agent_dataset.jsonl' file and formats it as a JSON object using the 'jq' utility. It's useful for quickly inspecting the structure and content of dataset entries.

```bash
head -n 1 my_first_agent_dataset.jsonl | jq .
```

--------------------------------

### Apply Formatters for Agent Training Data (YAML)

Source: https://github.com/lukehinds/deepfabric/blob/main/docs/guide/instruction-formats/agent-tool-calling/tutorials/getting-started.md

This YAML configuration defines a 'tool_calling' formatter for agent training data. It specifies the template, output file, and includes a system prompt for the model, indicating whether to include tools in the system message.

```yaml
dataset:
  formatters:
    - name: "tool_calling"
      template: "builtin://tool_calling"
      output: "agent_tool_calling_format.jsonl"
      config:
        system_prompt: "You are a function calling AI model."
        include_tools_in_system: true

```

--------------------------------

### Quick Prototype Configuration (YAML)

Source: https://github.com/lukehinds/deepfabric/blob/main/docs/examples/basic-usage.md

A minimal YAML configuration designed for rapid experimentation and testing. This setup quickly generates small datasets using OpenAI's GPT-4 Turbo model, suitable for initial proof-of-concept work or format validation.

```yaml
# prototype.yaml
dataset_system_prompt: "You are creating test examples for development."

topic_tree:
  topic_prompt: "Basic test concepts"
  topic_system_prompt: "You are creating test examples for development."
  degree: 2
  depth: 1
  temperature: 0.5
  provider: "openai"
  model: "gpt-4-turbo"
  save_as: "test_topics.jsonl"

data_engine:
  instructions: "Create a simple example for testing purposes."
  generation_system_prompt: "You are creating test examples for development."
  provider: "openai"
  model: "gpt-4-turbo"
  temperature: 0.5
  max_retries: 2

dataset:
  creation:
    num_steps: 5
    batch_size: 1
    provider: "openai"
    model: "gpt-4-turbo"
    sys_msg: false
  save_as: "test_dataset.jsonl"
```

--------------------------------

### Reference Custom Tools in Agent Configuration (YAML)

Source: https://github.com/lukehinds/deepfabric/blob/main/docs/guide/instruction-formats/agent-tool-calling/tutorials/getting-started.md

This YAML snippet shows how to update the agent's configuration to include custom tools. It specifies the path to the tool registry and lists the available tools, including the custom 'book_restaurant' tool.

```yaml
data_engine:
  tool_registry_path: "custom_tools.yaml"
  available_tools:
    - "get_weather"
    - "book_restaurant"  # Your custom tool

```

--------------------------------

### Check Python Environment and Imports for DeepFabric

Source: https://github.com/lukehinds/deepfabric/blob/main/docs/guide/instruction-formats/chain-of-thought/reference/troubleshooting.md

A Python command to verify that the DeepFabric library can be imported successfully, indicating a correct installation and environment setup.

```python
python -c "from deepfabric import DataSetGenerator; print('✅ Imports working')"
```

--------------------------------

### DeepFabric Imports and Configuration Setup

Source: https://github.com/lukehinds/deepfabric/blob/main/notebooks/trl_sft_training.ipynb

Imports core components from the DeepFabric library and defines the configuration dictionary for dataset generation. This includes settings for the dataset's system prompt, topic tree generation, data engine, and dataset formatting.

```python
import asyncio
import json
import os

from deepfabric.config import DeepFabricConfig
from deepfabric.dataset import Dataset
from deepfabric.generator import DataSetGenerator
from deepfabric.tree import Tree
```

```python
config = {
    "dataset_system_prompt": """You are a helpful AI assistant with access to various tools.
Use tools when needed to answer questions accurately.""",

    "topic_tree": {
        "topic_prompt": "Real-world tasks requiring AI assistant tool usage",
        "provider": "openai",
        "model": "gpt-4o-mini",
        "temperature": 0.7,
        "depth": 2,
        "degree": 3,
    },

    "data_engine": {
        "generation_system_prompt": """Generate realistic AI assistant scenarios with tool usage.
Show clear reasoning about tool selection and parameter construction.""",
        "provider": "openai",
        "model": "gpt-4o-mini",
        "temperature": 0.8,
        "conversation_type": "agent_cot_tools",
        "available_tools": ["web_search", "calculator", "get_weather"],
        "max_tools_per_query": 2,
    },

    "dataset": {
        "creation": {
            "num_steps": 5,
            "batch_size": 2,
            "sys_msg": True
        },
        "save_as": "trl_raw.jsonl",
        "formatters": [
            {
                "name": "trl_sft",
                "template": "builtin://trl_sft_tools",
                "output": "trl_formatted.jsonl",
                "config": {
                    "include_system_prompt": True,
                    "validate_tool_schemas": True,
                    "remove_available_tools_field": False,
                }
            }
        ]
    }
}

print("📊 Configuration:")
print(f"  - Topic depth: {config['topic_tree']['depth']}")
print(f"  - Topic degree: {config['topic_tree']['degree']}")
print(f"  - Samples: {config['dataset']['creation']['num_steps'] * config['dataset']['creation']['batch_size']}")
print(f"  - Conversation type: {config['data_engine']['conversation_type']}")
```

--------------------------------

### Install DeepFabric and Dependencies

Source: https://github.com/lukehinds/deepfabric/blob/main/notebooks/trl_sft_training.ipynb

Installs the necessary Python packages for DeepFabric, HuggingFace Transformers, TRL, Datasets, PEFT, and Accelerate. This is a prerequisite for running the tutorial.

```bash
pip install deepfabric transformers trl datasets peft accelerate
```

--------------------------------

### Sample DeepFabric Training Dataset Example (JSON)

Source: https://github.com/lukehinds/deepfabric/blob/main/docs/getting-started/first-dataset.md

This JSON snippet demonstrates the structure of a single training example within the generated dataset. It follows a conversational format with 'system', 'user', and 'assistant' roles, including the system prompt for context.

```json
{
  "messages": [
    {
      "role": "system",
      "content": "You are a Python programming instructor providing clear, educational content for intermediate developers."
    },
    {
      "role": "user",
      "content": "Can you explain list comprehensions in Python with a practical example?"
    },
    {
      "role": "assistant",
      "content": "List comprehensions provide a concise way to create lists in Python. Here's how they work: [detailed explanation with code examples]"
    }
  ]
}
```

--------------------------------

### DeepFabric Format Command Examples (Bash)

Source: https://github.com/lukehinds/deepfabric/blob/main/docs/cli/format.md

Provides practical examples of using the 'deepfabric format' command with different options, such as specifying a formatter, setting a custom output path, and using a configuration file.

```bash
# Apply the chatml formatter with default settings:
deepfabric format dataset.jsonl -f chatml

# Use a custom output path:
deepfabric format dataset.jsonl -f alpaca -o training_data.jsonl

# Use a configuration file for more control:
deepfabric format dataset.jsonl -c formatter_config.yaml
```

--------------------------------

### Check DeepFabric Installation

Source: https://github.com/lukehinds/deepfabric/blob/main/docs/guide/instruction-formats/chain-of-thought/reference/troubleshooting.md

Provides bash commands to verify if the 'deepfabric' package is installed and to check its version. This is a first step in diagnosing import errors.

```bash
pip list | grep deepfabric
# or
pip show deepfabric
```

--------------------------------

### Load and Initialize Deepfabric Trees

Source: https://context7.com/lukehinds/deepfabric/llms.txt

Demonstrates loading an existing tree from a JSONL file with specified parameters and initializing a new tree with a local Ollama model for quantum computing concepts.

```python
from deepfabric import Tree

# Load existing tree
loaded_tree = Tree.from_jsonl(
    "topics_tree.jsonl",
    {
        "topic_prompt": "Data science",
        "model_name": "gpt-4o-mini",
        "provider": "openai",
        "degree": 4,
        "depth": 3,
        "temperature": 0.8
    }
)

# Use with local Ollama model
local_tree = Tree(
    topic_prompt="Quantum computing concepts",
    model_name="qwen3:8b",
    provider="ollama",
    degree=3,
    depth=2,
    temperature=0.7
)
local_tree.build_tree()
```

--------------------------------

### Educational Content - Data Generation Prompt

Source: https://github.com/lukehinds/deepfabric/blob/main/docs/guide/system-prompts.md

System prompt for data generation tailored for educational content. It guides the AI to act as an experienced educator, providing clear explanations, examples, and addressing common misconceptions.

```yaml
data_engine:
  generation_system_prompt: "You are an experienced educator teaching [SUBJECT] to [LEVEL] students. Provide clear, step-by-step explanations with concrete examples. Avoid jargon unless you explain it. Include common misconceptions and how to avoid them. Always provide practical examples that students can relate to."
```

--------------------------------

### Setup TRL SFTTrainer Components (Python)

Source: https://github.com/lukehinds/deepfabric/blob/main/notebooks/trl_sft_training.ipynb

Initializes essential components for training a causal language model using the TRL library. This includes loading a pre-trained tokenizer and model, configuring LoRA for efficient fine-tuning, and setting up training arguments. Dependencies: `transformers`, `peft`, `trl` libraries. Inputs: `model_name` (string), `hf_dataset` (HuggingFace Dataset), `peft_config` (LoraConfig object), `training_args` (SFTConfig object). Outputs: Initialized `SFTTrainer` object.

```python
from peft import LoraConfig
from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import SFTConfig, SFTTrainer
import os

print("Setting up training components...")

# Select Model
model_name = "Qwen/Qwen2.5-0.5B-Instruct"  # Small model for testing

print(f"Model: {model_name}")
print("\nLoading tokenizer and model...")

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    device_map="auto",
    trust_remote_code=True,
)

print("✓ Model loaded")

# Configure LoRA
peft_config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
)

print("LoRA Configuration:")
print(f"  - r: {peft_config.r}")
print(f"  - alpha: {peft_config.lora_alpha}")
print(f"  - target modules: {peft_config.target_modules}")

# Training Configuration
training_args = SFTConfig(
    output_dir="./trl_tool_calling_model",
    num_train_epochs=3,
    per_device_train_batch_size=2,
    gradient_accumulation_steps=4,
    learning_rate=2e-4,
    logging_steps=10,
    save_steps=100,
    warmup_steps=50,
    max_seq_length=2048,
    # Tool calling specific
    dataset_text_field=None,  # We'll handle formatting
    packing=False,
)

print("Training Configuration:")
print(f"  - Epochs: {training_args.num_train_epochs}")
print(f"  - Batch size: {training_args.per_device_train_batch_size}")
print(f"  - Learning rate: {training_args.learning_rate}")
print(f"  - Max sequence length: {training_args.max_seq_length}")

# Initialize Trainer
print("Initializing SFTTrainer...")

trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=hf_dataset, # Assuming hf_dataset is loaded from previous step
    tokenizer=tokenizer,
    peft_config=peft_config,
)

print("✓ Trainer initialized")
```

--------------------------------

### YAML Examples for Topic Tree Degree and Depth

Source: https://github.com/lukehinds/deepfabric/blob/main/docs/guide/instruction-formats/chain-of-thought/configuration/yaml-config.md

Illustrates the configuration of `degree` and `depth` parameters for controlling the structure of the topic tree. Examples show settings for small, medium, and large topic datasets.

```yaml
# Small dataset (4-9 topics)
degree: 2
depth: 2

# Medium dataset (8-27 topics)
degree: 3
depth: 2

# Large dataset (16-64 topics)
degree: 4
depth: 3
```

--------------------------------

### Override DeepFabric Settings via CLI

Source: https://github.com/lukehinds/deepfabric/blob/main/examples/mcp/README.md

Example of overriding default configuration settings when starting DeepFabric dataset generation using command-line arguments. Supports parameters like model, temperature, batch size, and number of steps.

```bash
deepfabric start examples/mcp/firecrawl_config.yaml \
  --model gpt-4 \
  --temperature 0.9 \
  --batch-size 10 \
  --num-steps 20
```

--------------------------------

### Execute DeepFabric Dataset Generation Command

Source: https://github.com/lukehinds/deepfabric/blob/main/docs/examples/basic-usage.md

This command-line instruction demonstrates how to initiate the dataset generation process using a specified YAML configuration file. It's a fundamental step for running any of the DeepFabric examples.

```bash
deepfabric generate programming-basics.yaml
```

--------------------------------

### Tool Planning Step Example

Source: https://github.com/lukehinds/deepfabric/blob/main/docs/guide/instruction-formats/agent-tool-calling/single-turn.md

Shows a single step within the tool_planning component, detailing the reasoning, selected tool, parameter reasoning, and expected result for a specific action.

```json
{
  "step_number": 1,
  "reasoning": "Need current weather data for London as specified in the user's request",
  "selected_tool": {
    "name": "get_weather",
    "description": "Get current weather conditions for a location",
    "parameters": [
      {
        "name": "location",
        "type": "str",
        "description": "City name or location",
        "required": true
      }
    ],
    "returns": "Weather data including temperature and conditions",
    "category": "information"
  },
  "parameter_reasoning": {
    "location": "User explicitly mentioned London as first city"
  },
  "expected_result": "Current weather conditions for London including temperature, conditions, and other relevant data"
}
```

--------------------------------

### Hybrid CoT Configuration Example (YAML)

Source: https://github.com/lukehinds/deepfabric/blob/main/docs/guide/instruction-formats/chain-of-thought/configuration/yaml-config.md

An example YAML configuration for hybrid Chain of Thought datasets, designed for complex scientific problems requiring multi-modal reasoning. It sets the system prompt for a scientific expert persona.

```yaml
# hybrid-scientific-reasoning.yaml
# Optimized for complex scientific problems requiring multi-modal reasoning

dataset_system_prompt: "You are a scientific expert who combines intuitive insights with rigorous systematic analysis."


```

--------------------------------

### Run DeepFabric Examples with Python

Source: https://github.com/lukehinds/deepfabric/blob/main/examples/README.md

Instructions for running DeepFabric's Python examples directly from the command line. These scripts can be executed as-is or modified for custom use.

```bash
# Run Python examples directly
python examples/code/01-quickstart.py

# Or with modifications
python examples/code/02-programmatic-config.py
```

--------------------------------

### Utilize Example Data for Few-Shot Learning in DataSetGenerator

Source: https://github.com/lukehinds/deepfabric/blob/main/docs/api/generator.md

Demonstrates using the 'example_data' parameter to provide a pre-loaded Dataset for few-shot learning. This helps guide the DataSetGenerator to follow a specific style or format based on the provided examples, improving the quality and consistency of generated content.

```python
from deepfabric import Dataset

example_dataset = Dataset()
example_dataset.load("examples.jsonl")

generator = DataSetGenerator(
    instructions="Follow the style of the provided examples",
    generation_system_prompt="You are an expert instructor",
    provider="openai",
    model_name="gpt-4",
    example_data=example_dataset
)
```

--------------------------------

### Structured CoT Configuration Example (YAML)

Source: https://github.com/lukehinds/deepfabric/blob/main/docs/guide/instruction-formats/chain-of-thought/configuration/yaml-config.md

An example YAML configuration for generating structured computer science educational dialogues. It defines topic generation using a graph structure, data generation with systematic reasoning, and HuggingFace Hub upload with detailed metadata.

```yaml
# structured-cs-education.yaml
# Optimized for computer science educational dialogues

dataset_system_prompt: "You are a computer science instructor who guides students through systematic problem-solving approaches."

# Topic generation using Graph structure (interconnected)
topic_graph:
  topic_prompt: "Computer science fundamentals including algorithms, data structures, programming concepts, and computational thinking"

  # LLM settings
  provider: "openai"
  model: "gpt-4o-mini"
  temperature: 0.6

  # Graph structure (more complex than tree)
  degree: 2
  depth: 3

  # Output
  save_as: "cs_topics_graph.json"

# Data generation with structured conversations
data_engine:
  instructions: "Create educational conversations where students learn computer science concepts through guided discovery and systematic reasoning."
  generation_system_prompt: "You are a CS instructor creating realistic teaching dialogues with explicit reasoning steps."

  # LLM settings
  provider: "openai"
  model: "gpt-4o-mini"  # Consider gpt-4o for more complex conversations
  temperature: 0.4
  max_retries: 3

  # Structured CoT settings
  conversation_type: "cot_structured"
  reasoning_style: "logical"

# Dataset assembly
dataset:
  creation:
    num_steps: 8          # Fewer steps due to complexity
    batch_size: 1
    sys_msg: true         # Include system messages in conversations

  save_as: "cs_education_structured.jsonl"

# HuggingFace upload with rich metadata
huggingface:
  repository: "deepfabric/cs-education-structured-cot"
  tags:
    - "computer-science"
    - "education"
    - "algorithms"
    - "structured-reasoning"
    - "chain-of-thought"
    - "deepfabric"
  description: "Educational computer science conversations with structured reasoning traces"

```

--------------------------------

### Configure Cost-Effective Generation Setup (YAML)

Source: https://github.com/lukehinds/deepfabric/blob/main/docs/guide/instruction-formats/chain-of-thought/reference/troubleshooting.md

This YAML configuration outlines a budget-friendly setup for data generation. It specifies cost-effective choices like the `gpt-4o-mini` model, the simplest `cot_freetext` format, a lower temperature to reduce retries, and limited `max_retries`. It also configures a smaller topic tree structure.

```yaml
# Budget-friendly setup
data_engine:
  provider: "openai"
  model: "gpt-4o-mini"           # Most cost-effective
  conversation_type: "cot_freetext"  # Simplest format
  temperature: 0.3               # Reduce retries
  max_retries: 2                 # Limit retries

topic_tree:
  degree: 2                      # Smaller topic tree
  depth: 2
```

--------------------------------

### DeepFabric Agent Tool-Calling Sample Generation

Source: https://github.com/lukehinds/deepfabric/blob/main/notebooks/trl_sft_training.ipynb

Generates agent tool-calling training samples using the configured data engine and the previously built topic tree. This function creates the core dataset examples.

```python
print("Generating agent tool-calling samples...")

engine_params = df_config.get_engine_params()
generator = DataSetGenerator(**engine_params)

dataset = generator.create_data(
    num_steps=df_config.dataset.creation.num_steps,
    batch_size=df_config.dataset.creation.batch_size,
    sys_msg=df_config.dataset.creation.sys_msg,
    topic_model=tree,
)

print(f"✓ Generated {len(dataset)} samples")
```

--------------------------------

### Reinstall or Upgrade DeepFabric

Source: https://github.com/lukehinds/deepfabric/blob/main/docs/guide/instruction-formats/chain-of-thought/reference/troubleshooting.md

Shows bash commands to uninstall and then reinstall or upgrade the 'deepfabric' package. This can resolve issues caused by corrupted installations.

```bash
pip uninstall deepfabric
pip install deepfabric
# or
pip install --upgrade deepfabric
```

--------------------------------

### Load Environment Variables from .env

Source: https://github.com/lukehinds/deepfabric/blob/main/docs/getting-started/installation.md

Loads variables defined in a `.env` file into the current shell session. This is a common practice for managing application configurations and secrets.

```bash
set -a && source .env && set +a
```

--------------------------------

### Generate Educational Programming Dataset with OpenAI

Source: https://github.com/lukehinds/deepfabric/blob/main/docs/examples/basic-usage.md

Generates a dataset of approximately 25 educational programming examples with clear explanations and beginner-friendly code samples using OpenAI's GPT-4 Turbo. It employs a hierarchical topic tree and a data engine for content creation, saving the output to 'programming_examples.jsonl'.

```yaml
# programming-basics.yaml
dataset_system_prompt: "You are a programming instructor creating educational content for beginners."

topic_tree:
  topic_prompt: "Basic programming concepts for new developers"
  topic_system_prompt: "You are a programming instructor creating educational content for beginners."
  degree: 3
  depth: 2
  temperature: 0.7
  provider: "openai"
  model: "gpt-4-turbo"
  save_as: "programming_topics.jsonl"

data_engine:
  instructions: "Create a clear explanation with a simple code example that a beginner could understand and follow."
  generation_system_prompt: "You are a programming instructor creating educational content for beginners."
  provider: "openai"
  model: "gpt-4-turbo"
  temperature: 0.8
  max_retries: 3

dataset:
  creation:
    num_steps: 25
    batch_size: 3
    provider: "openai"
    model: "gpt-4-turbo"
    sys_msg: true
  save_as: "programming_examples.jsonl"
```

--------------------------------

### Free-text CoT Configuration Example (YAML)

Source: https://github.com/lukehinds/deepfabric/blob/main/docs/guide/instruction-formats/chain-of-thought/configuration/yaml-config.md

An example YAML configuration file optimized for mathematical word problems using free-text reasoning. It details settings for topic generation (tree structure), data generation (LLM parameters, CoT type), dataset assembly, and optional HuggingFace Hub upload.

```yaml
# free-text-math-reasoning.yaml
# Optimized for mathematical word problems with natural language reasoning

dataset_system_prompt: "You are a helpful math tutor who explains problems step-by-step with clear reasoning."

# Topic generation using Tree structure (hierarchical)
topic_tree:
  topic_prompt: "Elementary and middle school mathematics including arithmetic, basic algebra, geometry, and word problems"

  # LLM settings for topic generation
  provider: "openai"
  model: "gpt-4o-mini"
  temperature: 0.7        # Higher creativity for topic diversity

  # Tree structure parameters
  degree: 3               # 3 subtopics per node
  depth: 2                # 2 levels deep

  # Output
  save_as: "math_topics.jsonl"

# Data generation engine
data_engine:
  instructions: "Create clear mathematical word problems that require step-by-step analytical thinking to solve."
  generation_system_prompt: "You are a mathematics educator creating practice problems with detailed reasoning explanations."

  # LLM settings for data generation
  provider: "openai"
  model: "gpt-4o-mini"
  temperature: 0.3        # Lower temperature for consistent reasoning
  max_retries: 4

  # Chain of Thought specific settings
  conversation_type: "cot_freetext"
  reasoning_style: "mathematical"

# Dataset assembly
dataset:
  creation:
    num_steps: 15         # Generate 15 examples
    batch_size: 1         # Process one at a time
    sys_msg: false        # Free-text CoT doesn't use system messages

  save_as: "math_reasoning_freetext.jsonl"

# Optional: Upload to HuggingFace Hub
# huggingface:
#   repository: "username/math-reasoning-cot"
#   tags: ["mathematics", "reasoning", "chain-of-thought", "deepfabric"]

```

--------------------------------

### Bash Commands for DeepFabric Dataset Generation

Source: https://github.com/lukehinds/deepfabric/blob/main/docs/guide/instruction-formats/chain-of-thought/tutorials/math-reasoning.md

These bash commands initiate the dataset generation process using DeepFabric based on a YAML configuration file. The first command ('deepfabric generate math-reasoning-config.yaml') starts the generation, while the second command ('tail -f math_reasoning_dataset.jsonl | wc -l') monitors the progress by displaying the number of lines (generated examples) in the output file in real-time.

```bash
# Generate the dataset
deepfabric generate math-reasoning-config.yaml

# Monitor progress (in another terminal if desired)
tail -f math_reasoning_dataset.jsonl | wc -l
```

--------------------------------

### Advanced Prompt Engineering - Persona Definition

Source: https://github.com/lukehinds/deepfabric/blob/main/docs/guide/system-prompts.md

Example of defining a specific persona for a system prompt, detailing expertise level and background for advanced prompt engineering.

```yaml
system_prompt: "You are a senior machine learning engineer with 10+ years in production ML systems at tech companies like Google and Netflix..."
```

--------------------------------

### JSON Example: Calculation and Information Lookup Tool Planning

Source: https://github.com/lukehinds/deepfabric/blob/main/docs/guide/instruction-formats/agent-tool-calling/single-turn.md

Provides a JSON example for planning a calculation task, such as compound interest. It details the reasoning, selected tool, and parameter construction for the calculation.

```json
{
  "question": "If I invest $5000 at 7% annual interest for 3 years, how much will I have?",
  "tool_planning": [
    {
      "reasoning": "Need to calculate compound interest with given parameters",
      "selected_tool": {...calculator_tool...},
      "parameter_reasoning": {
        "expression": "Compound interest formula: P(1+r)^t where P=5000, r=0.07, t=3"
      },
      "expected_result": "Final investment value after 3 years"
    }
  ]
}

```

--------------------------------

### Comprehensive DeepFabric Data Generation Example

Source: https://github.com/lukehinds/deepfabric/blob/main/docs/cli/generate.md

An example command demonstrating a full synthetic data generation pipeline with multiple overrides for saving paths, AI provider and model, topic structure parameters, dataset size, batch processing, temperature, and system message inclusion.

```bash
deepfabric generate research-dataset.yaml \
  --save-tree research_topics.jsonl \
  --dataset-save-as research_examples.jsonl \
  --provider anthropic \
  --model claude-sonnet-4-5 \
  --degree 4 \
  --depth 3 \
  --num-steps 200 \
  --batch-size 8 \
  --temperature 0.8 \
  --sys-msg true
```

--------------------------------

### Configure base_delay (YAML)

Source: https://github.com/lukehinds/deepfabric/blob/main/docs/operation/rate-limiting.md

This YAML example shows how to set the initial delay in seconds before the first retry attempt. The `base_delay` parameter influences the starting point of the backoff strategy.

```yaml
rate_limit:
  base_delay: 2.0  # Wait 2 seconds before first retry
```

--------------------------------

### Verify DeepFabric API Key Format

Source: https://github.com/lukehinds/deepfabric/blob/main/docs/guide/instruction-formats/chain-of-thought/reference/troubleshooting.md

Bash commands using `grep` to verify that the `OPENAI_API_KEY` and `ANTHROPIC_API_KEY` environment variables follow the expected starting patterns ('sk-' and 'sk-ant-').

```bash
# OpenAI keys start with "sk-"
echo $OPENAI_API_KEY | grep "^sk-"

# Anthropic keys start with "sk-ant-"
echo $ANTHROPIC_API_KEY | grep "^sk-ant-"
```

--------------------------------

### Example YAML Configuration for DeepFabric Formatter

Source: https://github.com/lukehinds/deepfabric/blob/main/docs/cli/format.md

Illustrates a sample YAML configuration file for the DeepFabric formatter. This file allows for detailed customization of formatter settings, including template paths and output formats.

```yaml
dataset:
  formatters:
    - name: "chatml_training"
      template: "builtin://chatml.py"
      output: "formatted_output.jsonl"
      config:
        output_format: "text"
        start_token: "<|im_start|>"
        end_token: "<|im_end|>"
```

--------------------------------

### Configure Hugging Face Token for Uploads

Source: https://github.com/lukehinds/deepfabric/blob/main/docs/getting-started/installation.md

Sets the Hugging Face token as an environment variable, which is required for uploading datasets to Hugging Face Hub. Alternatively, the `huggingface-cli login` command can be used.

```bash
export HF_TOKEN="your-huggingface-token"
```

--------------------------------

### Configure Anthropic API Key

Source: https://github.com/lukehinds/deepfabric/blob/main/docs/getting-started/installation.md

Sets the Anthropic API key as an environment variable, allowing DeepFabric to use Claude models. Ensure you replace 'your-api-key-here' with your valid Anthropic API key.

```bash
export ANTHROPIC_API_KEY="your-api-key-here"
```

--------------------------------

### Set API Key Environment Variables (Bash)

Source: https://github.com/lukehinds/deepfabric/blob/main/docs/guide/instruction-formats/agent-tool-calling/tutorials/getting-started.md

Sets the necessary API key as an environment variable for either OpenAI or Anthropic, required for DeepFabric to authenticate with the chosen provider.

```bash
# For OpenAI
export OPENAI_API_KEY="your-api-key-here"

# Or for Anthropic
export ANTHROPIC_API_KEY="your-api-key-here"

```

--------------------------------

### Instruction Format Example (JSON)

Source: https://github.com/lukehinds/deepfabric/blob/main/docs/guide/dataset-generation.md

Illustrates the JSON structure for the Instruction Format, ideal for instruction-following model training. It consists of a single instruction and its corresponding response.

```json
{
  "instruction": "Explain the concept of recursion with a simple example",
  "response": "Recursion is a programming technique where a function calls itself..."
}
```

--------------------------------

### Configure DeepFabric Provider and Schema for CoT Hybrid

Source: https://github.com/lukehinds/deepfabric/blob/main/docs/guide/instruction-formats/chain-of-thought/reference/troubleshooting.md

YAML configuration example for DeepFabric's `data_engine`, specifying a provider and model compatible with complex schemas required for `cot_hybrid` conversation types.

```yaml
# Some providers have different schema requirements
data_engine:
  provider: "openai"
  model: "gpt-4o-mini"     # ✅ Supports structured output
  # model: "gpt-4-turbo"  # ⚠️  Limited schema support

  conversation_type: "cot_hybrid"  # Complex schema
```

--------------------------------

### DeepFabric YAML Configuration for Python Tutorial

Source: https://github.com/lukehinds/deepfabric/blob/main/docs/getting-started/first-dataset.md

This YAML configuration file defines the parameters for generating a synthetic dataset about Python programming fundamentals. It specifies system prompts, topic tree generation settings (degree, depth, temperature, provider, model), data engine instructions, and dataset creation parameters.

```yaml
dataset_system_prompt: "You are a Python programming instructor providing clear, educational content for intermediate developers."

topic_tree:
  topic_prompt: "Python programming fundamentals"
  topic_system_prompt: "You are a Python programming instructor providing clear, educational content for intermediate developers."
  degree: 3
  depth: 2
  temperature: 0.7
  provider: "ollama"
  model: "mistral"
  save_as: "python_topics.jsonl"

data_engine:
  instructions: "Create a Python code example with detailed explanation suitable for intermediate developers."
  generation_system_prompt: "You are a Python programming instructor providing clear, educational content for intermediate developers."
  provider: "ollama"
  model: "mistral"
  temperature: 0.8
  max_retries: 3

dataset:
  creation:
    num_steps: 10
    batch_size: 2
    provider: "ollama"
    model: "mistral"
    sys_msg: true
  save_as: "python_tutorial_dataset.jsonl"
```