Draccus

https://pypi.org/project/draccus/
Admin
Draccus is a Python framework for dataclass-based configuration management with support for...
Tokens:7,081
Snippets:94
Trust Score:10
Update:5 months ago
Show doc for...
Tokens:
Raw
### Command-line Parsing Example

Source: https://pypi.org/project/draccus/0.11.5

Demonstrates parsing configuration directly from the command line using the `draccus.parse` function with a dataclass.

```bash
$ python train_model.py --exp_name=my_first_model
Training my_first_model with 8 workers...

```

--------------------------------

### YAML Configuration Example

Source: https://pypi.org/project/draccus/0.11.5

Demonstrates the YAML format for configuration files, which Draccus uses.

```yaml
compute:
  worker_inds: [0,2,3]

```

--------------------------------

### YAML Configuration File Example

Source: https://pypi.org/project/draccus/0.11.5

An example of a YAML configuration file that can be used with Draccus. This file defines experiment name, worker count, and nested model configuration.

```yaml
exp_name: my_yaml_exp
workers: 42

model:
  type: bert
  num_layers: 24
  num_heads: 24
  hidden_size: 1024
  dropout: 0.2


```

--------------------------------

### Dataclass Configuration with Post-Init and Properties

Source: https://pypi.org/project/draccus

Define a dataclass for training configuration, utilizing `__post_init__` for default value processing and `@property` for derived attributes. This example uses `draccus.wrap` for parsing.

```python
from dataclasses import dataclass, field
from pathlib import Path
from typing import Optional
import draccus


@dataclass
class TrainConfig:
    """ Training config for Machine Learning """
    # The number of workers for training
    workers: int = field(default=8)
    # The number of workers for evaluation
    eval_workers: Optional[int] = field(default=None)
    # The experiment name
    exp_name: str = field(default='default_exp')
    # The experiment root folder path
    exp_root: Path = field(default=Path('/share/experiments'))

    def __post_init__(self):
        # A builtin method of dataclasses, used for post-processing our configuration.
        self.eval_workers = self.eval_workers or self.workers

    @property
    def exp_dir(self) -> Path:
        # Properties are great for arguments that can be derived from existing ones
        return self.exp_root / self.exp_name


@draccus.wrap()
def main(cfg: TrainConfig):
    print(f'Training {cfg.exp_name}...)
    print(f'	Using {cfg.workers} workers and {cfg.eval_workers} evaluation workers')
    print(f'	Saving to {cfg.exp_dir}')

```

```bash
$ python -m train_model.py --exp_name=my_second_exp --workers=42
Training my_second_exp...
    Using 42 workers and 42 evaluation workers
    Saving to /share/experiments/my_second_exp


```

--------------------------------

### Serialize Dataclass to YAML with pyrallis.dump

Source: https://pypi.org/project/draccus

This example shows how to serialize a dataclass configuration object into a YAML file using `pyrallis.dump`. The output YAML structure is also displayed.

```yaml
compute:
  eval_workers: 2
  workers: 8
log:
  exp_name: my_third_exp
  exp_root: /share/experiments


```

--------------------------------

### Basic Draccus Dataclass Configuration

Source: https://pypi.org/project/draccus/0.11.5

Use this snippet to parse command-line arguments or YAML files into a dataclass. Ensure draccus is installed and imported.

```python
from dataclasses import dataclass
import draccus


@dataclass
class TrainConfig:
    """Training Config for Machine Learning"""
    workers: int = 8               # The number of workers for training
    exp_name: str = 'default_exp'  # The experiment name


@draccus.wrap()
def main(cfg: TrainConfig):
    print(f"Training {cfg.exp_name} with {cfg.workers} workers...")

if __name__ == "__main__":
    main()

```

--------------------------------

### Including Config Files in Draccus (model_config.yaml)

Source: https://pypi.org/project/draccus/0.11.5

Example of a configuration file that can be included in another configuration file using Draccus's `!include` directive.

```yaml
# model_config.yaml
type: bert
num_layers: 24
num_heads: 24
hidden_size: 1024
dropout: 0.2


```

--------------------------------

### Registering Custom Decoding Functions

Source: https://pypi.org/project/draccus/0.11.5

Example of how to register a custom decoding function when an underlying error occurs for a specific type.

```python
Underlying error: No decoding function for type ~KT, consider using pyrallis.decode.register

```

--------------------------------

### Setting Configuration Type

Source: https://pypi.org/project/draccus

Example of how to globally set the configuration file type to JSON using pyrallis. This can be done either by calling a function or using a context manager.

```python
pyrallis.set_config_type('json')

```

```python
with pyrallis.config_type('json'):

```

--------------------------------

### Including Config Files in Draccus (train_config.yaml)

Source: https://pypi.org/project/draccus/0.11.5

Example of a main configuration file that includes another configuration file (`model_config.yaml`) using the `!include` directive.

```yaml
# train_config.yaml
exp_name: my_yaml_exp
workers: 42
model: !include model_config.yaml


```

--------------------------------

### Load Configuration with Command-Line Arguments

Source: https://pypi.org/project/draccus/0.11.5

Load configuration by providing arguments directly on the command line, including specifying a config path.

```python
python my_script.py --log.exp_name=readme_exp --config_path=/share/configs/config.yaml
```

--------------------------------

### Including Config Files via Command Line Arguments

Source: https://pypi.org/project/draccus/0.11.5

Demonstrates using the `include` keyword in command-line arguments to include a configuration file for a nested structure.

```bash
python train_model.py --config_path=train_config.yaml --model="include model_config.yaml"
```

--------------------------------

### Including Config Files via Command Line Arguments

Source: https://pypi.org/project/draccus

Shows how to use the `include` keyword in command-line arguments to dynamically include configuration files, overriding existing values.

```yaml
# model_config.yaml
type: bert_cli
num_layers: 48
num_heads: 48


```

```yaml
# train_config.yaml
exp_name: my_yaml_exp
workers: 42
model: 
  type: bert
  num_layers: 24
  num_heads: 24


```

--------------------------------

### Setting Configuration Type

Source: https://pypi.org/project/draccus/0.11.5

Demonstrates how to set the configuration file type to JSON globally.

```python
pyrallis.set_config_type('json')

```

--------------------------------

### Using Configuration Type Context Manager

Source: https://pypi.org/project/draccus/0.11.5

Shows how to use a context manager to specify the configuration file type for a specific block of code.

```python
with pyrallis.config_type('json'):

```

--------------------------------

### Command-line Argument Parsing (YAML Syntax)

Source: https://pypi.org/project/draccus/0.11.5

Shows how to pass configuration values using command-line arguments with YAML-like syntax.

```bash
python my_app.py --compute.worker_inds=[0,2,3]

```

--------------------------------

### Using Draccus with YAML Configuration Files

Source: https://pypi.org/project/draccus/0.11.5

Demonstrates how to specify configuration values using a YAML file. The file content is merged with command-line arguments.

```bash
$ python train_model.py --config_path=some_config.yaml --exp_name=my_first_exp
Training my_first_exp with 42 workers...

```

--------------------------------

### Dataclass Configuration with Post-Init and Properties

Source: https://pypi.org/project/draccus/0.11.5

Define a dataclass for training configuration, utilizing `field` for default values and `Optional` types. Implement `__post_init__` for post-processing and `@property` for derived attributes.

```python
from dataclasses import dataclass, field
from pathlib import Path
from typing import Optional
import draccus


@dataclass
class TrainConfig:
    """ Training config for Machine Learning """
    # The number of workers for training
    workers: int = field(default=8)
    # The number of workers for evaluation
    eval_workers: Optional[int] = field(default=None)
    # The experiment name
    exp_name: str = field(default='default_exp')
    # The experiment root folder path
    exp_root: Path = field(default=Path('/share/experiments'))

    def __post_init__(self):
        # A builtin method of dataclasses, used for post-processing our configuration.
        self.eval_workers = self.eval_workers or self.workers

    @property
    def exp_dir(self) -> Path:
        # Properties are great for arguments that can be derived from existing ones
        return self.exp_root / self.exp_name


@draccus.wrap()
def main(cfg: TrainConfig):
    print(f'Training {cfg.exp_name}...)
    print(f'	Using {cfg.workers} workers and {cfg.eval_workers} evaluation workers')
    print(f'	Saving to {cfg.exp_dir}')

```

--------------------------------

### Command-line Execution of Dataclass Configuration

Source: https://pypi.org/project/draccus/0.11.5

Execute the Python script with command-line arguments to override default configuration values. Observe the updated output reflecting the provided arguments.

```bash
$ python -m train_model.py --exp_name=my_second_exp --workers=42
Training my_second_exp...
    Using 42 workers and 42 evaluation workers
    Saving to /share/experiments/my_second_exp

```

--------------------------------

### Use pyrallis.wrap Decorator for Configuration

Source: https://pypi.org/project/draccus/0.11.5

The pyrallis.wrap decorator simplifies configuration parsing by automatically using type hints. This is the preferred syntax for the tutorial.

```python
@pyrallis.wrap()
def main(cfg: TrainConfig):
    # The decorator automagically uses the type hint to parsers arguments into TrainConfig
    print(f'Training {cfg.exp_name} with {cfg.workers} workers...)

```

--------------------------------

### Hierarchical Configurations with Nested Dataclasses

Source: https://pypi.org/project/draccus

Demonstrates how to build complex configurations using nested dataclasses for compute and logging. The `pyrallis.wrap` decorator handles parsing of nested arguments.

```python
@dataclass
class ComputeConfig:
    """ Config for training resources """
    # The number of workers for training
    workers: int = field(default=8)
    # The number of workers for evaluation
    eval_workers: Optional[int] = field(default=None)

    def __post_init__(self):
        # A builtin method of dataclasses, used for post-processing our configuration.
        self.eval_workers = self.eval_workers or self.workers


@dataclass
class LogConfig:
    """ Config for logging arguments """
    # The experiment name
    exp_name: str = field(default='default_exp')
    # The experiment root folder path
    exp_root: Path = field(default=Path('/share/experiments'))

    @property
    def exp_dir(self) -> Path:
        # Properties are great for arguments that can be derived from existing ones
        return self.exp_root / self.exp_name

# TrainConfig will be our main configuration class.
# Notice that default_factory is the standard way to initialize a class argument in dataclasses

@dataclass
class TrainConfig:
    log: LogConfig = field(default_factory=LogConfig)
    compute: ComputeConfig = field(default_factory=ComputeConfig)

@pyrallis.wrap()
def main(cfg: TrainConfig):
    print(f'Training {cfg.log.exp_name}...)
    print(f'	Using {cfg.compute.workers} workers and {cfg.compute.eval_workers} evaluation workers')
    print(f'	Saving to {cfg.log.exp_dir}')

```

```bash
$ python train_model.py --log.exp_name=my_third_exp --compute.eval_workers=2
Training my_third_exp...
    Using 8 workers and 2 evaluation workers
    Saving to /share/experiments/my_third_exp


```

--------------------------------

### Command-line Argument Parsing (Standard Syntax)

Source: https://pypi.org/project/draccus/0.11.5

Illustrates the more standard argparse syntax for passing list values as separate arguments.

```bash
python my_app.py --compute.worker_inds 0 2 3

```

--------------------------------

### Main Config for Command Line Inclusion

Source: https://pypi.org/project/draccus/0.11.5

The main configuration file that references a nested configuration to be included via command-line arguments.

```yaml
# train_config.yaml
exp_name: my_yaml_exp
workers: 42
model: 
  type: bert
  num_layers: 24
  num_heads: 24


```

--------------------------------

### Load Configuration Directly from YAML

Source: https://pypi.org/project/draccus/0.11.5

Load configuration directly from a YAML file into a specified configuration class without command-line parsing.

```python
cfg = pyrallis.load(TrainConfig, '/share/configs/config.yaml')
```

--------------------------------

### Including Configs at Top Level with YAML Merge

Source: https://pypi.org/project/draccus/0.11.5

Shows how to use PyYAML's `<<` merge key with `!include` to incorporate a base configuration file into the main configuration.

```yaml
# base_config.yaml
type: bert
lr: 0.001

# train_config.yaml
<<: !include base_config.yaml
exp_name: my_yaml_exp


```

--------------------------------

### Load Configuration with Pyrallis Decorator

Source: https://pypi.org/project/draccus/0.11.5

Use the @pyrallis.wrap decorator to load configuration from a specified YAML file.

```python
@pyrallis.wrap(config_path='/share/configs/config.yaml')
```

--------------------------------

### Nested Config File for Command Line Inclusion

Source: https://pypi.org/project/draccus/0.11.5

A configuration file used for inclusion via command-line arguments, demonstrating how nested configurations can be dynamically loaded.

```yaml
# model_config.yaml
type: bert_cli
num_layers: 48
num_heads: 48


```

--------------------------------

### Command-line Parsing of Hierarchical Configurations

Source: https://pypi.org/project/draccus/0.11.5

Parse hierarchical configurations using dot notation for nested arguments on the command line. The output reflects the nested structure of the configuration.

```bash
$ python train_model.py --log.exp_name=my_third_exp --compute.eval_workers=2
Training my_third_exp...
    Using 8 workers and 2 evaluation workers
    Saving to /share/experiments/my_third_exp

```

--------------------------------

### Parse Configuration with pyrallis.parse

Source: https://pypi.org/project/draccus/0.11.5

Use pyrallis.parse to load configuration from a dataclass. Ensure the config_class is defined.

```python
def main():
    cfg = pyrallis.parse(config_class=TrainConfig)
    print(f'Training {cfg.exp_name} with {cfg.workers} workers...')

```

--------------------------------

### Including Config Files at Top Level with YAML Merge

Source: https://pypi.org/project/draccus

Demonstrates using PyYAML's `<<` merge key with `!include` to combine a base configuration file with a main configuration file.

```yaml
# base_config.yaml
type: bert
lr: 0.001


```

```yaml
# train_config.yaml
<<: !include base_config.yaml
exp_name: my_yaml_exp


```

--------------------------------

### Automatic --help Generation

Source: https://pypi.org/project/draccus/0.11.5

Pyrallis automatically generates a `--help` string from dataclass comments, providing detailed information about available arguments and their descriptions.

```bash
$ python train_model.py --help
usage: train_model.py [-h] [--config_path str] [--workers int] [--exp_name str]

optional arguments:
  -h, --help      show this help message and exit
  --config_path str    Path for a config file to parse with pyrallis (default:
                  None)

TrainConfig:
   Training config for Machine Learning

  --workers int   The number of workers for training (default: 8)
  --exp_name str  The experiment name (default: default_exp)


```

--------------------------------

### Parse Configuration from YAML File

Source: https://pypi.org/project/draccus/0.11.5

Load a dataclass configuration from a YAML file using `pyrallis.parse` by specifying the `config_path`. This can be combined with command-line arguments.

```python
cfg = pyrallis.parse(config_class=TrainConfig,
                              config_path='/share/configs/config.yaml')

```

--------------------------------

### Hierarchical Configuration with Nested Dataclasses

Source: https://pypi.org/project/draccus/0.11.5

Define nested dataclasses for complex configurations, such as `ComputeConfig` and `LogConfig`, within a main `TrainConfig`. Use `default_factory` for initializing nested dataclass instances.

```python
@dataclass
class ComputeConfig:
    """ Config for training resources """
    # The number of workers for training
    workers: int = field(default=8)
    # The number of workers for evaluation
    eval_workers: Optional[int] = field(default=None)

    def __post_init__(self):
        # A builtin method of dataclasses, used for post-processing our configuration.
        self.eval_workers = self.eval_workers or self.workers


@dataclass
class LogConfig:
    """ Config for logging arguments """
    # The experiment name
    exp_name: str = field(default='default_exp')
    # The experiment root folder path
    exp_root: Path = field(default=Path('/share/experiments'))

    @property
    def exp_dir(self) -> Path:
        # Properties are great for arguments that can be derived from existing ones
        return self.exp_root / self.exp_name

# TrainConfig will be our main configuration class.
# Notice that default_factory is the standard way to initialize a class argument in dataclasses

@dataclass
class TrainConfig:
    log: LogConfig = field(default_factory=LogConfig)
    compute: ComputeConfig = field(default_factory=ComputeConfig)

@pyrallis.wrap()
def main(cfg: TrainConfig):
    print(f'Training {cfg.log.exp_name}...)
    print(f'	Using {cfg.compute.workers} workers and {cfg.compute.eval_workers} evaluation workers')
    print(f'	Saving to {cfg.log.exp_dir}')

```

--------------------------------

### Serialize Dataclass Configuration to YAML

Source: https://pypi.org/project/draccus/0.11.5

Use `pyrallis.dump` to serialize a dataclass configuration object into a YAML file. This function extends `yaml.dump` and uses the same syntax.

```yaml
compute:
  eval_workers: 2
  workers: 8
log:
  exp_name: my_third_exp
  exp_root: /share/experiments

```

--------------------------------

### Parse Config with pyrallis.wrap Decorator

Source: https://pypi.org/project/draccus

The pyrallis.wrap decorator provides a more concise way to parse configurations. It automatically infers the configuration class from the type hint of the function argument.

```python
@pyrallis.wrap()
def main(cfg: TrainConfig):
    # The decorator automagically uses the type hint to parsers arguments into TrainConfig
    print(f'Training {cfg.exp_name} with {cfg.workers} workers...')

```

--------------------------------

### Define Model Choices with ChoiceRegistry

Source: https://pypi.org/project/draccus/0.11.5

Use ChoiceRegistry to define a selection of model types that can be chosen at runtime. Register subclasses using the `@ModelConfig.register_subclass` decorator.

```python
from dataclasses import dataclass
import draccus


# Choice Registry lets you define a choice of implementations that can be selected at runtime
@dataclass
class ModelConfig(draccus.ChoiceRegistry):
    pass


@ModelConfig.register_subclass('gpt')
@dataclass
class GPTConfig(ModelConfig):
    """GPT Model Config"""
    num_layers: int = 12
    num_heads: int = 12
    hidden_size: int = 768


@ModelConfig.register_subclass('bert')
@dataclass
class BERTConfig(ModelConfig):
    """BERT Model Config"""
    num_layers: int = 12
    num_heads: int = 12
    hidden_size: int = 768
    dropout: float = 0.1


@dataclass
class TrainConfig:
    """Training Config for Machine Learning"""
    workers: int = 8                  # The number of workers for training
    exp_name: str = 'default_exp'     # The experiment name

    model: ModelConfig = GPTConfig()  # The model configuration


@draccus.wrap()
def main(cfg: TrainConfig):
    print(f"Training {cfg.exp_name} with {cfg.workers} workers...")

```

--------------------------------

### Parse Dataclass with Pyrallis

Source: https://pypi.org/project/draccus/0.11.5

Use `draccus.parse` to create an argparse configuration from a predefined dataclass. This function parses arguments directly from the command line.

```python
from dataclasses import dataclass, field
import draccus


@dataclass
class TrainConfig:
    """ Training config for Machine Learning """
    # The number of workers for training
    workers: int = field(default=8)
    # The experiment name
    exp_name: str = field(default='default_exp')


def main():
    cfg = draccus.parse(config_class=TrainConfig)
    print(f'Training {cfg.exp_name} with {cfg.workers} workers...')


if __name__ == '__main__':
    main()

```

--------------------------------

### Register Custom Type for Decoding

Source: https://pypi.org/project/draccus/0.11.5

Register a custom function to decode a specific type, such as numpy arrays, from command-line or YAML.

```python
# For decoding from cmd/yaml
pyrallis.decode.register(np.ndarray,np.asarray)
```

--------------------------------

### Pyrallis Field for Mutable Defaults

Source: https://pypi.org/project/draccus/0.11.5

Use pyrallis.field with is_mutable=True to handle mutable default values in dataclasses, avoiding common pitfalls.

```python
from draccus import field

worker_inds: List[int] = field(default=[1, 2, 3], is_mutable=True)
```

--------------------------------

### Register Custom Type for Decoding

Source: https://pypi.org/project/draccus

Register a custom function to decode NumPy arrays from YAML or command-line arguments.

```python
pyrallis.decode.register(np.ndarray,np.asarray)
```

--------------------------------

### Register Custom Type for Encoding

Source: https://pypi.org/project/draccus

Register a custom function to encode NumPy arrays to YAML.

```python
pyrallis.encode.register(np.ndarray, lambda x: x.tolist())
```

--------------------------------

### Register Custom Type for Encoding with Decorator

Source: https://pypi.org/project/draccus/0.11.5

Register a custom encoding function for a type using the @pyrallis.encode.register decorator.

```python
# Or with the wrapper version instead
@pyrallis.encode.register
def encode_array(arr : np.ndarray) -> str:
    return arr.tolist()
```

--------------------------------

### Register Custom Type for Encoding

Source: https://pypi.org/project/draccus/0.11.5

Register a custom function to encode a specific type, such as numpy arrays, to YAML.

```python
# For encoding to yaml
pyrallis.encode.register(np.ndarray, lambda x: x.tolist())
```
Tokens:
Raw