Metaflow

https://docs.metaflow.org
Admin
Metaflow makes it easy to build and manage real-life data science, AI, and ML projects.
Tokens:117,263
Snippets:1,496
Trust Score:9.6
Update:3 months ago
Show doc for...
Tokens:
Raw
### Start the Dev Stack

Source: https://docs.metaflow.org/getting-started/devstack

Initiate the Metaflow development stack setup, which includes downloading Minikube and deploying components using Tilt.

```bash
metaflow-dev up
```

--------------------------------

### Install Notebook

Source: https://docs.metaflow.org/getting-started/tutorials/season-1-the-local-experience/episode01

Install the Jupyter Notebook environment before running this episode.

```bash
python -m pip install notebook
```

--------------------------------

### Install Notebook and Matplotlib

Source: https://docs.metaflow.org/getting-started/tutorials/season-1-the-local-experience/episode02

Install the necessary libraries for running Jupyter notebooks and creating plots. These are prerequisites for this episode.

```bash
python -m pip install notebook
python -m pip install matplotlib
```

--------------------------------

### Install Metaflow

Source: https://docs.metaflow.org/getting-started/install

Use this command to install the latest version of Metaflow from PyPI.

```bash
pip install metaflow
```

--------------------------------

### TOML Configuration File Example

Source: https://docs.metaflow.org/metaflow/configuring-flows/parsing-configs

An example of a TOML file structure that can be parsed by the `TomlConfigFlow`.

```toml
[general]
a = 5

[resources]
cpu = 1

```

--------------------------------

### YAML Configuration File Example

Source: https://docs.metaflow.org/metaflow/configuring-flows/parsing-configs

An example of a YAML file structure that can be parsed by Metaflow.

```yaml
resources:
  cpu: 1
foo: bar

```

--------------------------------

### List and Info for Metaflow Tutorials

Source: https://docs.metaflow.org/getting-started/tutorials

Commands to list available tutorials or get information about a specific tutorial.

```bash
metaflow tutorials list  

```

```bash
metaflow tutorials info 00-helloworld  

```

--------------------------------

### Install metaflow-card-html Template

Source: https://docs.metaflow.org/metaflow/visualizing-results/advanced-shareable-cards-with-card-templates

Install the metaflow-card-html template using pip. This template converts HTML stored in an artifact to a static card.

```bash
pip install metaflow-card-html  

```

--------------------------------

### Launch Jupyter Notebook

Source: https://docs.metaflow.org/getting-started/tutorials/season-1-the-local-experience/episode01

Start a Jupyter Notebook server to explore the playlist tutorial interactively.

```bash
jupyter-notebook 01-playlist/playlist.ipynb
```

--------------------------------

### Run the Hello Cloud Tutorial

Source: https://docs.metaflow.org/getting-started/tutorials/season-2-scaling-out-and-up/episode05

Execute the main Python script for the Hello Cloud tutorial to start a Metaflow run.

```bash
python 05-hello-cloud/hello-cloud.py run
```

--------------------------------

### Install Metaflow MPI Extension

Source: https://docs.metaflow.org/scaling/remote-tasks/distributed-computing

Command to install the necessary Metaflow extension for using MPI.

```bash
pip install metaflow-mpi  

```

--------------------------------

### Start Local Card Server in Spin Mode

Source: https://docs.metaflow.org/metaflow/visualizing-results/effortless-task-inspection-with-default-cards

Use the `--mode spin` flag to start the local card server for viewing ephemeral spin results. This is useful for developing cards quickly.

```bash
python defaultcard.py --mode spin card server  

```

--------------------------------

### Start Local Card Viewer

Source: https://docs.metaflow.org/metaflow/visualizing-results/dynamic-cards

This command starts a local card viewer that polls for updates at a specified interval, useful for observing dynamic card changes in real-time.

```bash
python clockflow.py card server --poll-interval 1
```

--------------------------------

### Start Jupyter Notebook

Source: https://docs.metaflow.org/getting-started/tutorials/season-2-scaling-out-and-up/episode06

Launch the Jupyter notebook server for the statistics redux tutorial. This command assumes you are in the metaflow-tutorials directory.

```bash
jupyter-notebook 06-statistics-redux/stats.ipynb
```

--------------------------------

### Dataset Configuration File

Source: https://docs.metaflow.org/metaflow/composing-flows/mutators

Example JSON configuration file to specify the URL and filter for the dataset.

```json
{
   "url": "https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2020-01.parquet",
   "filter": "tip_amount > fare_amount"
}

```

--------------------------------

### Start Local Card Server

Source: https://docs.metaflow.org/metaflow/visualizing-results/effortless-task-inspection-with-default-cards

Run this command in a terminal to start the local card server. Navigate to localhost:8324 in your browser to view the latest cards.

```bash
python defaultcard.py card server  

```

--------------------------------

### Configure Timeout for Start Step

Source: https://docs.metaflow.org/metaflow/configuring-flows/basic-configuration

This example demonstrates how to use a JSON config file to set the timeout for a Metaflow step. The config value is accessed both in the decorator and within the step method.

```python
import time
from metaflow import FlowSpec, step, Config, timeout

class TimeoutConfigFlow(FlowSpec):
    config = Config("config", default="myconfig.json")

    @timeout(seconds=config.timeout)
    @step
    def start(self):
        print(f"timing out after {self.config.timeout} seconds")
        time.sleep(7)
        print("success")
        self.next(self.end)

    @step
    def end(self):
        print("full config", self.config)

if __name__ == "__main__":
    TimeoutConfigFlow()

```

--------------------------------

### Upload Data to S3 for Listing Example

Source: https://docs.metaflow.org/scaling/data

Prepares S3 with sample data for demonstrating listing operations. Uses `put_many` to upload multiple objects.

```python
from metaflow import S3

many = {'first/a/object1': 'data',
        'first/b/x/object2': 'data',
        'second/c/object3': 'data'}

with S3(s3root='s3://my-bucket/savin/tmp/s3demo_list/') as s3:
    s3.put_many(many.items())
```

--------------------------------

### Install Metaflow Checkpoint Extension

Source: https://docs.metaflow.org/scaling/checkpoint/introduction

Install the `@checkpoint` extension using pip in your development and deployment environments. Metaflow automatically packages extensions for remote execution.

```bash
pip install metaflow-checkpoint
```

--------------------------------

### Install Conda-Forge Channel

Source: https://docs.metaflow.org/getting-started/tutorials/season-1-the-local-experience/episode04

Before playing this episode, ensure you have the conda package manager installed and the conda-forge channel added. This command adds the specified channel to your conda configuration.

```bash
conda config --add channels conda-forge
```

--------------------------------

### Specify Compute Layer with Batch

Source: https://docs.metaflow.org/api/step-decorators/resources

Example of how to run a Metaflow flow on Batch using the requirements specified in @resources.

```bash
python myflow.py run --with batch
```

--------------------------------

### Install Metaflow Kubeflow Extension

Source: https://docs.metaflow.org/production/scheduling-metaflow-flows/scheduling-with-kubeflow

Install the Metaflow extension for Kubeflow integration using pip. This is required on the deployment environment, not on execution containers.

```bash
pip install metaflow-kubeflow  

```

--------------------------------

### Specify Compute Layer with Kubernetes

Source: https://docs.metaflow.org/api/step-decorators/resources

Example of how to run a Metaflow flow on Kubernetes using the requirements specified in @resources.

```bash
python myflow.py run --with kubernetes
```

--------------------------------

### Run Metaflow Flow with Configuration

Source: https://docs.metaflow.org/metaflow/configuring-flows/basic-configuration

Standard command to run a Metaflow flow. This example assumes the flow is saved as timeout_config.py.

```bash
python timeout_config.py run

```

--------------------------------

### View Card for Photo Flow

Source: https://docs.metaflow.org/metaflow/configuring-flows/basic-configuration

Generates and views the card for the `start` step of the `photo_config` flow.

```bash
python photo_config.py card view start    

```

--------------------------------

### Resume from an Arbitrary Step

Source: https://docs.metaflow.org/metaflow/debugging

Initiate a resume operation from a specified step (e.g., 'start') instead of the default failed step.

```bash
python debug.py resume start  

```

--------------------------------

### Instantiating and Running a Flow with Runner

Source: https://docs.metaflow.org/api/runner

Example of instantiating the Runner with a flow file and executing a run with specific parameters. The result object provides access to run details.

```python
with Runner('slowflow.py', pylint=False) as runner:
    result = runner.run(alpha=5, tags=["abc", "def"], max_workers=5)
    print(result.run.finished)
```

--------------------------------

### Example Metaflow Flow for Kubeflow Deployment

Source: https://docs.metaflow.org/production/scheduling-metaflow-flows/scheduling-with-kubeflow

A sample Metaflow flow demonstrating parameters and steps. Save this code in a Python file to deploy to Kubeflow Pipelines.

```python
from metaflow import FlowSpec, Parameter, step  
  
class ParameterFlow(FlowSpec):  
    alpha = Parameter('alpha',  
                      help='Learning rate',  
                      default=0.01)  
  
    @step  
    def start(self):  
        print('alpha is %f' % self.alpha)  
        self.next(self.end)  
  
    @step  
    def end(self):  
        print('alpha is still %f' % self.alpha)  
  
if __name__ == '__main__':  
    ParameterFlow()  


```

--------------------------------

### Running Flow with Step Mutator via --with

Source: https://docs.metaflow.org/metaflow/composing-flows/mutators

Example command to run the flow, applying the dataset step mutator using the `--with` argument and specifying a URL.

```bash
python datasetflow.py --environment=pypi run --with 'dataset.dataset:url=SOME_URL'  

```

--------------------------------

### Run Metaflow Flow with PyPI Environment

Source: https://docs.metaflow.org/metaflow/configuring-flows/basic-configuration

Execute a Metaflow flow that installs dependencies on the fly using the `--environment=pypi` flag.

```bash
python toplevel.py --environment=pypi run

```

--------------------------------

### Using YAML for Configuration

Source: https://docs.metaflow.org/metaflow/configuring-flows/parsing-configs

This snippet demonstrates how to configure a Metaflow flow to load settings from a YAML file using `yaml.full_load` as the parser. Ensure the `pyyaml` package is installed.

```python
config = Config("config", default="myconfig.yml", parser="yaml.full_load")

```

--------------------------------

### Example Metaflow Flow with Custom Decorator

Source: https://docs.metaflow.org/metaflow/composing-flows/custom-decorators

A complete Metaflow flow demonstrating the usage of a custom decorator ('my_profile') on a specific step.

```python
import time
from metaflow import FlowSpec, step

from myprofiler import my_profile

class WaiterFlow(FlowSpec):

    @step
    def start(self):
        self.waiters = list(range(5))
        self.next(self.wait, foreach='waiters')

    @my_profile
    @step
    def wait(self):
        self.duration = self.input / 10
        print(f"💤 Sleeping for {self.duration}s")
        time.sleep(self.duration)
        self.next(self.join)

    @step
    def join(self, inputs):
        self.total = sum(inp.duration for inp in inputs)
        print(f"Slept {self.total}s in total")
        self.next(self.end)

    @step
    def end(self):
        pass

if __name__ == '__main__':
    WaiterFlow()
```

--------------------------------

### Run the Checkpoint Counter Flow

Source: https://docs.metaflow.org/scaling/checkpoint/introduction

Execute the Metaflow flow after installing the `metaflow-checkpoint` extension. This command initiates the flow defined in the Python script.

```bash
python checkpoint_counter.py run
```

--------------------------------

### Following a Chain of Triggered Runs

Source: https://docs.metaflow.org/production/event-triggering/inspect-events

Use MetaflowTrigger to navigate backwards through a chain of runs connected by `@trigger_on_finish`. This example shows how to get the triggering ETLFlow run from the latest PredictionFlow run.

```python
etl_run = Flow('PredictionFlow').latest_run.trigger.run.trigger.run
```

--------------------------------

### Navigate to Tutorial Directory

Source: https://docs.metaflow.org/getting-started/tutorials/season-1-the-local-experience/episode01

Change your current directory to the metaflow-tutorials folder to access the episode files.

```bash
cd metaflow-tutorials
```

--------------------------------

### Metaflow Flow with Disabled Conda Environment

Source: https://docs.metaflow.org/scaling/dependencies/libraries

Demonstrates disabling isolated Conda environments for specific steps in a Metaflow flow, allowing access to system-wide packages. The `peekaboo` step uses a system-wide Python installation, while `start` and `end` use isolated environments.

```python
import sys  
from metaflow import FlowSpec, step, conda_base, conda  
  
@conda_base(python='3.9.13')  
class PeekabooFlow(FlowSpec):  
  
    @step  
    def start(self):  
        print(sys.executable)  
        self.next(self.peekaboo)  
  
    @conda(disabled=True)  
    @step  
    def peekaboo(self):  
        print(sys.executable)  
        self.next(self.end)  
  
    @step  
    def end(self):  
        print(sys.executable)  
  
if __name__ == '__main__':  
    PeekabooFlow()  

```

--------------------------------

### Open Hello Cloud Notebook

Source: https://docs.metaflow.org/getting-started/tutorials/season-2-scaling-out-and-up/episode05

Launch the Jupyter notebook associated with the Hello Cloud tutorial to interact with the flow and its results.

```bash
jupyter-notebook 05-hello-cloud/hello-cloud.ipynb
```

--------------------------------

### Initialize and Add Packages with uv

Source: https://docs.metaflow.org/scaling/dependencies/uv

Initialize a new Metaflow project using uv and add necessary packages like pandas and metaflow.

```bash
uv init myflow
cd myflow
uv add pandas metaflow
```

--------------------------------

### Show Tutorial Code

Source: https://docs.metaflow.org/getting-started/tutorials/season-1-the-local-experience/episode02

Execute the Python script to display the tutorial's code without running it. This is useful for reviewing the implementation.

```bash
python 02-statistics/stats.py show
```

--------------------------------

### Access Config Artifact using Client API

Source: https://docs.metaflow.org/metaflow/configuring-flows/basic-configuration

Example of how to retrieve a config artifact from a previous Metaflow run using the Client API in a notebook environment.

```python
Flow('TimeoutConfigFlow').latest_run.data.config

```

--------------------------------

### Display Available Parameters

Source: https://docs.metaflow.org/metaflow/basics

Use the --help flag to see all available parameters for your flow and their descriptions.

```bash
python parameter_flow.py run --help
```

--------------------------------

### Pull Metaflow Tutorials

Source: https://docs.metaflow.org/getting-started/tutorials

Use this command to download all Metaflow tutorials to your local directory.

```bash
metaflow tutorials pull  

```

--------------------------------

### Monitor Task Progress with Dynamic Cards

Source: https://docs.metaflow.org/metaflow/visualizing-results/dynamic-cards

This example demonstrates how to use `Markdown` and `ProgressBar` components to show real-time updates during a task's execution. It appends initial components, then iteratively updates their content and refreshes the card.

```python
from metaflow import step, FlowSpec, current, card
from metaflow.cards import Markdown, ProgressBar

class ClockFlow(FlowSpec):
    @card(type="blank", refresh_interval=1)
    @step
    def start(self):
        from datetime import datetime
        import time

        m = Markdown("# Clock is starting 🕒")
        p = ProgressBar(max=30, label="Seconds passed")
        current.card.append(m)
        current.card.append(p)
        current.card.refresh()
        for i in range(31):
            t = datetime.now().strftime("%H:%M:%S")
            m.update(f"# Time is {t}")
            p.update(i)
            current.card.refresh()
            print(t)
            time.sleep(1)
        m.update("# ⏰ ring ring!")
        self.next(self.end)

    @step
    def end(self):
        pass

if __name__ == "__main__":
    ClockFlow()
```

--------------------------------

### Installing PyTorch with CUDA using @conda_base

Source: https://docs.metaflow.org/scaling/remote-tasks/installing-drivers-and-frameworks

Use the `@conda_base` decorator to specify PyTorch and its CUDA toolkit version for your Metaflow flow. This installs the necessary libraries on-the-fly on top of a default image.

```python
from metaflow import FlowSpec, step, resources, conda_base  
  
@conda_base(  
    libraries={  
        "pytorch::pytorch": "2.0.1",  
        "pytorch::pytorch-cuda": "11.8"  
    },  
    python="3.9"  
)  
class GPUCondaFlow(FlowSpec):  
  
    @resources(gpu=1)  
    @step  
    def start(self):  
        import torch # pylint: disable=import-error  
        if torch.cuda.is_available():  
            print('Cuda found 🙌')  
            for d in range(torch.cuda.device_count()):  
                print(f"GPU device {d}:", torch.cuda.get_device_name(d))  
        else:  
            print('No CUDA 😭')  
        self.next(self.end)  
  
    @step  
    def end(self):  
        pass  
  
if __name__ == '__main__':  
    GPUCondaFlow()  

```

--------------------------------

### Define a Sample Metaflow Flow

Source: https://docs.metaflow.org/metaflow/managing-flows/runner

This Python script defines a simple Metaflow flow named SlowFlow with two steps: start and end. The start step includes a delay based on a parameter.

```python
import time
from metaflow import FlowSpec, Parameter, step

class SlowFlow(FlowSpec):

    secs = Parameter('seconds', default=3)

    @step
    def start(self):
        for i in range(self.secs):
            print(f"{i} seconds passed")
            time.sleep(1)
        self.next(self.end)

    @step
    def end(self):
        pass

if __name__ == '__main__':
    SlowFlow()

```

--------------------------------

### ArgoWorkflowsTriggeredRun.status

Source: https://docs.metaflow.org/api/deployer

Gets the status of a triggered run on Argo Workflows.

```APIDOC
## ArgoWorkflowsTriggeredRun.status

### Description
Get the status of the triggered run.

### Returns
- _str, optional_ - The status of the workflow considering the run object, or None if the status could not be retrieved.
```

--------------------------------

### Metaflow Flow Demonstrating AI Debug Decorator

Source: https://docs.metaflow.org/metaflow/composing-flows/advanced-custom-decorators

This Metaflow flow, 'FailFlow', uses the custom '@ai_debug' decorator on its 'start' step to catch and report errors. The 'start' step is designed to fail intentionally to test the decorator's functionality. Ensure the 'ai_debug.py' module is available and OPENAI_API_KEY is set.

```python
import math
from metaflow import FlowSpec, step

from ai_debug import ai_debug

class FailFlow(FlowSpec):

    @ai_debug
    @step
    def start(self):
        x = 3
        for i in range(5):
            math.sqrt(x - i)
        self.next(self.end)

    @step
    def end(self):
        pass

if __name__ == '__main__':
    FailFlow()

```

--------------------------------

### Instantiate Deployer

Source: https://docs.metaflow.org/metaflow/managing-flows/deployer

Instantiate the `Deployer` class by providing the path to the flow file you intend to deploy. This is the first step in programmatically deploying a flow.

```python
from metaflow import Deployer
deployer = Deployer('helloflow.py')
```

--------------------------------

### Show Flow Steps

Source: https://docs.metaflow.org/metaflow/basics

Use this command to print the steps of your flow, helping you assess the overall structure and readability.

```bash
python myflow.py show
```

--------------------------------

### Accessing the run ID

Source: https://docs.metaflow.org/api/current

Get the unique identifier for the current run with 'current.run_id'.

```python
run_id = current.run_id
```

--------------------------------

### Upgrade Metaflow

Source: https://docs.metaflow.org/getting-started/install

Use this command to upgrade an existing Metaflow installation to the latest version.

```bash
pip install --upgrade metaflow
```

--------------------------------

### Runner.async_run

Source: https://docs.metaflow.org/api/runner

Starts a flow run asynchronously and returns immediately. This method must be awaited.

```APIDOC
## Runner.async_run(self, **kwargs)

### Description
Non-blocking execution of the run. This method will return as soon as the run has launched.
Note that this method is asynchronous and needs to be `await`ed.

### Parameters
- **kwargs** (Any) - Additional arguments that you would pass to `python myflow.py` after the `run` command, in particular, any parameters accepted by the flow.

### Returns
- _ExecutingRun_ - ExecutingRun representing the run that was started.
```

--------------------------------

### Accessing the username

Source: https://docs.metaflow.org/api/current

Get the name of the user who initiated the run, if available, using 'current.username'.

```python
username = current.username
```

--------------------------------

### Using S3 Client as a Context Manager

Source: https://docs.metaflow.org/api/S3

This snippet demonstrates the recommended way to use the S3 client. The context manager automatically handles the creation and deletion of a temporary directory.

```python
with S3() as s3:
    data = [obj.blob for obj in s3.get_many(urls)]
print(data)
```

--------------------------------

### Run OmegaConf Flow with Overrides

Source: https://docs.metaflow.org/metaflow/configuring-flows/parsing-configs

Execute the OmegaConf-integrated flow, demonstrating how to override specific configuration values at runtime using the `overrides` parameter and OmegaConf's dot-list format.

```bash
python omega_config.py run --overrides dataset.name=newfile.txt,training.epochs=50

```

--------------------------------

### Runner.run

Source: https://docs.metaflow.org/api/runner

Starts a flow run and blocks until it completes. Accepts keyword arguments that are passed to the flow.

```APIDOC
## Runner.run(self, **kwargs)

### Description
Blocking execution of the run. This method will wait until the run has completed execution.

### Parameters
- **kwargs** (Any) - Additional arguments that you would pass to `python myflow.py` after the `run` command, in particular, any parameters accepted by the flow.

### Returns
- _ExecutingRun_ - ExecutingRun containing the results of the run.
```

--------------------------------

### Instantiating the Metaflow root object

Source: https://docs.metaflow.org/metaflow/client

Shows how to instantiate the Metaflow root object, which serves as the entry point for accessing all other Metaflow objects.

```python
from metaflow import Metaflow  
mf = Metaflow()  
```

--------------------------------

### Add a Tag Before a Run

Source: https://docs.metaflow.org/scaling/tagging

Annotate a run with a tag using the `--tag` option when starting the run.

```bash
python helloworld.py run --tag crazy_test  
```

--------------------------------

### Applying Decorators via Command Line

Source: https://docs.metaflow.org/api/step-decorators

Demonstrates how to attach decorators like `@retry` and `@batch` to all steps of a flow using the `--with` option on the command line.

```bash
python myflow.py run --with retry --with batch:cpu=4  

```

--------------------------------

### Runner.async_resume

Source: https://docs.metaflow.org/api/runner

Resumes a previously started flow run asynchronously and returns immediately. This method must be awaited.

```APIDOC
## Runner.async_resume(self, **kwargs)

### Description
Non-blocking resume execution of the run. This method will return as soon as the resume has launched.
Note that this method is asynchronous and needs to be `await`ed.

### Parameters
- **kwargs** (Any) - Additional arguments that you would pass to `python myflow.py` after the `resume` command.

### Returns
- _ExecutingRun_ - ExecutingRun representing the resumed run that was started.
```

--------------------------------

### Specifying Custom Conda Channel

Source: https://docs.metaflow.org/scaling/dependencies/libraries

Example of how to specify a custom channel for a Conda package, such as 'pytorch' from the 'pytorch' channel.

```python
@conda(
    python='3.9.13',
    packages={'pytorch::pytorch': '2.1.2'}
)

```

--------------------------------

### Open Jupyter Notebook

Source: https://docs.metaflow.org/getting-started/tutorials/season-1-the-local-experience/episode02

Launch the Jupyter Notebook associated with this episode to visualize the computed statistics and explore the data.

```bash
jupyter-notebook 02-statistics/stats.ipynb
```

--------------------------------

### Runner.resume

Source: https://docs.metaflow.org/api/runner

Resumes a previously started flow run and blocks until it completes. Accepts keyword arguments that are passed to the flow.

```APIDOC
## Runner.resume(self, **kwargs)

### Description
Blocking resume execution of the run. This method will wait until the resumed run has completed execution.

### Parameters
- **kwargs** (Any) - Additional arguments that you would pass to `python ./myflow.py` after the `resume` command.

### Returns
- _ExecutingRun_ - ExecutingRun containing the results of the resumed run.
```

--------------------------------

### current.project_flow_name

Source: https://docs.metaflow.org/api/current

Gets the flow name prefixed with the current project and branch. This is used to identify the deployment on a production scheduler.

```APIDOC
## current.project_flow_name

### Description
The flow name prefixed with the current project and branch. This name identifies the deployment on a production scheduler.

### Returns
- **str**: Flow name prefixed with project information.
```

--------------------------------

### Create Production Deployment

Source: https://docs.metaflow.org/production/coordinating-larger-metaflow-projects

Deploys the project flow as the official production version to AWS Step Functions. It will be named 'example_project.prod.ProjectFlow'.

```bash
python project_flow.py --production step-functions create  

```

--------------------------------

### Add Tag via Command Line

Source: https://docs.metaflow.org/metaflow/client

Provides an example of adding a tag to a run using the Metaflow command-line interface.

```bash
python helloflow.py tag add --run-id 2 one_tag
```

--------------------------------

### AttributeError in ShowDoc

Source: https://docs.metaflow.org/api/cards

An example of an AttributeError encountered when using ShowDoc, indicating an issue with string object attributes during signature formatting.

```python
    ---------------------------------------------------------------------------
  
    AttributeError                            Traceback (most recent call last)
  
    File ~/mambaforge/envs/docs/lib/python3.11/site-packages/IPython/core/formatters.py:344, in BaseFormatter.__call__(self, obj)
        342     method = get_real_method(obj, self.print_method)
        343     if method is not None:
    --> 344         return method()
        345     return None
        346 else:
  
  
    File ~/mambaforge/envs/docs/lib/python3.11/site-packages/nbdoc/showdoc.py:218, in ShowDoc._repr_html_(self)
        216 def _repr_html_(self):
        217     """This method controls what is displayed in Jupyter Notebooks."""
    --> 218     return f'<HTMLRemove>\n{self.nbhtml}\n</HTMLRemove>\n{self.jsx}'
  
  
    File ~/mambaforge/envs/docs/lib/python3.11/site-packages/nbdoc/showdoc.py:253, in ShowDoc.jsx(self)
        251 nm = f'<DocSection type="{self.typ}" name="{self.objnm}" module="{self.modnm}" show_import="{self.show_import}" heading_level="{self.hd_lvl}"{self._src_link_attr}>'
        252 spoof = '...' if self.decorator else self.spoofstr
    --> 253 sp = get_sig_section(self.obj, spoofstr=spoof)
        254 return f'{nm}\n{sp}\n' + self.npdocs + '\n</DocSection>'
  
  
    File ~/mambaforge/envs/docs/lib/python3.11/site-packages/nbdoc/showdoc.py:115, in get_sig_section(obj, spoofstr)
        113         return ''
        114     params = sig.parameters.items()
    --> 115     jsx_params = [fmt_sig_param(p) for _, p in params]
        116 else:
        117     jsx_params = [f'<SigArg name="{spoofstr}" />']
  
  
    File ~/mambaforge/envs/docs/lib/python3.11/site-packages/nbdoc/showdoc.py:115, in <listcomp>(.0)
        113         return ''
        114     params = sig.parameters.items()
    --> 115     jsx_params = [fmt_sig_param(p) for _, p in params]
        116 else:
        117     jsx_params = [f'<SigArg name="{spoofstr}" />']
  
  
    File ~/mambaforge/envs/docs/lib/python3.11/site-packages/nbdoc/showdoc.py:99, in fmt_sig_param(p)
         96 prefix = f'<SigArg name="{name}" '
         98 if p.annotation != inspect._empty:
    ---> 99     prefix += f'type="{p.annotation.__name__}" '
        100 if p.default != inspect._empty:
        101     prefix += f'default="{p.default}" '
  
  
    AttributeError: 'str' object has no attribute '__name__'
  
  
  
  
  
    <nbdoc.showdoc.ShowDoc object at 0x10a0ccb10>

```

--------------------------------

### Configuring Metaflow Metadata Provider

Source: https://docs.metaflow.org/metaflow/client

Demonstrates how to fetch the current metadata provider and configure it for local or remote use. Changing the provider is a global operation affecting all subsequent client calls.

```python
from metaflow import get_metadata, metadata

# Fetch currently configured metadata provider
get_metadata()

# Configure Client to use local metadata provider globally
metadata('/Users/bob/metaflow')

# Configure Client to use remote metadata provider globally
metadata('https://localhost:5000/mymetaflowservice')

```

--------------------------------

### MetaflowCard Render Runtime Example

Source: https://docs.metaflow.org/api/cards

This snippet shows the `render_runtime` method of the MetaflowCard class, which is used for rendering cards during runtime.

```python
#### MetaflowCard.render_runtime

```

--------------------------------

### Run Flow with Config and Parameter Overrides

Source: https://docs.metaflow.org/metaflow/configuring-flows/basic-configuration

Runs the `ConfigurablePhotoFlow` locally, overriding configuration defaults with command-line arguments for both config values and parameters.

```bash
python photo_config.py \
   --config-value config '{"id": 1, "size": 1, "style": {"grayscale": false, "blur": 0}}' \
   run --id 50 --size 600  

```

--------------------------------

### View Metaflow Card

Source: https://docs.metaflow.org/metaflow/configuring-flows/custom-parsers

Command to view the generated card for the 'start' step of the Metaflow flow, which displays the Git information.

```bash
python git_config.py card view start

```

--------------------------------

### Instantiating Objects

Source: https://docs.metaflow.org/api/client

You can instantiate specific objects at any level of the hierarchy by providing a corresponding pathspec. Examples include instantiating Metaflow, Flow, Run, Step, Task, and DataArtifact objects.

```APIDOC
## Instantiating Objects

### Description
Instantiate a specific object at any level of the hierarchy by providing a corresponding `pathspec`.

### Methods

- `Metaflow()`
- `Flow('FlowName')`
- `Run('FlowName/RunID')`
- `Step('FlowName/RunID/StepName')`
- `Task('FlowName/RunID/StepName/TaskID')`
- `DataArtifact('FlowName/RunID/StepName/TaskID/ArtifactName')`
```

--------------------------------

### Run Tutorial Flow

Source: https://docs.metaflow.org/getting-started/tutorials/season-1-the-local-experience/episode02

Execute the Metaflow flow defined in the stats.py script. This will process the data and compute statistics.

```bash
python 02-statistics/stats.py run
```

--------------------------------

### List Prefixes in Parallel with `list_paths`

Source: https://docs.metaflow.org/scaling/data

Demonstrates parallel listing of prefixes by providing a list of starting points to `list_paths`. Useful for querying multiple branches of an S3 hierarchy simultaneously.

```python
from metaflow import S3

with S3(s3root='s3://my-bucket/savin/tmp/s3demo_list/') as s3:
    for key in s3.list_paths(['first', 'second']):
        print key.key

a
b
c
```

--------------------------------

### Get Trigger Information

Source: https://docs.metaflow.org/api/current

Retrieve a `MetaflowTrigger` object if the current run was initiated by an event. This allows inspection of the triggering event.

```python
current.trigger
```

--------------------------------

### Using TOML for Configuration

Source: https://docs.metaflow.org/metaflow/configuring-flows/parsing-configs

This snippet shows how to configure a Metaflow flow to load settings from a TOML file using `tomllib.loads` as the parser. Ensure you are using Python 3.11 or newer.

```python
import pprint
from metaflow import FlowSpec, step, Config, resources

class TomlConfigFlow(FlowSpec):
    config = Config("config", default="myconfig.toml", parser="tomllib.loads")

    @resources(cpu=config.resources.cpu)
    @step
    def start(self):
        print("Config loaded:")
        pprint.pp(self.config)
        self.next(self.end)

    @step
    def end(self):
        pass


if __name__ == "__main__":
    TomlConfigFlow()

```

--------------------------------

### Retrieve Branch Name

Source: https://docs.metaflow.org/api/current

Get the name of the current branch, typically set using the `--branch=X` flag during deployment.

```python
current.branch_name
```

--------------------------------

### Using ExecutingRun as a Context Manager

Source: https://docs.metaflow.org/api/runner

Demonstrates the recommended pattern for using ExecutingRun with a Runner. This pattern allows you to manage the lifecycle of a Metaflow run within a Python context.

```python
with Runner(...).run() as running:
    ...
```

--------------------------------

### Graceful Spot Instance Interruption Handling

Source: https://docs.metaflow.org/scaling/remote-tasks/spot-instances

Implement a long-running task that checks for spot instance termination notices and performs cleanup. This example uses the `@kubernetes` decorator to schedule tasks on spot instances and `current.spot_termination_notice` to detect interruptions.

```python
import os  
import time  
from metaflow import FlowSpec, step, kubernetes, current  
  
  
class SpotInterruptionFlow(FlowSpec):  
  
    @kubernetes(node_selector={"node_type": "spot"})  
    @step  
    def start(self):  
        print("Starting long-running task...")  
        start_time = time.time()  
  
        try:  
            # Run for 5 minutes or until interrupted  
            while time.time() - start_time < 300:  # 5 minutes  
                if os.path.exists(current.spot_termination_notice):  
                    print("SPOT TERMINATION DETECTED...")  
                    break  
  
                seconds_elapsed = (time.time() - start_time)  
                print(f"Task running for {seconds_elapsed:.1f} seconds...")  
  
                time.sleep(1)  # Status update every 1 second  
        except KeyboardInterrupt:  
            print("Task was interrupted!")  
  
        print("Task completed or interrupted.")  
        self.next(self.end)  
  
    @step  
    def end(self):  
        pass  
  
if __name__ == '__main__':  
    SpotInterruptionFlow()  

```

--------------------------------

### MetaflowCard Refresh Method Example

Source: https://docs.metaflow.org/api/cards

This snippet shows the `refresh` method of the MetaflowCard class, which is used for refreshing the card's content.

```python
#### MetaflowCard.refresh

```

--------------------------------

### SQL Script for Populating Table

Source: https://docs.metaflow.org/scaling/dependencies/project-structure

This SQL script defines a table and inserts sample data. It is intended to be packaged with a Metaflow flow.

```sql
CREATE TABLE IF NOT EXISTS movie(title, year, score)  
INSERT INTO movie VALUES ('Moana', 2016, 7.6), ('Scream', 2022, 6.3)  

```

--------------------------------

### Equivalent @trigger with Dictionary Parameter Mapping

Source: https://docs.metaflow.org/api/flow-decorators/trigger

This example shows the dictionary-based equivalent of the list-based parameter mapping, clarifying how 'common_name' is mapped to itself.

```python
from metaflow import trigger

@trigger(event={'name':'foo', 'parameters':{'common_name': 'common_name', 'flow_param': 'event_field'}})
```
Tokens:
Raw