### Quick Start Development Setup Source: https://github.com/netflix/metaflow/blob/master/CONTRIBUTING.md Follow these steps to set up the Metaflow development environment, install dependencies, and configure pre-commit hooks. ```bash # 1. Fork and clone the repository git clone https://github.com/YOUR_USERNAME/metaflow.git cd metaflow # 2. Install in editable mode pip install -e . # 3. Set up pre-commit hooks (formats code automatically) pip install pre-commit pre-commit install # 4. Make your changes and add tests! # 5. Run tests cd test/unit python -m pytest -v ``` -------------------------------- ### Run Local Development Server Source: https://github.com/netflix/metaflow/blob/master/metaflow/plugins/cards/ui/README.md Starts a local server to preview a single card using example data. Changes to Svelte/CSS files will be reflected live. ```bash npm run dev ``` -------------------------------- ### Basic Development Setup Source: https://github.com/netflix/metaflow/blob/master/CONTRIBUTING.md Installs Metaflow in editable mode and sets up pre-commit hooks for local development. Assumes Metaflow repository has been cloned. ```bash cd metaflow pip install -e . pip install pre-commit && pre-commit install ``` -------------------------------- ### Install Metaflow R Package Source: https://github.com/netflix/metaflow/blob/master/R/README.md Install the Metaflow R package from GitHub and then install the Metaflow environment. This is the initial setup step for using Metaflow in R. ```R devtools::install_github("Netflix/metaflow", subdir="R") metaflow::install_metaflow() ``` -------------------------------- ### Install Metaflow and Pytest in a Virtual Environment Source: https://github.com/netflix/metaflow/blob/master/test/ux/README.md Create a virtual environment and install Metaflow and pytest for testing. Ensure Metaflow is installed in editable mode. ```bash python3 -m venv /tmp/metaflow-test-venv /tmp/metaflow-test-venv/bin/pip install -e /path/to/metaflow pytest ``` -------------------------------- ### Install Notebook Source: https://github.com/netflix/metaflow/blob/master/metaflow/tutorials/01-playlist/README.md Install the Jupyter Notebook environment to interact with Metaflow. ```bash python -m pip install notebook ``` -------------------------------- ### Install Dependencies Source: https://github.com/netflix/metaflow/blob/master/metaflow/plugins/cards/ui/README.md Installs the necessary Node.js dependencies for local development and building the UI. ```bash npm install ``` -------------------------------- ### Start Services for SFN + Batch Backend Source: https://github.com/netflix/metaflow/blob/master/devtools/README.md Starts the required services for the SFN + Batch backend using 'make up' with a service override. Includes MinIO, PostgreSQL, metadata-service, localbatch, ddb-local, and sfn-local. ```bash SERVICES_OVERRIDE=minio,postgresql,metadata-service,localbatch,ddb-local,sfn-local make up ``` -------------------------------- ### Client-Server Connection Steps Source: https://github.com/netflix/metaflow/blob/master/docs/Environment escape.md Establishes the client-server connection by determining socket paths, starting the server, loading overrides, and querying for proxied objects. The server then loads its overrides and waits for requests. ```text - [Client] Determines a path to the UNIX socket to use (a combination of PID and emulated module) - [Client] Start the server - [Client] Read the local overrides - [Client] Wait for the socket to be up and connect to it - [Client] Query the server asking for all the objects that will be proxied. Only the server knows because the file defining the whitelisted objects includes the library that the client cannot load. - [Server] Read the server overrides as well as the whitelisted information. This process is somewhat involved due to the way we handle exceptions (allowing for hierarchy information in exceptions). - [Server] Setting up handlers - [Server] Opening the UNIX socket and waiting for a connection - [Server] Once a connection is established, waiting for request. The server is single threaded by design (it is an extension of the client which is single threaded). ``` -------------------------------- ### Start Services for Airflow Backend Source: https://github.com/netflix/metaflow/blob/master/devtools/README.md Starts the required services for the Airflow backend using 'make up' with a service override. Includes MinIO, PostgreSQL, metadata-service, and Airflow. ```bash SERVICES_OVERRIDE=minio,postgresql,metadata-service,airflow make up ``` -------------------------------- ### Launch Jupyter Notebook Source: https://github.com/netflix/metaflow/blob/master/metaflow/tutorials/01-playlist/README.md Start a Jupyter Notebook server to explore the tutorial's notebook file. ```bash jupyter-notebook 01-playlist/playlist.ipynb ``` -------------------------------- ### Install and Configure Pre-commit Hooks Source: https://github.com/netflix/metaflow/blob/master/CONTRIBUTING.md Install pre-commit to automatically format Python code upon committing. Run this command once to set up the hooks. ```bash pip install pre-commit pre-commit install ``` -------------------------------- ### Install Notebook and Matplotlib Source: https://github.com/netflix/metaflow/blob/master/metaflow/tutorials/02-statistics/README.md Install the necessary libraries for running Jupyter Notebooks and plotting with Matplotlib. ```bash python -m pip install notebook ``` ```bash python -m pip install matplotlib ``` -------------------------------- ### Import Metaflow Client and Get Metadata Source: https://github.com/netflix/metaflow/blob/master/metaflow/tutorials/01-playlist/playlist.ipynb Imports the necessary Metaflow client components and prints the current metadata provider. Ensure Metaflow is installed and configured. ```python from metaflow import Flow, get_metadata print("Current metadata provider: %s" % get_metadata()) ``` -------------------------------- ### Start Specific Metaflow Devstack Services Source: https://github.com/netflix/metaflow/blob/master/devtools/README.md Customize the Metaflow Devstack startup by specifying a subset of services using the 'SERVICES_OVERRIDE' variable with 'make up'. For example, to start only 'localbatch' and 'minio'. ```bash SERVICES_OVERRIDE=localbatch,minio make up ``` -------------------------------- ### Install Metaflow Stubs Source: https://github.com/netflix/metaflow/blob/master/stubs/README.md Install the Metaflow Stubs package using pip. This provides type hints for Metaflow. ```sh pip install metaflow-stubs ``` -------------------------------- ### Start Metaflow Devstack Interactively Source: https://github.com/netflix/metaflow/blob/master/devtools/README.md Navigate to the devtools directory and run 'make up' to interactively select services and start the Metaflow Devstack. This command initiates the development environment with your chosen components. ```bash cd devtools make up ``` -------------------------------- ### Start Services for Argo Workflows Backend Source: https://github.com/netflix/metaflow/blob/master/devtools/README.md Starts the required services for the Argo Workflows backend using 'make up' with a service override. Ensure MinIO, PostgreSQL, metadata-service, and Argo Workflows are available. ```bash SERVICES_OVERRIDE=minio,postgresql,metadata-service,argo-workflows make up ``` -------------------------------- ### Run Playlist Redux Tutorial Source: https://github.com/netflix/metaflow/blob/master/metaflow/tutorials/03-playlist-redux/README.md Execute the playlist generator script to start the tutorial. This command initiates the process of generating movie recommendations based on box office data. ```bash cd metaflow-tutorials ``` ```bash python 03-playlist-redux/playlist.py show ``` ```bash python 03-playlist-redux/playlist.py run ``` -------------------------------- ### Metaflow Run Card Viewer JavaScript Setup Source: https://github.com/netflix/metaflow/blob/master/metaflow/plugins/cards/card_viewer/viewer.html Initializes constants, state, and event listeners for the card viewer, including handling selector changes. ```javascript (() => { // setup constants let DATA_UPDATE_POLL_INTERVAL = 5000; let RUN_UPDATE_POLL_INTERVAL = 10000; // setup state let currentCardsLength = 0; let currentRunId = ""; let selectedCard = ""; // setup selector const selector = document.getElementById("selector"); selector?.addEventListener("change", (event) => { selectedCard = event.target.value; handleLoadCard(event.target.name, event.target.value); }); ``` -------------------------------- ### Run Devstack Locally Source: https://github.com/netflix/metaflow/blob/master/devtools/README.md Starts the development stack locally using the same script as CI. Specify services like minio, postgresql, or metadata-service. ```bash cd devtools SERVICES=minio,postgresql,metadata-service ci/start-devstack.sh ``` -------------------------------- ### Install Metaflow using pip Source: https://github.com/netflix/metaflow/blob/master/README.md Install Metaflow in your Python environment using pip from PyPI. This is the standard method for adding Metaflow to your project. ```sh pip install metaflow ``` -------------------------------- ### Install Metaflow Dev Dependencies Source: https://github.com/netflix/metaflow/blob/master/devtools/README.md Installs Metaflow in development mode along with necessary testing dependencies like pytest and omegaconf. ```bash pip install -e ".[dev]" pytest kubernetes omegaconf ``` -------------------------------- ### Import Metaflow Client and Get Metadata Provider Source: https://github.com/netflix/metaflow/blob/master/metaflow/tutorials/07-worldview/worldview.ipynb Import necessary components from the Metaflow library and print the current metadata provider. ```python from metaflow import Metaflow, Flow, get_metadata, namespace print("Current metadata provider: %s" % get_metadata()) ``` -------------------------------- ### Start All Metaflow Devstack Services Source: https://github.com/netflix/metaflow/blob/master/devtools/README.md To launch all available services in the Metaflow Devstack without interactive selection, use the 'make all-up' command. This ensures the entire development environment is running. ```bash make all-up ``` -------------------------------- ### Enter Metaflow Development Shell Source: https://github.com/netflix/metaflow/blob/master/CONTRIBUTING.md Starts an interactive shell within the development environment, automatically configuring flows to use local MinIO. ```bash metaflow-dev shell ``` -------------------------------- ### Install Metaflow using conda Source: https://github.com/netflix/metaflow/blob/master/README.md Install Metaflow using conda from the conda-forge channel. This is an alternative installation method for users who prefer conda environments. ```sh conda install -c conda-forge metaflow ``` -------------------------------- ### Create a Custom Card Using Metaflow Components Source: https://github.com/netflix/metaflow/blob/master/docs/cards.md This example demonstrates creating a custom card that reuses Metaflow's `BlankCard`, `Artifact`, and `Table` components. The `render` method is overridden to include a table of artifacts. ```python from metaflow.cards import BlankCard from metaflow.cards import Artifact,Table class MyCustomCard(BlankCard): type = 'my_custom_card' def render(self, task): art_com = [ Table( [[Artifact(k.data,k.id)] for k in task] ).render() ] return super().render(task,components=[art_com]) CARDS = [MyCustomCard] ``` -------------------------------- ### Import Metaflow Client and Get Metadata Provider Source: https://github.com/netflix/metaflow/blob/master/metaflow/tutorials/05-hello-cloud/hello-cloud.ipynb Import necessary components from the Metaflow library and print the current metadata provider. This is a foundational step for interacting with Metaflow. ```python from metaflow import Flow, get_metadata, namespace print("Current metadata provider: %s" % get_metadata()) ``` -------------------------------- ### Example Good Commit Message Source: https://github.com/netflix/metaflow/blob/master/CONTRIBUTING.md Illustrates a well-structured commit message, including a concise summary, detailed explanation, and issue reference. ```git Fix symlink traversal edge case in packaging The symlink detection was happening globally across branches, causing some modules to be skipped when encountered through different paths. Now tracks symlinks per-branch to handle this correctly. Fixes #2751 ``` -------------------------------- ### Numpydoc Style Function Docstring Example Source: https://github.com/netflix/metaflow/blob/master/CONTRIBUTING.md Illustrates the numpydoc format for Python docstrings, including parameters, return values, and examples. Adhere to this style for function documentation. ```python def example_function(param1, param2): """ Brief description of the function. Parameters ---------- param1 : str Description of param1 param2 : int Description of param2 Returns ------- bool Description of return value Examples -------- >>> example_function("test", 42) True """ ``` -------------------------------- ### Custom MetaflowCardComponent Implementation Source: https://github.com/netflix/metaflow/blob/master/docs/cards.md Demonstrates how to create custom card components by extending MetaflowCardComponent and implementing the render method. This example shows how to define Title and Text components and use them within a CustomCard. ```python from metaflow.cards import MetaflowCard,MetaflowCardComponent class Title(MetaflowCardComponent): def __init__(self,text): self._text = text def render(self): return "

%s

"%self._text class Text(MetaflowCardComponent): def __init__(self,text): self._text = text def render(self): return "

%s

"%self._text class CustomCard(MetaflowCard): type = "custom_card" HTML = "{data}" def __init__(self, options={"no_header": True}, graph=None, components=[], flow=None, **kwargs): super().__init__() self._no_header = True self._graph = graph if "no_header" in options: self._no_header = options["no_header"] def render(self, task): pt = self._get_mustache() data = '\n'.join([ Title("Title 1").render(), Text("some text comes here").render(), Title("Title 2").render(), Text("some text comes here again").render(), ]) data = dict( data = data ) html_template = self.HTML return pt.render(html_template,data) ``` -------------------------------- ### Generate and Display Gantt Chart for Flow Steps Source: https://github.com/netflix/metaflow/blob/master/metaflow/tutorials/08-autopilot/autopilot.ipynb Creates a Gantt chart visualizing the steps of a Metaflow run, showing their start and finish times. Requires the run object to be successfully loaded. ```python step_data = [dict(Task=step.id, Start=step.created_at, Finish=step.finished_at, Status='timeline') for step in run] fig = ff.create_gantt(step_data, title="Steps of %s/%s" % (run.parent.id, run.id), colors=COLORS, index_col='Status', show_colorbar=True) fig.show() ``` -------------------------------- ### Implement a Custom Metaflow Card Source: https://github.com/netflix/metaflow/blob/master/docs/cards.md Example of creating a custom Metaflow card by subclassing `MetaflowCard` and implementing the `render` function. This custom card uses a Mustache template for rendering HTML. ```python from metaflow.cards import MetaflowCard # path to the custom html file which is a `mustache` template. PATH_TO_CUSTOM_HTML = 'myhtml.html' class CustomCard(MetaflowCard): type = "custom_card" def __init__(self, options={"no_header": True}, graph=None, components=[], flow=None, **kwargs): super().__init__() self._no_header = True self._graph = graph if "no_header" in options: self._no_header = options["no_header"] def render(self, task): pt = self._get_mustache() data = dict( graph = self._graph, header = self._no_header ) html_template = None with open(PATH_TO_CUSTOM_HTML) as f: html_template = f.read() return pt.render(html_template,data) ``` -------------------------------- ### Example Usage of Environment Escape Source: https://github.com/netflix/metaflow/blob/master/docs/Environment escape.md Illustrates how to use a package like 'data_accessor' that is not available in the primary Conda environment. Stub objects are used on the client side to represent server-side objects, and method calls are forwarded for execution. ```python import data_accessor as da sql = 'select * from %s order by int' % name.replace('/', '.') job = da.SqlJob()\ .script(sql)\ .headers()\ .execute() job.wait() job.raise_for_status() result = job.pandas().to_dict() ``` -------------------------------- ### Navigate to Tutorial Directory Source: https://github.com/netflix/metaflow/blob/master/R/inst/tutorials/06-worldview/README.md Change the current directory to the 06-worldview tutorial folder. ```bash cd tutorials/06-worldview/ ``` -------------------------------- ### Show Playlist Tutorial Source: https://github.com/netflix/metaflow/blob/master/metaflow/tutorials/01-playlist/README.md Execute the playlist script to display its help information. ```bash python 01-playlist/playlist.py show ``` -------------------------------- ### Example Unit Test for Symlink Traversal Source: https://github.com/netflix/metaflow/blob/master/CONTRIBUTING.md This is an example of a unit test that checks symlink traversal logic, specifically addressing circular references. It includes setup, implementation, and assertion steps. ```python def test_symlink_traversal_handles_circular_references(): """Test that symlink detection works correctly when modules are encountered through different paths.""" # Test case for issue #2751 # Setup: Create circular symlink structure # ... test implementation # Assert: Verify all modules are included correctly ``` -------------------------------- ### Get Metaflow Version Source: https://github.com/netflix/metaflow/blob/master/CONTRIBUTING.md Use this command to retrieve the installed Metaflow version for bug reports. ```python import metaflow print(metaflow.__version__) ``` -------------------------------- ### Navigate to Tutorial Directory Source: https://github.com/netflix/metaflow/blob/master/metaflow/tutorials/02-statistics/README.md Change the current directory to the metaflow-tutorials folder to access the episode files. ```bash cd metaflow-tutorials ``` -------------------------------- ### Run Playlist Tutorial Source: https://github.com/netflix/metaflow/blob/master/metaflow/tutorials/01-playlist/README.md Run the Metaflow playlist tutorial flow. ```bash python 01-playlist/playlist.py run ``` -------------------------------- ### Run Metaflow Tutorial Source: https://github.com/netflix/metaflow/blob/master/R/inst/tutorials/00-helloworld/README.md Commands to navigate to the tutorial directory and execute the R script for showing and running the Metaflow flow. ```bash cd tutorials/00-helloworld ``` ```bash Rscript helloworld.R show ``` ```bash Rscript helloworld.R run ``` -------------------------------- ### Execute the Metaflow Playlist Flow Source: https://github.com/netflix/metaflow/blob/master/R/inst/tutorials/01-playlist/README.md Run the Metaflow playlist tutorial. Use 'show' to display the flow, 'run' to execute it with default parameters, or 'run --genre ' to specify a movie genre. ```bash cd tutorials/01-playlist/ ``` ```bash Rscript playlist.R show ``` ```bash Rscript playlist.R run ``` ```bash Rscript playlist.R run --genre comedy ``` -------------------------------- ### Run Playlist Tutorial with Genre Parameter Source: https://github.com/netflix/metaflow/blob/master/metaflow/tutorials/01-playlist/README.md Run the Metaflow playlist tutorial flow, specifying a 'comedy' genre. ```bash python 01-playlist/playlist.py run --genre comedy ``` -------------------------------- ### Pull Metaflow Tutorials Source: https://github.com/netflix/metaflow/blob/master/R/README.md Access Metaflow tutorials to learn how to use the package. This command downloads available tutorial resources. ```R metaflow::pull_tutorials() ``` -------------------------------- ### View or Get Card using Metaflow CLI Source: https://github.com/netflix/metaflow/blob/master/docs/cards.md CLI commands to view a card in the browser or get its HTML content. Supports specifying a hash, card type, and retrieving cards for resumed tasks. ```sh # python myflow.py card view --hash --type python myflow.py card view 100/stepname/1000 --hash ads34 --type default --follow-resumed ``` -------------------------------- ### Get Python Version Source: https://github.com/netflix/metaflow/blob/master/CONTRIBUTING.md Use this command to check your Python version for bug reports. ```bash python --version ``` -------------------------------- ### Configure Metaflow Metadata Provider Source: https://github.com/netflix/metaflow/blob/master/R/inst/tutorials/03-playlist-redux/README.md Set up the Metaflow default metadata provider to 'local'. This is a prerequisite for running the episode. ```bash $mkdir -p /path/to/home/.metaflow $export METAFLOW_DEFAULT_METADATA=local ``` -------------------------------- ### Run Unit Tests Source: https://github.com/netflix/metaflow/blob/master/CONTRIBUTING.md Execute all unit tests locally. Navigate to the 'test/unit' directory before running. ```bash cd test/unit python -m pytest -v ``` -------------------------------- ### Load a Scheduled Run by ID Source: https://github.com/netflix/metaflow/blob/master/metaflow/tutorials/08-autopilot/autopilot.ipynb Loads a specific run of a Metaflow flow using its run ID. Handles potential KeyErrors if the run has not yet started or the ID is incorrect. ```python # When you triggered your flow on Argo Workflows using `argo-workflows trigger`, you would have # seen an output similar to - # # ... # Workflow MovieStatsFlow triggered on Argo Workflows (run-id argo-moviestatsflow-68z2h). # ... # # Paste the run-id below (run_id = 'argo-moviestatsflow-68z2h'). run_id = 'argo-moviestatsflow-68z2h' flow_name = 'MovieStatsFlow' try: run = Flow(flow_name)[run_id] except KeyError: print('The run %s for flow %s might not have started yet. \nThis can happen ' 'when the underlying Kubernetes resources are not immediately available. ' '\nPlease wait for a few moments before trying again or check the run_id ' 'for any typos.' % (run_id, flow_name)) ``` -------------------------------- ### Open Jupyter Notebook Source: https://github.com/netflix/metaflow/blob/master/metaflow/tutorials/02-statistics/README.md Launch a Jupyter Notebook server to interactively explore the results and create plots. ```bash jupyter-notebook 02-statistics/stats.ipynb ``` -------------------------------- ### Open R Markdown File Source: https://github.com/netflix/metaflow/blob/master/R/inst/tutorials/06-worldview/README.md Open the worldview.Rmd file in RStudio on your local computer to begin the tutorial. ```bash worldview.Rmd ``` -------------------------------- ### Run All Tests with Default Contexts Source: https://github.com/netflix/metaflow/blob/master/test/README.md Execute all valid combinations of contexts, tests, graphs, and checkers. Suitable for CI systems. ```bash cd metaflow/test/core PYTHONPATH=`pwd`/../../ python run_tests.py --debug --contexts dev-local ``` -------------------------------- ### Run all inheritance tests Source: https://github.com/netflix/metaflow/blob/master/test/unit/inheritance/README.md Execute all tests within the inheritance directory. Use this to run the full suite of inheritance-related tests. ```bash pytest unit/inheritance/ ``` -------------------------------- ### Tear Down Development Environment Source: https://github.com/netflix/metaflow/blob/master/devtools/README.md Stops Tilt, deletes the Minikube cluster, and removes generated files under .devtools/. ```bash make down ``` -------------------------------- ### Run Specific Unit Test Source: https://github.com/netflix/metaflow/blob/master/CONTRIBUTING.md Execute a single unit test file. Replace 'test_your_feature.py' with the target file. ```bash python -m pytest test/unit/test_your_feature.py -v ``` -------------------------------- ### Configure Metaflow Metadata Provider Source: https://github.com/netflix/metaflow/blob/master/R/inst/tutorials/02-statistics/README.md Sets the Metaflow metadata provider to local. This is a prerequisite for running the episode. ```bash mkdir -p /path/to/home/.metaflow export METAFLOW_DEFAULT_METADATA=local ``` -------------------------------- ### Run Integration Tests Source: https://github.com/netflix/metaflow/blob/master/CONTRIBUTING.md Execute integration tests. Ensure the PYTHONPATH is set correctly to include the parent directory. ```bash cd test/core PYTHONPATH=`pwd`/../../ python run_tests.py --debug --contexts dev-local ``` -------------------------------- ### Open Metaflow Development Shell Source: https://github.com/netflix/metaflow/blob/master/devtools/README.md Waits for the local stack to be ready and opens a sub-shell with Metaflow configured for the local environment. Ensure necessary environment variables like METAFLOW_HOME and METAFLOW_PROFILE are set. ```bash METAFLOW_HOME=.devtools METAFLOW_PROFILE=local # loads .devtools/config_local.json AWS_CONFIG_FILE=.devtools/aws_config # MinIO credentials (if minio is running) ``` -------------------------------- ### Show Playlist Generator Configuration Source: https://github.com/netflix/metaflow/blob/master/metaflow/tutorials/04-playlist-plus/README.md Use this command to display the configuration of the playlist generator script, utilizing conda for environment management. ```bash python 04-playlist-plus/playlist.py --environment=conda show ``` -------------------------------- ### Build UI for Metaflow Source: https://github.com/netflix/metaflow/blob/master/metaflow/plugins/cards/ui/README.md Builds the Javascript and CSS files that Metaflow will use to render cards. Includes a linting step to ensure code quality. ```bash npm run lint npm run build ``` -------------------------------- ### Run PlaylistReduxFlow with Rscript Source: https://github.com/netflix/metaflow/blob/master/R/inst/tutorials/03-playlist-redux/README.md Execute the playlist R script to either show the flow or run it. Ensure you are in the 'tutorials/03-playlist-redux' directory. ```bash cd tutorials/03-playlist-redux ``` ```bash Rscript playlist.R show ``` ```bash Rscript playlist.R run ``` -------------------------------- ### Import Metaflow Client and Utilities Source: https://github.com/netflix/metaflow/blob/master/metaflow/tutorials/06-statistics-redux/stats.ipynb Imports necessary components from the metaflow library and matplotlib for plotting. It also prints the current metadata provider. ```python from metaflow import Flow, get_metadata, namespace import matplotlib.pyplot as plt print("Current metadata provider: %s" % get_metadata()) ``` -------------------------------- ### Run Playlist Generator with Hint Source: https://github.com/netflix/metaflow/blob/master/metaflow/tutorials/04-playlist-plus/README.md Run the playlist generator script, providing a 'hint' parameter to suggest a bonus movie. This utilizes conda for environment management and string edit distance for movie selection. ```bash python 04-playlist-plus/playlist.py --environment=conda run --hint "Data Science Strikes Back" ``` -------------------------------- ### Run tests with verbose output Source: https://github.com/netflix/metaflow/blob/master/test/unit/inheritance/README.md Execute inheritance tests and display verbose output, showing each test case as it runs. This is helpful for monitoring test progress and identifying failures. ```bash pytest unit/inheritance/ -v ``` -------------------------------- ### Access Metaflow Cards in Notebooks Source: https://github.com/netflix/metaflow/blob/master/docs/cards.md Python code demonstrating how to access Metaflow cards programmatically within a notebook environment using the `Task` and `get_cards` functions. ```python from metaflow import Task from metaflow.cards import get_cards taskspec = 'MyFlow/1000/stepname/100' task = Task(taskspec) card_iterator = get_cards(task) # you can even call `get_cards(taskspec)` # view card in browser card = card_iterator[0] card.view() # Get HTML of card html = card_iterator[0].get() ``` -------------------------------- ### Run PlaylistReduxFlow with RStudio Source: https://github.com/netflix/metaflow/blob/master/R/inst/tutorials/03-playlist-redux/README.md Alternatively, if using RStudio, you can execute the 'playlist.R' script directly using source(). ```r source("playlist.R") ``` -------------------------------- ### Run Specific Integration Test Source: https://github.com/netflix/metaflow/blob/master/CONTRIBUTING.md Execute a specific integration test by name. Use the '--tests' flag followed by the test name. ```bash PYTHONPATH=`pwd`/../../ python run_tests.py --debug --contexts dev-local --tests YourTestName ``` -------------------------------- ### Run a Single Test Case Source: https://github.com/netflix/metaflow/blob/master/test/README.md Execute a single test case by specifying the context, graph, and test name. Useful for developing new tests. ```bash cd metaflow/test/core PYTHONPATH=`pwd`/../../ python run_tests.py --debug --contexts dev-local --graphs single-linear-step --tests BasicArtifactTest ``` -------------------------------- ### Import Metaflow Client and Plotting Libraries Source: https://github.com/netflix/metaflow/blob/master/metaflow/tutorials/08-autopilot/autopilot.ipynb Import necessary modules from Metaflow and Plotly for flow management and visualization. Sets the metadata provider and namespace. ```python from metaflow import Flow, get_metadata, namespace import plotly.figure_factory as ff print("Current metadata provider: %s" % get_metadata()) namespace(None) COLORS = {'timeline': 'rgb(0, 255, 0)'} ``` -------------------------------- ### Run UX Tests in Deployer Mode Source: https://github.com/netflix/metaflow/blob/master/test/ux/README.md Execute UX tests in deployer mode against a remote scheduler like Argo Workflows. Replace with your actual cluster name. ```bash pytest test/ux/ --scheduler-type argo-workflows --cluster ``` -------------------------------- ### Manually Format Python Code with Black Source: https://github.com/netflix/metaflow/blob/master/CONTRIBUTING.md Execute Black to format all Python files in the current directory and its subdirectories. Use this if automatic formatting is insufficient. ```bash black . ``` -------------------------------- ### Open Metaflow Shell Source: https://github.com/netflix/metaflow/blob/master/devtools/README.md Use 'make shell' to open a shell environment pre-configured with Metaflow settings. This is useful for running Metaflow commands within the devstack context. ```bash make shell ``` -------------------------------- ### Run a Metaflow Flow with Cards Source: https://github.com/netflix/metaflow/blob/master/metaflow/plugins/cards/ui/README.md Executes a Metaflow flow that includes card generation. The `--with card` flag enables card creation. ```bash python MYCARD.py run --with card ``` -------------------------------- ### Create AWS Step Functions Flow Source: https://github.com/netflix/metaflow/blob/master/R/inst/tutorials/07-autopilot/README.md Use this command to schedule your Metaflow flow on AWS Step Functions. Ensure you are in the correct tutorial directory. ```bash cd tutorials/02-statistics/ Rscript stats.R --package-suffixes=.R,.csv step-functions create --max-workers 4 ``` -------------------------------- ### Create Argo Workflow Source: https://github.com/netflix/metaflow/blob/master/metaflow/tutorials/08-autopilot/README.md Use this command to create an Argo Workflow for your Metaflow script. Specify the script path and the '--with kubernetes argo-workflows create' option to enable cloud scheduling. Adjust '--max-workers' as needed for your compute resources. ```bash 02-statistics/stats.py --with kubernetes argo-workflows create --max-workers 4 ``` -------------------------------- ### Run Data/S3 Tests Source: https://github.com/netflix/metaflow/blob/master/CONTRIBUTING.md Execute tests related to data handling and S3 interactions. This command includes verbose output and skips benchmarking. ```bash cd test/data/ PYTHONPATH=`pwd`/../../ python3 -m pytest -x -s -v --benchmark-skip ``` -------------------------------- ### Test with Local S3 (MinIO) Source: https://github.com/netflix/metaflow/blob/master/CONTRIBUTING.md Runs S3-dependent tests using the local MinIO instance. Configure the 'METAFLOW_S3_TEST_ROOT' environment variable to point to your MinIO bucket. ```bash cd test/data/s3 METAFLOW_S3_TEST_ROOT=s3://metaflow-test/test python -m pytest -v ``` -------------------------------- ### Run UX Tests with Pytest Source: https://github.com/netflix/metaflow/blob/master/test/ux/README.md Execute specific UX test files using pytest from the repository root. This command assumes the virtual environment is activated. ```bash /tmp/metaflow-test-venv/bin/pytest test/ux/core/test_basic.py -v /tmp/metaflow-test-venv/bin/pytest test/ux/core/test_config.py -v ```