### Quick Start Development Setup Source: https://github.com/netflix/metaflow/blob/master/CONTRIBUTING.md Follow these steps to set up the Metaflow development environment, install dependencies, and configure pre-commit hooks. ```bash # 1. Fork and clone the repository git clone https://github.com/YOUR_USERNAME/metaflow.git cd metaflow # 2. Install in editable mode pip install -e . # 3. Set up pre-commit hooks (formats code automatically) pip install pre-commit pre-commit install # 4. Make your changes and add tests! # 5. Run tests cd test/unit python -m pytest -v ``` -------------------------------- ### Run Local Development Server Source: https://github.com/netflix/metaflow/blob/master/metaflow/plugins/cards/ui/README.md Starts a local server to preview a single card using example data. Changes to Svelte/CSS files will be reflected live. ```bash npm run dev ``` -------------------------------- ### Basic Development Setup Source: https://github.com/netflix/metaflow/blob/master/CONTRIBUTING.md Installs Metaflow in editable mode and sets up pre-commit hooks for local development. Assumes Metaflow repository has been cloned. ```bash cd metaflow pip install -e . pip install pre-commit && pre-commit install ``` -------------------------------- ### Install Metaflow R Package Source: https://github.com/netflix/metaflow/blob/master/R/README.md Install the Metaflow R package from GitHub and then install the Metaflow environment. This is the initial setup step for using Metaflow in R. ```R devtools::install_github("Netflix/metaflow", subdir="R") metaflow::install_metaflow() ``` -------------------------------- ### Install Metaflow and Pytest in a Virtual Environment Source: https://github.com/netflix/metaflow/blob/master/test/ux/README.md Create a virtual environment and install Metaflow and pytest for testing. Ensure Metaflow is installed in editable mode. ```bash python3 -m venv /tmp/metaflow-test-venv /tmp/metaflow-test-venv/bin/pip install -e /path/to/metaflow pytest ``` -------------------------------- ### Install Notebook Source: https://github.com/netflix/metaflow/blob/master/metaflow/tutorials/01-playlist/README.md Install the Jupyter Notebook environment to interact with Metaflow. ```bash python -m pip install notebook ``` -------------------------------- ### Install Dependencies Source: https://github.com/netflix/metaflow/blob/master/metaflow/plugins/cards/ui/README.md Installs the necessary Node.js dependencies for local development and building the UI. ```bash npm install ``` -------------------------------- ### Start Services for SFN + Batch Backend Source: https://github.com/netflix/metaflow/blob/master/devtools/README.md Starts the required services for the SFN + Batch backend using 'make up' with a service override. Includes MinIO, PostgreSQL, metadata-service, localbatch, ddb-local, and sfn-local. ```bash SERVICES_OVERRIDE=minio,postgresql,metadata-service,localbatch,ddb-local,sfn-local make up ``` -------------------------------- ### Client-Server Connection Steps Source: https://github.com/netflix/metaflow/blob/master/docs/Environment escape.md Establishes the client-server connection by determining socket paths, starting the server, loading overrides, and querying for proxied objects. The server then loads its overrides and waits for requests. ```text - [Client] Determines a path to the UNIX socket to use (a combination of PID and emulated module) - [Client] Start the server - [Client] Read the local overrides - [Client] Wait for the socket to be up and connect to it - [Client] Query the server asking for all the objects that will be proxied. Only the server knows because the file defining the whitelisted objects includes the library that the client cannot load. - [Server] Read the server overrides as well as the whitelisted information. This process is somewhat involved due to the way we handle exceptions (allowing for hierarchy information in exceptions). - [Server] Setting up handlers - [Server] Opening the UNIX socket and waiting for a connection - [Server] Once a connection is established, waiting for request. The server is single threaded by design (it is an extension of the client which is single threaded). ``` -------------------------------- ### Start Services for Airflow Backend Source: https://github.com/netflix/metaflow/blob/master/devtools/README.md Starts the required services for the Airflow backend using 'make up' with a service override. Includes MinIO, PostgreSQL, metadata-service, and Airflow. ```bash SERVICES_OVERRIDE=minio,postgresql,metadata-service,airflow make up ``` -------------------------------- ### Launch Jupyter Notebook Source: https://github.com/netflix/metaflow/blob/master/metaflow/tutorials/01-playlist/README.md Start a Jupyter Notebook server to explore the tutorial's notebook file. ```bash jupyter-notebook 01-playlist/playlist.ipynb ``` -------------------------------- ### Install and Configure Pre-commit Hooks Source: https://github.com/netflix/metaflow/blob/master/CONTRIBUTING.md Install pre-commit to automatically format Python code upon committing. Run this command once to set up the hooks. ```bash pip install pre-commit pre-commit install ``` -------------------------------- ### Install Notebook and Matplotlib Source: https://github.com/netflix/metaflow/blob/master/metaflow/tutorials/02-statistics/README.md Install the necessary libraries for running Jupyter Notebooks and plotting with Matplotlib. ```bash python -m pip install notebook ``` ```bash python -m pip install matplotlib ``` -------------------------------- ### Import Metaflow Client and Get Metadata Source: https://github.com/netflix/metaflow/blob/master/metaflow/tutorials/01-playlist/playlist.ipynb Imports the necessary Metaflow client components and prints the current metadata provider. Ensure Metaflow is installed and configured. ```python from metaflow import Flow, get_metadata print("Current metadata provider: %s" % get_metadata()) ``` -------------------------------- ### Start Specific Metaflow Devstack Services Source: https://github.com/netflix/metaflow/blob/master/devtools/README.md Customize the Metaflow Devstack startup by specifying a subset of services using the 'SERVICES_OVERRIDE' variable with 'make up'. For example, to start only 'localbatch' and 'minio'. ```bash SERVICES_OVERRIDE=localbatch,minio make up ``` -------------------------------- ### Install Metaflow Stubs Source: https://github.com/netflix/metaflow/blob/master/stubs/README.md Install the Metaflow Stubs package using pip. This provides type hints for Metaflow. ```sh pip install metaflow-stubs ``` -------------------------------- ### Start Metaflow Devstack Interactively Source: https://github.com/netflix/metaflow/blob/master/devtools/README.md Navigate to the devtools directory and run 'make up' to interactively select services and start the Metaflow Devstack. This command initiates the development environment with your chosen components. ```bash cd devtools make up ``` -------------------------------- ### Start Services for Argo Workflows Backend Source: https://github.com/netflix/metaflow/blob/master/devtools/README.md Starts the required services for the Argo Workflows backend using 'make up' with a service override. Ensure MinIO, PostgreSQL, metadata-service, and Argo Workflows are available. ```bash SERVICES_OVERRIDE=minio,postgresql,metadata-service,argo-workflows make up ``` -------------------------------- ### Run Playlist Redux Tutorial Source: https://github.com/netflix/metaflow/blob/master/metaflow/tutorials/03-playlist-redux/README.md Execute the playlist generator script to start the tutorial. This command initiates the process of generating movie recommendations based on box office data. ```bash cd metaflow-tutorials ``` ```bash python 03-playlist-redux/playlist.py show ``` ```bash python 03-playlist-redux/playlist.py run ``` -------------------------------- ### Metaflow Run Card Viewer JavaScript Setup Source: https://github.com/netflix/metaflow/blob/master/metaflow/plugins/cards/card_viewer/viewer.html Initializes constants, state, and event listeners for the card viewer, including handling selector changes. ```javascript (() => { // setup constants let DATA_UPDATE_POLL_INTERVAL = 5000; let RUN_UPDATE_POLL_INTERVAL = 10000; // setup state let currentCardsLength = 0; let currentRunId = ""; let selectedCard = ""; // setup selector const selector = document.getElementById("selector"); selector?.addEventListener("change", (event) => { selectedCard = event.target.value; handleLoadCard(event.target.name, event.target.value); }); ``` -------------------------------- ### Run Devstack Locally Source: https://github.com/netflix/metaflow/blob/master/devtools/README.md Starts the development stack locally using the same script as CI. Specify services like minio, postgresql, or metadata-service. ```bash cd devtools SERVICES=minio,postgresql,metadata-service ci/start-devstack.sh ``` -------------------------------- ### Install Metaflow using pip Source: https://github.com/netflix/metaflow/blob/master/README.md Install Metaflow in your Python environment using pip from PyPI. This is the standard method for adding Metaflow to your project. ```sh pip install metaflow ``` -------------------------------- ### Install Metaflow Dev Dependencies Source: https://github.com/netflix/metaflow/blob/master/devtools/README.md Installs Metaflow in development mode along with necessary testing dependencies like pytest and omegaconf. ```bash pip install -e ".[dev]" pytest kubernetes omegaconf ``` -------------------------------- ### Import Metaflow Client and Get Metadata Provider Source: https://github.com/netflix/metaflow/blob/master/metaflow/tutorials/07-worldview/worldview.ipynb Import necessary components from the Metaflow library and print the current metadata provider. ```python from metaflow import Metaflow, Flow, get_metadata, namespace print("Current metadata provider: %s" % get_metadata()) ``` -------------------------------- ### Start All Metaflow Devstack Services Source: https://github.com/netflix/metaflow/blob/master/devtools/README.md To launch all available services in the Metaflow Devstack without interactive selection, use the 'make all-up' command. This ensures the entire development environment is running. ```bash make all-up ``` -------------------------------- ### Enter Metaflow Development Shell Source: https://github.com/netflix/metaflow/blob/master/CONTRIBUTING.md Starts an interactive shell within the development environment, automatically configuring flows to use local MinIO. ```bash metaflow-dev shell ``` -------------------------------- ### Install Metaflow using conda Source: https://github.com/netflix/metaflow/blob/master/README.md Install Metaflow using conda from the conda-forge channel. This is an alternative installation method for users who prefer conda environments. ```sh conda install -c conda-forge metaflow ``` -------------------------------- ### Create a Custom Card Using Metaflow Components Source: https://github.com/netflix/metaflow/blob/master/docs/cards.md This example demonstrates creating a custom card that reuses Metaflow's `BlankCard`, `Artifact`, and `Table` components. The `render` method is overridden to include a table of artifacts. ```python from metaflow.cards import BlankCard from metaflow.cards import Artifact,Table class MyCustomCard(BlankCard): type = 'my_custom_card' def render(self, task): art_com = [ Table( [[Artifact(k.data,k.id)] for k in task] ).render() ] return super().render(task,components=[art_com]) CARDS = [MyCustomCard] ``` -------------------------------- ### Import Metaflow Client and Get Metadata Provider Source: https://github.com/netflix/metaflow/blob/master/metaflow/tutorials/05-hello-cloud/hello-cloud.ipynb Import necessary components from the Metaflow library and print the current metadata provider. This is a foundational step for interacting with Metaflow. ```python from metaflow import Flow, get_metadata, namespace print("Current metadata provider: %s" % get_metadata()) ``` -------------------------------- ### Example Good Commit Message Source: https://github.com/netflix/metaflow/blob/master/CONTRIBUTING.md Illustrates a well-structured commit message, including a concise summary, detailed explanation, and issue reference. ```git Fix symlink traversal edge case in packaging The symlink detection was happening globally across branches, causing some modules to be skipped when encountered through different paths. Now tracks symlinks per-branch to handle this correctly. Fixes #2751 ``` -------------------------------- ### Numpydoc Style Function Docstring Example Source: https://github.com/netflix/metaflow/blob/master/CONTRIBUTING.md Illustrates the numpydoc format for Python docstrings, including parameters, return values, and examples. Adhere to this style for function documentation. ```python def example_function(param1, param2): """ Brief description of the function. Parameters ---------- param1 : str Description of param1 param2 : int Description of param2 Returns ------- bool Description of return value Examples -------- >>> example_function("test", 42) True """ ``` -------------------------------- ### Custom MetaflowCardComponent Implementation Source: https://github.com/netflix/metaflow/blob/master/docs/cards.md Demonstrates how to create custom card components by extending MetaflowCardComponent and implementing the render method. This example shows how to define Title and Text components and use them within a CustomCard. ```python from metaflow.cards import MetaflowCard,MetaflowCardComponent class Title(MetaflowCardComponent): def __init__(self,text): self._text = text def render(self): return "
%s
"%self._text class CustomCard(MetaflowCard): type = "custom_card" HTML = "{data}" def __init__(self, options={"no_header": True}, graph=None, components=[], flow=None, **kwargs): super().__init__() self._no_header = True self._graph = graph if "no_header" in options: self._no_header = options["no_header"] def render(self, task): pt = self._get_mustache() data = '\n'.join([ Title("Title 1").render(), Text("some text comes here").render(), Title("Title 2").render(), Text("some text comes here again").render(), ]) data = dict( data = data ) html_template = self.HTML return pt.render(html_template,data) ``` -------------------------------- ### Generate and Display Gantt Chart for Flow Steps Source: https://github.com/netflix/metaflow/blob/master/metaflow/tutorials/08-autopilot/autopilot.ipynb Creates a Gantt chart visualizing the steps of a Metaflow run, showing their start and finish times. Requires the run object to be successfully loaded. ```python step_data = [dict(Task=step.id, Start=step.created_at, Finish=step.finished_at, Status='timeline') for step in run] fig = ff.create_gantt(step_data, title="Steps of %s/%s" % (run.parent.id, run.id), colors=COLORS, index_col='Status', show_colorbar=True) fig.show() ``` -------------------------------- ### Implement a Custom Metaflow Card Source: https://github.com/netflix/metaflow/blob/master/docs/cards.md Example of creating a custom Metaflow card by subclassing `MetaflowCard` and implementing the `render` function. This custom card uses a Mustache template for rendering HTML. ```python from metaflow.cards import MetaflowCard # path to the custom html file which is a `mustache` template. PATH_TO_CUSTOM_HTML = 'myhtml.html' class CustomCard(MetaflowCard): type = "custom_card" def __init__(self, options={"no_header": True}, graph=None, components=[], flow=None, **kwargs): super().__init__() self._no_header = True self._graph = graph if "no_header" in options: self._no_header = options["no_header"] def render(self, task): pt = self._get_mustache() data = dict( graph = self._graph, header = self._no_header ) html_template = None with open(PATH_TO_CUSTOM_HTML) as f: html_template = f.read() return pt.render(html_template,data) ``` -------------------------------- ### Example Usage of Environment Escape Source: https://github.com/netflix/metaflow/blob/master/docs/Environment escape.md Illustrates how to use a package like 'data_accessor' that is not available in the primary Conda environment. Stub objects are used on the client side to represent server-side objects, and method calls are forwarded for execution. ```python import data_accessor as da sql = 'select * from %s order by int' % name.replace('/', '.') job = da.SqlJob()\ .script(sql)\ .headers()\ .execute() job.wait() job.raise_for_status() result = job.pandas().to_dict() ``` -------------------------------- ### Navigate to Tutorial Directory Source: https://github.com/netflix/metaflow/blob/master/R/inst/tutorials/06-worldview/README.md Change the current directory to the 06-worldview tutorial folder. ```bash cd tutorials/06-worldview/ ``` -------------------------------- ### Show Playlist Tutorial Source: https://github.com/netflix/metaflow/blob/master/metaflow/tutorials/01-playlist/README.md Execute the playlist script to display its help information. ```bash python 01-playlist/playlist.py show ``` -------------------------------- ### Example Unit Test for Symlink Traversal Source: https://github.com/netflix/metaflow/blob/master/CONTRIBUTING.md This is an example of a unit test that checks symlink traversal logic, specifically addressing circular references. It includes setup, implementation, and assertion steps. ```python def test_symlink_traversal_handles_circular_references(): """Test that symlink detection works correctly when modules are encountered through different paths.""" # Test case for issue #2751 # Setup: Create circular symlink structure # ... test implementation # Assert: Verify all modules are included correctly ``` -------------------------------- ### Get Metaflow Version Source: https://github.com/netflix/metaflow/blob/master/CONTRIBUTING.md Use this command to retrieve the installed Metaflow version for bug reports. ```python import metaflow print(metaflow.__version__) ``` -------------------------------- ### Navigate to Tutorial Directory Source: https://github.com/netflix/metaflow/blob/master/metaflow/tutorials/02-statistics/README.md Change the current directory to the metaflow-tutorials folder to access the episode files. ```bash cd metaflow-tutorials ``` -------------------------------- ### Run Playlist Tutorial Source: https://github.com/netflix/metaflow/blob/master/metaflow/tutorials/01-playlist/README.md Run the Metaflow playlist tutorial flow. ```bash python 01-playlist/playlist.py run ``` -------------------------------- ### Run Metaflow Tutorial Source: https://github.com/netflix/metaflow/blob/master/R/inst/tutorials/00-helloworld/README.md Commands to navigate to the tutorial directory and execute the R script for showing and running the Metaflow flow. ```bash cd tutorials/00-helloworld ``` ```bash Rscript helloworld.R show ``` ```bash Rscript helloworld.R run ``` -------------------------------- ### Execute the Metaflow Playlist Flow Source: https://github.com/netflix/metaflow/blob/master/R/inst/tutorials/01-playlist/README.md Run the Metaflow playlist tutorial. Use 'show' to display the flow, 'run' to execute it with default parameters, or 'run --genre