### Project Setup and Dependency Installation Source: https://github.com/marcoeg/bauplan/blob/main/06-near-real-time/README.md Commands to clone the example repository and install necessary Python dependencies using pip. ```bash git clone https://github.com/BauplanLabs/wap_with_bauplan_and_prefect.git cd wap_with_bauplan_and_prefect pip install -r requirements.txt ``` -------------------------------- ### Get Branches Example Source: https://github.com/marcoeg/bauplan/blob/main/Documentation/Module_Documentation.md Example of how to list available branches using the Bauplan client. ```python import bauplan client = bauplan.Client() for branch in client.get_branches(): print(branch.name, branch.hash) ``` -------------------------------- ### Get Branch Example Source: https://github.com/marcoeg/bauplan/blob/main/Documentation/Module_Documentation.md Example of how to retrieve a branch using the Bauplan client. ```python import bauplan client = bauplan.Client() branch = client.get_branch('my_branch_name') print(branch.hash) ``` -------------------------------- ### Define a Bauplan Input Model with Pushdown Source: https://github.com/marcoeg/bauplan/blob/main/01-quick-start/README.md Example of defining an input model in Bauplan that scans an existing table with column selection and filter pushdown for pickup time. ```python @bauplan.model() def trips(): # Scans 'taxi_fhvhv' table with column selection and filter pushdown on pickup time pass ``` -------------------------------- ### Setup: Clone Repository Source: https://github.com/marcoeg/bauplan/blob/main/08-playlist-recomendations-mongodb/README.md Clones the example repository for the recommender system project. ```bash git clone https://github.com/BauplanLabs/wap_with_bauplan_and_prefect.git cd wap_with_bauplan_and_prefect ``` -------------------------------- ### Project Setup and Execution Commands Source: https://github.com/marcoeg/bauplan/blob/main/10-pdf-analysis-with-openai/README.md Provides bash commands for setting up the project, installing dependencies, configuring environment variables, and running the Prefect pipeline. ```bash git clone https://github.com/BauplanLabs/wap_with_bauplan_and_prefect.git cd wap_with_bauplan_and_prefect pip install bauplan prefect PyPDF2 pandas openai export BAUPLAN_API_KEY= export AWS_ACCESS_KEY_ID= export AWS_SECRET_ACCESS_KEY= export OPENAI_API_KEY= prefect server start prefect deploy -f pdf_analysis_pipeline.py prefect run -n pdf-analysis-pipeline bauplan logs ``` -------------------------------- ### Setup Commands for Bauplan and Prefect Project Source: https://github.com/marcoeg/bauplan/blob/main/13-data-products/README.md Bash commands for setting up the project environment, including cloning the repository, installing dependencies, and configuring environment variables for Bauplan API key and AWS credentials. ```bash git clone https://github.com/BauplanLabs/wap_with_bauplan_and_prefect.git cd wap_with_bauplan_and_prefect pip install bauplan prefect pandas export BAUPLAN_API_KEY= export AWS_ACCESS_KEY_ID= export AWS_SECRET_ACCESS_KEY= ``` -------------------------------- ### Define a Bauplan Join Model Source: https://github.com/marcoeg/bauplan/blob/main/01-quick-start/README.md Example of a Bauplan model that joins two tables and combines chunks. This model specifies its execution environment and dependencies. ```python @bauplan.model() @bauplan.python(python_version='3.11') def trips_and_zones(trips, zones): # Joins trips data with zones table based on PULocationID and LocationID # Combines chunks post-join pass ``` -------------------------------- ### Setup: Install Dependencies Source: https://github.com/marcoeg/bauplan/blob/main/08-playlist-recomendations-mongodb/README.md Installs the necessary Python libraries for the project, including bauplan, pymongo, sentence-transformers, and pandas. ```bash pip install bauplan pymongo sentence-transformers pandas ``` -------------------------------- ### Project Setup and Configuration Commands Source: https://github.com/marcoeg/bauplan/blob/main/09-unstructured-to-structured-with-llm/README.md Provides essential bash commands for setting up the project, including cloning the repository, installing dependencies, and configuring environment variables for API keys and AWS credentials. ```bash git clone https://github.com/BauplanLabs/wap_with_bauplan_and_prefect.git cd wap_with_bauplan_and_prefect pip install -r requirements.txt export BAUPLAN_API_KEY= export AWS_ACCESS_KEY_ID= export AWS_SECRET_ACCESS_KEY= export LLM_API_KEY= ``` -------------------------------- ### Project Setup and Configuration Source: https://github.com/marcoeg/bauplan/blob/main/07-entity-matching-with-llm/README.md Provides bash commands for setting up the project, including cloning the repository, installing dependencies, and configuring necessary environment variables for Bauplan, AWS, and LLM API keys. ```bash git clone https://github.com/BauplanLabs/wap_with_bauplan_and_prefect.git cd wap_with_bauplan_and_prefect pip install -r requirements.txt export BAUPLAN_API_KEY= export AWS_ACCESS_KEY_ID= export AWS_SECRET_ACCESS_KEY= export LLM_API_KEY= ``` -------------------------------- ### Define a Python Bauplan Model Source: https://github.com/marcoeg/bauplan/blob/main/01-quick-start/README.md Example of defining a Python model for data transformation using Bauplan. This model specifies its Python environment and dependencies, including pandas and numpy. ```python @bauplan.model() @bauplan.python(python_version='3.11', packages=['pandas==1.5.3', 'numpy==1.23.2']) def normalized_taxi_trips(trips_and_zones): # Convert Arrow to Pandas # Apply timestamp filters # Remove trips with 0 or excessive miles # Log-transform trip_miles pass ``` -------------------------------- ### Get Tables Example Source: https://github.com/marcoeg/bauplan/blob/main/Documentation/Module_Documentation.md Illustrates how to retrieve a list of tables and views from a specified branch using the `get_tables` method. The example shows iterating through the results and printing table names and kinds. ```python import bauplan client = bauplan.Client() for table in client.get_tables('my_ref_or_branch_name'): print(table.name, table.kind) ``` -------------------------------- ### Run Bauplan Pipeline Source: https://github.com/marcoeg/bauplan/blob/main/01-quick-start/README.md Commands to create a branch, checkout, and run the Bauplan pipeline. These commands orchestrate the execution of the defined data transformation DAG. ```bash bauplan branch create . bauplan checkout . bauplan run ``` -------------------------------- ### Get Table Data Example Source: https://github.com/marcoeg/bauplan/blob/main/Documentation/Module_Documentation.md Demonstrates how to fetch table data and metadata using the `get_table` method. It shows how to iterate through table fields and access records. ```python import bauplan client = bauplan.Client() table = client.get_table( table='my_table_name', ref='my_ref_or_branch_name', ) for c in table.fields: print(c.name, c.required, c.type) print(table.records) ``` -------------------------------- ### Get Tag Example Source: https://github.com/marcoeg/bauplan/blob/main/Documentation/Module_Documentation.md Provides an example of how to retrieve a specific tag using its name with the `get_tag` method and access its hash. ```python import bauplan client = bauplan.Client() tag = client.get_tag('my_tag_name') print(tag.hash) ``` -------------------------------- ### Clone Example Repository Source: https://github.com/marcoeg/bauplan/blob/main/11-RAG-service-support-agent/README.md Clones the example repository for the RAG pipeline using Bauplan and Prefect. ```bash git clone https://github.com/BauplanLabs/wap_with_bauplan_and_prefect.git cd wap_with_bauplan_and_prefect ``` -------------------------------- ### Revert Table Example Source: https://github.com/marcoeg/bauplan/blob/main/Documentation/Module_Documentation.md Example of how to revert a table to a previous state using the Bauplan client. ```python import bauplan client = bauplan.Client() assert client.revert_table( table='my_table_name', source_ref='my_ref_or_branch_name', into_branch='main', ) ``` -------------------------------- ### Shortened Trips and Zones Model Source: https://github.com/marcoeg/bauplan/blob/main/Documentation/Code_Examples.md A concise definition for the 'trips_and_zones' model using Bauplan and Python, suitable for quick setup or examples. ```python @bauplan.model() @bauplan.python('3.11') def trips_and_zones(...): ``` -------------------------------- ### Prefect Pipeline Deployment and Execution Source: https://github.com/marcoeg/bauplan/blob/main/06-near-real-time/README.md Commands to start the Prefect server, deploy the defined pipeline, and then run an instance of the pipeline. ```bash prefect server start prefect deploy -f nrt_pipeline.py prefect run -n nrt-pipeline ``` -------------------------------- ### Python Example: Query to Generator Source: https://github.com/marcoeg/bauplan/blob/main/Documentation/Module_Documentation.md Demonstrates how to use the `query_to_generator` method of the Bauplan Client to execute a SQL query and iterate over the results. ```python import bauplan client = bauplan.Client() res = client.query_to_generator( query='SELECT c1 FROM my_table', ref='my_ref_or_branch_name', ) for row in res: # do logic ``` -------------------------------- ### Install Dependencies Source: https://github.com/marcoeg/bauplan/blob/main/11-RAG-service-support-agent/README.md Installs the necessary Python libraries for the RAG pipeline, including Bauplan, Prefect, sentence-transformers, pandas, faiss-cpu, and openai. ```bash pip install bauplan prefect sentence-transformers pandas faiss-cpu openai ``` -------------------------------- ### Delete Table Example Source: https://github.com/marcoeg/bauplan/blob/main/Documentation/Module_Documentation.md Example of how to delete a table using the Bauplan client. ```python import bauplan client = bauplan.Client() assert client.delete_table( table='my_table_name', branch='my_branch_name', ) ``` -------------------------------- ### Example Usage of query_to_parquet_file Source: https://github.com/marcoeg/bauplan/blob/main/Documentation/Module_Documentation.md Demonstrates how to use the `query_to_parquet_file` function to export query results to a Parquet file. ```python import bauplan client = bauplan.Client() client.query_to_parquet_file( path='./my.parquet', query='SELECT c1 FROM my_table', ref='my_ref_or_branch_name', ) ``` -------------------------------- ### Delete Namespace Example Source: https://github.com/marcoeg/bauplan/blob/main/Documentation/Module_Documentation.md Example of how to delete a namespace using the Bauplan client. ```python import bauplan client = bauplan.Client() assert client.delete_namespace( namespace='my_namespace_name', branch='my_branch_name', ) ``` -------------------------------- ### Python Example: Query to JSON File Source: https://github.com/marcoeg/bauplan/blob/main/Documentation/Module_Documentation.md Illustrates how to use the `query_to_json_file` method of the Bauplan Client to export query results to a JSON file. ```python import bauplan client = bauplan.Client() client.query_to_json_file( path='./my.json', query='SELECT c1 FROM my_table', ref='my_ref_or_branch_name', ) ``` -------------------------------- ### Match Products with LLM Source: https://github.com/marcoeg/bauplan/blob/main/Documentation/Code_Examples.md Matches products from two lists by iterating through inputs, formatting prompts, and calling an OpenAI client. This is suitable for tasks requiring semantic matching or comparison. ```python def match_with_llm(_product_a_list, _product_b_list, _llm_client): # loop over inputs, format prompts, call OpenAI pass ``` -------------------------------- ### Python Environment with DuckDB 0.10.3 Source: https://github.com/marcoeg/bauplan/blob/main/Documentation/Environment_Setups.md Sets up a Python 3.11 environment with DuckDB version 0.10.3. This is beneficial for efficient in-process analytical data management. ```python @bauplan.python('3.11', pip={'duckdb': '0.10.3'}) ``` -------------------------------- ### Check Namespace Existence Example Source: https://github.com/marcoeg/bauplan/blob/main/Documentation/Module_Documentation.md Shows how to verify the existence of a namespace within a specific ref or branch using the `has_namespace` method. ```python import bauplan client = bauplan.Client() assert client.has_namespace( namespace='my_namespace_name', ref='my_ref_or_branch_name', ) ``` -------------------------------- ### Create Public S3 Bucket Source: https://github.com/marcoeg/bauplan/blob/main/Documentation/Code_Examples.md Creates an S3 bucket and applies a public read policy. It checks for existing buckets before creation and uses a predefined bucket policy. ```python import json def create_public_bucket(s3_client, bucket_name: str): if not does_bucket_exist(s3_client, bucket_name): s3_client.create_bucket(Bucket=bucket_name) bucket_policy = {...} s3_client.put_bucket_policy(Bucket=bucket_name, Policy=json.dumps(bucket_policy)) ``` -------------------------------- ### Create Tag in Bauplan Source: https://github.com/marcoeg/bauplan/blob/main/Documentation/Module_Documentation.md Provides an example of creating a new tag in Bauplan, referencing a specific branch or ref. This is useful for marking specific points in the project history. ```python import bauplan client = bauplan.Client() assert client.create_tag( tag='my_tag', from_ref='my_ref_or_branch_name', ) ``` -------------------------------- ### Serialized Walmart Products with Pandas Source: https://github.com/marcoeg/bauplan/blob/main/Documentation/Code_Examples.md A placeholder function for serializing Walmart product data. It uses Pandas and is defined within the Bauplan framework. ```python @bauplan.python('3.11', pip={'pandas': '2.2.0'}) @bauplan.model() def serialized_walmart_products(...) ``` -------------------------------- ### Python Environment with DuckDB and OpenAI Source: https://github.com/marcoeg/bauplan/blob/main/Documentation/Environment_Setups.md Establishes a Python 3.11 environment with DuckDB (1.0.0) and OpenAI (1.57.2). This configuration supports data analysis integrated with AI/ML workflows. ```python @bauplan.python('3.11', pip={'duckdb': '1.0.0', 'openai': '1.57.2'}) ``` -------------------------------- ### Generate Tip Predictions with Pandas and Scikit-learn Source: https://github.com/marcoeg/bauplan/blob/main/Documentation/Code_Examples.md Generates tip predictions using a pre-trained linear regression model loaded via bauplan.store. Requires pandas and scikit-learn. ```python @bauplan.model(materialization_strategy='REPLACE') @bauplan.python('3.11', pip={'scikit-learn': '1.3.2', 'pandas': '2.1.0'}) def tip_predictions(data=bauplan.Model('train_regression_model')): from bauplan.store import load_obj reg = load_obj("regression") test_set = data.to_pandas() y_hat = reg.predict(test_set[['log_trip_miles', 'base_passenger_fare', 'trip_time']]) prediction_df = test_set.copy() prediction_df['predictions'] = y_hat return prediction_df ``` -------------------------------- ### Monitoring Prefect and Bauplan Source: https://github.com/marcoeg/bauplan/blob/main/06-near-real-time/README.md Instructions on how to access the Prefect UI for monitoring and how to view Bauplan logs. ```bash prefect ui bauplan logs ``` -------------------------------- ### Prefect Pipeline Deployment and Execution Source: https://github.com/marcoeg/bauplan/blob/main/07-entity-matching-with-llm/README.md Demonstrates how to deploy and run a Prefect pipeline. It includes commands to start the Prefect server, deploy the defined flow, and execute the pipeline. ```bash prefect server start prefect deploy -f llm_pipeline.py prefect run -n llm-pipeline ``` -------------------------------- ### Plan Table Creation in Bauplan Source: https://github.com/marcoeg/bauplan/blob/main/Documentation/Module_Documentation.md Illustrates planning a table creation from S3 data using bauplan. This involves providing the table name, S3 search URI, and optional parameters like branch and partitioning. ```python import bauplan client = bauplan.Client() plan_state = client.plan_table_creation( table='my_table_name', search_uri='s3://path/to/my/files/*.parquet', ref='my_ref_or_branch_name', ) if plan_state.error: plan_error_action(...) success_action(plan_state.plan) ``` -------------------------------- ### Running the Pipeline: Deploy Source: https://github.com/marcoeg/bauplan/blob/main/08-playlist-recomendations-mongodb/README.md Deploys the Prefect recommender pipeline. This command makes the pipeline available for execution within the Prefect environment. ```bash prefect deploy -f recommender_pipeline.py ``` -------------------------------- ### Delete Tag Example Source: https://github.com/marcoeg/bauplan/blob/main/Documentation/Module_Documentation.md Example of how to delete a tag using the Bauplan client. ```python import bauplan client = bauplan.Client() assert client.delete_tag('my_tag_name') ``` -------------------------------- ### Running Prefect Pipelines Source: https://github.com/marcoeg/bauplan/blob/main/13-data-products/README.md Bash commands to manage and run Prefect pipelines. This includes starting the Prefect server, deploying a pipeline defined in a Python file, and executing a deployed pipeline. ```bash prefect server start prefect deploy -f dataproduct_pipeline.py prefect run -n dataproduct-pipeline ``` -------------------------------- ### Bauplan Client Initialization and Namespace Retrieval Source: https://github.com/marcoeg/bauplan/blob/main/Documentation/Module_Documentation.md Demonstrates how to initialize the Bauplan client and retrieve a specific namespace using its name and a reference. Includes error handling for non-existent namespaces or refs. ```python import bauplan client = bauplan.Client() namespace = client.get_namespace( namespace='my_namespace_name', ref='my_ref_or_branch_name', ) ``` -------------------------------- ### Simple Bauplan Model Definition Source: https://github.com/marcoeg/bauplan/blob/main/Documentation/Code_Examples.md A basic example of defining a Bauplan model that takes a table as input and returns it, serving as a template for transformations. ```python @bauplan.model() def my_model(data=bauplan.Model('some_table')): # Transformation logic here return data ``` -------------------------------- ### Bauplan Client Initialization and Query Source: https://github.com/marcoeg/bauplan/blob/main/Documentation/Module_Documentation.md Demonstrates how to initialize the bauplan client and perform a query, returning the result as an Arrow Table, which can then be converted to a pandas DataFrame. ```python import bauplan client = bauplan.Client() # query the table and return result set as an arrow Table my_table = client.query('SELECT sum(trips) trips FROM travel_table', branch_name='main') # efficiently cast the table to a pandas DataFrame df = my_table.to_pandas() ``` -------------------------------- ### Python Environment with DuckDB 1.0.0 Source: https://github.com/marcoeg/bauplan/blob/main/Documentation/Environment_Setups.md Configures a Python 3.11 environment with DuckDB version 1.0.0. This setup is recommended for advanced analytical queries and data processing. ```python @bauplan.python('3.11', pip={'duckdb': '1.0.0'}) ``` -------------------------------- ### Materialization Strategy Example Source: https://github.com/marcoeg/bauplan/blob/main/Documentation/Code_Examples.md Demonstrates setting a materialization strategy for a Bauplan model, specifying 'REPLACE' to overwrite existing data. This is often used with data warehousing or batch processing. ```arrow @bauplan.model(materialization_strategy='REPLACE') ``` -------------------------------- ### Prefect Pipeline Deployment and Execution Source: https://github.com/marcoeg/bauplan/blob/main/09-unstructured-to-structured-with-llm/README.md Demonstrates how to deploy and run the Prefect pipeline. It includes starting the Prefect server, deploying the defined flow, and executing the pipeline, along with commands for monitoring logs. ```bash prefect server start prefect deploy -f llm_tabular_pipeline.py prefect run -n llm-tabular-pipeline # Monitoring # Access Prefect UI: http://localhost:4200 # Check Bauplan logs: bauplan logs ``` -------------------------------- ### Python Environment with Pandas 2.2.0 Source: https://github.com/marcoeg/bauplan/blob/main/Documentation/Environment_Setups.md Configures a Python 3.11 environment with Pandas version 2.2.0. This setup is suitable for projects that require the latest features or bug fixes in Pandas. ```python @bauplan.python('3.11', pip={'pandas': '2.2.0'}) ``` -------------------------------- ### Create Table in Bauplan Source: https://github.com/marcoeg/bauplan/blob/main/Documentation/Module_Documentation.md Illustrates the process of creating a table in Bauplan from S3 data. It shows how to specify the table name, the S3 URI for data, and optionally the branch and partitioning. ```python import bauplan client = bauplan.Client() table = client.create_table( table='my_table_name', search_uri='s3://path/to/my/files/*.parquet', ref='my_ref_or_branch_name', ) ``` -------------------------------- ### Get Item Recommendations Source: https://github.com/marcoeg/bauplan/blob/main/08-playlist-recomendations-mongodb/README.md This Python code snippet demonstrates how to get recommendations for an item by querying a MongoDB collection using pre-computed embeddings. It utilizes the SentenceTransformer library to generate embeddings and a custom `query_recommendations` function. Ensure MongoDB URI is set in environment variables. ```python from sentence_transformers import SentenceTransformer import os sample_embedding = SentenceTransformer('all-MiniLM-L6-v2').encode("Sample item description").tolist() results = query_recommendations( mongodb_uri=os.getenv("MONGODB_URI"), db_name="recommender_db", collection_name="items", embedding=sample_embedding, top_k=5 ) for result in results: print(result['name'], result['description']) ``` -------------------------------- ### Product LLM Matches with DuckDB and OpenAI Source: https://github.com/marcoeg/bauplan/blob/main/Documentation/Code_Examples.md A placeholder function for matching products using LLM, likely involving DuckDB for data querying and OpenAI for LLM capabilities. It's configured for internet access. ```python @bauplan.python('3.11', pip={'duckdb': '1.0.0', 'openai': '1.57.2'}) @bauplan.model(materialization_strategy='REPLACE', internet_access=True) def product_llm_matches(...) ``` -------------------------------- ### Evaluate Regression Model Source: https://github.com/marcoeg/bauplan/blob/main/Documentation/Code_Examples.md Calculates the score (e.g., R-squared) of a trained regression model on test data. ```python reg.score(X_test, y_test) ``` -------------------------------- ### Bauplan Client Parameters Source: https://github.com/marcoeg/bauplan/blob/main/Documentation/Module_Documentation.md Defines the configuration parameters for initializing the Bauplan client, including API keys, branches, namespaces, caching, debugging, and endpoint configurations. ```APIDOC Parameters: - profile: `str | None = None` (optional) The Bauplan config profile name to use to determine `api_key`. - api_key: `str | None = None` (optional) Your unique Bauplan API key; mutually exclusive with `profile`. If not provided, fetch precedence is 1) environment `BAUPLAN_API_KEY` 2) `.bauplan/config.yml`. - branch: `str | None = None` (optional) The default branch to use for queries and runs. If not provided, `active_branch` from the profile is used. - namespace: `str | None = None` (optional) The default namespace to use for queries and runs. - cache: `'on' | 'off' | None = None` (optional) Whether to enable or disable caching for all requests. - debug: `bool | None = None` (optional) Whether to enable or disable debug mode for all requests. - verbose: `bool | None = None` (optional) Whether to enable or disable verbose mode for all requests. - args: `dict[str, str] | None = None` (optional) Additional arguments to pass to all requests. - api_endpoint: `str | None = None` (optional) The Bauplan API endpoint to use. If not provided, fetch precedence is 1) environment `BAUPLAN_API_ENDPOINT` 2) `.bauplan/config.yml`. - catalog_endpoint: `str | None = None` (optional) The Bauplan catalog endpoint to use. If not provided, fetch precedence is 1) environment `BAUPLAN_CATALOG_ENDPOINT` 2) `.bauplan/config.yml`. - catalog_max_records: `int | None = None` (optional) The maximum number of records to fetch, per page, from the catalog. - client_timeout: `int | None = None` (optional) The client timeout in seconds for all requests. - env: `str | None = None` (optional) The environment to use for all requests. Default: `'prod'`. - config_file_path: `str | Path | None = None` (optional) The path to the Bauplan config file to use. If not provided, fetch precedence is 1) environment `BAUPLAN_CONFIG_PATH` 2) `~/.bauplan/config.yml`. *Note*: Based on our previous discussion, if you’re using a `.env` file to set `BAUPLAN_CONFIG_PATH`, ensure it’s in the correct project directory to avoid loading a default `~/.bauplan/config.yml`. - user_session_token: `str | None = None` (optional) Your unique Bauplan user session token. - feature_flags: `dict[str, Any] | None = None` (optional) Feature flags to enable experimental features. ``` -------------------------------- ### Check Branch Existence Example Source: https://github.com/marcoeg/bauplan/blob/main/Documentation/Module_Documentation.md Demonstrates how to check if a branch exists using the `has_branch` method, asserting its presence. ```python import bauplan client = bauplan.Client() assert client.has_branch('my_branch_name') ``` -------------------------------- ### Running the Pipeline: Run Locally Source: https://github.com/marcoeg/bauplan/blob/main/08-playlist-recomendations-mongodb/README.md Executes the Prefect recommender pipeline locally. This is useful for testing and development before deploying to a remote environment. ```bash prefect run -n recommender-pipeline ``` -------------------------------- ### Convert Arrow to Pandas DataFrame Source: https://github.com/marcoeg/bauplan/blob/main/Documentation/Code_Examples.md Converts an Arrow DataFrame to a Pandas DataFrame. This is a common operation for data manipulation. ```python df = data.to_pandas() ``` -------------------------------- ### Load Model Object using bauplan.store Source: https://github.com/marcoeg/bauplan/blob/main/Documentation/Code_Examples.md Retrieves a previously saved model object from storage using the bauplan.store.load_obj function. ```python from bauplan.store import load_obj reg = load_obj("regression") ``` -------------------------------- ### Bauplan Client Authentication Methods Source: https://github.com/marcoeg/bauplan/blob/main/Documentation/Module_Documentation.md Illustrates various ways to authenticate the bauplan client, prioritizing API key over profile configuration, and showing direct specification of API key or profile. ```python # by default, authenticate from BAUPLAN_API_KEY >> BAUPLAN_PROFILE >> ~/.bauplan/config.yml client = bauplan.Client() # client used ~/.bauplan/config.yml profile 'default' os.environ['BAUPLAN_PROFILE'] = "someprofile" client = bauplan.Client() # >> client now uses profile 'someprofile' os.environ['BAUPLAN_API_KEY'] = "mykey" client = bauplan.Client() # >> client now authenticates with api_key value "mykey", because api key > profile # specify authentication directly - this supercedes BAUPLAN_API_KEY in the environment client = bauplan.Client(api_key='MY_KEY') # specify a profile from ~/.bauplan/config.yml - this supercedes BAUPLAN_PROFILE in the environment client = bauplan.Client(profile='default') ``` -------------------------------- ### Train Linear Regression Model Source: https://github.com/marcoeg/bauplan/blob/main/Documentation/Code_Examples.md Trains a linear regression model using provided training data (X_train, y_train). ```python reg = LinearRegression().fit(X_train, y_train) ``` -------------------------------- ### Setup: Configure Environment Variables Source: https://github.com/marcoeg/bauplan/blob/main/08-playlist-recomendations-mongodb/README.md Sets up essential environment variables for API keys and database URIs. These include Bauplan API key, AWS credentials, and MongoDB Atlas URI. ```bash export BAUPLAN_API_KEY= export AWS_ACCESS_KEY_ID= export AWS_SECRET_ACCESS_KEY= export MONGODB_URI= ``` -------------------------------- ### Prepare Training Dataset with Pandas and Scikit-learn Source: https://github.com/marcoeg/bauplan/blob/main/Documentation/Code_Examples.md Prepares a training dataset by dropping null values, applying log transformation to trip miles, and scaling features using StandardScaler. Requires pandas and scikit-learn. ```python @bauplan.model() @bauplan.python('3.10', pip={'pandas': '1.5.3', 'scikit-learn': '1.3.2'}) def training_dataset(data=bauplan.Model('clean_taxi_trips')): df = data.to_pandas() df = df.dropna() df['log_trip_miles'] = np.log10(df['trip_miles']) scaler = StandardScaler() scaled_features = scaler.fit_transform(df[['log_trip_miles', 'base_passenger_fare', 'trip_time']]) scaled_df = pd.DataFrame(scaled_features, columns=['log_trip_miles', 'base_passenger_fare', 'trip_time']) return scaled_df ``` -------------------------------- ### Shortened Taxi FHVHV Model Source: https://github.com/marcoeg/bauplan/blob/main/Documentation/Code_Examples.md A compact definition for the 'taxi_fhvhv' model in Bauplan, demonstrating filter pushdown and column pruning for performance. ```python trips=bauplan.Model('taxi_fhvhv', columns=[...], filter='...') ``` -------------------------------- ### Configure Environment Variables Source: https://github.com/marcoeg/bauplan/blob/main/11-RAG-service-support-agent/README.md Sets up essential environment variables for Bauplan API key, AWS credentials, OpenAI API key, and optionally a vector database URL. ```bash export BAUPLAN_API_KEY= export AWS_ACCESS_KEY_ID= export AWS_SECRET_ACCESS_KEY= export OPENAI_API_KEY= export VECTOR_DB_URL= # Optional, depending on vector DB ``` -------------------------------- ### Serialized Amazon Products with Pandas Source: https://github.com/marcoeg/bauplan/blob/main/Documentation/Code_Examples.md A placeholder function for serializing Amazon product data. It uses Pandas and is defined within the Bauplan framework. ```python @bauplan.python('3.11', pip={'pandas': '2.2.0'}) @bauplan.model() def serialized_amazon_products(...) ```