### Set up Local Virtual Environment and Install Dependencies

Source: https://github.com/gokumohandas/made-with-ml/blob/main/README.md

Sets up a Python virtual environment named 'venv' and installs project dependencies from 'requirements.txt'. It also configures the PYTHONPATH and installs pre-commit hooks for code quality.

```bash
export PYTHONPATH=$PYTHONPATH:$PWD
python3 -m venv venv
source venv/bin/activate
python3 -m pip install --upgrade pip setuptools wheel
python3 -m pip install -r requirements.txt
pre-commit install
pre-commit autoupdate
```

--------------------------------

### Install Python Packages

Source: https://github.com/gokumohandas/made-with-ml/blob/main/notebooks/benchmarks.ipynb

Installs necessary Python packages for the project, including openai and tqdm. The '-q' flag ensures a quiet installation process.

```python
!pip install openai==0.27.8 tqdm==4.65.0 -q
```

--------------------------------

### Set up Anyscale Virtual Environment and Install Dependencies

Source: https://github.com/gokumohandas/made-with-ml/blob/main/README.md

Configures the PYTHONPATH and installs pre-commit hooks for the Anyscale environment. The Python version and core libraries are assumed to be pre-configured by the Anyscale Workspace.

```bash
export PYTHONPATH=$PYTHONPATH:$PWD
pre-commit install
pre-commit autoupdate
```

--------------------------------

### MLflow Experiment Tracking Setup and Usage

Source: https://context7.com/gokumohandas/made-with-ml/llms.txt

This section details how to set up and use MLflow for experiment tracking. It includes commands to export the model registry path, start the MLflow server, and programmatic access to search for runs and retrieve artifact URIs.

```bash
# Get model registry path
export MODEL_REGISTRY=$(python -c "from madewithml import config; print(config.MODEL_REGISTRY)")

# Start MLflow server
mlflow server -h 0.0.0.0 -p 8080 --backend-store-uri $MODEL_REGISTRY

# Access UI at http://localhost:8080/
```

```python
# Programmatic access to experiments
from madewithml.config import mlflow

# Search for best run
sorted_runs = mlflow.search_runs(
    experiment_names=["llm"],
    order_by=["metrics.val_loss ASC"],
)
best_run_id = sorted_runs.iloc[0].run_id
print(f"Best run: {best_run_id}")

# Get run artifacts
run = mlflow.get_run(best_run_id)
artifact_uri = run.info.artifact_uri
print(f"Artifacts: {artifact_uri}")
```

--------------------------------

### Start and Stop Local Ray Server

Source: https://github.com/gokumohandas/made-with-ml/blob/main/README.md

Commands to start and stop a local Ray cluster for serving ML models. This involves initializing Ray and then shutting it down.

```bash
ray start --head

...

ray stop  # shutdown
```

--------------------------------

### Start Jupyter Notebook Locally

Source: https://github.com/gokumohandas/made-with-ml/blob/main/README.md

Launches a JupyterLab instance to interactively explore the machine learning workloads defined in the 'madewithml.ipynb' notebook.

```bash
jupyter lab notebooks/madewithml.ipynb
```

--------------------------------

### Set Up and Serve ML Model Locally

Source: https://github.com/gokumohandas/made-with-ml/blob/main/README.md

Sets up the environment variables and starts the ML model serving application locally using Python scripts. It identifies the best run ID based on a specified metric and mode.

```bash
export EXPERIMENT_NAME="llm"
export RUN_ID=$(python madewithml/predict.py get-best-run-id --experiment-name $EXPERIMENT_NAME --metric val_loss --mode ASC)
python madewithml/serve.py --run_id $RUN_ID
```

--------------------------------

### Start MLflow Server using Bash

Source: https://github.com/gokumohandas/made-with-ml/blob/main/README.md

This command starts an MLflow tracking server. It retrieves the model registry location using a Python command and then launches the MLflow server, making it accessible on port 8080. This is useful for tracking experiments locally or within a managed environment like Anyscale.

```bash
export MODEL_REGISTRY=$(python -c "from madewithml import config; print(config.MODEL_REGISTRY)")
mlflow server -h 0.0.0.0 -p 8080 --backend-store-uri $MODEL_REGISTRY
```

--------------------------------

### Set up Ray Tune Configuration

Source: https://github.com/gokumohandas/made-with-ml/blob/main/notebooks/madewithml.ipynb

Configures the Ray Tune tuner with the optimization metric, mode, search algorithm, scheduler, and the number of samples to run. This setup is for minimizing the validation loss.

```python
import ray.tune as tune

# Assuming search_alg, scheduler, num_runs are defined elsewhere
tune_config = tune.TuneConfig(
    metric="val_loss",
    mode="min",
    search_alg=search_alg,
    scheduler=scheduler,
    num_samples=num_runs,
)

```

--------------------------------

### Start MLFlow Server

Source: https://github.com/gokumohandas/made-with-ml/blob/main/notebooks/madewithml.ipynb

Starts the MLFlow server to host the experiment tracking dashboard. It binds to all network interfaces and listens on port 8080. The backend store URI is specified to point to the MLFlow data directory.

```bash
mlflow server -h 0.0.0.0 -p 8080 --backend-store-uri $EFS_DIR/mlflow
```

--------------------------------

### Initialize HyperOptSearch with Initial Parameters

Source: https://github.com/gokumohandas/made-with-ml/blob/main/notebooks/madewithml.ipynb

Sets up the HyperOptSearch algorithm for hyperparameter optimization, providing initial parameter sets to start the search. It also limits the concurrency of trials.

```python
from ray.tune.search.hyperopt import HyperOptSearch
from ray.tune.search import ConcurrencyLimiter

# Initial hyperparameters to evaluate
initial_params = [{"train_loop_config": {"dropout_p": 0.5, "lr": 1e-4, "lr_factor": 0.8, "lr_patience": 3}}]
search_alg = HyperOptSearch(points_to_evaluate=initial_params)
search_alg = ConcurrencyLimiter(search_alg, max_concurrent=2)  # Limit concurrent trials

```

--------------------------------

### Install Jupyter Kernel for Virtual Environment (Bash)

Source: https://github.com/gokumohandas/made-with-ml/blob/main/README.md

This command installs a Jupyter kernel for the current virtual environment, allowing notebooks to use the environment's packages. It requires Python 3 and the `ipykernel` package to be installed.

```bash
python3 -m ipykernel install --user --name=venv
```

--------------------------------

### Initiate Model Training with Trainer

Source: https://github.com/gokumohandas/made-with-ml/blob/main/notebooks/madewithml.ipynb

This snippet shows the initiation of the model training process using a 'trainer' object. It calls the 'fit()' method to start the training. The output indicates the use of distributed workers and GPU allocation.

```python
results = trainer.fit()

```

--------------------------------

### Model Serving with Ray Serve

Source: https://context7.com/gokumohandas/made-with-ml/llms.txt

This section demonstrates how to deploy machine learning models as a REST API using Ray Serve, integrated with FastAPI. It covers starting Ray, deploying a model using its run ID, and interacting with the API endpoints for health checks, run ID retrieval, predictions, and evaluation.

```bash
# Start Ray and deploy model
ray start --head
export EXPERIMENT_NAME="llm"
export RUN_ID=$(python madewithml/predict.py get-best-run-id \
    --experiment-name $EXPERIMENT_NAME \
    --metric val_loss \
    --mode ASC)
python madewithml/serve.py --run_id $RUN_ID --threshold 0.9
```

```bash
# cURL example for prediction
curl -X POST "http://127.0.0.1:8000/predict/" \
    -H "Content-Type: application/json" \
    -d '{"title": "Transfer learning with transformers", "description": "Using transformers for transfer learning."}'

# Shutdown Ray
ray stop
```

--------------------------------

### Set up MLflow Logger Callback

Source: https://github.com/gokumohandas/made-with-ml/blob/main/notebooks/madewithml.ipynb

Configures an MLflow logger callback for tracking experiments. This requires MLflow to be installed and a tracking URI to be set. It saves artifacts during training.

```python
from ray.ml.integrations.mlflow import MLflowLoggerCallback

# Assuming MLFLOW_TRACKING_URI and experiment_name are defined elsewhere
mlflow_callback = MLflowLoggerCallback(
    tracking_uri=MLFLOW_TRACKING_URI,
    experiment_name=experiment_name,
    save_artifact=True)

```

--------------------------------

### Batch Inference Example

Source: https://github.com/gokumohandas/made-with-ml/blob/main/notebooks/madewithml.ipynb

This snippet demonstrates how to perform batch inference using Ray Data. It preprocesses test data and then uses a predictor with an ActorPoolStrategy for parallel processing.

```APIDOC
## Batch Inference

### Description
This section shows an example of how to perform batch inference using Ray Data. It involves preprocessing a dataset and then mapping a predictor function across batches in parallel.

### Method
N/A (Code Example)

### Endpoint
N/A (Code Example)

### Parameters
N/A

### Request Example
```python
preprocessed_ds = preprocessor.transform(test_ds)
compute = ActorPoolStrategy(min_size=1, max_size=2)
outputs = preprocessed_ds.map_batches(predictor, batch_size=128, compute=compute)
np.array([d["output"] for d in outputs.take_all()])
```

### Response
#### Success Response (200)
N/A (Code Example)

#### Response Example
```python
# Example output format (actual values will vary)
array([0, 3, 3, 0, 2, 0, 0, 0, 0, 2, 0, 0, 2, 3, 0, 0, 0, 2, 3, 2, 3, 0,
       3, 2, 0, 0, 2, 1, 1, 2, 2, 2, 2, 2, 2, 0, 0, 0, 0, 0, 1, 2, 2, 0,
       0, 3, 1, 2, 0, 2, 2, 3, 3, 0, 2, 1, 2, 3, 3, 3, 3, 2, 0, 0, 0, 2,
       2, 3, 2, 1, 0, 2, 3, 1, 0, 2, 0, 2, 2, 2, 0, 0, 2, 1, 1, 0, 0, 0,
       0, 3, 0, 0, 2, 0, 2, 2, 3, 2, 0, 2, 0, 2, 2, 0, 1, 0, 0, 0, 0, 2,
       0, 0, 1, 2, 2, 2, 3, 0, 2, 0, 2, 3, 2, 3, 3, 3, 2, 0, 2, 2, 2, 2,
       0, 2, 2, 2, 0, 1, 2, 2, 2, 2, 2, 1, 2, 0, 3, 0, 2, 2, 1, 1, 2, 0,
       2, 0, 0, 0, 0, 2, 2, 2, 0, 2, 1, 2, 2, 0, 0, 1, 2, 3, 2, 2, 2, 0,
       0, 2, 0, 2, 1, 3, 0, 2, 2, 0, 1, 2, 1, 2, 2])
```
```

--------------------------------

### Initiate Model Tuning with Ray Tune (Python)

Source: https://github.com/gokumohandas/made-with-ml/blob/main/notebooks/madewithml.ipynb

This Python snippet executes the `fit()` method on a `tuner` object, which is responsible for starting a model tuning or training process. It typically orchestrates hyperparameter search and distributed execution, returning the final results upon completion.

```python
results = tuner.fit()
```

--------------------------------

### Tokenize Text and Get Embeddings (Python)

Source: https://github.com/gokumohandas/made-with-ml/blob/main/notebooks/madewithml.ipynb

Demonstrates how to tokenize a sample text using a tokenizer, convert the output to PyTorch tensors, and then pass it through the pre-trained LLM to obtain sequence and pooled embeddings. The output shape indicates the batch size, sequence length, and embedding dimension.

```python
# Sample
text = "Transfer learning with transformers for text classification."
batch = tokenizer([text], return_tensors="np", padding="longest")
batch = {k:torch.tensor(v) for k,v in batch.items()}  # convert to torch tensors
seq, pool = llm(input_ids=batch["input_ids"], attention_mask=batch["attention_mask"])
np.shape(seq), np.shape(pool)
```

--------------------------------

### Initialize and Apply Custom Preprocessor

Source: https://github.com/gokumohandas/made-with-ml/blob/main/notebooks/madewithml.ipynb

This snippet demonstrates initializing a CustomPreprocessor, fitting it to the training dataset, and then transforming both training and validation datasets. It also includes materializing the datasets after transformation.

```python
preprocessor = CustomPreprocessor()
preprocessor = preprocessor.fit(train_ds)
train_ds = preprocessor.transform(train_ds)
val_ds = preprocessor.transform(val_ds)
train_ds = train_ds.materialize()
val_ds = val_ds.materialize()
```

--------------------------------

### Anyscale Production Deployment Commands

Source: https://context7.com/gokumohandas/made-with-ml/llms.txt

This section outlines the command-line interface (CLI) commands for deploying and managing models using Anyscale. It covers submitting training jobs, rolling out services, querying deployed services, rolling back to previous versions, and terminating services.

```bash
# Submit training job
anyscale job submit deploy/jobs/workloads.yaml

# Deploy service
anyscale service rollout -f deploy/services/serve_model.yaml

# Query production service
curl -X POST -H "Content-Type: application/json" \
    -H "Authorization: Bearer $SECRET_TOKEN" \
    -d '{
      "title": "Transfer learning with transformers",
      "description": "Using transformers for transfer learning on text classification tasks."
    }' $SERVICE_ENDPOINT/predict/

# Rollback to previous version
anyscale service rollback -f $SERVICE_CONFIG --name $SERVICE_NAME

# Terminate service
anyscale service terminate --name $SERVICE_NAME
```

--------------------------------

### Inference and Prediction CLI

Source: https://context7.com/gokumohandas/made-with-ml/llms.txt

Command-line interface for getting the best run ID from MLflow experiments and making predictions on new data.

```APIDOC
## Get best run ID and evaluate on holdout set

This section describes how to retrieve the best run ID from an MLflow experiment and then use it to evaluate the model on a holdout dataset.

### Method

`GET` (Implicitly through script execution)

### Endpoint

`madewithml/predict.py` and `madewithml/evaluate.py`

### Parameters

#### Script: `madewithml/predict.py` (get-best-run-id)

- **--experiment-name** (string) - Required - The name of the MLflow experiment.
- **--metric** (string) - Required - The metric to use for determining the best run (e.g., `val_loss`).
- **--mode** (string) - Required - The mode for selecting the best run (`ASC` for ascending, `DESC` for descending).

#### Script: `madewithml/evaluate.py`

- **--run-id** (string) - Required - The ID of the MLflow run to evaluate.
- **--dataset-loc** (string) - Required - The location (URL or path) of the dataset for evaluation.
- **--results-fp** (string) - Required - The file path to save the evaluation results.

### Request Example

```bash
export EXPERIMENT_NAME="llm"
export RUN_ID=$(python madewithml/predict.py get-best-run-id \
    --experiment-name $EXPERIMENT_NAME \
    --metric val_loss \
    --mode ASC)
export HOLDOUT_LOC="https://raw.githubusercontent.com/GokuMohandas/Made-With-ML/main/datasets/holdout.csv"

python madewithml/evaluate.py \
    --run-id $RUN_ID \
    --dataset-loc $HOLDOUT_LOC \
    --results-fp results/evaluation_results.json
```

### Response

#### Success Response (File Output)

The evaluation results are saved to the specified `--results-fp`.

#### Response Example (results/evaluation_results.json):

```json
{
  "timestamp": "June 09, 2023 09:26:18 AM",
  "run_id": "6149e3fec8d24f1492d4a4cabd5c06f6",
  "overall": {
    "precision": 0.9076136428670714,
    "recall": 0.9057591623036649,
    "f1": 0.9046792827719773,
    "num_samples": 191.0
  },
  "per_class": {
    "natural-language-processing": {"precision": 0.92, "recall": 0.94, "f1": 0.93},
    "computer-vision": {"precision": 0.89, "recall": 0.88, "f1": 0.88}
  },
  "slices": {
    "nlp_llm": {"precision": 0.95, "recall": 0.94, "f1": 0.94, "num_samples": 25},
    "short_text": {"precision": 0.78, "recall": 0.75, "f1": 0.76, "num_samples": 18}
  }
}
```
```

--------------------------------

### Set Random Seeds for Reproducibility

Source: https://github.com/gokumohandas/made-with-ml/blob/main/notebooks/madewithml.ipynb

Ensures reproducibility by setting random seeds for various libraries. This is a common practice before starting training or experiments.

```python
# Set up
set_seeds()
```

--------------------------------

### Zero-Shot Prediction with GPT-3.5 (Python)

Source: https://github.com/gokumohandas/made-with-ml/blob/main/notebooks/benchmarks.ipynb

Performs zero-shot learning evaluation using the 'gpt-3.5-turbo-0613' model. It sets up system content to guide the model and then calls the evaluate function.

```python
system_content = f"""
    You are a NLP prediction service that predicts the label given an input's title and description. 
    You must choose between one of the following labels for each input: {tags}. 
    Only respond with the label name and nothing else.
    """
# Zero-shot with GPT 3.5
method = "zero_shot"
model = "gpt-3.5-turbo-0613"
y_pred[method][model], performance[method][model] = evaluate(
    test_df=test_df, model=model, system_content=system_content,
    assistant_content="", tags=tags)
```

--------------------------------

### Hyperparameter Tuning with Ray Tune

Source: https://context7.com/gokumohandas/made-with-ml/llms.txt

Performs hyperparameter optimization using Ray Tune with HyperOptSearch and AsyncHyperBandScheduler. This snippet demonstrates setting up environment variables and running the tuning process to find optimal parameters for dropout, learning rate, and scheduler. It searches over a defined parameter space.

```bash
# Set environment variables
export EXPERIMENT_NAME="llm"
export DATASET_LOC="https://raw.githubusercontent.com/GokuMohandas/Made-With-ML/main/datasets/dataset.csv"
export TRAIN_LOOP_CONFIG='{"dropout_p": 0.5, "lr": 1e-4, "lr_factor": 0.8, "lr_patience": 3}'
export INITIAL_PARAMS="[{"train_loop_config": $TRAIN_LOOP_CONFIG}]"

# Run hyperparameter tuning
python madewithml/tune.py \
    --experiment-name "$EXPERIMENT_NAME" \
    --dataset-loc "$DATASET_LOC" \
    --initial-params "$INITIAL_PARAMS" \
    --num-runs 2 \
    --num-workers 1 \
    --cpu-per-worker 3 \
    --gpu-per-worker 1 \
    --num-epochs 10 \
    --batch-size 256 \
    --results-fp results/tuning_results.json

# Tuning searches over parameter space:
# - dropout_p: uniform(0.3, 0.9)
# - lr: loguniform(1e-5, 5e-4)
# - lr_factor: uniform(0.1, 0.9)
# - lr_patience: uniform(1, 10)
```

--------------------------------

### Build and Deploy Anyscale Cluster Environment

Source: https://github.com/gokumohandas/made-with-ml/blob/main/README.md

Builds and deploys a cluster environment configuration for Anyscale using a specified YAML file. This defines the OS and dependencies for the cluster.

```bash
export CLUSTER_ENV_NAME="madewithml-cluster-env"
anyscale cluster-env build deploy/cluster_env.yaml --name $CLUSTER_ENV_NAME
```

--------------------------------

### Get Ray Train Device in Python

Source: https://github.com/gokumohandas/made-with-ml/blob/main/notebooks/madewithml.ipynb

Retrieves the appropriate device (CPU or GPU) for Ray Train operations. This is essential for ensuring that computations are performed on the correct hardware.

```python
from ray.train.torch import get_device
```

--------------------------------

### Zero-Shot Prediction with GPT-4 (Python)

Source: https://github.com/gokumohandas/made-with-ml/blob/main/notebooks/benchmarks.ipynb

Performs zero-shot learning evaluation using the 'gpt-4-0613' model. Similar to the GPT-3.5 example, it configures the system prompt and executes the evaluation pipeline.

```python
# Zero-shot with GPT 4
method = "zero_shot"
model = "gpt-4-0613"
y_pred[method][model], performance[method][model] = evaluate(
    test_df=test_df, model=model, system_content=system_content,
    assistant_content="", tags=tags)
```

--------------------------------

### Run Ray Serve Service

Source: https://github.com/gokumohandas/made-with-ml/blob/main/notebooks/madewithml.ipynb

This snippet initializes and runs a Ray Serve application, binding a specific model deployment with a given threshold. It sets the route prefix for the service. Dependencies include the 'serve' library and a 'ModelDeploymentRobust' class.

```python
serve.run(ModelDeploymentRobust.bind(run_id=run_id, threshold=0.9), route_prefix="/")
```

--------------------------------

### Initialize and Configure Ray

Source: https://github.com/gokumohandas/made-with-ml/blob/main/notebooks/madewithml.ipynb

Initializes a local Ray instance, shutting it down if already running. It also loads environment variables and sets up autoreload for development. The output shows the Ray dashboard URL and version information.

```python
import sys; sys.path.append("..")
import warnings; warnings.filterwarnings("ignore")
from dotenv import load_dotenv; load_dotenv()
%load_ext autoreload
%autoreload 2
```

```python
# Initialize Ray
if ray.is_initialized():
    ray.shutdown()
ray.init()
```

--------------------------------

### Get Most Common Items with Counter

Source: https://github.com/gokumohandas/made-with-ml/blob/main/notebooks/madewithml.ipynb

Uses the Counter object to find and return the most frequent items from a list or iterable. This is commonly used to analyze the distribution of tags or other categorical features.

```python
# Most common tags
all_tags = Counter(df.tag)
all_tags.most_common()
```

--------------------------------

### Custom Preprocessor Fit and Transform

Source: https://github.com/gokumohandas/made-with-ml/blob/main/notebooks/madewithml.ipynb

This snippet demonstrates how to initialize, fit, and transform datasets using a CustomPreprocessor. It first creates an instance of the preprocessor, fits it to the training dataset, and then applies the transformation to both training and validation datasets. Finally, it materializes the transformed datasets.

```python
preprocessor = CustomPreprocessor()
preprocessor =  preprocessor.fit(train_ds)
train_ds = preprocessor.transform(train_ds)
val_ds = preprocessor.transform(val_ds)
train_ds = train_ds.materialize()
val_ds = val_ds.materialize()
```

--------------------------------

### Configure Anyscale Service Runtime Environment

Source: https://github.com/gokumohandas/made-with-ml/blob/main/README.md

Sets up the runtime environment for Anyscale Services, defining the import path for the service entrypoint, the working directory, and an S3 upload path for service artifacts. It also configures environment variables like GITHUB_USERNAME.

```yaml
ray_serve_config:
  import_path: deploy.services.serve_model:entrypoint
  runtime_env:
    working_dir: .
    upload_path: s3://madewithml/$GITHUB_USERNAME/services  # <--- CHANGE USERNAME (case-sensitive)
    env_vars:
      GITHUB_USERNAME: $GITHUB_USERNAME  # <--- CHANGE USERNAME (case-sensitive)
```

--------------------------------

### Clone Repository using Git

Source: https://github.com/gokumohandas/made-with-ml/blob/main/README.md

Clones the Made-With-ML repository from GitHub to the local machine. This command initializes the project directory with all necessary code and configuration files.

```bash
git clone https://github.com/GokuMohandas/Made-With-ML.git .
```

--------------------------------

### Configure Ray Worker Nodes

Source: https://github.com/gokumohandas/made-with-ml/blob/main/notebooks/madewithml.ipynb

Defines the number of worker nodes and their resource allocation (CPU and GPU). This configuration is essential for setting up distributed training or computation, with specific examples for GPU and CPU-only environments.

```python
# Workers (1 g4dn.xlarge)
num_workers = 1
resources_per_worker={"CPU": 3, "GPU": 1}
```

```python
# If you are running this on a local laptop (no GPU), use the CPU count from `ray.cluster_resources()` to set your resources. For example if your machine has 10 CPUs:
num_workers = 6  # prefer to do a few less than total available CPU (1 for head node + 1 for background tasks)
resources_per_worker={"CPU": 1, "GPU": 0}
```

--------------------------------

### Query Deployed Model Service for Predictions

Source: https://github.com/gokumohandas/made-with-ml/blob/main/notebooks/madewithml.ipynb

Sends a POST request to the deployed Ray Serve model endpoint to get predictions. It formats the input data as JSON and includes a title and description for the prediction task.

```python
import json
import requests

title = "Transfer learning with transformers"
description = "Using transformers for transfer learning on text classification tasks."
json_data = json.dumps({"title": title, "description": description})
requests.post("http://127.0.0.1:8000/predict/", data=json_data).json()
```

--------------------------------

### Create Anyscale Compute Configuration

Source: https://github.com/gokumohandas/made-with-ml/blob/main/README.md

Creates a compute configuration for Anyscale workloads using a specified YAML file. This defines the resources (e.g., instance type) that the workloads will run on.

```bash
export CLUSTER_COMPUTE_NAME="madewithml-cluster-compute-g5.4xlarge"
anyscale cluster-compute create deploy/cluster_compute.yaml --name $CLUSTER_COMPUTE_NAME
```

--------------------------------

### Get Best Trial Hyperparameters - Python

Source: https://github.com/gokumohandas/made-with-ml/blob/main/notebooks/madewithml.ipynb

Extracts the configuration dictionary for the best trial, specifically focusing on the 'train_loop_config'. This code assumes the 'best_trial' object has a 'config' attribute which is a dictionary containing training loop parameters.

```python
# Best trial's hyperparameters
best_trial.config["train_loop_config"]
```

--------------------------------

### Checkpoint Creation During Training

Source: https://github.com/gokumohandas/made-with-ml/blob/main/notebooks/madewithml.ipynb

This section demonstrates the successful creation of checkpoints during the training process. It highlights the path where the checkpoints are saved and the associated metadata. The output also includes warnings about NumPy array writability.

```python
# Checkpoint successfully created at: Checkpoint(filesystem=local, path=/efs/shared_storage/madewithml/GokuMohandas/TorchTrainer_2023-09-17_22-42-09/TorchTrainer_1bd07_00000_0_2023-09-17_22-42-09/checkpoint_000000)

```

```python
# Checkpoint successfully created at: Checkpoint(filesystem=local, path=/efs/shared_storage/madewithml/GokuMohandas/TorchTrainer_2023-09-17_22-42-09/TorchTrainer_1bd07_00000_0_2023-09-17_22-42-09/checkpoint_000001)

```

```python
# Checkpoint successfully created at: Checkpoint(filesystem=local, path=/efs/shared_storage/madewithml/GokuMohandas/TorchTrainer_2023-09-17_22-42-09/TorchTrainer_1bd07_00000_0_2023-09-17_22-42-09/checkpoint_000002)

```

```python
# Checkpoint successfully created at: Checkpoint(filesystem=local, path=/efs/shared_storage/madewithml/GokuMohandas/TorchTrainer_2023-09-17_22-42-09/TorchTrainer_1bd07_00000_0_2023-09-17_22-42-09/checkpoint_000003)

```

--------------------------------

### Query OpenAI Chat Completion Endpoint

Source: https://github.com/gokumohandas/made-with-ml/blob/main/notebooks/benchmarks.ipynb

This snippet demonstrates how to send a chat completion request to the OpenAI API. It specifies system, assistant, and user content to guide the model's response. The response is then printed, extracting the content from the message.

```python
import openai

system_content = "you only answer in rhymes"  # system content (behavior)
assistant_content = ""  # assistant content (context)
user_content = "how are you"  # user content (message)
response = openai.ChatCompletion.create(
    model="gpt-3.5-turbo-0613",
    messages=[
        {"role": "system", "content": system_content},
        {"role": "assistant", "content": assistant_content},
        {"role": "user", "content": user_content},
    ],
)
print (response.to_dict()["choices"][0].to_dict()["message"]["content"])
```

--------------------------------

### Model Serving with Ray Serve

Source: https://context7.com/gokumohandas/made-with-ml/llms.txt

Deploying models as a REST API using Ray Serve with FastAPI integration, providing health check, run ID, evaluation, and prediction endpoints.

```APIDOC
## Model Serving with Ray Serve

This section describes how to deploy ML models as a REST API using Ray Serve and FastAPI.

### Method

`POST` (for deployment command)

### Endpoint

`madewithml/serve.py`

### Parameters

- **--run-id** (string) - Required - The MLflow run ID of the model to deploy.
- **--threshold** (float) - Optional - The confidence threshold for predictions. Defaults to 0.9.

### Request Example (Deployment)

```bash
# Start Ray and deploy model
ray start --head
export EXPERIMENT_NAME="llm"
export RUN_ID=$(python madewithml/predict.py get-best-run-id \
    --experiment-name $EXPERIMENT_NAME \
    --metric val_loss \
    --mode ASC)
python madewithml/serve.py --run_id $RUN_ID --threshold 0.9
```

### API Endpoints (Served via Ray Serve)

#### Health Check

- **Method**: `GET`
- **Endpoint**: `/`
- **Description**: Checks the health status of the deployed service.
- **Response Example**: `{"message": "OK", "status-code": 200, "data": {}}`

#### Get Run ID

- **Method**: `GET`
- **Endpoint**: `/run_id/`
- **Description**: Retrieves the MLflow run ID of the deployed model.
- **Response Example**: `{"run_id": "6149e3fec8d24f1492d4a4cabd5c06f6"}`

#### Predict

- **Method**: `POST`
- **Endpoint**: `/predict/`
- **Description**: Makes predictions on new input data.
- **Request Body**: JSON object with `title` and `description` fields.
  - **title** (string) - Required - The title of the input text.
  - **description** (string) - Required - The description of the input text.
- **Request Example (cURL)**:
  ```bash
  curl -X POST "http://127.0.0.1:8000/predict/" \
      -H "Content-Type: application/json" \
      -d '{"title": "Transfer learning with transformers", "description": "Using transformers for transfer learning."}'
  ```
- **Response Example**: `{"results": [{"prediction": "natural-language-processing", "probabilities": {...}}]}`

#### Evaluate

- **Method**: `POST`
- **Endpoint**: `/evaluate/`
- **Description**: Evaluates the model on a given dataset.
- **Request Body**: JSON object with a `dataset` field.
  - **dataset** (string) - Required - The location (URL or path) of the dataset for evaluation.
- **Request Example (Python Client)**:
  ```python
  import json
  import requests
  eval_data = json.dumps({"dataset": "https://raw.githubusercontent.com/.../holdout.csv"})
  response = requests.post("http://127.0.0.1:8000/evaluate/", data=eval_data)
  print(response.json())
  ```
- **Response Example**: `{"results": {"overall": {...}, "per_class": {...}, "slices": {...}}}`

### Shutdown Ray

```bash
ray stop
```
```

--------------------------------

### Load and Preprocess Data with Ray Datasets

Source: https://context7.com/gokumohandas/made-with-ml/llms.txt

Loads datasets into Ray Datasets, performs stratified splits, cleans text, tokenizes using SciBERT, and applies a custom preprocessor. This snippet demonstrates data loading from a URL, splitting, and transforming data for ML tasks.

```python
import ray
from madewithml import data

# Load dataset from CSV URL
dataset_loc = "https://raw.githubusercontent.com/GokuMohandas/Made-With-ML/main/datasets/dataset.csv"
ds = data.load_data(dataset_loc=dataset_loc, num_samples=100)

# Stratified split by tag column
train_ds, val_ds = data.stratify_split(
    ds=ds,
    stratify="tag",
    test_size=0.2,
    shuffle=True,
    seed=1234
)

# Get unique tags
tags = train_ds.unique(column="tag")
print(f"Classes: {tags}")
# Output: Classes: ['natural-language-processing', 'computer-vision', 'mlops', 'other']

# Create and fit preprocessor
preprocessor = data.CustomPreprocessor()
preprocessor = preprocessor.fit(train_ds)
print(f"Class mapping: {preprocessor.class_to_index}")
# Output: Class mapping: {'natural-language-processing': 0, 'computer-vision': 1, 'mlops': 2, 'other': 3}

# Transform datasets
train_ds = preprocessor.transform(train_ds)
val_ds = preprocessor.transform(val_ds)

# Clean text example
text = "Transfer Learning with Transformers! Using BERT for NLP tasks."
cleaned = data.clean_text(text)
print(f"Cleaned: {cleaned}")
# Output: Cleaned: transfer learning transformers using bert nlp tasks
```

--------------------------------

### Get Best Run ID and Evaluate Model

Source: https://context7.com/gokumohandas/made-with-ml/llms.txt

This script retrieves the best run ID from an MLflow experiment based on a specified metric and then evaluates the model associated with that run ID on a holdout dataset. The evaluation results are saved to a JSON file.

```bash
export EXPERIMENT_NAME="llm"
export RUN_ID=$(python madewithml/predict.py get-best-run-id \
    --experiment-name $EXPERIMENT_NAME \
    --metric val_loss \
    --mode ASC)
export HOLDOUT_LOC="https://raw.githubusercontent.com/GokuMohandas/Made-With-ML/main/datasets/holdout.csv"

python madewithml/evaluate.py \
    --run-id $RUN_ID \
    --dataset-loc $HOLDOUT_LOC \
    --results-fp results/evaluation_results.json
```

--------------------------------

### Configure Experiment Run Settings with Checkpointing

Source: https://github.com/gokumohandas/made-with-ml/blob/main/notebooks/madewithml.ipynb

This snippet sets up the RunConfig for an experiment, defining the experiment name ('llm'), checkpointing strategy (keeping the best 1 checkpoint based on validation loss), and the storage path for checkpoints using EFS_DIR.

```python
# Run config
checkpoint_config = CheckpointConfig(num_to_keep=1, checkpoint_score_attribute="val_loss", checkpoint_score_order="min")
run_config = RunConfig(name="llm", checkpoint_config=checkpoint_config, storage_path=EFS_DIR)
```

--------------------------------

### Define Hyperparameter Search Space

Source: https://github.com/gokumohandas/made-with-ml/blob/main/notebooks/madewithml.ipynb

Specifies the ranges and distributions for hyperparameters to be tuned. This example defines a search space for dropout probability, learning rate, learning rate factor, and learning rate patience within the training loop configuration.

```python
import ray.tune as tune

param_space = {
    "train_loop_config": {
        "dropout_p": tune.uniform(0.3, 0.9),
        "lr": tune.loguniform(1e-5, 5e-4),
        "lr_factor": tune.uniform(0.1, 0.9),
        "lr_patience": tune.uniform(1, 10),
    }
}

```

--------------------------------

### Get Best Trial Epochs Metrics - Python

Source: https://github.com/gokumohandas/made-with-ml/blob/main/notebooks/madewithml.ipynb

Retrieves the metrics dataframe for the best trial, identified by the minimum validation loss. This code assumes a 'results' object with a 'get_best_result' method and that the best trial's metrics are stored in a pandas DataFrame.

```python
# Best trial's epochs
best_trial = results.get_best_result(metric="val_loss", mode="min")
best_trial.metrics_dataframe
```

--------------------------------

### Ray Data Initialization with Python

Source: https://github.com/gokumohandas/made-with-ml/blob/main/notebooks/madewithml.ipynb

This code snippet demonstrates how to import necessary components from the Ray Data library in Python. It includes importing the main `ray.data` module and the `ActorPoolStrategy` for configuring execution. These are foundational for setting up data processing pipelines with Ray.

```python
import ray.data
from ray.data import ActorPoolStrategy
```

--------------------------------

### Initialize Ray Tune Tuner

Source: https://github.com/gokumohandas/made-with-ml/blob/main/notebooks/madewithml.ipynb

Initializes the Ray Tune Tuner, which orchestrates the hyperparameter search process. It takes the trainable component (trainer), run configuration, parameter space, and tune configuration as input.

```python
from ray.tune import Tuner

# Assuming trainer, run_config, param_space, tune_config are defined elsewhere
tuner = Tuner(
    trainable=trainer,
    run_config=run_config,
    param_space=param_space,
    tune_config=tune_config,
)

```

--------------------------------

### Initialize Pretrained LLM and Get Embedding Dimension (Python)

Source: https://github.com/gokumohandas/made-with-ml/blob/main/notebooks/madewithml.ipynb

Initializes a pre-trained BERT model from Hugging Face's transformers library and retrieves its hidden size, which represents the embedding dimension. This is a common first step for many NLP tasks.

```python
# Pretrained LLM
llm = BertModel.from_pretrained("allenai/scibert_scivocab_uncased", return_dict=False)
embedding_dim = llm.config.hidden_size
```

--------------------------------

### TorchTrainer Initialization

Source: https://github.com/gokumohandas/made-with-ml/blob/main/notebooks/madewithml.ipynb

Initializes a TorchTrainer for distributed training. It specifies the training loop, configuration, scaling, run settings, datasets, and metadata.

```python
# Trainer
trainer = TorchTrainer(
    train_loop_per_worker=train_loop_per_worker,
    train_loop_config=train_loop_config,
    scaling_config=scaling_config,
    run_config=run_config,
    datasets={"train": train_ds, "val": val_ds},
    dataset_config=dataset_config,
    metadata={"class_to_index": preprocessor.class_to_index}
)

```

--------------------------------

### FinetunedLLM Model Definition and Usage

Source: https://context7.com/gokumohandas/made-with-ml/llms.txt

Defines the `FinetunedLLM` class, a PyTorch model for multi-class text classification. It utilizes a pre-trained SciBERT model and adds a classification head. The example shows model instantiation, a forward pass with sample data, and prediction of class indices.

```python
import torch
from transformers import BertModel
from madewithml.models import FinetunedLLM

# Create model
llm = BertModel.from_pretrained("allenai/scibert_scivocab_uncased", return_dict=False)
model = FinetunedLLM(
    llm=llm,
    dropout_p=0.5,
    embedding_dim=llm.config.hidden_size,  # 768
    num_classes=4
)

# Forward pass with batch
batch = {
    "ids": torch.tensor([[101, 2478, 102, 0], [101, 3456, 7890, 102]]),
    "masks": torch.tensor([[1, 1, 1, 0], [1, 1, 1, 1]]),
    "targets": torch.tensor([0, 1])
}
logits = model(batch)
print(f"Logits shape: {logits.shape}")
# Output: Logits shape: torch.Size([2, 4])

# Predict class indices
predictions = model.predict(batch)
print(f"Predictions: {predictions}")
# Output: Predictions: [0 1]
```

--------------------------------

### Configure Run with MLflow Callback

Source: https://github.com/gokumohandas/made-with-ml/blob/main/notebooks/madewithml.ipynb

Sets up the run configuration for Ray Tune, including the MLflow callback for experiment tracking. It specifies checkpointing, storage paths, and local directories for the training run.

```python
from ray.tune.run_config import RunConfig

run_config = RunConfig(
    callbacks=[mlflow_callback],
    checkpoint_config=checkpoint_config,
    storage_path=EFS_DIR,
    local_dir=EFS_DIR
)
```

--------------------------------

### Get Single Tag Prediction from OpenAI

Source: https://github.com/gokumohandas/made-with-ml/blob/main/notebooks/benchmarks.ipynb

This Python function, `get_tag`, queries the OpenAI ChatCompletion API to predict a tag for a given input. It handles potential service unavailability and API errors by returning None. The function is designed to work with a specified model and message roles.

```python
import openai

def get_tag(model, system_content="", assistant_content="", user_content=""):
    try:
        # Get response from OpenAI
        response = openai.ChatCompletion.create(
            model=model,
            messages=[
                {"role": "system", "content": system_content},
                {"role": "assistant", "content": assistant_content},
                {"role": "user", "content": user_content},
            ],
        )
        predicted_tag = response.to_dict()["choices"][0].to_dict()["message"]["content"]
        return predicted_tag

    except (openai.error.ServiceUnavailableError, openai.error.APIError) as e:
        return None
```

--------------------------------

### Explain Text Instance with LIME

Source: https://github.com/gokumohandas/made-with-ml/blob/main/notebooks/madewithml.ipynb

Explains a single text instance using the initialized LIME explainer and the defined classifier function. It specifies the text to explain, the classifier function, and the number of top labels to consider. The result is displayed in the notebook.

```python
# Explain instance
text = "Using pretrained convolutional neural networks for object detection."
# Assuming explainer and classifier_fn are defined
explainer.explain_instance(text, classifier_fn=classifier_fn, top_labels=1).show_in_notebook(text=True)
```

--------------------------------

### Configure Anyscale CLI Credentials

Source: https://github.com/gokumohandas/made-with-ml/blob/main/README.md

Sets environment variables for Anyscale CLI authentication, specifying the Anyscale host and the CLI token retrieved from the Anyscale console.

```bash
export ANYSCALE_HOST=https://console.anyscale.com
export ANYSCALE_CLI_TOKEN=$YOUR_CLI_TOKEN  # retrieved from https://console.anyscale.com/o/madewithml/credentials
```

--------------------------------

### Get Best Checkpoint from MLFlow Run

Source: https://github.com/gokumohandas/made-with-ml/blob/main/notebooks/madewithml.ipynb

Retrieves the best checkpoint from a specified MLFlow run. It parses the artifact URI from the run information and loads the checkpoint using Ray Train's Result object. This function is essential for resuming training or deploying models from the best performing run.

```python
from ray.train import Result
from urllib.parse import urlparse
import mlflow

def get_best_checkpoint(run_id):
    artifact_dir = urlparse(mlflow.get_run(run_id).info.artifact_uri).path  # get path from mlflow
    results = Result.from_path(artifact_dir)
    return results.best_checkpoints[0][0]
```

--------------------------------

### Train Model using Bash

Source: https://github.com/gokumohandas/made-with-ml/blob/main/README.md

This script trains a machine learning model. It sets environment variables for the experiment name, dataset location, and training loop configuration, then executes the training script with specified hyperparameters and output paths. Dependencies include the 'madewithml' Python package.

```bash
export EXPERIMENT_NAME="llm"
export DATASET_LOC="https://raw.githubusercontent.com/GokuMohandas/Made-With-ML/main/datasets/dataset.csv"
export TRAIN_LOOP_CONFIG='{"dropout_p": 0.5, "lr": 1e-4, "lr_factor": 0.8, "lr_patience": 3}'
python madewithml/train.py \
    --experiment-name "$EXPERIMENT_NAME" \
    --dataset-loc "$DATASET_LOC" \
    --train-loop-config "$TRAIN_LOOP_CONFIG" \
    --num-workers 1 \
    --cpu-per-worker 3 \
    --gpu-per-worker 1 \
    --num-epochs 10 \
    --batch-size 256 \
    --results-fp results/training_results.json
```

--------------------------------

### Create .env file for Credentials

Source: https://github.com/gokumohandas/made-with-ml/blob/main/README.md

Creates a .env file to store environment-specific credentials, such as the GitHub username. This file should be kept private and not committed to version control.

```bash
touch .env
```

```bash
# Inside .env
GITHUB_USERNAME="CHANGE_THIS_TO_YOUR_USERNAME"  # ← CHANGE THIS
```

```bash
source .env
```

--------------------------------

### Ray Data Execution Log - Initial Processing

Source: https://github.com/gokumohandas/made-with-ml/blob/main/notebooks/madewithml.ipynb

This log details the initial execution flow of a Ray Data pipeline. It shows the DAG structure, starting from input data buffering, reading CSV, splitting blocks, shuffling, sorting, and applying batch transformations. The log also provides execution configuration and tips for verbose progress reporting.

```log
2023-09-18 21:57:51,011	INFO streaming_executor.py:93 -- Executing DAG InputDataBuffer[Input] -> TaskPoolMapOperator[ReadCSV->SplitBlocks(64)] -> AllToAllOperator[RandomShuffle] -> AllToAllOperator[Sort] -> AllToAllOperator[MapBatches(group_fn)->MapBatches(_filter_split)->RandomShuffle] -> LimitOperator[limit=1]
2023-09-18 21:57:51,012	INFO streaming_executor.py:94 -- Execution config: ExecutionOptions(resource_limits=ExecutionResources(cpu=None, gpu=None, object_store_memory=None), locality_with_output=False, preserve_order=True, actor_locality_enabled=True, verbose_progress=False)
2023-09-18 21:57:51,012	INFO streaming_executor.py:96 -- Tip: For detailed progress reporting, run `ray.data.DataContext.get_current().execution_options.verbose_progress = True`
```

--------------------------------

### Initialize Git Repository for CI/CD

Source: https://github.com/gokumohandas/made-with-ml/blob/main/README.md

Sets the remote origin URL for a Git repository and checks out a new branch named 'dev'. This is a preliminary step for automating CI/CD workflows.

```bash
git remote set-url origin https://github.com/$GITHUB_USERNAME/Made-With-ML.git  # <-- CHANGE THIS to your username
git checkout -b dev
```

--------------------------------

### Training Timing and Summary Log

Source: https://github.com/gokumohandas/made-with-ml/blob/main/notebooks/madewithml.ipynb

This log output provides a summary of the training run, including CPU times (user, system, total) and wall time. The wall time indicates the total elapsed time for the training process.

```log
CPU times: user 1.23 s, sys: 1.31 s, total: 2.53 s
Wall time: 1min 21s
```

--------------------------------

### Initialize LIME Text Explainer

Source: https://github.com/gokumohandas/made-with-ml/blob/main/notebooks/madewithml.ipynb

Initializes the LimeTextExplainer from the LIME library. This explainer is used to understand the importance of different words in a text for a given prediction. It requires the class names that the model predicts.

```python
from lime.lime_text import LimeTextExplainer
from sklearn.pipeline import make_pipeline

# Assuming class_to_index is defined elsewhere
explainer = LimeTextExplainer(class_names=list(class_to_index.keys()))
```

--------------------------------

### Configure Run Configuration for Training

Source: https://github.com/gokumohandas/made-with-ml/blob/main/notebooks/madewithml.ipynb

Defines the run configuration for the training process, including callbacks, checkpointing strategy, and storage directories. It specifies keeping only the best checkpoint based on validation loss.

```python
from ray.air.config import RunConfig, CheckpointConfig

# Assuming mlflow_callback and EFS_DIR are defined elsewhere
checkpoint_config = CheckpointConfig(num_to_keep=1, checkpoint_score_attribute="val_loss", checkpoint_score_order="min")
run_config = RunConfig(
    callbacks=[mlflow_callback],
    checkpoint_config=checkpoint_config,
    storage_path=EFS_DIR,
    local_dir=EFS_DIR)

```

--------------------------------

### Manage Anyscale Services

Source: https://github.com/gokumohandas/made-with-ml/blob/main/README.md

Provides commands for managing Anyscale Services, including rolling out new versions, querying predictions, rolling back to previous versions, and terminating services.

```bash
# Rollout service
anyscale service rollout -f deploy/services/serve_model.yaml

# Query
curl -X POST -H "Content-Type: application/json" -H "Authorization: Bearer $SECRET_TOKEN" -d '{
  "title": "Transfer learning with transformers",
  "description": "Using transformers for transfer learning on text classification tasks."
}' $SERVICE_ENDPOINT/predict/

# Rollback (to previous version of the Service)
anyscale service rollback -f $SERVICE_CONFIG --name $SERVICE_NAME

# Terminate
anyscale service terminate --name $SERVICE_NAME
```