### Install Hopsworks via Kubernetes Installer

Source: https://github.com/logicalclocks/hopsworks/blob/master/README.md

Downloads and executes the Hopsworks Kubernetes installation script. This command requires a machine with sudo privileges and internet access to fetch the installer.

```bash
curl -O https://raw.githubusercontent.com/logicalclocks/hopsworks-k8s-installer/master/install-hopsworks.py
python3 install-hopsworks.py
```

--------------------------------

### Start Deployed Iris Flower Classifier for Predictions

Source: https://github.com/logicalclocks/hopsworks/blob/master/hopsworks-IT/src/test/ruby/spec/auxiliary/ml/notebooks/serving/kserve/sklearn/model_serving_kserve_sklearn.ipynb

This Python command initiates the deployed 'irisflowerclassifier' model, making it ready to receive prediction requests. The `start()` method on the deployment object typically handles the necessary setup for the model server to become active.

```python
irisclassifier.start()
```

--------------------------------

### Spark Job Configuration Example

Source: https://github.com/logicalclocks/hopsworks/blob/master/docs/compute/jobs.md

This example demonstrates the JSON structure for configuring a PySpark job, including resource allocation, application path, and dependency management.

```APIDOC
## Spark Job Configuration

### Description
This section outlines the configuration parameters for Spark and PySpark jobs. These settings allow for fine-grained control over resource allocation, job execution, and dependency management.

### Method
N/A (Configuration structure)

### Endpoint
N/A (Configuration structure)

### Parameters
#### Request Body
- **type** (string) - Required - Specifies the type of job configuration, e.g., "sparkJobConfiguration".
- **amQueue** (string) - Required - The name of the YARN queue to use for the application master.
- **amMemory** (integer) - Required - Memory in megabytes for the Spark driver (application master).
- **amVCores** (integer) - Required - Virtual cores for the Spark driver (application master).
- **jobType** (string) - Required - Type of the Spark job ('SPARK' for JARs, 'PYSPARK' for Python scripts/notebooks).
- **appPath** (string) - Required - HDFS path to the application file (.jar, .py, or .ipynb).
- **mainClass** (string) - Required - The main class for JAR applications, or 'org.apache.spark.deploy.PythonRunner' for PySpark jobs.
- **spark.executor.instances** (integer) - Optional - Number of executor instances to use (static allocation).
- **spark.executor.cores** (integer) - Optional - Number of cores per executor.
- **spark.executor.memory** (integer) - Optional - Memory in megabytes per executor.
- **spark.executor.gpus** (integer) - Optional - Number of GPUs per executor.
- **spark.tensorflow.num.ps** (integer) - Optional - Number of parameter server instances for TensorFlow jobs.
- **spark.dynamicAllocation.enabled** (boolean) - Optional - Enables dynamic allocation of executors.
- **spark.dynamicAllocation.minExecutors** (integer) - Optional - Minimum number of executors for dynamic allocation.
- **spark.dynamicAllocation.maxExecutors** (integer) - Optional - Maximum number of executors for dynamic allocation.
- **spark.dynamicAllocation.initialExecutors** (integer) - Optional - Initial number of executors for dynamic allocation.
- **spark.blacklist.enabled** (boolean) - Optional - Enables executor blacklisting.
- **properties** (string) - Optional - New-line separated properties for the Spark application (e.g., environment variables).
- **spark.yarn.dist.pyFiles** (string) - Optional - Comma-separated paths to additional Python files.
- **spark.yarn.dist.files** (string) - Optional - Comma-separated paths to additional application files.
- **spark.yarn.dist.jars** (string) - Optional - Comma-separated paths to additional JAR files for the classpath.
- **spark.yarn.dist.archives** (string) - Optional - Comma-separated paths to additional archives to be extracted.
- **appName** (string) - Required - Name of the Spark job.
- **defaultArgs** (string) - Optional - A string containing the default arguments for the application.

### Request Example
```json
{
    "type": "sparkJobConfiguration",
    "amQueue": "default",
    "amMemory": 2048,
    "amVCores": 1,
    "jobType": "PYSPARK",
    "appPath": "hdfs:///Projects/{PROJECT_NAME}/{DATASET_NAME}/{APP_FILE}",
    "mainClass": "org.apache.spark.deploy.PythonRunner",
    "spark.executor.instances": 1,
    "spark.executor.cores": 1,
    "spark.executor.memory": 2048,
    "spark.executor.gpus": 0,
    "spark.tensorflow.num.ps": 1,
    "spark.dynamicAllocation.enabled": true,
    "spark.dynamicAllocation.minExecutors": 0,
    "spark.dynamicAllocation.maxExecutors": 1,
    "spark.dynamicAllocation.initialExecutors": 0,
    "spark.blacklist.enabled": false,
    "properties": "",
    "spark.yarn.dist.pyFiles": "",
    "spark.yarn.dist.files": "",
    "spark.yarn.dist.jars": "",
    "spark.yarn.dist.archives": "",
    "appName":"{JOB_NAME}",
    "defaultArgs": ""
}
```

### Response
#### Success Response (200)
- **status** (string) - Indicates the success of the operation.
- **message** (string) - A confirmation message.

#### Response Example
```json
{
  "status": "success",
  "message": "Spark job configuration updated successfully."
}
```
```

--------------------------------

### Configure Integration Test Environment

Source: https://github.com/logicalclocks/hopsworks/blob/master/hopsworks-IT/README.md

Sets up the local environment by copying the example configuration file and setting necessary shell variables.

```bash
cd hopsworks/hopsworks-IT/src/test/ruby/
cp .env.example .env
export GLASSFISH_HOST_NAME=localhost
export GLASSFISH_HTTP_PORT=8181
export GLASSFISH_ADMIN_PORT=4848
```

--------------------------------

### Start New Job Execution

Source: https://context7.com/logicalclocks/hopsworks/llms.txt

Initiates a new execution of a job. Custom arguments can be passed to the job during startup.

```bash
curl -X POST "https://{HOPSWORKS_HOST}/hopsworks-api/api/project/{projectId}/jobs/{jobName}/executions" \
  -H "Authorization: Bearer {JWT_TOKEN}" \
  -H "Content-Type: text/plain" \
  -d "--custom-arg value --another-arg value2"
```

--------------------------------

### Get Storage Connectors (GET)

Source: https://context7.com/logicalclocks/hopsworks/llms.txt

Retrieves information about storage connectors configured for external data sources within a feature store. Requires project, featurestore IDs, and JWT authentication.

```bash
curl -X GET "https://{HOPSWORKS_HOST}/hopsworks-api/api/project/{projectId}/featurestores/{featurestoreId}/storageconnectors" \
  -H "Authorization: Bearer {JWT_TOKEN}"
```

--------------------------------

### Retrieve and Validate Models

Source: https://github.com/logicalclocks/hopsworks/blob/master/hopsworks-IT/src/test/ruby/spec/auxiliary/test_hsml.ipynb

Demonstrates how to fetch specific model versions and validate their training metrics, schema, and input examples.

```python
exported_tf_model = mr.get_model("model_tf")
assert isinstance(exported_tf_model, hsml.tensorflow.model.Model)
assert 'accuracy' in exported_tf_model.training_metrics
assert len(exported_tf_model.model_schema['input_schema']['columnar_schema']) == 3

exported_tf_model_v3 = mr.get_model("model_tf", version=3)
assert exported_tf_model_v3.version == 3
```

--------------------------------

### Configure Alertmanager HA and HTTPS

Source: https://github.com/logicalclocks/hopsworks/blob/master/alerting/README.md

Examples for setting up multiple clients for High Availability clusters and enabling HTTPS for secure communication.

```java
AlertManagerClient client1 = new AlertManagerClient.Builder(ClientBuilder.newClient())
    .withTarget("alertmanager1.prometheus.service.consul")
    .build();
AlertManagerClient client2 = new AlertManagerClient.Builder(ClientBuilder.newClient())
    .withTarget("alertmanager2.prometheus.service.consul")
    .build();

AlertManagerClient httpsClient = new AlertManagerClient.Builder(ClientBuilder.newClient())
    .enableHttps()
    .build();
```

--------------------------------

### List Training Datasets (GET)

Source: https://context7.com/logicalclocks/hopsworks/llms.txt

Retrieves a list of all training datasets within a specified feature store. Requires project and featurestore IDs, along with JWT authentication.

```bash
curl -X GET "https://{HOPSWORKS_HOST}/hopsworks-api/api/project/{projectId}/featurestores/{featurestoreId}/trainingdatasets" \
  -H "Authorization: Bearer {JWT_TOKEN}"
```

--------------------------------

### Setup Kafka Consumer for Prediction Logs

Source: https://github.com/logicalclocks/hopsworks/blob/master/hopsworks-IT/src/test/ruby/spec/auxiliary/ml/notebooks/serving/kserve/python/model_serving_kserve_python.ipynb

Initializes a Kafka consumer and subscribes to the specific topic associated with the model's inference logs. It uses the default Hopsworks Kafka configuration.

```python
from hops import kafka
from confluent_kafka import Producer, Consumer, KafkaError

TOPIC_NAME = irisclassifier.inference_logger.kafka_topic.name

config = kafka.get_kafka_default_config()
config['default.topic.config'] = {'auto.offset.reset': 'earliest'}
consumer = Consumer(config)
topics = [TOPIC_NAME]
consumer.subscribe(topics)
```

--------------------------------

### List All Jobs (GET)

Source: https://context7.com/logicalclocks/hopsworks/llms.txt

Retrieves a list of all jobs defined within a project. Requires project ID and JWT authentication.

```bash
curl -X GET "https://{HOPSWORKS_HOST}/hopsworks-api/api/project/{projectId}/jobs" \
  -H "Authorization: Bearer {JWT_TOKEN}"
```

--------------------------------

### Deploy Model with Custom Transformer to KServe

Source: https://github.com/logicalclocks/hopsworks/blob/master/hopsworks-IT/src/test/ruby/spec/auxiliary/ml/notebooks/serving/kserve/tensorflow/model_serving_kserve_tensorflow.ipynb

This Python example demonstrates deploying a model to KServe while attaching a custom transformer script. The `transformer` argument in the `deploy` method takes a Transformer object, which points to a Python script or notebook for preprocessing and postprocessing.

```python
# from hsml.transformer import Transformer

# TRANSFORMER_SCRIPT_PATH = mr.project_path + "/Jupyter/serving/kserve/tensorflow/transformer.py" # or .ipynb

# mnistclassifier = best_model.deploy(serving_tool="KSERVE",
#                                     transformer=Transformer(script_file=TRANSFORMER_SCRIPT_PATH))
```

--------------------------------

### Get Feature Store Keywords (GET)

Source: https://context7.com/logicalclocks/hopsworks/llms.txt

Retrieves available keywords for the feature store, which can be useful for various configurations and operations. Requires project ID and JWT authentication.

```bash
curl -X GET "https://{HOPSWORKS_HOST}/hopsworks-api/api/project/{projectId}/featurestores/keywords" \
  -H "Authorization: Bearer {JWT_TOKEN}"
```

--------------------------------

### List Feature Views (GET)

Source: https://context7.com/logicalclocks/hopsworks/llms.txt

Lists all feature views available within a specified feature store. Requires project and featurestore IDs, along with JWT authentication.

```bash
curl -X GET "https://{HOPSWORKS_HOST}/hopsworks-api/api/project/{projectId}/featurestores/{featurestoreId}/featureview" \
  -H "Authorization: Bearer {JWT_TOKEN}"
```

--------------------------------

### Model Serving and Inference Operations

Source: https://context7.com/logicalclocks/hopsworks/llms.txt

Manage model deployment lifecycles including listing, creating, starting, stopping, and deleting serving instances. Also includes the endpoint for making real-time inference requests.

```bash
curl -X GET "https://{HOPSWORKS_HOST}/hopsworks-api/api/project/{projectId}/serving" -H "Authorization: Bearer {JWT_TOKEN}"
curl -X PUT "https://{HOPSWORKS_HOST}/hopsworks-api/api/project/{projectId}/serving" -H "Authorization: Bearer {JWT_TOKEN}" -H "Content-Type: application/json" -d '{"name": "fraud-detector-serving", "modelPath": "/Models/fraud_detector/1", "modelVersion": 1, "modelServer": "TENSORFLOW_SERVING", "servingTool": "KSERVE", "requestedInstances": 1, "batchingEnabled": true, "kafkaTopicDTO": {"name": "fraud_predictions_log"}}'
curl -X POST "https://{HOPSWORKS_HOST}/hopsworks-api/api/project/{projectId}/inference/models/{modelName}:predict" -H "Authorization: Bearer {JWT_TOKEN}" -H "Content-Type: application/json" -d '{"instances": [{"amount": 150.50, "merchant_category": 5, "customer_age": 35}]}'
```

--------------------------------

### PySpark Job Configuration JSON Example

Source: https://github.com/logicalclocks/hopsworks/blob/master/docs/compute/jobs.md

This JSON object provides an example configuration for a PySpark job when creating it programmatically in Hopsworks. It specifies essential parameters like YARN queue, driver resources (memory, virtual cores), job type, application path, main class, executor configurations, dynamic allocation settings, and paths for additional dependencies. It also includes fields for job name and default arguments.

```json
{
    "type": "sparkJobConfiguration",
    "amQueue": "default",
    "amMemory": 2048,
    "amVCores": 1,
    "jobType": "PYSPARK",
    "appPath": "hdfs:///Projects/{PROJECT_NAME}/{DATASET_NAME}/{APP_FILE}",
    "mainClass": "org.apache.spark.deploy.PythonRunner",
    "spark.executor.instances": 1,
    "spark.executor.cores": 1,
    "spark.executor.memory": 2048,
    "spark.executor.gpus": 0,
    "spark.tensorflow.num.ps": 1,
    "spark.dynamicAllocation.enabled": true,
    "spark.dynamicAllocation.minExecutors": 0,
    "spark.dynamicAllocation.maxExecutors": 1,
    "spark.dynamicAllocation.initialExecutors": 0,
    "spark.blacklist.enabled": false,
    "properties": "",
    "spark.yarn.dist.pyFiles": "",
    "spark.yarn.dist.files": "",
    "spark.yarn.dist.jars": "",
    "spark.yarn.dist.archives": "",
    "appName":"{JOB_NAME}",
    "defaultArgs": ""
}
```

--------------------------------

### RonDB Comprehensive Backup Script Reference (Shell)

Source: https://github.com/logicalclocks/hopsworks/blob/master/docs/admin/ha-dr/dr.md

This references a more comprehensive shell script for RonDB backups, which includes collecting partial backups and can be adapted for shipping backups off-cluster. It serves as a starting point for advanced backup strategies.

```shell
# Reference to a comprehensive backup script available at: https://github.com/logicalclocks/ndb-chef/blob/master/templates/default/native_ndb_backup.sh.erb
```

--------------------------------

### Exporting Models to Hopsworks Model Registry

Source: https://github.com/logicalclocks/hopsworks/blob/master/hopsworks-IT/src/test/ruby/spec/auxiliary/export_model.ipynb

Demonstrates how to initialize a connection to the Hopsworks Model Registry and export various model types (Python, Scikit-Learn, Torch) with input data examples. These functions validate the model instance type upon successful registration.

```python
def export_sklearn():
	import pandas
	import hsml
	connection = hsml.connection()
	mr = connection.get_model_registry()
	input_df = pandas.DataFrame({'int_column': [1, 2], 'string_column': ["John", "Jamie"], 'float_column': [0.5, 0.3]})
	exported_sklearn_model = sklearn_model(mr, "model_sklearn", input_example=input_df)
	assert isinstance(exported_sklearn_model, hsml.sklearn.model.Model)

def export_torch():
	import pandas
	import hsml
	connection = hsml.connection()
	mr = connection.get_model_registry()
	input_df = pandas.DataFrame({'int_column': [1, 2], 'string_column': ["John", "Jamie"], 'float_column': [0.5, 0.3]})
	exported_torch_model = torch_model(mr, "model_torch", input_example=input_df)
	assert isinstance(exported_torch_model, hsml.torch.model.Model)
```

--------------------------------

### Python Kafka Consumer Setup for Prediction Logs

Source: https://github.com/logicalclocks/hopsworks/blob/master/hopsworks-IT/src/test/ruby/spec/auxiliary/ml/notebooks/end_to_end_pipeline/sklearn/end_to_end_sklearn.ipynb

This snippet shows how to set up a Kafka consumer in Python to subscribe to a topic containing prediction logs. It configures the consumer with default settings and specifies an 'earliest' offset reset policy. It also demonstrates how to retrieve and convert the Kafka Avro schema for message parsing.

```python
TOPIC_NAME = irisclassifier.inference_logger.kafka_topic.name

config = kafka.get_kafka_default_config()
config['default.topic.config'] = {'auto.offset.reset': 'earliest'}
consumer = Consumer(config)
topics = [TOPIC_NAME]
consumer.subscribe(topics)

json_schema = kafka.get_schema(TOPIC_NAME)
avro_schema = kafka.convert_json_schema_to_avro(json_schema)
```

--------------------------------

### Send Prediction Requests to Deployed Model

Source: https://github.com/logicalclocks/hopsworks/blob/master/hopsworks-IT/src/test/ruby/spec/auxiliary/ml/notebooks/serving/kserve/tensorflow/model_serving_kserve_tensorflow.ipynb

Demonstrates how to use the predict method on a deployment metadata object to send inference requests. It iterates through a loop to send input examples to the model.

```python
for i in range(10):
    data = { "signature_name": "serving_default", "instances": [best_model.input_example] }
    predictions = mnistclassifier.predict(data)
    print(predictions)
```

--------------------------------

### Deploy Trained Model for Serving

Source: https://github.com/logicalclocks/hopsworks/blob/master/hopsworks-IT/src/test/ruby/spec/auxiliary/ml/notebooks/end_to_end_pipeline/sklearn/end_to_end_sklearn.ipynb

This Python code deploys the best version of a trained model from the Model Registry for serving. It specifies the path to the predictor script and optionally configures the serving tool. The deployment object can be used to start the model server and access its metadata.

```python
PREDICTOR_SCRIPT_PATH = best_model.version_path + "/" + PREDICTOR_SCRIPT_NAME

# Deploy the best version of the model
irisclassifier = best_model.deploy(script_file=PREDICTOR_SCRIPT_PATH)  # To serve models with KServe, set serving_tool to "KSERVE"
```

--------------------------------

### Python Job Configuration JSON

Source: https://github.com/logicalclocks/hopsworks/blob/master/docs/compute/jobs.md

Example JSON configuration for a Python job in Hopsworks. This allows programmatic job creation and overrides default project-scoped settings. It specifies application path, resources, and other parameters.

```json
{
    "type": "pythonJobConfiguration",
    "appPath": "hdfs:///Projects/{PROJECT_NAME}/{DATASET_NAME}/{APP_FILE}",
    "files": "", 
    "resourceConfig": {
        "type": "dockerResourcesConfiguration",
        "cores": 1,
        "memory": 1024, 
        "gpus": 0 
    },
    "logRedirection": true,
    "appName": "{JOB_NAME}", 
    "defaultArgs": "" 
}
```

--------------------------------

### Get Job by Name (GET)

Source: https://context7.com/logicalclocks/hopsworks/llms.txt

Retrieves details for a specific job by its name. Requires project ID, job name, and JWT authentication.

```bash
curl -X GET "https://{HOPSWORKS_HOST}/hopsworks-api/api/project/{projectId}/jobs/{jobName}" \
  -H "Authorization: Bearer {JWT_TOKEN}"
```

--------------------------------

### Create Training Dataset (POST)

Source: https://context7.com/logicalclocks/hopsworks/llms.txt

Creates a new training dataset from features defined in feature groups. This involves specifying dataset name, version, description, data format, splits, and the features to include. Requires project, featurestore IDs, JWT authentication, and a detailed JSON payload.

```bash
curl -X POST "https://{HOPSWORKS_HOST}/hopsworks-api/api/project/{projectId}/featurestores/{featurestoreId}/trainingdatasets" \
  -H "Authorization: Bearer {JWT_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "fraud_training_data",
    "version": 1,
    "description": "Training data for fraud detection model",
    "dataFormat": "TFRECORD",
    "splits": [
      {"name": "train", "percentage": 0.8},
      {"name": "test", "percentage": 0.2}
    ],
    "seed": 42,
    "features": [
      {"featureGroupId": 1, "name": "amount"},
      {"featureGroupId": 1, "name": "merchant_category"},
      {"featureGroupId": 2, "name": "customer_age"}
    ]
  }'
```

--------------------------------

### Get Feature Group Statistics (GET)

Source: https://context7.com/logicalclocks/hopsworks/llms.txt

Retrieves statistics for a specified feature group. Requires project, featurestore, and featuregroup IDs, along with JWT authentication.

```bash
curl -X GET "https://{HOPSWORKS_HOST}/hopsworks-api/api/project/{projectId}/featurestores/{featurestoreId}/featuregroups/{featuregroupId}/statistics" \
  -H "Authorization: Bearer {JWT_TOKEN}"
```

--------------------------------

### Launch Hopsworks Experiment

Source: https://github.com/logicalclocks/hopsworks/blob/master/hopsworks-IT/src/test/ruby/spec/auxiliary/run_multiple_experiments.ipynb

This snippet demonstrates how to initialize and launch an experiment in Hopsworks. It requires a wrapper function to define the training logic and a unique name for the experiment.

```python
from hops import experiment

def wrapper():
    pass

experiment.launch(wrapper, name="mnist")
```

--------------------------------

### Start MNIST Classifier Prediction Service

Source: https://github.com/logicalclocks/hopsworks/blob/master/hopsworks-IT/src/test/ruby/spec/auxiliary/ml/notebooks/serving/kserve/tensorflow/model_serving_kserve_tensorflow.ipynb

This Python command starts the prediction service for the deployed MNIST classifier. It initiates the model serving process, making it ready to receive inference requests.

```python
mnistclassifier.start()
```

--------------------------------

### Get Transformation Functions (GET)

Source: https://context7.com/logicalclocks/hopsworks/llms.txt

Retrieves a list of available transformation functions within a feature store. These functions can be used for data preprocessing and feature engineering. Requires project, featurestore IDs, and JWT authentication.

```bash
curl -X GET "https://{HOPSWORKS_HOST}/hopsworks-api/api/project/{projectId}/featurestores/{featurestoreId}/transformationfunctions" \
  -H "Authorization: Bearer {JWT_TOKEN}"
```

--------------------------------

### GET /model-tags

Source: https://context7.com/logicalclocks/hopsworks/llms.txt

Retrieves the tags associated with a specific model in the registry.

```APIDOC
## GET /hopsworks-api/api/project/{projectId}/modelregistries/{modelRegistryId}/models/{modelId}/tags

### Description
Retrieves the tags associated with a specific model in the registry.

### Method
GET

### Endpoint
/hopsworks-api/api/project/{projectId}/modelregistries/{modelRegistryId}/models/{modelId}/tags

### Parameters
#### Path Parameters
- **projectId** (string) - Required - The ID of the project
- **modelRegistryId** (string) - Required - The ID of the model registry
- **modelId** (string) - Required - The ID of the model

### Request Example
curl -X GET "https://{HOPSWORKS_HOST}/hopsworks-api/api/project/{projectId}/modelregistries/{modelRegistryId}/models/{modelId}/tags" -H "Authorization: Bearer {JWT_TOKEN}"
```

--------------------------------

### Get TensorBoard URLs

Source: https://context7.com/logicalclocks/hopsworks/llms.txt

Specifically retrieves the URLs for the TensorBoard interface associated with an application.

```bash
curl -X GET "https://{HOPSWORKS_HOST}/hopsworks-api/api/project/{projectId}/jobs/{appId}/tensorboard" \
  -H "Authorization: Bearer {JWT_TOKEN}"
```

--------------------------------

### GET /inference-logs

Source: https://github.com/logicalclocks/hopsworks/blob/master/hopsworks-IT/src/test/ruby/spec/auxiliary/ml/notebooks/serving/kserve/sklearn/model_serving_kserve_sklearn.ipynb

Consumes prediction request and response logs from a Kafka topic for monitoring purposes.

```APIDOC
## GET /inference-logs

### Description
Subscribes to a Kafka topic associated with a model's inference logger to retrieve real-time request and response data.

### Method
GET (via Kafka Consumer)

### Endpoint
Kafka Topic: [model_name]_inference_logs

### Parameters
#### Query Parameters
- **timeout** (integer) - Optional - Time in seconds to wait for a message from the Kafka topic.

### Response
#### Success Response (200)
- **servingId** (string) - Unique identifier for the serving instance.
- **modelName** (string) - Name of the model.
- **modelVersion** (integer) - Version of the model.
- **requestTimestamp** (integer) - Unix timestamp of the event.
- **inferenceId** (string) - Unique identifier for the specific inference request.
- **messageType** (string) - Type of message, either 'request' or 'response'.
- **payload** (object) - The actual input instances or output predictions.

#### Response Example
{
  "servingId": "8",
  "modelName": "irisflowerclassifier",
  "modelVersion": 1,
  "requestTimestamp": 1652885019,
  "inferenceId": "2d23b009-f968-4f6b-995a-c79b4f4fa8af",
  "messageType": "response",
  "payload": {"predictions": [0]}
}
```

--------------------------------

### Download Model Artifacts

Source: https://github.com/logicalclocks/hopsworks/blob/master/hopsworks-IT/src/test/ruby/spec/auxiliary/test_hsml.ipynb

Downloads the physical model files from the registry to the local filesystem for deployment or inference.

```python
import os
model_dir = best_tf_model.download()
assert 'saved_model.pb' in os.listdir(model_dir)
```

--------------------------------

### Execute Integration Tests

Source: https://github.com/logicalclocks/hopsworks/blob/master/hopsworks-IT/README.md

Commands to compile, deploy, and run integration tests using Maven or RSpec.

```bash
# Compile and deploy
cd hopsworks/
mvn clean install -Pjruby-tests

# Run tests
cd hopsworks/hopsworks-IT/src/test/ruby/
bundle install
rspec --format html --out ../target/test-report.html
```

--------------------------------

### Job Executions API

Source: https://context7.com/logicalclocks/hopsworks/llms.txt

Endpoints for managing job runs, including listing, starting, stopping, and retrieving execution details.

```APIDOC
## GET /api/project/{projectId}/jobs/{jobName}/executions

### Description
Lists all job executions for a given job.

### Method
GET

### Endpoint
/api/project/{projectId}/jobs/{jobName}/executions

### Parameters
#### Path Parameters
- **projectId** (integer) - Required - The ID of the project.
- **jobName** (string) - Required - The name of the job.

### Response
#### Success Response (200)
- **executions** (array) - A list of job execution objects.

### Response Example
```json
{
  "executions": [
    {
      "id": 1,
      "status": "RUNNING",
      "startTime": "2023-10-27T10:00:00Z"
    }
  ]
}
```

## POST /api/project/{projectId}/jobs/{jobName}/executions

### Description
Starts a new execution for a job.

### Method
POST

### Endpoint
/api/project/{projectId}/jobs/{jobName}/executions

### Parameters
#### Path Parameters
- **projectId** (integer) - Required - The ID of the project.
- **jobName** (string) - Required - The name of the job.

#### Request Body
- **customArgs** (string) - Optional - Custom arguments to pass to the job execution.

### Request Example
```text
--custom-arg value --another-arg value2
```

### Response
#### Success Response (200)
- **executionId** (integer) - The ID of the newly created execution.

### Response Example
```json
{
  "executionId": 2
}
```

## GET /api/project/{projectId}/jobs/{jobName}/executions/{executionId}

### Description
Retrieves details for a specific job execution by its ID.

### Method
GET

### Endpoint
/api/project/{projectId}/jobs/{jobName}/executions/{executionId}

### Parameters
#### Path Parameters
- **projectId** (integer) - Required - The ID of the project.
- **jobName** (string) - Required - The name of the job.
- **executionId** (integer) - Required - The ID of the execution.

### Response
#### Success Response (200)
- **executionDetails** (object) - Details of the job execution.

### Response Example
```json
{
  "executionDetails": {
    "id": 1,
    "status": "COMPLETED",
    "startTime": "2023-10-27T10:00:00Z",
    "endTime": "2023-10-27T11:00:00Z"
  }
}
```

## PUT /api/project/{projectId}/jobs/{jobName}/executions/{executionId}/status

### Description
Updates the status of a job execution (e.g., to STOPPED).

### Method
PUT

### Endpoint
/api/project/{projectId}/jobs/{jobName}/executions/{executionId}/status

### Parameters
#### Path Parameters
- **projectId** (integer) - Required - The ID of the project.
- **jobName** (string) - Required - The name of the job.
- **executionId** (integer) - Required - The ID of the execution.

#### Request Body
- **status** (string) - Required - The new status for the execution (e.g., "STOPPED").

### Request Example
```json
{
  "status": "STOPPED"
}
```

### Response
#### Success Response (200)
- **message** (string) - Confirmation message.

### Response Example
```json
{
  "message": "Execution status updated successfully."
}
```

## DELETE /api/project/{projectId}/jobs/{jobName}/executions/{executionId}

### Description
Deletes a specific job execution.

### Method
DELETE

### Endpoint
/api/project/{projectId}/jobs/{jobName}/executions/{executionId}

### Parameters
#### Path Parameters
- **projectId** (integer) - Required - The ID of the project.
- **jobName** (string) - Required - The name of the job.
- **executionId** (integer) - Required - The ID of the execution to delete.

### Response
#### Success Response (200)
- **message** (string) - Confirmation message.

### Response Example
```json
{
  "message": "Execution deleted successfully."
}
```

## GET /api/project/{projectId}/jobs/{appId}/ui/{isTensorBoard}

### Description
Retrieves UI URLs for Spark or TensorBoard.

### Method
GET

### Endpoint
/api/project/{projectId}/jobs/{appId}/ui/{isTensorBoard}

### Parameters
#### Path Parameters
- **projectId** (integer) - Required - The ID of the project.
- **appId** (string) - Required - The application ID.
- **isTensorBoard** (boolean) - Required - Set to `true` for TensorBoard, `false` for Spark UI.

### Response
#### Success Response (200)
- **uiUrl** (string) - The URL for the Spark or TensorBoard UI.

### Response Example
```json
{
  "uiUrl": "http://localhost:8080/proxy/application_1234567890/"
}
```

## GET /api/project/{projectId}/jobs/{appId}/tensorboard

### Description
Retrieves TensorBoard URLs for a specific application.

### Method
GET

### Endpoint
/api/project/{projectId}/jobs/{appId}/tensorboard

### Parameters
#### Path Parameters
- **projectId** (integer) - Required - The ID of the project.
- **appId** (string) - Required - The application ID.

### Response
#### Success Response (200)
- **tensorboardUrl** (string) - The URL for the TensorBoard.

### Response Example
```json
{
  "tensorboardUrl": "http://localhost:6006/"
}
```
```

--------------------------------

### Recreate Kafka Topics

Source: https://github.com/logicalclocks/hopsworks/blob/master/docs/admin/ha-dr/dr.md

Recreates Kafka topics with correct partitioning and replication factors using information from the RonDB backup. This script is deployed during platform deployment.

```shell
/srv/hops/kafka/bin/kafka-restore.sh
```

--------------------------------

### Backup RonDB Users and Privileges (Shell)

Source: https://github.com/logicalclocks/hopsworks/blob/master/docs/admin/ha-dr/dr.md

This command backs up user definitions and privileges for Hopsworks, HopsFS, Metastore, and the online feature store. It should be run as the 'mysql' user or with sudo privileges.

```shell
/srv/hops/mysql/bin/mysqlpump -S /srv/hops/mysql-cluster/mysql.sock --exclude-databases=% --exclude-users=root,mysql.sys,mysql.session,mysql.infoschema --users > users.sql
```

--------------------------------

### Get Specific Model by ID

Source: https://context7.com/logicalclocks/hopsworks/llms.txt

Retrieves details of a specific model identified by its ID within a model registry. The ID format is typically 'name_version'.

```bash
curl -X GET "https://{HOPSWORKS_HOST}/hopsworks-api/api/project/{projectId}/modelregistries/{modelRegistryId}/models/{modelId}" \
  -H "Authorization: Bearer {JWT_TOKEN}"
```

--------------------------------

### PUT /serving

Source: https://context7.com/logicalclocks/hopsworks/llms.txt

Creates or updates a model serving deployment.

```APIDOC
## PUT /hopsworks-api/api/project/{projectId}/serving

### Description
Creates or updates a model serving deployment for real-time inference.

### Method
PUT

### Endpoint
/hopsworks-api/api/project/{projectId}/serving

### Request Body
- **name** (string) - Required - Deployment name
- **modelPath** (string) - Required - Path to the model
- **modelVersion** (integer) - Required - Version of the model
- **modelServer** (string) - Required - Server type (e.g., TENSORFLOW_SERVING)

### Request Example
{
  "name": "fraud-detector-serving",
  "modelPath": "/Models/fraud_detector/1",
  "modelVersion": 1,
  "modelServer": "TENSORFLOW_SERVING"
}
```

--------------------------------

### List Git Repositories (Git Integration API)

Source: https://context7.com/logicalclocks/hopsworks/llms.txt

Lists all Git repositories integrated with the Hopsworks project. This API requires the project ID. The output is a list of repositories available for version control operations.

```bash
curl -X GET "https://{HOPSWORKS_HOST}/hopsworks-api/api/project/{projectId}/git" \
  -H "Authorization: Bearer {JWT_TOKEN}"
```

--------------------------------

### Get Job Execution by ID

Source: https://context7.com/logicalclocks/hopsworks/llms.txt

Retrieves details of a specific job execution using its unique ID. Provides information about a particular run of the job.

```bash
curl -X GET "https://{HOPSWORKS_HOST}/hopsworks-api/api/project/{projectId}/jobs/{jobName}/executions/{executionId}" \
  -H "Authorization: Bearer {JWT_TOKEN}"
```

--------------------------------

### Get Default Job Configuration

Source: https://context7.com/logicalclocks/hopsworks/llms.txt

Retrieves the default configuration for a specified job type (e.g., Spark). This helps in understanding the baseline settings for different job types.

```bash
curl -X GET "https://{HOPSWORKS_HOST}/hopsworks-api/api/project/{projectId}/jobs/spark/configuration" \
  -H "Authorization: Bearer {JWT_TOKEN}"
```

--------------------------------

### POST /api/project/{projectId}/featurestores/{featurestoreId}/trainingdatasets

Source: https://context7.com/logicalclocks/hopsworks/llms.txt

Creates a new training dataset by selecting features from existing feature groups.

```APIDOC
## POST /api/project/{projectId}/featurestores/{featurestoreId}/trainingdatasets

### Description
Creates a training dataset based on specified features and split configurations.

### Method
POST

### Endpoint
/api/project/{projectId}/featurestores/{featurestoreId}/trainingdatasets

### Parameters
#### Path Parameters
- **projectId** (string) - Required - The ID of the project
- **featurestoreId** (string) - Required - The ID of the feature store

#### Request Body
- **name** (string) - Required - Name of the training dataset
- **version** (integer) - Required - Version number
- **dataFormat** (string) - Required - Format (e.g., TFRECORD)
- **features** (array) - Required - List of features to include

### Request Example
{
  "name": "fraud_training_data",
  "version": 1,
  "dataFormat": "TFRECORD",
  "features": [{"featureGroupId": 1, "name": "amount"}]
}

### Response
#### Success Response (200)
- **id** (integer) - The ID of the created training dataset
```

--------------------------------

### Configure Server for Kerberos Authentication

Source: https://github.com/logicalclocks/hopsworks/blob/master/docs/admin/ldap/configure-server.md

This YAML configuration snippet details the parameters required for Kerberos integration with Hopsworks, which is used in conjunction with SPNEGO for secure authentication. It includes enabling Kerberos, specifying paths for the Kerberos configuration and keytab files, and defining SPNEGO server configurations, including the principal for authentication. LDAP attributes are also included as they are necessary for Kerberos configuration.

```yaml
kerberos:
    enabled: true
    krb_conf_path: "/etc/krb5.conf"
    krb_server_key_tab_path: "/etc/security/keytabs/service.keytab"
    krb_server_key_tab_name: "service.keytab"
    spnego_server_conf: '\nuseKeyTab=true\nprincipal="HTTP/server.hopsworks.ai@HOPSWORKS.AI"\nstoreKey=true\nisInitiator=false'
ldap:
    jndilookupname: "dc=hopsworks,dc=ai"
    provider_url: "ldap://193.10.66.104:1389"
    attr_binary_val: "objectGUID"
    security_auth: "none"
    security_principal: ""
    security_credentials: ""
    referral: "ignore"
    additional_props: ""
```

--------------------------------

### Inspect Python Program for Job Configuration

Source: https://context7.com/logicalclocks/hopsworks/llms.txt

Inspects a Python program file to get suggested job configuration. This is useful for determining optimal resource allocation and settings for a new job.

```bash
curl -X GET "https://{HOPSWORKS_HOST}/hopsworks-api/api/project/{projectId}/jobs/pyspark/inspection?path=/Projects/my_project/Resources/app.py" \
  -H "Authorization: Bearer {JWT_TOKEN}"
```

--------------------------------

### Post Alert via AlertManagerClient

Source: https://github.com/logicalclocks/hopsworks/blob/master/alerting/README.md

Shows how to instantiate an AlertManagerClient and send a postable alert with custom labels and annotations to the Alertmanager API.

```java
AlertManagerClient client = new AlertManagerClient.Builder(ClientBuilder.newClient())
    .withTarget("serviceFQDN")
    .build();
try {
    Map<String, String> labels = new HashMap<>();
    labels.put("alertname", "HopsworksAlert");
    labels.put("instance", "hopsworks0");
    labels.put("severity", "warning");
    labels.put("project", "project1");
    Map<String, String> annotations = new HashMap<>();
    annotations.put("info", "project1 info");
    annotations.put("summary", "Job finished.");
    PostableAlert alert = new PostableAlert(labels, annotations);
    Response res = client.alertsApi().postAlerts(alerts);
} finally {
    if (client != null) {
        client.close();
    }
}
```

--------------------------------

### Describe Model Deployment

Source: https://github.com/logicalclocks/hopsworks/blob/master/hopsworks-IT/src/test/ruby/spec/auxiliary/ml/notebooks/end_to_end_pipeline/sklearn/end_to_end_sklearn.ipynb

This Python snippet demonstrates how to retrieve and print the metadata description of a deployed model in Hopsworks. It uses the `.describe()` method on the deployment object to display configuration details.

```python
print("Description of " + irisclassifier.name)
irisclassifier.describe()
```

--------------------------------

### Export Models to HDFS Paths

Source: https://github.com/logicalclocks/hopsworks/blob/master/hopsworks-IT/src/test/ruby/spec/auxiliary/export_model.ipynb

Shows how to export Scikit-Learn, PyTorch, and Python models to HDFS. It covers different path configurations including absolute HDFS paths with/without prefixes and relative project paths.

```python
def sklearn_model(mr, name, metrics=None, description=None, input_example=None, model_schema=None):
    from hops import hdfs
    model_path = "hdfs:///Projects/{}/Resources".format(hdfs.project_name())
    sklearn_model = mr.sklearn.create_model(name, metrics=metrics, description=description, input_example=input_example, model_schema=model_schema)
    return sklearn_model.save(model_path)

def torch_model(mr, name, metrics=None, description=None, input_example=None, model_schema=None):
    from hops import hdfs
    model_path = "/Projects/{}/Resources".format(hdfs.project_name())
    torch_model = mr.torch.create_model(name, metrics=metrics, description=description, input_example=input_example, model_schema=model_schema)
    return torch_model.save(model_path)

def python_model(mr, name, metrics=None, description=None, input_example=None, model_schema=None):
    model_path = "Resources"
    python_model = mr.python.create_model(name, metrics=metrics, description=description, input_example=input_example, model_schema=model_schema)
    return python_model.save(model_path)
```

--------------------------------

### Launching Model Export Experiments

Source: https://github.com/logicalclocks/hopsworks/blob/master/hopsworks-IT/src/test/ruby/spec/auxiliary/export_model.ipynb

Uses the Hops experiment library to launch model export tasks as distributed experiments. This allows for tracking and managing model registration workflows from both local and HDFS file paths.

```python
from hops import experiment

# Export from local paths
experiment.launch(export_tf)

# Export from hdfs paths
experiment.launch(export_py)
experiment.launch(export_sklearn)
experiment.launch(export_torch)
```

--------------------------------

### Connect to Hopsworks and Get Model Registry

Source: https://github.com/logicalclocks/hopsworks/blob/master/hopsworks-IT/src/test/ruby/spec/auxiliary/ml/notebooks/serving/kserve/tensorflow/model_serving_kserve_tensorflow.ipynb

This Python snippet establishes a connection to Hopsworks using the hsml library and retrieves a handle to the Model Registry. This handle is essential for interacting with registered models.

```python
import hsml

# Connect with Hopsworks
conn = hsml.connection()

# Retrieve the model registry handle
mr = conn.get_model_registry()
```

--------------------------------

### Get Spark/TensorBoard UI URLs

Source: https://context7.com/logicalclocks/hopsworks/llms.txt

Retrieves the URLs for accessing the Spark or TensorBoard user interfaces for a given application ID. The boolean parameter likely controls whether to fetch Spark or TensorBoard URLs.

```bash
curl -X GET "https://{HOPSWORKS_HOST}/hopsworks-api/api/project/{projectId}/jobs/{appId}/ui/false" \
  -H "Authorization: Bearer {JWT_TOKEN}"
```

--------------------------------

### Print Hello World in Python

Source: https://github.com/logicalclocks/hopsworks/blob/master/hopsworks-IT/src/test/ruby/spec/auxiliary/test_attach_xattr_ipython_kernel.ipynb

A basic Python script to output text to the standard output. This is typically used to verify the execution environment of a notebook cell.

```python
print("Hello world")
```