TensorFlow Extended

https://www.tensorflow.org/tfx
Admin
TensorFlow Extended (TFX) is an end-to-end platform for deploying production ML pipelines,...
Tokens:833,956
Snippets:8,703
Trust Score:9.9
Update:3 months ago
Show doc for...
Tokens:
Raw
### Installation output

Source: https://www.tensorflow.org/tfx/tutorials/serving/rest_simple

Example output from the installation process.

```text
--2024-04-30 10:41:50--  http://storage.googleapis.com/tensorflow-serving-apt/pool/tensorflow-model-server-2.8.0/t/tensorflow-model-server/tensorflow-model-server_2.8.0_all.deb
Resolving storage.googleapis.com (storage.googleapis.com)... 142.251.172.207, 142.251.180.207, 142.251.183.207, ...
Connecting to storage.googleapis.com (storage.googleapis.com)|142.251.172.207|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 340152790 (324M) [application/x-debian-package]
Saving to: ‘tensorflow-model-server_2.8.0_all.deb’

tensorflow-model-se 100%[===================>] 324.39M   223MB/s    in 1.5s    

2024-04-30 10:41:51 (223 MB/s) - ‘tensorflow-model-server_2.8.0_all.deb’ saved [340152790/340152790]

dpkg: error: requested operation requires superuser privilege
Collecting tensorflow-serving-api==2.8.0
  Downloading tensorflow_serving_api-2.8.0-py2.py3-none-any.whl.metadata (1.8 kB)
Requirement already satisfied: grpcio<2,>=1.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow-serving-api==2.8.0) (1.26.0)
```

--------------------------------

### Install lab dependencies

Source: https://www.tensorflow.org/tfx/tutorials/tfx/airflow_workshop

Navigate to the setup directory and execute the demo setup script to configure the environment.

```bash
cd ~/tfx/tfx/examples/airflow_workshop/taxi/setup/
./setup_demo.sh
```

--------------------------------

### Example version output

Source: https://www.tensorflow.org/tfx/tutorials/tfx/gcp/vertex_pipelines_bq

Sample output showing the expected version information for the installed libraries.

```text
2024-05-08 09:24:47.541894: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-05-08 09:24:47.541946: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-05-08 09:24:47.543649: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
TensorFlow version: 2.15.1
TFX version: 1.15.0
KFP version: 1.8.22
```

--------------------------------

### Initialize Examples Artifact

Source: https://www.tensorflow.org/tfx/api_docs/python/tfx/v1/types/standard_artifacts/Examples

Constructor for the Examples artifact class.

```python
tfx.v1.types.standard_artifacts.Examples(
    *args, **kwargs
)
```

--------------------------------

### Full EvalConfig and Analysis Execution

Source: https://www.tensorflow.org/tfx/model_analysis/model_validations

Complete example showing EvalConfig definition, shared model setup, and running model analysis.

```python
# Run in a Jupyter Notebook.
from google.protobuf import text_format

eval_config = text_format.Parse("""
  model_specs {
    # This assumes a serving model with a "serving_default" signature.
    label_key: "label"
    example_weight_key: "weight"
  }
  metrics_spec {
    # Training Time metric thresholds
    thresholds {
      key: "auc"
      value: {
        value_threshold {
          lower_bound { value: 0.7 }
        }
        change_threshold {
          direction: HIGHER_IS_BETTER
          absolute { value: -1e-10 }
        }
      }
    }
    # Post Training metrics and their thesholds.
    metrics {
      # This assumes a binary classification model.
      class_name: "AUC"
      threshold {
        value_threshold {
          lower_bound { value: 0 }
        }
      }
    }
  }
  slicing_specs {}
  slicing_specs {
    feature_keys: ["age"]
  }
""", tfma.EvalConfig())

eval_shared_models = [
  tfma.default_eval_shared_model(
      model_name=tfma.CANDIDATE_KEY,
      eval_saved_model_path='/path/to/saved/candiate/model',
      eval_config=eval_config),
  tfma.default_eval_shared_model(
      model_name=tfma.BASELINE_KEY,
      eval_saved_model_path='/path/to/saved/baseline/model',
      eval_config=eval_config),
]

eval_result = tfma.run_model_analysis(
    eval_shared_models,
    eval_config=eval_config,
    # This assumes your data is a TFRecords file containing records in the
    # tf.train.Example format.
    data_location="/path/to/file/containing/tfrecords",
    output_path="/path/for/output")

tfma.view.render_slicing_metrics(eval_result)
tfma.load_validation_result(output_path)
```

--------------------------------

### Install TFX-Addons

Source: https://www.tensorflow.org/tfx/addons?hl=zh-tw

Install the latest version of the TFX-Addons library from PyPI.

```bash
pip install tfx-addons
```

--------------------------------

### Create Pipeline Examples

Source: https://www.tensorflow.org/tfx/guide/cli

Various examples for creating pipelines using different orchestrators or auto-detection.

```bash
tfx pipeline create --engine=kubeflow --pipeline_path=pipeline-path \
--iap_client_id=iap-client-id --namespace=namespace --endpoint=endpoint \
--build_image
```

```bash
tfx pipeline create --engine=local --pipeline_path=pipeline-path
```

```bash
tfx pipeline create --engine=vertex --pipeline_path=pipeline-path \
--build_image
```

```bash
tfx pipeline create --pipeline_path=pipeline-path
```

--------------------------------

### Serialize Batch of Examples

Source: https://www.tensorflow.org/tfx/tutorials/transform/census

Serializes a batch of examples into a tensor of strings.

```python
serialized_example_batch = tf.constant([
  encode_example(pandas_train.loc[i]).SerializeToString()
  for i in range(3)
])

serialized_example_batch
```

```text
<tf.Tensor: shape=(3,), dtype=string, numpy=
array([b'\n\xf9\x02\n\x0f\n\x03age\x12\x08\x12\x06\n\x04\x00\x00\x1cB\n\x12\n\x05label\x12\t\n\x07\n\x05<=50K\n\x1a\n\x0ehours-per-week\x12\x08\x12\x06\n\x04\x00\x00 B\n#\n\x0enative-country\x12\x11\n\x0f\n\rUnited-States\n\x1a\n\tworkclass\x12\r\n\x0b\n\tState-gov\n\x0f\n\x03sex\x12\x08\n\x06\n\x04Male\n\x18\n\x0ccapital-loss\x12\x08\x12\x06\n\x04\x00\x00\x00\x00\n\x19\n\reducation-num\x12\x08\x12\x06\n\x04\x00\x00PA\n!\n\x0crelationship\x12\x11\n\x0f\n\rNot-in-family\n\x1e\n\noccupation\x12\x10\n\x0e\n\x0cAdm-clerical\n#\n\x0emarital-status\x12\x11\n\x0f\n\rNever-married\n\x11\n\x04race\x12\t\n\x07\n\x05White\n\x1a\n\teducation\x12\r\n\x0b\n\tBachelors\n\x18\n\x0ccapital-gain\x12\x08\x12\x06\n\x04\x00\xe0\x07E',
       b'\n\x82\x03\n\x12\n\x05label\x12\t\n\x07\n\x05<=50K\n\x1a\n\teducation\x12\r\n\x0b\n\tBachelors\n\x18\n\x0ccapital-gain\x12\x08\x12\x06\n\x04\x00\x00\x00\x00\n(\n\x0emarital-status\x12\x16\n\x14\n\x12Married-civ-spouse\n\x1b\n\x0crelationship\x12\x0b\n\t\n\x07Husband\n\x19\n\reducation-num\x12\x08\x12\x06\n\x04\x00\x00PA\n\x11\n\x04race\x12\t\n\x07\n\x05White\n\x18\n\x0ccapital-loss\x12\x08\x12\x06\n\x04\x00\x00\x00\x00\n#\n\x0enative-country\x12\x11\n\x0f\n\rUnited-States\n!\n\noccupation\x12\x13\n\x11\n\x0fExec-managerial\n!\n\tworkclass\x12\x14\n\x12\n\x10Self-emp-not-inc\n\x0f\n\x03age\x12\x08\x12\x06\n\x04\x00\x00HB\n\x1a\n\x0ehours-per-week\x12\x08\x12\x06\n\x04\x00\x00PA\n\x0f\n\x03sex\x12\x08\n\x06\n\x04Male',
       b'\n\xf5\x02\n\x19\n\reducation-num\x12\x08\x12\x06\n\x04\x00\x00\x10A\n!\n\x0crelationship\x12\x11\n\x0f\n\rNot-in-family\n#\n\noccupation\x12\x15\n\x13\n\x11Handlers-cleaners\n\x0f\n\x03age\x12\x08\x12\x06\n\x04\x00\x00\x18B\n\x18\n\tworkclass\x12\x0b\n\t\n\x07Private\n\x18\n\x0ccapital-gain\x12\x08\x12\x06\n\x04\x00\x00\x00\x00\n\x18\n\x0ccapital-gain\x12\x08\x12\x06\n\x04\x00\x00\x00\x00\n\x12\n\x05label\x12\t\n\x07\n\x05<=50K\n\x0f\n\x03sex\x12\x08\n\x06\n\x04Male\n\x1a\n\x0ehours-per-week\x12\x08\x12\x06\n\x04\x00\x00 B\n\x18\n\teducation\x12\x0b\n\t\n\x07HS-grad\n\x11\n\x04race\x12\t\n\x07\n\x05White\n\x1e\n\x0emarital-status\x12\x0c\n\n\n\x08Divorced\n#\n\x0enative-country\x12\x11\n\x0f\n\rUnited-States'],
      dtype=object)>
```

--------------------------------

### Install TFX BSL

Source: https://www.tensorflow.org/tfx/transform/get_started?hl=ja

Install the required TFX BSL library.

```bash
pip install -U -q tfx_bsl
```

--------------------------------

### Create Run Examples by Orchestrator

Source: https://www.tensorflow.org/tfx/guide/cli

Examples for creating pipeline runs using Kubeflow, local, and Vertex orchestrators.

```bash
tfx run create --engine=kubeflow --pipeline_name=pipeline-name --iap_client_id=iap-client-id \
--namespace=namespace --endpoint=endpoint
```

```bash
tfx run create --engine=local --pipeline_name=pipeline-name
```

```bash
tfx run create --engine=vertex --pipeline_name=pipeline-name \
  --runtime_parameter=var_name=var_value \
  --project=gcp-project-id --region=gcp-region
```

--------------------------------

### Install KFP and Skaffold

Source: https://www.tensorflow.org/tfx/tutorials/tfx/penguin_template

Install the KFP Python package and download the Skaffold binary for pipeline deployment.

```bash
pip install --upgrade -q kfp

# Download skaffold and set it executable.
curl -Lo skaffold https://storage.googleapis.com/skaffold/releases/latest/skaffold-linux-amd64 && chmod +x skaffold
```

--------------------------------

### Channel Creation Example

Source: https://www.tensorflow.org/tfx/api_docs/python/tfx/v1/types/standard_artifacts/ExampleAnomalies

Example of using the artifact type to create a Channel.

```python
Channel(type=standard_artifacts.Model)
```

--------------------------------

### Install dependencies and import libraries

Source: https://www.tensorflow.org/tfx/tutorials/serving/rest_simple

Installs necessary packages and imports TensorFlow and helper libraries.

```python
# TensorFlow and tf.keras
print("Installing dependencies for Colab environment")
!pip install -Uq grpcio==1.26.0

import tensorflow as tf
from tensorflow import keras

# Helper libraries
import numpy as np
import matplotlib.pyplot as plt
import os
import subprocess

print('TensorFlow version: {}'.format(tf.__version__))
```

--------------------------------

### Install TFX and Recommender Dependencies

Source: https://www.tensorflow.org/tfx/tutorials/tfx/recommenders

Install the necessary libraries for TFX, TensorFlow Recommenders, and TensorFlow Datasets.

```bash
pip install -Uq tfx
pip install -Uq tensorflow-recommenders
pip install -Uq tensorflow-datasets
```

--------------------------------

### Install TFX and KFP packages

Source: https://www.tensorflow.org/tfx/tutorials/tfx/gcp/vertex_pipelines_bq

Upgrade pip and install the required TFX and KFP libraries for pipeline authoring.

```bash
# Use the latest version of pip.
pip install --upgrade pip
pip install --upgrade "tfx[kfp]<2"
```

--------------------------------

### SavedModel CLI output example

Source: https://www.tensorflow.org/tfx/tutorials/serving/rest_simple

Example output showing the signature definitions and concrete functions available in a SavedModel.

```text
2024-04-30 10:41:41.354485: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-04-30 10:41:41.354542: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-04-30 10:41:41.355947: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered

MetaGraphDef with tag-set: 'serve' contains the following SignatureDefs:

signature_def['__saved_model_init_op']:
  The given SavedModel SignatureDef contains the following input(s):
  The given SavedModel SignatureDef contains the following output(s):
    outputs['__saved_model_init_op'] tensor_info:
        dtype: DT_INVALID
        shape: unknown_rank
        name: NoOp
  Method name is: 

signature_def['serving_default']:
  The given SavedModel SignatureDef contains the following input(s):
    inputs['Conv1_input'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 28, 28, 1)
        name: serving_default_Conv1_input:0
  The given SavedModel SignatureDef contains the following output(s):
    outputs['Dense'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 10)
        name: StatefulPartitionedCall:0
  Method name is: tensorflow/serving/predict
The MetaGraph with tag set ['serve'] contains the following ops: {'SaveV2', 'Pack', 'ShardedFilename', 'NoOp', 'ReadVariableOp', 'Placeholder', 'Reshape', 'RestoreV2', 'StaticRegexFullMatch', 'Const', 'Identity', 'Select', 'Conv2D', 'AssignVariableOp', 'Relu', 'BiasAdd', 'MergeV2Checkpoints', 'DisableCopyOnRead', 'StringJoin', 'StatefulPartitionedCall', 'VarHandleOp', 'MatMul'}

Concrete Functions:
  Function Name: '__call__'
    Option #1
      Callable with:
        Argument #1
          Conv1_input: TensorSpec(shape=(None, 28, 28, 1), dtype=tf.float32, name='Conv1_input')
        Argument #2
          DType: bool
          Value: False
        Argument #3
          DType: NoneType
          Value: None
    Option #2
      Callable with:
        Argument #1
          Conv1_input: TensorSpec(shape=(None, 28, 28, 1), dtype=tf.float32, name='Conv1_input')
        Argument #2
          DType: bool
          Value: True
        Argument #3
          DType: NoneType
          Value: None

  Function Name: '_default_save_signature'
    Option #1
      Callable with:
        Argument #1
          Conv1_input: TensorSpec(shape=(None, 28, 28, 1), dtype=tf.float32, name='Conv1_input')

  Function Name: 'call_and_return_all_conditional_losses'
    Option #1
      Callable with:
        Argument #1
          Conv1_input: TensorSpec(shape=(None, 28, 28, 1), dtype=tf.float32, name='Conv1_input')
        Argument #2
          DType: bool
          Value: False
        Argument #3
          DType: NoneType
          Value: None
    Option #2
      Callable with:
        Argument #1
          Conv1_input: TensorSpec(shape=(None, 28, 28, 1), dtype=tf.float32, name='Conv1_input')
        Argument #2
          DType: bool
          Value: True
        Argument #3
          DType: NoneType
          Value: None
```

--------------------------------

### setup

Source: https://www.tensorflow.org/tfx/data_validation/api_docs/python/tfdv/CombinerStatsGenerator

Prepares the instance for combining.

```APIDOC
## setup()

### Description
Prepares an instance for combining. Subclasses should use this for costly initializations.
```

--------------------------------

### Get example weights

Source: https://www.tensorflow.org/tfx/model_analysis/api_docs/python/tfma/metrics/StandardMetricInputs?hl=tr

Returns the example weights extract.

```python
get_example_weights(
    model_name: Optional[str] = None, output_name: Optional[str] = None
) -> Optional[types.TensorValueMaybeMultiLevelDict]
```

--------------------------------

### setup

Source: https://www.tensorflow.org/tfx/model_analysis/api_docs/python/tfma/metrics/Preprocessor

Prepares an instance for processing bundles of elements.

```APIDOC
## setup()

### Description
Called to prepare an instance for processing bundles of elements. Useful for initializing transient in-memory resources like network connections.
```

--------------------------------

### Get Example Weights

Source: https://www.tensorflow.org/tfx/model_analysis/api_docs/python/tfma/metrics/StandardMetricInputs

Returns the example weights extract, optionally filtered by model and output name.

```python
get_example_weights(
    model_name: Optional[str] = None, output_name: Optional[str] = None
) -> Optional[types.TensorValueMaybeMultiLevelDict]

```

--------------------------------

### Implement setup

Source: https://www.tensorflow.org/tfx/data_validation/api_docs/python/tfdv/CombinerStatsGenerator

Method for performing costly initializations per worker.

```python
setup() -> None
```

--------------------------------

### setup

Source: https://www.tensorflow.org/tfx/model_analysis/api_docs/python/tfma/metrics/Preprocessor?hl=zh-cn

Called to prepare an instance for processing bundles of elements.

```APIDOC
## setup()

### Description
Called to prepare an instance for processing bundles of elements. This is a good place to initialize transient in-memory resources, such as network connections.
```

--------------------------------

### Install PyArrow

Source: https://www.tensorflow.org/tfx/transform/get_started?hl=ja

Install the PyArrow dependency.

```bash
pip install pyarrow
```

--------------------------------

### Setup lifecycle method

Source: https://www.tensorflow.org/tfx/model_analysis/api_docs/python/tfma/metrics/Preprocessor?hl=zh-cn

Initializes transient resources before processing bundles.

```python
setup()
```

--------------------------------

### Install TFX

Source: https://www.tensorflow.org/tfx/tutorials/tfx/penguin_simple

Installs the latest version of the TFX package.

```bash
pip install -U tfx
```

--------------------------------

### Install TFX

Source: https://www.tensorflow.org/tfx/guide/build_local_pipeline

Install the TFX package in your Python environment.

```bash
pip install tfx
```

--------------------------------

### Configure ExampleGen Parameters

Source: https://www.tensorflow.org/tfx/tutorials/tfx/penguin_template?hl=id

Defines input paths, split configurations, and output formats for the ExampleGen component.

```protobuf
          properties {
            key: "span"
            value: INT
          }
          properties {
            key: "split_names"
            value: STRING
          }
          properties {
            key: "version"
            value: INT
          }
          base_type: DATASET
        }
      }
    }
  }
}
parameters {
  parameters {
    key: "input_base"
    value {
      field_value {
        string_value: "/home/kbuilder/imported/my_pipeline/data"
      }
    }
  }
  parameters {
    key: "input_config"
    value {
      field_value {
        string_value: "{\n  \"splits\": [\n    {\n      \"name\": \"single_split\",\n      \"pattern\": \"*\"\n    }\n  ]\n}"
      }
    }
  }
  parameters {
    key: "output_config"
    value {
      field_value {
        string_value: "{\n  \"split_config\": {\n    \"splits\": [\n      {\n        \"hash_buckets\": 2,\n        \"name\": \"train\"\n      },\n      {\n        \"hash_buckets\": 1,\n        \"name\": \"eval\"\n      }\n    ]\n  }\n}"
      }
    }
  }
  parameters {
    key: "output_data_format"
    value {
      field_value {
        int_value: 6
      }
    }
  }
  parameters {
    key: "output_file_format"
    value {
      field_value {
        int_value: 5
      }
    }
  }
}
downstream_nodes: "StatisticsGen"
execution_options {
  caching_options {
  }
}
```

--------------------------------

### Example output format

Source: https://www.tensorflow.org/tfx/tutorials/tfx/recommenders?hl=zh-cn

The structure of the transformed example data.

```text
channel_name: transformed_examples, split_name: train ("Split-train"), num_examples: 1

features {
  feature {
    key: "movie_title"
    value {
      bytes_list {
        value: "One Flew Over the Cuckoo\'s Nest (1975)"
      }
    }
  }
  feature {
    key: "user_id"
    value {
      bytes_list {
        value: "138"
      }
    }
  }
}
```

--------------------------------

### Build from Source

Source: https://www.tensorflow.org/tfx/transform/install

Steps to create a virtual environment, clone the repository, and build the wheel file locally.

```bash
python3 -m venv <virtualenv_name>
source <virtualenv_name>/bin/activate
pip3 install setuptools wheel
git clone https://github.com/tensorflow/transform.git
cd transform
python3 setup.py bdist_wheel
```

--------------------------------

### Preview ExampleGen output

Source: https://www.tensorflow.org/tfx/tutorials/tfx/penguin_template

Displays the first two examples from each data split.

```python
preview_examples(examples_artifacts)
```

--------------------------------

### Install Package Output

Source: https://www.tensorflow.org/tfx/tutorials/transform/data_preprocessing_with_cloud

Expected output after installing the tensorflow-transform package.

```text
Successfully installed ...
Note: you may need to restart the kernel to use updated packages.
```

--------------------------------

### Initialize ExampleGen Components

Source: https://www.tensorflow.org/tfx/guide/examplegen

Basic instantiation of CsvExampleGen and ImportExampleGen for standard data ingestion.

```python
example_gen = CsvExampleGen(input_base='data_root')
```

```python
example_gen = ImportExampleGen(input_base=path_to_tfrecord_dir)
```

--------------------------------

### Install TensorFlow Transform

Source: https://www.tensorflow.org/tfx/transform/get_started?hl=ja

Install the required library for data transformation.

```bash
pip install -U tensorflow_transform
```

--------------------------------

### Initialize SetupContextParam

Source: https://www.tensorflow.org/tfx/model_analysis/api_docs/python/tfma/metrics/Preprocessor/SetupContextParam

Constructor for defining a context manager to manage DoFn-scoped parameters.

```python
tfma.metrics.Preprocessor.SetupContextParam(
    context_manager_constructor, args=(), kwargs=None, *, name=None
)
```

--------------------------------

### Interactive Context Initialization Logs

Source: https://www.tensorflow.org/tfx/tutorials/tfx/gpt2_finetuning_and_conversion

Example output showing the temporary directory and metadata database location for the interactive context.

```text
WARNING:absl:InteractiveContext pipeline_root argument not provided: using temporary directory /tmpfs/tmp/tfx-interactive-2024-06-19T10_25_02.997883-vluhbwr6 as root for pipeline outputs.
WARNING:absl:InteractiveContext metadata_connection_config not provided: using SQLite ML Metadata database at /tmpfs/tmp/tfx-interactive-2024-06-19T10_25_02.997883-vluhbwr6/metadata.sqlite.
```

--------------------------------

### Initialize and run ExampleValidator

Source: https://www.tensorflow.org/tfx/tutorials/tfx/gpt2_finetuning_and_conversion

Configures the component with statistics and schema inputs, optionally excluding specific data splits.

```python
example_validator = tfx.components.ExampleValidator(
    statistics=statistics_gen.outputs['statistics'],
    schema=schema_gen.outputs['schema'],
    exclude_splits=['eval'],
)
context.run(example_validator, enable_cache=False)
```

--------------------------------

### Install TFX

Source: https://www.tensorflow.org/tfx/tutorials/tfx/components

Installs a specific version of TFX compatible with current constraints.

```bash
# TFX has a constraint of 1.16 due to the removal of tf.estimator support.
pip install "tfx<1.16"
```

--------------------------------

### Install TensorFlow Model Analysis

Source: https://www.tensorflow.org/tfx/tutorials/model_analysis/chicago_taxi?hl=zh-cn

Upgrade pip and install the TFMA package.

```bash
# Upgrade pip to the latest, and install TFMA.
pip install -U pip
pip install tensorflow-model-analysis

```

--------------------------------

### Initialize ExampleProtoCoder

Source: https://www.tensorflow.org/tfx/transform/api_docs/python/tft/coders/ExampleProtoCoder

Instantiate the coder with a schema and specify whether the input consists of serialized Example protos.

```python
tft.coders.ExampleProtoCoder(
    schema, serialized=True
)
```

--------------------------------

### Verify TFDV Installation

Source: https://www.tensorflow.org/tfx/tutorials/data_validation/chicago_taxi

Checks and prints the installed versions of TensorFlow and TFDV.

```python
import tensorflow as tf
import tensorflow_data_validation as tfdv
print('TF version:', tf.__version__)
print('TFDV version:', tfdv.version.__version__)
```

--------------------------------

### Example Usage of create_container_component

Source: https://www.tensorflow.org/tfx/api_docs/python/tfx/v1/dsl/experimental/create_container_component

Demonstrates how to instantiate a container component with defined inputs, outputs, parameters, and command-line placeholders.

```python
  component = create_container_component(
      name='TrainModel',
      inputs={
          'training_data': Dataset,
      },
      outputs={
          'model': Model,
      },
      parameters={
          'num_training_steps': int,
      },
      image='gcr.io/my-project/my-trainer',
      command=[
          'python3', 'my_trainer',
          '--training_data_uri', InputUriPlaceholder('training_data'),
          '--model_uri', OutputUriPlaceholder('model'),
          '--num_training-steps', InputValuePlaceholder('num_training_steps'),
      ]
  )
```

--------------------------------

### Install TFDV Packages

Source: https://www.tensorflow.org/tfx/tutorials/data_validation/chicago_taxi

Installs the TensorFlow Data Validation library and its dependencies.

```python
print('Installing TensorFlow Data Validation')
!pip install --upgrade 'tensorflow_data_validation[visualization]<2'
```

--------------------------------

### Install TFMA via PyPI

Source: https://www.tensorflow.org/tfx/model_analysis/install

Standard installation command for the TFMA library.

```bash
pip install tensorflow-model-analysis
```

--------------------------------

### Initialize and Run ExampleValidator

Source: https://www.tensorflow.org/tfx/tutorials/tfx/components

Instantiate the component with statistics and schema inputs, then execute it within the TFX context.

```python
example_validator = tfx.components.ExampleValidator(
    statistics=statistics_gen.outputs['statistics'],
    schema=schema_gen.outputs['schema'])
context.run(example_validator)
```

--------------------------------

### ExampleDiff Component Initialization

Source: https://www.tensorflow.org/tfx/api_docs/python/tfx/v1/components/ExampleDiff

Initializes the ExampleDiff component with test and base example channels and a configuration object.

```python
tfx.v1.components.ExampleDiff(
    examples_test: tfx.v1.types.BaseChannel,
    examples_base: tfx.v1.types.BaseChannel,
    config: tfx.v1.proto.ExampleDiffConfig,
    include_split_pairs: Optional[List[Tuple[str, str]]] = None
)
```

--------------------------------

### Inspect Transformed Examples

Source: https://www.tensorflow.org/tfx/tutorials/tfx/recommenders?hl=zh-cn

Inspects the transformed examples generated by the Transform component.

```python
inspect_examples(movies_transform, channel_name='transformed_examples')
```

--------------------------------

### Usage Example for tft.scale_to_0_1_per_key

Source: https://www.tensorflow.org/tfx/transform/api_docs/python/tft/scale_to_0_1_per_key

Demonstrates how to integrate the scaling function within a preprocessing_fn and execute it using tft_beam.AnalyzeAndTransformDataset.

```python
def preprocessing_fn(inputs):
  return {
     'scaled': tft.scale_to_0_1_per_key(inputs['x'], inputs['s'])
  }
raw_data = [dict(x=1, s='a'), dict(x=0, s='b'), dict(x=3, s='a')]
feature_spec = dict(
    x=tf.io.FixedLenFeature([], tf.float32),
    s=tf.io.FixedLenFeature([], tf.string))
raw_data_metadata = tft.DatasetMetadata.from_feature_spec(feature_spec)
with tft_beam.Context(temp_dir=tempfile.mkdtemp()):
  transformed_dataset, transform_fn = (
      (raw_data, raw_data_metadata)
      | tft_beam.AnalyzeAndTransformDataset(preprocessing_fn))
transformed_data, transformed_metadata = transformed_dataset
transformed_data
[{'scaled': 0.0}, {'scaled': 0.5}, {'scaled': 1.0}]
```

--------------------------------

### Examples Artifact Properties

Source: https://www.tensorflow.org/tfx/api_docs/python/tfx/v1/types/standard_artifacts/Examples

The dictionary defining the standard properties for the Examples artifact.

```python
{
 'span': PropertyType.INT,
 'split_names': PropertyType.STRING,
 'version': PropertyType.INT
}
```

--------------------------------

### FinalStatusStr Usage Example

Source: https://www.tensorflow.org/tfx/api_docs/python/tfx/v1/orchestration/experimental/FinalStatusStr

Example of passing FinalStatusStr to an exit handler component.

```python
exit_handler = exit_handler_component(
    final_status=tfx.dsl.experimental.FinalStatusStr())
```

--------------------------------

### Environment Version Output

Source: https://www.tensorflow.org/tfx/tutorials/tfx/neural_structured_learning

Example output showing library versions and hardware availability.

```text
2024-08-02 09:07:42.541870: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-08-02 09:07:42.541914: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-08-02 09:07:42.543415: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
TF Version:  2.15.1
Eager mode:  True
GPU is available
NSL Version:  1.4.0
TFX Version:  1.15.1
TFDV version:  1.15.1
TFT version:  1.15.0
TFMA version:  0.46.0
Hub version:  0.15.0
Beam version:  2.57.0
```

--------------------------------

### Install TFX and TFMA

Source: https://www.tensorflow.org/tfx/tutorials/tfx/penguin_template

Installs the necessary TFX and TensorFlow Model Analysis packages.

```bash
pip install -U tfx tensorflow-model-analysis
```

--------------------------------

### Install Built Wheel

Source: https://www.tensorflow.org/tfx/transform/install

Command to install the generated wheel file from the dist directory.

```bash
cd dist
pip3 install tensorflow_transform-<version>-py3-none-any.whl
```

--------------------------------

### Create a ServerCore instance

Source: https://www.tensorflow.org/tfx/serving/api_docs/cc/class/tensorflow/serving/server-core?hl=zh-cn

Initializes a ServerCore instance based on the provided options and waits for statically configured models to load.

```cpp
Status Create(
  Options options,
  std::unique_ptr< ServerCore > *core
)
```

--------------------------------

### Install via PyPI

Source: https://www.tensorflow.org/tfx/transform/install

The standard method for installing the tensorflow-transform package using pip.

```bash
pip install tensorflow-transform
```

--------------------------------

### Installing and Updating ModelServer

Source: https://www.tensorflow.org/tfx/serving/setup

Update the package list and install the TensorFlow ModelServer binary.

```bash
apt-get update && apt-get install tensorflow-model-server
```

--------------------------------

### Test serving model with batch

Source: https://www.tensorflow.org/tfx/tutorials/transform/census

Instantiate the serving model and perform a test run on a batch of serialized examples.

```python
serving_model = ServingModel(model, output_dir)

serving_model(serialized_example_batch)
```

```text
{'classes': <tf.Tensor: shape=(3, 2), dtype=string, numpy=
 array([[b'0', b'1'],
        [b'0', b'1'],
        [b'0', b'1']], dtype=object)>,
 'scores': <tf.Tensor: shape=(3, 2), dtype=float32, numpy=
 array([[-1.6049761e+00,  9.7535902e-01],
        [-5.3329688e-01, -1.6330201e-03],
        [-1.8765860e+00,  1.5198938e+00]], dtype=float32)>}
```

--------------------------------

### Install Built Pip Package

Source: https://www.tensorflow.org/tfx/data_validation/install

Installs the generated wheel file from the dist directory.

```bash
pip install dist/*.whl
```

--------------------------------

### Prepare Monitored Directory

Source: https://www.tensorflow.org/tfx/serving/serving_advanced

Create the directory structure and copy the initial model version for the server to monitor.

```bash
mkdir /tmp/monitored
cp -r /tmp/mnist/1 /tmp/monitored
```

--------------------------------

### Install TFDV via PyPI

Source: https://www.tensorflow.org/tfx/data_validation/install

Standard installation command for the stable TFDV package.

```bash
pip install tensorflow-data-validation
```

--------------------------------

### Run the built binary

Source: https://www.tensorflow.org/tfx/serving/setup

Execute the compiled model server binary from the bazel-bin directory.

```bash
bazel-bin/tensorflow_serving/model_servers/tensorflow_model_server
```

--------------------------------

### Kubeflow Status Command Example

Source: https://www.tensorflow.org/tfx/guide/cli

Example usage of the status command for a Kubeflow pipeline.

```bash
tfx run status --engine=kubeflow --run_id=run-id --pipeline_name=pipeline-name \
--iap_client_id=iap-client-id --namespace=namespace --endpoint=endpoint
```

--------------------------------

### Instantiate and train the model

Source: https://www.tensorflow.org/tfx/tutorials/transform/simple?hl=zh-cn

Initializes the model and demonstrates the training and inference workflow.

```python
trained_model = TrainedModel()
```

```python
trained_model.compile(loss=..., optimizer='adam')
trained_model.fit(...)
```

```python
trained_model_output = trained_model(transformed_batch)
trained_model_output.shape
```

```python
TensorShape([3, 10])
```

--------------------------------

### Initialize ExampleCount

Source: https://www.tensorflow.org/tfx/model_analysis/api_docs/python/tfma/metrics/ExampleCount

Constructor for the ExampleCount metric class.

```python
tfma.metrics.ExampleCount(
    name: str = EXAMPLE_COUNT_NAME
)
```

--------------------------------

### Package installation output

Source: https://www.tensorflow.org/tfx/tutorials/transform/data_preprocessing_with_cloud

Expected output after installing Apache Beam or TensorFlow Transform packages.

```text
Successfully installed ...
```

```text
Successfully installed ...
Note: you may need to restart the kernel to use updated packages.
```

--------------------------------

### Install TFMA notebook extension

Source: https://www.tensorflow.org/tfx/tutorials/tfx/penguin_template

Installs the required TensorFlow Model Analysis extension for JupyterLab.

```bash
# Install TFMA notebook extension.
jupyter labextension install tensorflow_model_analysis@{tfma.__version__}
```
Tokens:
Raw