### Run Pipeline with Dataset and Metrics Source: https://github.com/datamllab/tods/blob/master/docs/source/getting_started.md This command demonstrates how to execute a pre-built pipeline using the run_pipeline.py script. It takes the pipeline description file, dataset path, evaluation metric, and target index as arguments. This is useful for applying anomaly detection models to time-series data. ```bash python examples/run_pipeline.py --pipeline_path example_pipeline.json --table_path datasets/NAB/realTweets/labeled_Twitter_volume_IBM.csv --metric F1_MACRO --target_index 2 ``` -------------------------------- ### Change Directory to Sk Examples Source: https://github.com/datamllab/tods/blob/master/examples/Demo Notebook/TODS Official Demo Notebook.ipynb This command navigates the user to the 'examples/sk_examples' directory, which likely contains example scripts for scikit-learn compatible anomaly detection algorithms within the Tods library. ```bash %cd examples/sk_examples ``` -------------------------------- ### Output Pipeline Description to JSON Source: https://github.com/datamllab/tods/blob/master/docs/source/getting_started.md This Python code snippet demonstrates how to convert a pipeline description object to a JSON string and save it to a file. It then prints the JSON data to the console. Dependencies include the pipeline_description object. ```python data = pipeline_description.to_json() with open('example_pipeline.json', 'w') as f: f.write(data) print(data) ``` -------------------------------- ### Install TODS Package from Git Repository Source: https://github.com/datamllab/tods/blob/master/examples/Demo Notebook/TODS Official Demo Notebook.ipynb This snippet illustrates how to install the TODS package directly from a Git repository using pip. The command clones the repository and installs the package in editable mode. It also lists dependencies already satisfied. ```python !pip install -e git+https://github.com/datamllab/tods.git#egg=tods ``` -------------------------------- ### List Directory Contents Source: https://github.com/datamllab/tods/blob/master/examples/Demo Notebook/TODS Official Demo Notebook.ipynb This command lists the contents of the current directory. In this context, it shows the available example data files and Python test scripts for various anomaly detection algorithms. ```bash !ls ``` -------------------------------- ### Outlier Detection Pipeline with Autoencoder on NAB Dataset (Python) Source: https://github.com/datamllab/tods/blob/master/docs/source/getting_started.md This Python code defines a d3m pipeline for point-wise outlier detection on the NAB dataset using an Autoencoder. It includes steps for data transformation, column parsing, feature extraction, scaling, and prediction construction. The pipeline relies on various d3m primitives and specific semantic types for data interpretation. ```python from d3m import index from d3m.metadata.base import ArgumentType from d3m.metadata.pipeline import Pipeline, PrimitiveStep # Creating pipeline pipeline_description = Pipeline() pipeline_description.add_input(name='inputs') # Step 0: dataset_to_dataframe step_0 = PrimitiveStep(primitive=index.get_primitive('d3m.primitives.data_transformation.dataset_to_dataframe.Common')) step_0.add_argument(name='inputs', argument_type=ArgumentType.CONTAINER, data_reference='inputs.0') step_0.add_output('produce') pipeline_description.add_step(step_0) # Step 1: column_parser step_1 = PrimitiveStep(primitive=index.get_primitive('d3m.primitives.data_transformation.column_parser.Common')) step_1.add_argument(name='inputs', argument_type=ArgumentType.CONTAINER, data_reference='steps.0.produce') step_1.add_output('produce') pipeline_description.add_step(step_1) # Step 2: extract_columns_by_semantic_types(attributes) step_2 = PrimitiveStep(primitive=index.get_primitive('d3m.primitives.data_transformation.extract_columns_by_semantic_types.Common')) step_2.add_argument(name='inputs', argument_type=ArgumentType.CONTAINER, data_reference='steps.1.produce') step_2.add_output('produce') step_2.add_hyperparameter(name='semantic_types', argument_type=ArgumentType.VALUE, data=['https://metadata.datadrivendiscovery.org/types/Attribute']) pipeline_description.add_step(step_2) # Step 3: extract_columns_by_semantic_types(targets) step_3 = PrimitiveStep(primitive=index.get_primitive('d3m.primitives.data_transformation.extract_columns_by_semantic_types.Common')) step_3.add_argument(name='inputs', argument_type=ArgumentType.CONTAINER, data_reference='steps.0.produce') step_3.add_output('produce') step_3.add_hyperparameter(name='semantic_types', argument_type=ArgumentType.VALUE, data=['https://metadata.datadrivendiscovery.org/types/TrueTarget']) pipeline_description.add_step(step_3) attributes = 'steps.2.produce' targets = 'steps.3.produce' # Step 4: processing step_4 = PrimitiveStep(primitive=index.get_primitive('d3m.primitives.tods.timeseries_processing.transformation.axiswise_scaler')) step_4.add_argument(name='inputs', argument_type=ArgumentType.CONTAINER, data_reference=attributes) step_4.add_output('produce') pipeline_description.add_step(step_4) # Step 5: algorithm step_5 = PrimitiveStep(primitive=index.get_primitive('d3m.primitives.tods.detection_algorithm.pyod_ae')) step_5.add_argument(name='inputs', argument_type=ArgumentType.CONTAINER, data_reference='steps.4.produce') step_5.add_output('produce') pipeline_description.add_step(step_5) # Step 6: Predictions step_6 = PrimitiveStep(primitive=index.get_primitive('d3m.primitives.data_transformation.construct_predictions.Common')) step_6.add_argument(name='inputs', argument_type=ArgumentType.CONTAINER, data_reference='steps.5.produce') step_6.add_argument(name='reference', argument_type=ArgumentType.CONTAINER, data_reference='steps.1.produce') step_6.add_output('produce') pipeline_description.add_step(step_6) # Final Output pipeline_description.add_output(name='output predictions', data_reference='steps.6.produce') ``` -------------------------------- ### Setup Argument Parser for Pipeline Execution Source: https://github.com/datamllab/tods/blob/master/examples/Demo Notebook/TODS Notebook Master-Branch.ipynb Initializes an ArgumentParser for configuring TODS pipeline execution parameters. Defines arguments for data path, target column index, evaluation metric, and pipeline JSON path. Includes helpful descriptions and default values for each parameter. ```python parser = argparse.ArgumentParser(description='Arguments for running predefined pipelin.') parser.add_argument('--table_path', type=str, default=default_data_path, help='Input the path of the input data table') parser.add_argument('--target_index', type=int, default=6, help='Index of the ground truth (for evaluation)') parser.add_argument('--metric',type=str, default='F1_MACRO', help='Evaluation Metric (F1, F1_MACRO)') parser.add_argument('--pipeline_path', default=os.path.join(this_path, 'autoencoder_pipeline.json'), help='Input the path of the pre-built pipeline description') ``` -------------------------------- ### Automated Machine Learning (AutoML) Pipeline Search Source: https://github.com/datamllab/tods/blob/master/docs/source/index.md This example showcases using AutoML to automatically find a suitable pipeline for a given dataset. It utilizes the `BruteForceSearch` algorithm to search for the best pipeline based on a specified time limit and a metric like F1-MACRO. Axolotl is used as the backend for this search. ```python import pandas as pd from axolotl.backend.simple import SimpleRunner from tods.utils import generate_dataset_problem from tods.search import BruteForceSearch # Some information #table_path = 'datasets/NAB/realTweets/labeled_Twitter_volume_GOOG.csv' # The path of the dataset #target_index = 2 # what column is the target table_path = 'datasets/yahoo_sub_5.csv' target_index = 6 # what column is the target time_limit = 30 # How many seconds you wanna search metric = 'F1_MACRO' # F1 on both label 0 and 1 # Read data and generate dataset and problem df = pd.read_csv(table_path) dataset= generate_dataset(df, target_index=target_index) problem_description = generate_problem(dataset, metric) # Start backend backend = SimpleRunner(random_seed=0) # Start search algorithm search = BruteForceSearch(problem_description=problem_description, backend=backend) # Find the best pipeline best_runtime, best_pipeline_result = search.search_fit(input_data=[dataset], time_limit=time_limit) best_pipeline = best_runtime.pipeline best_output = best_pipeline_result.output # Evaluate the best pipeline best_scores = search.evaluate(best_pipeline).scores ``` -------------------------------- ### Loading CSV Data with Pandas (Python) Source: https://github.com/datamllab/tods/blob/master/examples/Demo Notebook/TODS Official Demo Notebook.ipynb This example shows how to load data from a CSV file using `pd.read_csv`. Importantly, it specifies the path to the CSV file. The shape and the first 5 rows of the DataFrame are printed. ```python import pandas as pd data_yahoo = pd.read_csv('../../datasets/anomaly/raw_data/yahoo_sub_5.csv') ``` ```python print("shape:", data_yahoo.shape) ``` ```python print("First 5 rows:\n", data_yahoo[:5]) ``` -------------------------------- ### Check Python Version Source: https://github.com/datamllab/tods/blob/master/examples/Demo Notebook/TODS Notebook Master-Branch.ipynb Verifies the installed Python version, ensuring it meets the project's requirements (Python 3.6). This is a crucial step for environment setup. ```python !python -V # Make sure python version is 3.6 ``` -------------------------------- ### Custom Hyperparameter Tuning for Anomaly Detection Source: https://context7.com/datamllab/tods/llms.txt This snippet illustrates the setup for custom hyperparameter tuning of anomaly detection algorithms within a pipeline. It involves loading data and preparing it for pipeline construction, which can then be integrated with a hyperparameter optimization framework. ```python from tods.utils import build_pipeline from tods import generate_dataset, evaluate_pipeline import pandas as pd # Load data df = pd.read_csv('datasets/anomaly/raw_data/yahoo_sub_5.csv') dataset = generate_dataset(df, target_index=6) # Configuration for pipeline building (to be used with tuning) # config = { ... } ``` -------------------------------- ### Initialize Backend Source: https://github.com/datamllab/tods/blob/master/examples/Demo Notebook/TODS Official Demo Notebook.ipynb This initializes a SimpleRunner backend for the machine learning pipeline. ```Python backend = SimpleRunner(random_seed=0) ``` -------------------------------- ### Initialize SimpleRunner Backend Source: https://github.com/datamllab/tods/blob/master/examples/Demo Notebook/TODSBlockchainNotebook.ipynb This code initializes the 'SimpleRunner' backend from the Axolotl library. The 'SimpleRunner' is used to execute D3M pipelines. A 'random_seed' is set for reproducibility of results. ```python # Start backend backend = SimpleRunner(random_seed=0) ``` -------------------------------- ### Initialize BruteForceSearch in Python Source: https://github.com/datamllab/tods/blob/master/examples/Demo Notebook/TODSBlockchainNotebook.ipynb Creates a BruteForceSearch instance with specified problem description and backend. This is likely part of a search or optimization algorithm implementation. ```python search = BruteForceSearch(problem_description=problem_description, backend=backend) ``` -------------------------------- ### Initialize TODS Pipeline and Add Data Processing Steps Source: https://github.com/datamllab/tods/blob/master/examples/Demo Notebook/TODS Official Demo Notebook.ipynb This code creates a basic TODS pipeline, adds an input, and appends primitive steps for dataset_to_dataframe and column_parser. It uses the d3m index for primitives and may encounter dependency warnings like networkx version mismatches. Purpose is pipeline construction for anomaly detection; inputs are primitive references, outputs are step connections, with limitations from unresolved requirements. ```python pipeline_description = Pipeline() pipeline_description.add_input(name='inputs') step_0 = PrimitiveStep(primitive=index.get_primitive('d3m.primitives.tods.data_processing.dataset_to_dataframe')) step_0.add_argument(name='inputs', argument_type=ArgumentType.CONTAINER, data_reference='inputs.0') step_0.add_output('produce') pipeline_description.add_step(step_0) step_1 = PrimitiveStep(primitive=index.get_primitive('d3m.primitives.tods.data_processing.column_parser')) step_1.add_argument(name='inputs', argument_type=ArgumentType.CONTAINER, data_reference='steps.0.produce') step_1.add_output('produce') pipeline_description.add_step(step_1) ``` -------------------------------- ### Evaluate Pipeline Scores in Python Source: https://github.com/datamllab/tods/blob/master/examples/Demo Notebook/TODSBlockchainNotebook.ipynb Evaluates the best pipeline and retrieves its scores. This snippet shows how to use the evaluate method to get performance metrics for a pipeline. ```python best_scores = search.evaluate(best_pipeline).scores ``` -------------------------------- ### Display satoshis and blocks data sample Source: https://github.com/datamllab/tods/blob/master/examples/Demo Notebook/TODSBlockchainNotebook.ipynb Shows the first few rows of the Bitcoin satoshis and blocks data DataFrame to verify successful data retrieval and structure. ```python df_2.head() ``` -------------------------------- ### Importing tods Modules (Python) Source: https://github.com/datamllab/tods/blob/master/examples/Demo Notebook/TODS Official Demo Notebook.ipynb This snippet demonstrates importing essential modules from the `tods` library for generating datasets, pipelines, and searchers. It imports `generate_dataset`, `load_pipeline`, `evaluate_pipeline`, and `BruteForceSearch`. ```python from tods import generate_dataset, load_pipeline, evaluate_pipeline from tods.searcher import BruteForceSearch ``` -------------------------------- ### Import libraries for data analysis and visualization Source: https://github.com/datamllab/tods/blob/master/examples/Demo Notebook/TODSBlockchainNotebook.ipynb Imports essential Python libraries for data manipulation, machine learning, visualization, and Google Cloud BigQuery interaction. Includes scipy, sklearn, matplotlib, and other data science tools. ```python from google.cloud import bigquery from scipy.stats.mstats import zscore from sklearn.metrics import precision_recall_curve from sklearn.metrics import accuracy_score from sklearn.metrics import confusion_matrix from sklearn.metrics import classification_report from tods.sk_interface.detection_algorithm.Telemanom_skinterface import TelemanomSKI from tods.sk_interface.detection_algorithm.DeepLog_skinterface import DeepLogSKI from sklearn.preprocessing import MinMaxScaler, QuantileTransformer from sklearn.cluster import KMeans from sklearn.manifold import TSNE from mpl_toolkits.mplot3d import Axes3D import matplotlib.pyplot as plt plt.style.use('fivethirtyeight') import matplotlib as mpl from pathlib import Path from sklearn.decomposition import PCA from sklearn.preprocessing import StandardScaler import seaborn as sns import datetime as dt from datetime import datetime,tzinfo import scipy, json, csv, time, pytz from pytz import timezone import numpy as np import pandas as pd seed = 135 %config InlineBackend.figure_format = 'retina' %matplotlib inline import os ``` -------------------------------- ### Predict Labels and Scores using TelemanomSKI Source: https://github.com/datamllab/tods/blob/master/examples/Demo Notebook/TODSBlockchainNotebook.ipynb These snippets demonstrate how to use the trained TelemanomSKI regressor to predict anomaly labels and scores for the given data. The `predict` method generates the anomaly labels, and the `predict_score` method generates the anomaly scores. The output shows the structure and some example values of these predictions. ```python prediction_labels_TL = transformer_TL.predict(data) ``` ```python prediction_score_TL = transformer_TL.predict_score(data) ``` ```python print("Prediction Labels\n", prediction_labels_TL) print("Prediction Score\n", prediction_score_TL) ``` -------------------------------- ### Import D3M and ToDS Libraries Source: https://github.com/datamllab/tods/blob/master/examples/Demo Notebook/TODSBlockchainNotebook.ipynb This snippet imports necessary modules from the D3M and ToDS libraries for pipeline creation, data generation, and search operations. It includes imports for Pipeline, PrimitiveStep, SimpleRunner, generate_dataset, generate_problem, and BruteForceSearch. ```python from d3m import index from d3m.metadata.base import ArgumentType from d3m.metadata.pipeline import Pipeline, PrimitiveStep from axolotl.backend.simple import SimpleRunner from tods import generate_dataset, generate_problem from tods.searcher import BruteForceSearch from tods import generate_dataset, load_pipeline, evaluate_pipeline ``` -------------------------------- ### Reshaping Data with NumPy (Python) Source: https://github.com/datamllab/tods/blob/master/examples/Demo Notebook/TODS Official Demo Notebook.ipynb This example demonstrates reshaping a NumPy array by adding an extra dimension using `np.expand_dims`. This is often necessary for machine learning models that expect multi-dimensional input. It splits the data into training and testing sets. ```python X_train = np.expand_dims(data[:10000], axis=1) ``` ```python X_test = np.expand_dims(data[10000:], axis=1) ``` -------------------------------- ### Display transaction data sample Source: https://github.com/datamllab/tods/blob/master/examples/Demo Notebook/TODSBlockchainNotebook.ipynb Shows the first few rows of the Bitcoin transaction data DataFrame to verify successful data retrieval and structure. ```python df_1.head() ``` -------------------------------- ### Advanced Anomaly Detection Pipeline with Feature Engineering Source: https://context7.com/datamllab/tods/llms.txt This example details the creation of an advanced anomaly detection pipeline that integrates time series preprocessing, diverse feature extraction techniques (time, frequency, latent domains), and an ensemble detection algorithm. It also includes a rule-based reinforcement filter. ```python from tods.utils import build_pipeline from tods import generate_dataset, evaluate_pipeline import pandas as pd # Load data df = pd.read_csv('datasets/anomaly/raw_data/yahoo_sub_5.csv') dataset = generate_dataset(df, target_index=6) # Advanced pipeline configuration config = { 'timeseries_processing': [ ['power_transformer', {'standardize': True}], ['holt_winters_exponential_smoothing', { 'trend': 'add', 'seasonal': 'add', 'seasonal_periods': 12 }] ], 'feature_analysis': [ # Time domain features ('statistical_mean', {'window_size': 10}), ('statistical_std', {'window_size': 10}), ('statistical_abs_energy', None), # Frequency domain features ('fft', {'n_fft': 64}), ('auto_correlation', {'lag': 20}), # Latent features ('wavelet_transform', {'wavelet': 'db4'}) ], 'detection_algorithm': [ ['ensemble', { 'base_estimators': ['pyod_ae', 'pyod_lof', 'pyod_iforest'], 'combination': 'average' }] ], 'reinforcement': [ ('rule_based_filter', { 'rule': "#0# > 100 and #1# < 50", # Business rules 'use_columns': [0, 1] }) ] } # Build and evaluate pipeline = build_pipeline(config) pipeline_result = evaluate_pipeline(dataset, pipeline, metric='F1_MACRO') print("Advanced pipeline scores:", pipeline_result.scores) ``` -------------------------------- ### Connect to Google Cloud BigQuery Source: https://github.com/datamllab/tods/blob/master/examples/Demo Notebook/TODSBlockchainNotebook.ipynb Establishes connection to Google Cloud BigQuery using service account credentials. Requires a valid JSON key file for authentication. ```python #Connecting to Google datastore (use path to ur private key) os.environ['GOOGLE_APPLICATION_CREDENTIALS']="blockchain-316820-a21123305db7.json" client = bigquery.Client() ``` -------------------------------- ### Autoencoder Deep Learning Anomaly Detection using SK-Interface in Python Source: https://context7.com/datamllab/tods/llms.txt Utilize a deep learning-based autoencoder for anomaly detection with TODS's scikit-learn compatible interface. This involves preparing multivariate time series data, configuring the AutoEncoderSKI model with parameters like hidden layer architecture and activation functions, training the detector, and then detecting anomalies to get binary labels and continuous anomaly scores. ```python from tods.sk_interface.detection_algorithm import AutoEncoderSKI import numpy as np # Prepare multivariate time series data # Shape: (n_samples, n_features) X_train = np.random.randn(5000, 10) # 5000 time points, 10 features X_test = np.random.randn(1000, 10) # 1000 test points # Configure autoencoder detector = AutoEncoderSKI( contamination=0.1, # Expected proportion of outliers hidden_neurons=[32, 16, 8, 16, 32], # Encoder-decoder architecture hidden_activation='relu', output_activation='sigmoid', epochs=100, batch_size=32, dropout_rate=0.2, validation_size=0.1 ) # Train detector detector.fit(X_train) # Detect anomalies predictions = detector.predict(X_test) # Binary labels (0/1) anomaly_scores = detector.predict_score(X_test) # Continuous scores (0-1) print(f"Detected {np.sum(predictions)} anomalies out of {len(predictions)} samples") print(f"Anomaly score range: {anomaly_scores.min():.3f} - {anomaly_scores.max():.3f}") ``` -------------------------------- ### Data Loading and Problem Definition Source: https://github.com/datamllab/tods/blob/master/examples/Demo Notebook/TODS Official Demo Notebook.ipynb This snippet shows the data loading and problem definition steps. It reads a CSV file into a Pandas DataFrame, generates a dataset object and then creates a problem description based on the dataset and metric. ```Python df = pd.read_csv(table_path) dataset = generate_dataset(df, target_index=target_index) problem_description = generate_problem(dataset, metric) ``` -------------------------------- ### Define Table Path and Target Index Source: https://github.com/datamllab/tods/blob/master/examples/Demo Notebook/TODSBlockchainNotebook.ipynb This code sets up configuration variables for data loading and processing. It defines the 'table_path' for the input CSV file and the 'target_index' which specifies the column containing the target variable for analysis. ```python table_path = 'nodateblock.csv' target_index = 3 # what column is the target time_limit = 30 # How many seconds you wanna search ``` -------------------------------- ### Import D3M and Axolotl Libraries Source: https://github.com/datamllab/tods/blob/master/examples/Demo Notebook/TODS Official Demo Notebook.ipynb This code block imports necessary components from the D3M (Data Mining Discoveries) library for pipeline management and metadata handling, along with the SimpleRunner from Axolotl for backend execution. These are typically used for more complex experiment management and execution frameworks. ```python from d3m import index from d3m.metadata.base import ArgumentType from d3m.metadata.pipeline import Pipeline, PrimitiveStep from axolotl.backend.simple import SimpleRunner ``` -------------------------------- ### Load Pipeline (Python) Source: https://github.com/datamllab/tods/blob/master/examples/Demo Notebook/TODS Official Demo Notebook.ipynb This snippet shows how to load a pre-trained pipeline using the `load_pipeline` function. It takes the pipeline path as input and returns the loaded pipeline object. The `pipeline_path` variable holds the path to the pipeline file. ```python pipeline = load_pipeline(pipeline_path) ``` -------------------------------- ### Checkout Git Branch Source: https://github.com/datamllab/tods/blob/master/examples/Demo Notebook/TODS Official Demo Notebook.ipynb This command checks out a specific Git branch named 'wangyanghe'. It sets up the branch to track the corresponding remote branch from 'origin' and switches the working directory to this new branch. This is useful for working with specific versions or feature branches. ```bash !git checkout wangyanghe ``` -------------------------------- ### Print Best Pipeline Details in Python Source: https://github.com/datamllab/tods/blob/master/examples/Demo Notebook/TODSBlockchainNotebook.ipynb Outputs the details of the best pipeline, including its ID, JSON structure, and scores. This can be used to share or reload the pipeline in other systems. ```python print('Best pipeline:') print('-' * 52) print('Pipeline id:', best_pipeline.id) print('Pipeline json:', best_pipeline.to_json()) print('Output:') print(best_output) print('Scores:') print(best_scores) ``` -------------------------------- ### Pipeline Initialization and Input Addition (Python) Source: https://github.com/datamllab/tods/blob/master/examples/Demo Notebook/TODS Notebook Master-Branch.ipynb This snippet demonstrates how to initialize a Pipeline object and add an input named 'inputs' to it. The output indicates the name of the added input. Dependencies include the Pipeline class. ```python # Creating pipeline pipeline_description = Pipeline() pipeline_description.add_input(name='inputs') ``` -------------------------------- ### Parse Command-Line Arguments Source: https://github.com/datamllab/tods/blob/master/examples/Demo Notebook/TODS Official Demo Notebook.ipynb This Python code uses the 'argparse' module to define and parse command-line arguments for running predefined pipelines. It sets up arguments for input table path, target index, evaluation metric, and pipeline description file path, with default values provided. The snippet shows how to create an argument parser and define several common arguments. ```python import argparse import os this_path = os.path.dirname(os.path.abspath("__file__")) default_data_path = os.path.join(this_path, '../../datasets/anomaly/raw_data/yahoo_sub_5.csv') parser = argparse.ArgumentParser(description='Arguments for running predefined pipelin.') parser.add_argument('--table_path', type=str, default=default_data_path, help='Input the path of the input data table') parser.add_argument('--target_index', type=int, default=6, help='Index of the ground truth (for evaluation)') parser.add_argument('--metric',type=str, default='F1_MACRO', help='Evaluation Metric (F1, F1_MACRO)') parser.add_argument('--pipeline_path', default=os.path.join(this_path, 'autoencoder_pipeline.json'), help='Input the path of the pre-built pipeline description') ``` -------------------------------- ### Execute Data Pipeline and Print Results (Python) Source: https://github.com/datamllab/tods/blob/master/examples/Demo Notebook/TODS Official Demo Notebook.ipynb This Python snippet demonstrates how to evaluate a data pipeline using a given dataset and metric, and then prints the resulting evaluation. It includes a commented-out line for raising an error if pipeline execution fails. ```python pipeline_result = evaluate_pipeline(dataset, pipeline, metric) print(pipeline_result) #raise pipeline_result.error[0] ``` -------------------------------- ### Import Core Python Libraries Source: https://github.com/datamllab/tods/blob/master/examples/Demo Notebook/TODS Official Demo Notebook.ipynb This code block imports essential Python libraries for data manipulation (sys, argparse, os, numpy, pandas), machine learning evaluation (sklearn.metrics), and plotting (matplotlib.pyplot). These are foundational for most data science and machine learning tasks. ```python import sys import argparse import os import numpy as np import pandas as pd from sklearn.metrics import precision_recall_curve from sklearn.metrics import accuracy_score from sklearn.metrics import confusion_matrix from sklearn.metrics import classification_report import matplotlib.pyplot as plt from sklearn import metrics ``` -------------------------------- ### Detection Algorithm (PyOD AE) in Python Source: https://github.com/datamllab/tods/blob/master/examples/Demo Notebook/TODS Official Demo Notebook.ipynb This step applies a detection algorithm, specifically using the PyOD Autoencoder primitive. It takes the output from the previous processing step as input. ```python # Step 5: algorithm` step_5 = PrimitiveStep(primitive=index.get_primitive('d3m.primitives.tods.detection_algorithm.pyod_ae')) step_5.add_argument(name='inputs', argument_type=ArgumentType.CONTAINER, data_reference='steps.4.produce') step_5.add_output('produce') pipeline_description.add_step(step_5) ``` -------------------------------- ### Initializing the Brute-Force Search Algorithm Source: https://github.com/datamllab/tods/blob/master/examples/Demo Notebook/TODS Notebook Master-Branch.ipynb Initializes the BruteForceSearch algorithm provided by D3M. It requires the previously generated problem description and the initialized backend runner. This sets up the search strategy for finding anomaly detection solutions. ```python # Start search algorithm search = BruteForceSearch(problem_description=problem_description, backend=backend) ``` -------------------------------- ### Generate Dataset and Problem Description Source: https://github.com/datamllab/tods/blob/master/examples/Demo Notebook/TODSBlockchainNotebook.ipynb This Python code reads data from a CSV file into a Pandas DataFrame, then uses functions from the ToDS library to generate a dataset object and a problem description. These are essential steps for preparing data for the D3M pipeline execution. ```python # Read data and generate dataset and problem df = pd.read_csv(table_path) dataset = generate_dataset(df, target_index=target_index) problem_description = generate_problem(dataset, metric) ``` -------------------------------- ### Build Pipeline from Configuration - Python Source: https://context7.com/datamllab/tods/llms.txt Constructs TODS detection pipelines using simple configuration dictionaries instead of verbose primitive definitions. The config-based approach allows quick pipeline prototyping with nested lists defining timeseries processing, feature analysis, and detection algorithm stages. Supports hyperparameter customization through dictionaries and None values for defaults. Returns a pipeline object that can be directly evaluated on datasets, providing a more readable alternative to manual primitive construction. ```python from tods.utils import build_pipeline from tods import evaluate_pipeline import pandas as pd # Define pipeline configuration as dictionary config = { 'timeseries_processing': [ ['standard_scaler', {'with_mean': True}] ], 'feature_analysis': [ ('statistical_maximum', {'window_size': 3}), ('statistical_minimum', None) # Use default hyperparameters ], 'detection_algorithm': [ ['pyod_ae', {'hidden_neurons': [32, 16, 8, 16, 32]}] ] } # Build pipeline from config pipeline = build_pipeline(config) # Load data and evaluate df = pd.read_csv('datasets/anomaly/raw_data/yahoo_sub_5.csv') dataset = generate_dataset(df, target_index=6) pipeline_result = evaluate_pipeline(dataset, pipeline, metric='F1_MACRO') print("Scores:", pipeline_result.scores) # Output: {'F1_MACRO': 0.85, 'accuracy': 0.92, ...} ``` -------------------------------- ### Import Project Dependencies Source: https://github.com/datamllab/tods/blob/master/examples/Demo Notebook/TODS Notebook Master-Branch.ipynb Imports various libraries and modules essential for TODS functionality. This includes standard Python libraries, scikit-learn for machine learning, D3M for pipeline management, and TODS-specific modules. ```python import sys import argparse import os import numpy as np import pandas as pd from sklearn.metrics import precision_recall_curve from sklearn.metrics import accuracy_score from sklearn.metrics import confusion_matrix from sklearn.metrics import classification_report import matplotlib.pyplot as plt from sklearn import metrics from d3m import index from d3m.metadata.base import ArgumentType from d3m.metadata.pipeline import Pipeline, PrimitiveStep from axolotl.backend.simple import SimpleRunner from tods import generate_dataset, generate_problem from tods.searcher import BruteForceSearch from tods import generate_dataset, load_pipeline, evaluate_pipeline from tods.sk_interface.detection_algorithm.DeepLog_skinterface import DeepLogSKI from tods.sk_interface.detection_algorithm.Telemanom_skinterface import TelemanomSKI ``` -------------------------------- ### Check Python Version Source: https://github.com/datamllab/tods/blob/master/examples/Demo Notebook/TODS Official Demo Notebook.ipynb This snippet shows how to check the Python version using the command line. It executes the `python -V` command and displays the output including the Python version and environment details. ```python !python -V ``` -------------------------------- ### Initialize and Fit TelemanomSKI Model Source: https://github.com/datamllab/tods/blob/master/examples/Demo Notebook/TODS Official Demo Notebook.ipynb This snippet initializes the TelemanomSKI transformer with specified parameters for anomaly detection, fits it on training data, and generates predictions and scores on test data. It relies on the TODS library and assumes pre-loaded X_train and X_test datasets. Outputs include binary labels and anomaly scores; limitations include potential deprecation warnings from scikit-learn dependencies. ```python transformer = TelemanomSKI(l_s= 2, n_predictions= 1) transformer.fit(X_train) prediction_labels_train = transformer.predict(X_train) prediction_labels_test = transformer.predict(X_test) prediction_score = transformer.predict_score(X_test) ``` -------------------------------- ### Standardize Data and Prepare for Outlier Detection Source: https://github.com/datamllab/tods/blob/master/examples/Demo Notebook/TODSBlockchainNotebook.ipynb Selects key features ('Output_Satoshis', 'Blocks', 'Transactions') from the result DataFrame, standardizes them using StandardScaler, and converts the scaled data into a NumPy array. This step is crucial for algorithms that are sensitive to the scale of input features, such as anomaly detection models. ```python # select the three most important features (Transactions, Blocks, Output Satoshis) from the data data = result[['Output_Satoshis','Blocks','Transactions']] outliers_fraction=0.05 scaler = StandardScaler() np_scaled = scaler.fit_transform(data) data = pd.DataFrame(np_scaled).to_numpy() print(data) ``` -------------------------------- ### Configuration for Anomaly Detection Data Loading Source: https://github.com/datamllab/tods/blob/master/examples/Demo Notebook/TODS Notebook Master-Branch.ipynb Sets the path to the CSV file containing the dataset and specifies the index of the target column and the time limit for the search. These configurations are essential for loading and processing the data correctly. ```python table_path = '../../datasets/anomaly/raw_data/yahoo_sub_5.csv' target_index = 6 # column of the target label time_limit = 30 # How many seconds you wanna search ``` -------------------------------- ### Configure Pipeline Output in Python Source: https://github.com/datamllab/tods/blob/master/examples/Demo Notebook/TODS Notebook Master-Branch.ipynb Adds an output step to a TODS pipeline description object. Specifies the data reference from step 6's produce method and names the output. Returns the output reference identifier. ```python pipeline_description.add_output(name='output predictions', data_reference='steps.6.produce') ``` -------------------------------- ### Inspecting Reshaped Data (Python) Source: https://github.com/datamllab/tods/blob/master/examples/Demo Notebook/TODS Official Demo Notebook.ipynb This snippet prints the first 5 rows of the reshaped training and testing data to verify the reshaping operation. This allows for a quick visual confirmation that the data has been prepared correctly for subsequent analysis or model training. ```python print("First 5 rows train:\n", X_train[:5]) ``` ```python print("First 5 rows test:\n", X_test[:5]) ``` -------------------------------- ### DeepLog Model Initialization and Prediction - Python Source: https://github.com/datamllab/tods/blob/master/examples/Demo Notebook/TODS Official Demo Notebook.ipynb Initializes the DeepLog anomaly detection model, fits it to training data, and generates predictions and anomaly scores for both training and testing datasets. This is a core step in applying the DeepLog algorithm. ```python transformer = DeepLogSKI() transformer.fit(X_train) prediction_labels_train = transformer.predict(X_train) prediction_labels_test = transformer.predict(X_test) prediction_score = transformer.predict_score(X_test) ``` -------------------------------- ### Evaluate Pipeline in Python Source: https://github.com/datamllab/tods/blob/master/docs/source/index.md This snippet demonstrates evaluating a pre-defined pipeline on a time-series dataset using the `evaluate_pipeline` function. It loads data, generates a dataset and problem description, loads a default pipeline, and then runs the pipeline to produce a result. ```python import pandas as pd from tods import schemas as schemas_utils from tods.utils import generate_dataset_problem, evaluate_pipeline table_path = 'datasets/yahoo_sub_5.csv' target_index = 6 # what column is the target metric = 'F1_MACRO' # F1 on both label 0 and 1 # Read data and generate dataset and problem df = pd.read_csv(table_path) dataset, problem_description = generate_dataset_problem(df, target_index) # Load the default pipeline pipeline = schemas_utils.load_default_pipeline() # Run the pipeline pipeline_result = evaluate_pipeline(dataset, pipeline, metric) ``` -------------------------------- ### Search and Fit Best Pipeline using TODs Source: https://github.com/datamllab/tods/blob/master/examples/Demo Notebook/TODS Notebook Master-Branch.ipynb This code snippet uses the TODs library's search module to find and fit the best pipeline for a given dataset within a specified time limit. It requires the search module and a prepared dataset object. Inputs include the dataset and time limit; outputs are the runtime, pipeline result, extracted pipeline, and output. Limitations include potential hyper-parameter warnings and runtime errors if pipelines fail. ```python # Find the best pipeline best_runtime, best_pipeline_result = search.search_fit(input_data=[dataset], time_limit=time_limit) best_pipeline = best_runtime.pipeline best_output = best_pipeline_result.output ``` -------------------------------- ### Read CSV and Generate Dataset (Python) Source: https://github.com/datamllab/tods/blob/master/examples/Demo Notebook/TODS Official Demo Notebook.ipynb This snippet demonstrates how to read data from a CSV file using pandas and generate a dataset. It assumes the existence of functions `generate_dataset` and pre-defined variables `table_path` and `target_index`. The generated dataset is likely used for machine learning tasks. ```python df = pd.read_csv(table_path) dataset = generate_dataset(df, target_index) ``` -------------------------------- ### Construct Predictions in Python Source: https://github.com/datamllab/tods/blob/master/examples/Demo Notebook/TODS Official Demo Notebook.ipynb This step constructs the final predictions by combining the output of the detection algorithm with a reference input. It uses the `construct_predictions` primitive. ```python # Step 6: Predictions step_6 = PrimitiveStep(primitive=index.get_primitive('d3m.primitives.tods.data_processing.construct_predictions')) step_6.add_argument(name='inputs', argument_type=ArgumentType.CONTAINER, data_reference='steps.5.produce') step_6.add_argument(name='reference', argument_type=ArgumentType.CONTAINER, data_reference='steps.1.produce') step_6.add_output('produce') pipeline_description.add_step(step_6) ``` -------------------------------- ### Search Best Pipeline in Python Source: https://github.com/datamllab/tods/blob/master/examples/Demo Notebook/TODSBlockchainNotebook.ipynb This Python code snippet uses the search object's search_fit method to identify the optimal pipeline from input datasets within a specified time limit, likely part of the TODS library for automated machine learning on time series data. It depends on the search instance, dataset, and time_limit variables being defined; inputs are input_data as a list of datasets and time_limit as an integer, outputs are best_runtime and best_pipeline_result. A limitation is the assumption that the search object is already configured and fitted appropriately for the given data. ```python # Find the best pipeline best_runtime, best_pipeline_result = search.search_fit(input_data=[dataset], time_limit=time_limit) ``` -------------------------------- ### Print Yahoo Dataset Info Source: https://github.com/datamllab/tods/blob/master/examples/Demo Notebook/TODS Notebook Master-Branch.ipynb Prints the shape and the first five rows of the loaded Yahoo dataset. This provides an initial understanding of the dataset's dimensions and the structure of its data points. ```python print("shape:", data_yahoo.shape) print("First 5 rows:\n", data_yahoo[:5]) ``` -------------------------------- ### Query Bitcoin satoshis and block data from BigQuery Source: https://github.com/datamllab/tods/blob/master/examples/Demo Notebook/TODSBlockchainNotebook.ipynb Retrieves daily Bitcoin block counts and total satoshis spent from the public BigQuery Bitcoin blockchain dataset. Converts satoshis to standard units and filters for the same date range. ```python # The query to get sum of all satoshis spent each day and number of blocks query_2 = """ SELECT o.Date, COUNT(o.block) AS Blocks, SUM(o.output_price) AS Output_Satoshis FROM ( SELECT DATE(TIMESTAMP_MILLIS(timestamp)) AS Date, output.output_satoshis AS output_price, block_id AS block FROM `bigquery-public-data.bitcoin_blockchain.transactions`, UNNEST(outputs) AS output ) AS o GROUP BY o.date HAVING o.date >= '2009-01-09' AND o.date <= '2020-06-12' ORDER BY o.date, blocks """ query_job_2 = client.query(query_2) # Waits for the query to finish iterator_2 = query_job_2.result(timeout=30) rows_2 = list(iterator_2) df_2 = pd.DataFrame(data=[list(x.values()) for x in rows_2], columns=list(rows_2[0].keys())) df_2["Output_Satoshis"]= df_2["Output_Satoshis"].apply(lambda x: float(x/100000000)) ``` -------------------------------- ### Output Pipeline Description to JSON Source: https://github.com/datamllab/tods/blob/master/examples/Demo Notebook/TODS Official Demo Notebook.ipynb This Python code snippet demonstrates how to convert a pipeline description object to a JSON string and write it to a file. It ensures the JSON data is saved and also printed to the console. No external libraries are explicitly imported here, assuming 'pipeline_description' is defined elsewhere. ```python data = pipeline_description.to_json() with open('autoencoder_pipeline.json', 'w') as f: f.write(data) print(data) ``` -------------------------------- ### Print Search History in Python Source: https://github.com/datamllab/tods/blob/master/examples/Demo Notebook/TODSBlockchainNotebook.ipynb Prints the search history of pipelines with their IDs and performance scores. Useful for analyzing the performance of different pipelines. ```python print('Search History:') for pipeline_result in search.history: print('-' * 52) print('Pipeline id:', pipeline_result.pipeline.id) print(pipeline_result.scores) ``` -------------------------------- ### Parse CLI Arguments for Pipeline Configuration Source: https://github.com/datamllab/tods/blob/master/examples/Demo Notebook/TODS Notebook Master-Branch.ipynb Processes command-line arguments using parse_known_args() to extract pipeline configuration values. Assigns parsed values to variables for table path, target index, pipeline path, and evaluation metric. The method allows unknown arguments without throwing errors. ```python args, unknown = parser.parse_known_args() table_path = args.table_path target_index = args.target_index # what column is the target pipeline_path = args.pipeline_path metric = args.metric # F1 on both label 0 and 1 ``` -------------------------------- ### Data Preparation Pipeline Hyperparameter Warning (Log Output) Source: https://github.com/datamllab/tods/blob/master/examples/Demo Notebook/TODS Official Demo Notebook.ipynb This entry represents a log output indicating that not all provided hyperparameters for a data preparation pipeline were utilized. It lists the specific hyperparameters that were not used. ```log Not all provided hyper-parameters for the data preparation pipeline 79ce71bd-db96-494b-a455-14f2e2ac5040 were used: ['method', 'number_of_folds', 'randomSeed', 'shuffle', 'stratified'] ``` -------------------------------- ### Run Pipeline Evaluation Source: https://github.com/datamllab/tods/blob/master/examples/Demo Notebook/TODS Notebook Master-Branch.ipynb This Python code executes a data pipeline and prints the results. It assumes the dataset, pipeline, and metric are pre-defined. The output is the evaluation result of the pipeline. ```python pipeline_result = evaluate_pipeline(dataset, pipeline, metric) print(pipeline_result) ``` -------------------------------- ### Neural Network Model Summary and Training Progress (Log Output) Source: https://github.com/datamllab/tods/blob/master/examples/Demo Notebook/TODS Official Demo Notebook.ipynb This log output details the architecture of a sequential neural network model, including layer types, output shapes, and parameter counts. It also shows the progress of the model's training over epochs, displaying loss and validation loss for each step. ```log Model: "sequential_2" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= dense_2 (Dense) (None, 12) 156 _________________________________________________________________ dropout_2 (Dropout) (None, 12) 0 _________________________________________________________________ dense_3 (Dense) (None, 12) 156 _________________________________________________________________ dropout_3 (Dropout) (None, 12) 0 _________________________________________________________________ dense_4 (Dense) (None, 1) 13 _________________________________________________________________ dropout_4 (Dropout) (None, 1) 0 _________________________________________________________________ dense_5 (Dense) (None, 4) 8 _________________________________________________________________ dropout_5 (Dropout) (None, 4) 0 _________________________________________________________________ dense_6 (Dense) (None, 1) 5 _________________________________________________________________ dropout_6 (Dropout) (None, 1) 0 _________________________________________________________________ dense_7 (Dense) (None, 12) 24 ================================================================= Total params: 362 Trainable params: 362 Non-trainable params: 0 _________________________________________________________________ None Epoch 1/100 40/40 [==============================] - 0s 6ms/step - loss: 2.1020 - val_loss: 1.3966 Epoch 2/100 40/40 [==============================] - 0s 1ms/step - loss: 1.8250 - val_loss: 1.2834 Epoch 3/100 40/40 [==============================] - 0s 1ms/step - loss: 1.7095 - val_loss: 1.2056 Epoch 4/100 40/40 [==============================] - 0s 1ms/step - loss: 1.6036 - val_loss: 1.1504 Epoch 5/100 40/40 [==============================] - 0s 1ms/step - loss: 1.5416 - val_loss: 1.1075 Epoch 6/100 40/40 [==============================] - 0s 1ms/step - loss: 1.4905 - val_loss: 1.0713 Epoch 7/100 40/40 [==============================] - 0s 1ms/step - loss: 1.4248 - val_loss: 1.0404 Epoch 8/100 40/40 [==============================] - 0s 1ms/step - loss: 1.4080 - val_loss: 1.0133 Epoch 9/100 40/40 [==============================] - 0s 1ms/step - loss: 1.3664 - val_loss: 0.9888 Epoch 10/100 40/40 [==============================] - 0s 2ms/step - loss: 1.3319 - val_loss: 0.9664 Epoch 11/100 40/40 [==============================] - 0s 2ms/step - loss: 1.2825 - val_loss: 0.9456 Epoch 12/100 40/40 [==============================] - 0s 1ms/step - loss: 1.2695 - val_loss: 0.9260 Epoch 13/100 40/40 [==============================] - 0s 1ms/step - loss: 1.2545 - val_loss: 0.9075 Epoch 14/100 40/40 [==============================] - 0s 1ms/step - loss: 1.2153 - val_loss: 0.8899 Epoch 15/100 40/40 [==============================] - 0s 1ms/step - loss: 1.2071 - val_loss: 0.8733 Epoch 16/100 40/40 [==============================] - 0s 1ms/step - loss: 1.1693 - val_loss: 0.8575 Epoch 17/100 40/40 [==============================] - 0s 1ms/step - loss: 1.1569 - val_loss: 0.8424 Epoch 18/100 40/40 [==============================] - 0s 1ms/step - loss: 1.1470 - val_loss: 0.8280 Epoch 19/100 40/40 [==============================] - 0s 1ms/step - loss: 1.1229 - val_loss: 0.8143 Epoch 20/100 40/40 [==============================] - 0s 2ms/step - loss: 1.1088 - val_loss: 0.8011 Epoch 21/100 40/40 [==============================] - 0s 2ms/step - loss: 1.0923 - val_loss: 0.7885 Epoch 22/100 40/40 [==============================] - 0s 2ms/step - loss: 1.0745 - val_loss: 0.7764 Epoch 23/100 40/40 [==============================] - 0s 2ms/step - loss: 1.0592 - val_loss: 0.7648 Epoch 24/100 40/40 [==============================] - 0s 2ms/step - loss: 1.0476 - val_loss: 0.7537 Epoch 25/100 40/40 [==============================] - 0s 1ms/step - loss: 1.0341 - val_loss: 0.7430 Epoch 26/100 40/40 [==============================] - 0s 1ms/step - loss: 1.0216 - val_loss: 0.7328 Epoch 27/100 40/40 [==============================] - 0s 1ms/step - loss: 1.0110 - val_loss: 0.7230 Epoch 28/100 40/40 [==============================] - 0s 1ms/step - loss: 0.9972 - val_loss: 0.7136 Epoch 29/100 ``` -------------------------------- ### Best F1-Score and Threshold Identification - Python Source: https://github.com/datamllab/tods/blob/master/examples/Demo Notebook/TODS Official Demo Notebook.ipynb Identifies and prints the optimal threshold for classification based on the F1 scores and the corresponding best F1-score achieved. This helps in selecting the best operating point for the model. ```python print('Best threshold: ', thresholds[np.argmax(f1_scores)]) print('Best F1-Score: ', np.max(f1_scores)) ``` -------------------------------- ### Build System-wise Anomaly Detection Pipeline Source: https://context7.com/datamllab/tods/llms.txt This snippet shows how to construct a system-wise anomaly detection pipeline using TODS utilities. It involves loading system data, configuring a pipeline with feature analysis and detection algorithms, and then evaluating its performance. ```python from tods.utils import build_system_pipeline from tods import generate_dataset, evaluate_pipeline import pandas as pd # Load system-wise data df = pd.read_csv('datasets/anomaly/system_wise/sample/train.csv') system_dir = 'datasets/anomaly/system_wise/sample/systems' dataset = generate_dataset(df, target_index=2, system_dir=system_dir) # Configure system-wise pipeline config = { 'feature_analysis': [ ('statistical_maximum', None), ('statistical_mean', {'window_size': 5}) ], 'detection_algorithm': [ ('pyod_ocsvm', {'contamination': 0.1}) ] } # Build system pipeline (includes denormalization step) pipeline = build_system_pipeline(config) # Evaluate pipeline_result = evaluate_pipeline(dataset, pipeline, metric='F1_MACRO') if pipeline_result.status == 'ERRORED': raise pipeline_result.error[0] else: print("System-wise detection scores:", pipeline_result.scores) ``` -------------------------------- ### Change Directory to Tods Source Source: https://github.com/datamllab/tods/blob/master/examples/Demo Notebook/TODS Official Demo Notebook.ipynb This command changes the current working directory to the src/tods directory within the project. This is often a prerequisite for running scripts or commands related to the Tods library. ```bash %cd src/tods ```