### Manual Development Environment Setup (Bash)

Source: https://github.com/databrickslabs/tempo/blob/master/CONTRIBUTING.md

Provides alternative manual steps for setting up the development environment if 'make init' is not used. It includes installing development tools and creating a virtual environment.

```bash
make setup
make venv
```

--------------------------------

### Install Tempo from Source using pip (Bash)

Source: https://github.com/databrickslabs/tempo/blob/master/docs/index.md

Shows how to install the Tempo library by cloning the latest source code from its GitHub repository. This method is useful for accessing the most recent features or for development purposes. It uses pip with an editable install flag.

```bash
$ pip install -e git+https://github.com/databrickslabs/tempo.git#"egg=dbl-tempo&subdirectory=python"
```

--------------------------------

### Install Tempo Library

Source: https://github.com/databrickslabs/tempo/blob/master/Tempo QuickStart - Python.ipynb

Installs the Tempo library from a Git repository using pip. This is a prerequisite for using Tempo's functionalities.

```python
%pip install -e git+https://github.com/databrickslabs/tempo.git@baf0d7b9b53b2c46f4452d86bcc487ae0b72b708#egg=tempo&subdirectory=python
```

--------------------------------

### Install Tempo using pip

Source: https://github.com/databrickslabs/tempo/blob/master/docs/about/user-guide.md

Installs the dbl-tempo library using pip. This is the primary method for adding Tempo to your Python environment.

```bash
python -m pip install dbl-tempo
```

--------------------------------

### Install Tempo using pip (Bash)

Source: https://github.com/databrickslabs/tempo/blob/master/docs/index.md

Provides the command to install the Tempo library directly from PyPI using pip. This is the standard method for adding the library to your Python environment.

```bash
$ pip install dbl-tempo
```

--------------------------------

### Troubleshoot Dependency Issues (Bash)

Source: https://github.com/databrickslabs/tempo/blob/master/CONTRIBUTING.md

Reinstalls development tools to resolve potential dependency issues encountered during setup or command execution.

```bash
make setup
```

--------------------------------

### Install Python Versions with Pyenv (Bash)

Source: https://github.com/databrickslabs/tempo/blob/master/CONTRIBUTING.md

Installs specific Python versions (3.9, 3.10, 3.11) required for testing using the pyenv version manager.

```bash
pyenv install 3.9 3.10 3.11
```

--------------------------------

### Install dbl-tempo Package

Source: https://context7.com/databrickslabs/tempo/llms.txt

Installs the dbl-tempo Python package using pip. This is the primary method for obtaining the Tempo library.

```bash
pip install dbl-tempo
```

--------------------------------

### Install Tempo Python Package

Source: https://github.com/databrickslabs/tempo/blob/master/python/README.md

Installs the dbl-tempo Python package in Databricks notebooks or locally. This is the first step to using Tempo's time series utilities.

```shell
%pip install dbl-tempo
```

```shell
pip install dbl-tempo
```

--------------------------------

### Troubleshoot Python Version Issues (Bash)

Source: https://github.com/databrickslabs/tempo/blob/master/CONTRIBUTING.md

Installs all required Python versions for the project, which can help resolve Python version-related problems.

```bash
make setup-python-versions
```

--------------------------------

### Slice TSDF by specific timestamps in Python

Source: https://github.com/databrickslabs/tempo/blob/master/docs/about/user-guide.md

Demonstrates various methods to slice a TSDF object based on timestamps. This includes selecting data at a specific time, before or after a time, within an interval, and retrieving the earliest or latest records.

```python
target_time = '2015-02-23T13:03:53.919+0000'
at_target_tsdf = phone_accel_tsdf.at(target_time)
display(at_target_tsdf)
```

```python
before_tsdf = phone_accel_tsdf.before(target_time)
at_or_after_tsdf = phone_accel_tsdf.atOrAfter(target_time)
```

```python
start_ts = '2015-02-23T13:03:53.909+0000'
end_ts = target_time
interval_inclusive = phone_accel_tsdf.between(start_ts, end_ts)
interval_exclusive = phone_accel_tsdf.between(start_ts, end_ts, inclusive=False)
```

```python
n = 5
oldest_five_tsdf = phone_accel_tsdf.earliest(n)
latest_five_tsdf = phone_accel_tsdf.latest(n)
```

```python
as_of_tsdf = phone_accel_tsdf.priorTo(target_time)
next_five_tsdf = phone_accel_tsdf.subsequentTo(target_time, n=5)
```

--------------------------------

### Resample and visualize TSDF data in Python

Source: https://github.com/databrickslabs/tempo/blob/master/docs/about/user-guide.md

Resamples the TSDF data to a specified frequency ('min' in this case) using a 'floor' aggregation function. It then converts the resampled data to a Pandas DataFrame for visualization with Plotly, showing 'z' accelerometer values over time, grouped by 'User'.

```python
# ts_col = timestamp column on which to sort fact and source table
# partition_cols - columns to use for partitioning the TSDF into more granular time series for windowing and sorting

resampled_sdf = phone_accel_tsdf.resample(freq='min', func='floor')
resampled_pdf = resampled_sdf.df.filter(col('event_ts').cast("date") == "2015-02-23").toPandas()

import plotly.graph_objs as go
import plotly.express as px
import pandas as pd

# Plotly figure 1
fig = px.line(resampled_pdf, x='event_ts', y='z',
color="User",
line_group="User", hover_name="User")
fig.update_layout(title='Phone Accelerometer Usage' , showlegend=False)

fig.show()
```

--------------------------------

### Initialize Development Environment (Bash)

Source: https://github.com/databrickslabs/tempo/blob/master/CONTRIBUTING.md

Sets up the complete development environment for new contributors. This is the recommended first step after cloning the repository.

```bash
cd tempo/python
make init
```

--------------------------------

### As-Of Join with Partitioning and Fraction

Source: https://github.com/databrickslabs/tempo/blob/master/Tempo QuickStart - Python.ipynb

Performs an as-of join with additional parameters for time partitioning and fraction. This allows for more controlled and potentially faster joins on large datasets by specifying join strategy.

```python
joined_df = watch_accel_tsdf.asofJoin(phone_accel_tsdf, right_prefix="watch_accel", tsPartitionVal = 10, fraction = 0.1).df
display(joined_df)
```

--------------------------------

### Project Cleaning and Initialization (Bash)

Source: https://github.com/databrickslabs/tempo/blob/master/CONTRIBUTING.md

Provides commands to clean and reinitialize the project environment. `make clean` removes generated files, `make deep-clean` removes all artifacts including virtual environments and caches, and `make init` reinstalls all dependencies.

```bash
make clean          # Remove all generated files
make deep-clean     # Remove everything (venvs, caches, artifacts)
make init           # Reinstall everything
```

--------------------------------

### Clone Repository and Navigate (Bash)

Source: https://github.com/databrickslabs/tempo/blob/master/CONTRIBUTING.md

Clones the Tempo repository from GitHub and navigates into the Python development directory. This is a prerequisite for setting up the development environment.

```bash
git clone https://github.com/databrickslabs/tempo.git
cd tempo/python
```

--------------------------------

### Interpolate Time-Series Data with Tempo

Source: https://github.com/databrickslabs/tempo/blob/master/docs/about/user-guide.md

Interpolates missing values in a time-series DataFrame (TSDF) based on specified partition columns, timestamp column, frequency, interpolation function, method, and target columns. The `show_interpolated` parameter can be set to True to indicate which rows have been interpolated.

```python
interpolated_tsdf = input_tsdf.interpolate(
    partition_cols=["partition_c"],
    ts_col="other_event_ts"
    freq="30 seconds",
    func="mean",
    method="linear",
    target_cols= ["columnA","columnB"],
    show_interpolated=True,
)
```

--------------------------------

### Build Project Artifacts (Bash)

Source: https://github.com/databrickslabs/tempo/blob/master/CONTRIBUTING.md

Commands to build various artifacts for the Tempo project, including distribution packages and documentation.

```bash
make build
make build-dist
make build-docs
```

--------------------------------

### Interpolation Methods for Time Series in Tempo

Source: https://github.com/databrickslabs/tempo/blob/master/docs/about/user-guide.md

Interpolates a time series to fill missing values using various methods like zero fill, null fill, backwards fill, forwards fill, and linear fill. It can be used independently or after resampling, and supports specifying target columns, frequency, aggregation function, and overriding default TSDF parameters.

```python
# Create instance of the TSDF class
input_tsdf = TSDF(
            input_df,
            partition_cols=["partition_a", "partition_b"],
            ts_col="event_ts",
        )


# Aggregate all valid numeric columns using mean into 30 second intervals and interpolate missing values using linear fill
interpolated_tsdf = input_tsdf.resample(freq="30 seconds", func="mean").interpolate(
    method="linear"
)
```

```python
# Aggregate specified columns using mean into 30 second intervals and interpolate missing values using linear fill
interpolated_tsdf = input_tsdf.interpolate(
    freq="30 seconds",
    func="mean",
    target_cols= ["columnA","columnB"],
    method="linear"

)
```

```python
# Override default TSDF parameters for interpolation
interpolated_tsdf = input_tsdf.interpolate(
    partition_cols=["partition_c"],
    ts_col="other_event_ts"
    freq="30 seconds",
    func="mean",
    target_cols= ["columnA","columnB"],
    method="linear"

)
```

--------------------------------

### Run Complete CI Pipeline Locally (Bash)

Source: https://github.com/databrickslabs/tempo/blob/master/CONTRIBUTING.md

Executes the full Continuous Integration (CI) pipeline locally, mimicking the checks performed in the GitHub Actions workflow. This includes quality checks, testing all DBR versions, building artifacts, and generating a coverage report.

```bash
make ci
make all
```

--------------------------------

### Load and Prepare Watch Accelerometer Data

Source: https://github.com/databrickslabs/tempo/blob/master/Tempo QuickStart - Python.ipynb

Loads watch accelerometer data from a CSV file, similar to the phone data. It casts relevant columns to appropriate types and adds an event timestamp column.

```python
from pyspark.sql.functions import *

watch_accel_df = spark.read.format("csv").option("header", "true").load("dbfs:/home/tempo/Watch_accelerometer").withColumn("event_ts", (col("Arrival_Time").cast("double")/1000).cast("timestamp")).withColumn("x", col("x").cast("double")).withColumn("y", col("y").cast("double")).withColumn("z", col("z").cast("double")).withColumn("event_ts_dbl", col("event_ts").cast("double"))

#13062475
display(watch_accel_df)
```

--------------------------------

### Run Tests Against All DBR Versions (Bash)

Source: https://github.com/databrickslabs/tempo/blob/master/CONTRIBUTING.md

Runs the test suite against all supported Databricks Runtime (DBR) versions.

```bash
make test-all
```

--------------------------------

### Describe Watch Accelerometer Time Series Data

Source: https://github.com/databrickslabs/tempo/blob/master/Tempo QuickStart - Python.ipynb

Generates descriptive statistics for the watch accelerometer time series DataFrame using Tempo's describe() method. This helps in understanding the characteristics of the watch sensor data.

```python
watch_accel_tsdf = TSDF(watch_accel_df, ts_col="event_ts", partition_cols = ["User"])
display(watch_accel_tsdf.describe())
```

--------------------------------

### Describe Phone Accelerometer Time Series Data

Source: https://github.com/databrickslabs/tempo/blob/master/Tempo QuickStart - Python.ipynb

Generates descriptive statistics for the phone accelerometer time series DataFrame using Tempo's describe() method. This provides insights into the distribution and characteristics of the time series data.

```python
display(phone_accel_tsdf.describe())
```

--------------------------------

### Run All Code Quality Checks (Bash)

Source: https://github.com/databrickslabs/tempo/blob/master/CONTRIBUTING.md

Executes all code quality checks (formatting, linting, and type-checking) sequentially.

```bash
make quality
```

--------------------------------

### Set Local Python Version with Pyenv (Bash)

Source: https://github.com/databrickslabs/tempo/blob/master/CONTRIBUTING.md

Creates a .python-version file within the current directory to specify the Python versions pyenv should use for this project.

```bash
pyenv local 3.9 3.10 3.11
```

--------------------------------

### Initialize Time Series DataFrame (Python)

Source: https://github.com/databrickslabs/tempo/blob/master/docs/index.md

Demonstrates how to load and prepare a CSV file into a Spark DataFrame, cast columns to appropriate types (timestamp, double), and then initialize a Tempo Time Series DataFrame (TSDF). This involves specifying the timestamp column and partitioning columns for efficient time series operations.

```python
from pyspark.sql.functions import *
phone_accel_df = spark.read.format("csv").option("header", "true").load("dbfs:/home/tempo/Phones_accelerometer").withColumn("event_ts", (col("Arrival_Time").cast("double")/1000).cast("timestamp")).withColumn("x", col("x").cast("double")).withColumn("y", col("y").cast("double")).withColumn("z", col("z").cast("double")).withColumn("event_ts_dbl", col("event_ts").cast("double"))
from tempo import *
phone_accel_tsdf = TSDF(phone_accel_df, ts_col="event_ts", partition_cols = ["User"])
display(phone_accel_tsdf)
```

--------------------------------

### Check Development Environment Status (Bash)

Source: https://github.com/databrickslabs/tempo/blob/master/CONTRIBUTING.md

Commands to quickly check the status or perform a comprehensive health check of the development environment.

```bash
make status
make doctor
```

--------------------------------

### Describe Delta Table History

Source: https://github.com/databrickslabs/tempo/blob/master/Tempo QuickStart - Python.ipynb

Displays the history of the 'delta_silver_phone_accel_tsdf' Delta Lake table using SQL. This command shows past operations like writes, updates, and deletes performed on the table.

```sql
%sql describe history delta_silver_phone_accel_tsdf
```

--------------------------------

### Load and Prepare Phone Accelerometer Data

Source: https://github.com/databrickslabs/tempo/blob/master/Tempo QuickStart - Python.ipynb

Loads phone accelerometer data from a CSV file, casts relevant columns to appropriate types (timestamp, double), and converts the Spark DataFrame to a Pandas DataFrame. It also adds an event timestamp column.

```python
from pyspark.sql.functions import *

phone_accel_df = spark.read.format("csv").option("header", "true").load("dbfs:/home/tempo/Phones_accelerometer").withColumn("event_ts", (col("Arrival_Time").cast("double")/1000).cast("timestamp")).withColumn("x", col("x").cast("double")).withColumn("y", col("y").cast("double")).withColumn("z", col("z").cast("double")).withColumn("event_ts_dbl", col("event_ts").cast("double"))

series = phone_accel_df.toPandas()

#13062475
display(phone_accel_df)
```

--------------------------------

### Query Aggregated Data from Delta Table

Source: https://github.com/databrickslabs/tempo/blob/master/Tempo QuickStart - Python.ipynb

Executes a SQL query on the 'delta_silver_phone_accel_tsdf' Delta Lake table to calculate the count of records and aggregate statistics (min x, min y, avg z).

```sql
%sql

select count(1), min(x), min(y), avg(z) 
from delta_silver_phone_accel_tsdf
```

--------------------------------

### Resample and Plot Phone Accelerometer Data

Source: https://github.com/databrickslabs/tempo/blob/master/Tempo QuickStart - Python.ipynb

Resamples the phone accelerometer time series data to a minimum frequency using the 'closest_lead' function and filters for a specific date. It then uses Plotly to visualize the 'z' acceleration component over time, colored by user.

```python
resampled_sdf = phone_accel_tsdf.resample(freq='min', func='closest_lead')
resampled_pdf = resampled_sdf.df.filter(col('event_ts').cast("date") == "2015-02-23").toPandas()

import plotly.graph_objs as go
import plotly.express as px
import pandas as pd

# Plotly figure 1
fig = px.line(resampled_pdf, x='event_ts', y='z',
              color="User",
              line_group="User", hover_name="User")
fig.update_layout(title='Phone Accelerometer Usage' , showlegend=False)

fig.show()
```

--------------------------------

### Run Individual Code Quality Checks (Bash)

Source: https://github.com/databrickslabs/tempo/blob/master/CONTRIBUTING.md

Executes individual code quality checks, including code formatting with Black, linting, and type checking.

```bash
make format
make lint
make type-check
```

--------------------------------

### Generate Test Coverage Report (Bash)

Source: https://github.com/databrickslabs/tempo/blob/master/CONTRIBUTING.md

Generates a test coverage report. This command requires that tests have been run previously.

```bash
make coverage-report
```

--------------------------------

### Create and Manipulate IntervalsDF

Source: https://context7.com/databrickslabs/tempo/llms.txt

Shows how to create an IntervalsDF from raw data, make overlapping intervals disjoint, and convert stacked metrics into a pivoted format. It also demonstrates converting back to a stacked format.

```python
from tempo.intervals import IntervalsDF

# Create interval data
data = [
    ("2024-01-01 10:00:00", "2024-01-01 10:05:00", "device_1", 100, 50),
    ("2024-01-01 10:03:00", "2024-01-01 10:08:00", "device_1", 110, 55),
    ("2024-01-01 10:00:00", "2024-01-01 10:04:00", "device_2", 200, 80),
]

df = spark.createDataFrame(
    data,
    "start_ts STRING, end_ts STRING, device_id STRING, metric_1 INT, metric_2 INT"
)

# Create IntervalsDF
idf = IntervalsDF(
    df,
    start_ts="start_ts",
    end_ts="end_ts",
    series_ids=["device_id"]
)

# Make overlapping intervals disjoint
disjoint_idf = idf.make_disjoint()
disjoint_idf.df.show()

# Create from stacked metrics format
stacked_data = [
    ("2024-01-01 10:00:00", "2024-01-01 10:05:00", "device_1", "temperature", 25),
    ("2024-01-01 10:00:00", "2024-01-01 10:05:00", "device_1", "humidity", 60),
    ("2024-01-01 10:00:00", "2024-01-01 10:04:00", "device_2", "temperature", 22),
]

stacked_df = spark.createDataFrame(
    stacked_data,
    "start_ts STRING, end_ts STRING, device_id STRING, metric_name STRING, metric_value INT"
)

# Pivot stacked metrics into columns
pivoted_idf = IntervalsDF.fromStackedMetrics(
    stacked_df,
    start_ts="start_ts",
    end_ts="end_ts",
    series=["device_id"],
    metrics_name_col="metric_name",
    metrics_value_col="metric_value",
    metric_names=["temperature", "humidity"]
)
pivoted_idf.df.show()

# Convert back to DataFrame with stacked metrics
stacked_output = pivoted_idf.toDF(stack=True)
stacked_output.show()
```

--------------------------------

### Prepare Feature Data for Time Series Cross-Validation

Source: https://context7.com/databrickslabs/tempo/llms.txt

This snippet demonstrates how to prepare time series data for machine learning by creating a Spark DataFrame, converting a string column to timestamps, and assembling features using VectorAssembler. It then sets up a TimeSeriesCrossValidator for model evaluation, specifying time series columns, gap, number of folds, estimator, and evaluator.

```python
from pyspark.sql.functions import to_timestamp
from pyspark.ml.feature import VectorAssembler
from pyspark.ml.regression import LinearRegression
from pyspark.ml.evaluation import RegressionEvaluator
from tempo.ml import TimeSeriesCrossValidator

# Prepare sample data
data = [
    ("series_1", "2024-01-01 10:00:00", 1.0, 10.0),
    ("series_1", "2024-01-01 10:01:00", 2.0, 12.0),
    ("series_1", "2024-01-01 10:02:00", 3.0, 14.0),
    ("series_1", "2024-01-01 10:03:00", 4.0, 16.0),
    ("series_1", "2024-01-01 10:04:00", 5.0, 18.0),
    ("series_1", "2024-01-01 10:05:00", 6.0, 20.0),
]

df = spark.createDataFrame(data, ["series_id", "event_ts", "feature", "label"])
    .withColumn("event_ts", to_timestamp("event_ts"))

# Prepare features
assembler = VectorAssembler(inputCols=["feature"], outputCol="features")
prepared_df = assembler.transform(df)

# Create time series cross-validator
ts_cv = TimeSeriesCrossValidator(
    timeSeriesCol="event_ts",
    seriesIdCols=["series_id"],
    gap=1,                    # Gap between train and test sets
    numFolds=3,
    estimator=LinearRegression(featuresCol="features", labelCol="label"),
    evaluator=RegressionEvaluator(labelCol="label", metricName="rmse")
)

# Fit model with time series cross-validation
cv_model = ts_cv.fit(prepared_df)
print(f"Best model RMSE: {cv_model.avgMetrics}")
```

--------------------------------

### Query Filtered Data from Delta Table

Source: https://github.com/databrickslabs/tempo/blob/master/Tempo QuickStart - Python.ipynb

Selects all columns from the 'delta_silver_phone_accel_tsdf' Delta Lake table, filtering records based on user, event date, and a time range for the event timestamp.

```sql
%sql

select * from delta_silver_phone_accel_tsdf
where user = 'g' 
and event_dt = '2015-02-23'
and event_time between 103400 and 103500
```

--------------------------------

### Group Time-Series Data with Statistics (Python)

Source: https://github.com/databrickslabs/tempo/blob/master/python/README.md

This snippet demonstrates how to use the `withGroupedStats` method to group time-series data by a specified frequency and compute metrics on selected columns. It requires the Tempo library and a time-series DataFrame.

```python
grouped_stats = watch_accel_tsdf.withGroupedStats(metricCols = ["y"], freq="1 minute")
display(grouped_stats)
```

--------------------------------

### Write Time Series Data to Delta Lake

Source: https://github.com/databrickslabs/tempo/blob/master/Tempo QuickStart - Python.ipynb

Writes the processed phone accelerometer time series DataFrame to a Delta Lake table named 'delta_silver_phone_accel_tsdf'. This enables persistent storage and querying of the time series data.

```python
phone_accel_tsdf.write(spark, "delta_silver_phone_accel_tsdf")
```

--------------------------------

### As-Of Join Watch and Phone Accelerometer Data

Source: https://github.com/databrickslabs/tempo/blob/master/Tempo QuickStart - Python.ipynb

Performs an as-of join between watch and phone accelerometer time series DataFrames. This operation aligns data points based on the nearest preceding timestamp in the right DataFrame.

```python
joined_df = watch_accel_tsdf.asofJoin(phone_accel_tsdf, right_prefix="phone_accel").df
display(joined_df)
```

--------------------------------

### Run Tests with Specific DBR Version (Bash)

Source: https://github.com/databrickslabs/tempo/blob/master/CONTRIBUTING.md

Executes tests against a specific Databricks Runtime (DBR) version. The default DBR version is 154.

```bash
make test
make test DBR=143
```

--------------------------------

### Create Tempo Time Series DataFrame (Phone Accelerometer)

Source: https://github.com/databrickslabs/tempo/blob/master/Tempo QuickStart - Python.ipynb

Initializes a Tempo Time Series DataFrame (TSDF) from a Spark DataFrame. It specifies the timestamp column and partitioning columns for efficient time series operations.

```python
from tempo import *

phone_accel_tsdf = TSDF(phone_accel_df, ts_col="event_ts", partition_cols = ["User"])
```

--------------------------------

### Interpolate Time Series Data with Tempo

Source: https://context7.com/databrickslabs/tempo/llms.txt

Demonstrates various interpolation methods (ffill, bfill, zero, linear) for filling missing values in a time series DataFrame. It allows specifying frequency, aggregation function, target columns, and an option to flag interpolated rows.

```python
from tempo import TSDF

# Forward fill - propagate last known value forward
interpolated_ffill = tsdf.interpolate(
    freq="30 seconds",
    func="mean",
    method="ffill",
    target_cols=["value"]
)
interpolated_ffill.df.orderBy("event_ts").show()

# Backward fill - fill with next known value
interpolated_bfill = tsdf.interpolate(
    freq="30 seconds",
    func="mean",
    method="bfill",
    target_cols=["value"]
)
interpolated_bfill.df.orderBy("event_ts").show()

# Zero fill - fill missing values with zeros
interpolated_zero = tsdf.interpolate(
    freq="30 seconds",
    func="mean",
    method="zero",
    target_cols=["value"]
)
interpolated_zero.df.orderBy("event_ts").show()

# Chain resample with interpolate
interpolated_chained = tsdf.resample(
    freq="30 seconds",
    func="mean"
).interpolate(method="linear")
interpolated_chained.df.orderBy("event_ts").show()

# Show which rows were interpolated
interpolated_with_flag = tsdf.interpolate(
    freq="30 seconds",
    func="mean",
    method="linear",
    target_cols=["value"],
    show_interpolated=True  # Adds is_interpolated columns
)
interpolated_with_flag.df.orderBy("event_ts").show()
```

--------------------------------

### Read JSON Data from File (Python)

Source: https://github.com/databrickslabs/tempo/blob/master/python/tests/unit_test_data/json-fixer.ipynb

Reads JSON data from a specified file path into a Python dictionary. This is typically used for loading test case configurations or shared data.

```python
import json

with open('./resample_tests.json', 'r') as file:
    before = json.load(file)
```

--------------------------------

### Set Global Python Version with Pyenv (Bash)

Source: https://github.com/databrickslabs/tempo/blob/master/CONTRIBUTING.md

Sets the global Python version using pyenv. This determines the default Python interpreter used in the terminal.

```bash
pyenv global 3.9
```

--------------------------------

### Skew Join Optimized AS OF Join with Tempo TSDF

Source: https://github.com/databrickslabs/tempo/blob/master/python/README.md

Performs a skew-optimized AS OF join to merge the latest source record onto a fact table based on partition and timestamp columns. It requires timestamp and partition columns, a partition value for time brackets, an overlap fraction, and a prefix for source columns.

```python
joined_df = watch_accel_tsdf.asofJoin(phone_accel_tsdf, right_prefix="watch_accel", tsPartitionVal = 10, fraction = 0.1)
display(joined_df)
```

--------------------------------

### Combine Transformed Test Data (Python)

Source: https://github.com/databrickslabs/tempo/blob/master/python/tests/unit_test_data/json-fixer.ipynb

Combines two transformed data dictionaries, 'after' and 'after_2', using the dictionary union operator '|'. This merges the results of processing general test cases and shared data into a single 'combined' dictionary.

```python
combined = after | after_2
combined
```

--------------------------------

### Describe Time Series Data with Tempo TSDF

Source: https://context7.com/databrickslabs/tempo/llms.txt

This Python snippet shows how to use the `describe()` method of a Tempo TSDF object to generate comprehensive statistics. It includes global summaries like unique time series count and timestamp ranges, as well as per-column statistics such as count, mean, standard deviation, min, max, and missing value percentages. The output is displayed using `summary.show()`.

```python
from tempo import TSDF
from pyspark.sql.functions import to_timestamp

# Create sample data
data = [
    ("sensor_1", "2024-01-01 10:00:00", 25.5, 60.2),
    ("sensor_1", "2024-01-01 10:01:00", 25.7, None),  # Missing humidity
    ("sensor_2", "2024-01-01 10:00:00", 22.1, 55.3),
    ("sensor_2", "2024-01-01 10:01:00", None, 55.8),  # Missing temperature
]

df = spark.createDataFrame(data, ["sensor_id", "event_ts", "temperature", "humidity"])
    .withColumn("event_ts", to_timestamp("event_ts"))

tsdf = TSDF(df, ts_col="event_ts", partition_cols=["sensor_id"])

# Get comprehensive statistics
summary = tsdf.describe()
summary.show(truncate=False)
```

--------------------------------

### Process Shared Data for Test Cases (Python)

Source: https://github.com/databrickslabs/tempo/blob/master/python/tests/unit_test_data/json-fixer.ipynb

Processes the '__SharedData' section of the loaded JSON, extracting configuration details for test cases into the 'after_2' dictionary. This logic appears to handle a different structure within the shared data compared to the general test cases.

```python
after_2 = {}
for i in before.keys(): # i is test class
    if i != "__SharedData":
        continue
    after_2[i] = {}
    for j in before[i].keys(): # j is test method
        tsdf = {}
        update_dict(tsdf, "ts_col", before[i][j].get("ts_col", None))
        update_dict(tsdf, "other_ts_cols", before[i][j].get("other_ts_cols", None))
        update_dict(tsdf, "partition_cols", before[i][j].get("partition_cols", None))
        update_dict(tsdf,  "sequence_col", before[i][j].get("sequence_col", None))
        update_dict(tsdf, "start_ts", before[i][j].get("start_ts", None))
        update_dict(tsdf, "end_ts", before[i][j].get("end_ts", None))
        update_dict(tsdf, "series", before[i][j].get("series", None))
        sdf = {}
        update_dict(sdf, "schema", before[i][j].get("schema", None))
        update_dict(sdf, "ts_convert", before[i][j].get("ts_convert", None))
        update_dict(sdf, "data", before[i][j].get("data", None))
        after_2[i][j] = {
                    "tsdf": tsdf,
                    "df": sdf,
                }
```

--------------------------------

### Create TSDF Object from Spark DataFrame (Python)

Source: https://github.com/databrickslabs/tempo/blob/master/python/README.md

Demonstrates how to create a TSDF (Time Series Data Frame) object from a Spark DataFrame. The TSDF wraps the DataFrame and requires a timestamp column for sorting. Optional partition and sequence number columns can also be specified.

```python
from pyspark.sql.functions import *

phone_accel_df = spark.read.format("csv").option("header", "true").load("dbfs:/home/tempo/Phones_accelerometer").withColumn("event_ts", (col("Arrival_Time").cast("double")/1000).cast("timestamp")).withColumn("x", col("x").cast("double")).withColumn("y", col("y").cast("double")).withColumn("z", col("z").cast("double")).withColumn("event_ts_dbl", col("event_ts").cast("double"))

from tempo import *

phone_accel_tsdf = TSDF(phone_accel_df, ts_col="event_ts", partition_cols = ["User"])

display(phone_accel_tsdf)
```

--------------------------------

### Perform AS OF Join on Time Series Data (Python)

Source: https://github.com/databrickslabs/tempo/blob/master/python/README.md

Performs an AS OF join between two TSDF objects. This join selects the latest record from the right table for each record in the left table, based on the timestamp. It's useful for merging related time series data.

```python
from pyspark.sql.functions import *

watch_accel_df = spark.read.format("csv").option("header", "true").load("dbfs:/home/tempo/Watch_accelerometer").withColumn("event_ts", (col("Arrival_Time").cast("double")/1000).cast("timestamp")).withColumn("x", col("x").cast("double")).withColumn("y", col("y").cast("double")).withColumn("z", col("z").cast("double")).withColumn("event_ts_dbl", col("event_ts").cast("double"))

watch_accel_tsdf = TSDF(watch_accel_df, ts_col="event_ts", partition_cols = ["User"])

# Applying AS OF join to TSDF datasets
joined_df = watch_accel_tsdf.asofJoin(phone_accel_tsdf, right_prefix="phone_accel")

display(joined_df)
```

--------------------------------

### Compute Volume Weighted Average Price (VWAP)

Source: https://context7.com/databrickslabs/tempo/llms.txt

Calculates the Volume Weighted Average Price (VWAP) for time series data. This method groups data by a specified frequency and computes the weighted average price based on volume.

```python
from tempo import TSDF
from pyspark.sql.functions import to_timestamp

# Create trade data
data = [
    ("AAPL", "2024-01-01 10:00:05", 150.0, 1000),
    ("AAPL", "2024-01-01 10:00:20", 151.0, 500),
    ("AAPL", "2024-01-01 10:00:45", 149.5, 1500),
    ("AAPL", "2024-01-01 10:01:10", 152.0, 800),
    ("AAPL", "2024-01-01 10:01:30", 153.0, 1200),
]

df = spark.createDataFrame(data, ["symbol", "event_ts", "price", "volume"])
    .withColumn("event_ts", to_timestamp("event_ts"))

tsdf = TSDF(df, ts_col="event_ts", partition_cols=["symbol"])

# Compute minute-level VWAP
vwap_result = tsdf.vwap(
    frequency="m",        # 'm' for minute, 'H' for hour, 'D' for day
    volume_col="volume",
    price_col="price"
)

vwap_result.df.show()
```

--------------------------------

### Extract State Intervals from TSDF

Source: https://context7.com/databrickslabs/tempo/llms.txt

Demonstrates how to extract intervals where a specific status or metric remains constant or changes according to a defined condition. It takes a TSDF object and a state definition as input.

```python
from tempo import TSDF
from pyspark.sql.functions import to_timestamp

# Create status data
data = [
    ("machine_1", "2024-01-01 10:00:00", "running", 100),
    ("machine_1", "2024-01-01 10:01:00", "running", 100),
    ("machine_1", "2024-01-01 10:02:00", "running", 100),
    ("machine_1", "2024-01-01 10:03:00", "stopped", 0),
    ("machine_1", "2024-01-01 10:04:00", "stopped", 0),
    ("machine_1", "2024-01-01 10:05:00", "running", 100),
]

df = spark.createDataFrame(data, ["machine_id", "event_ts", "status", "power"])
    .withColumn("event_ts", to_timestamp("event_ts"))

tsdf = TSDF(df, ts_col="event_ts", partition_cols=["machine_id"])

# Extract intervals where status remained constant
intervals = tsdf.extractStateIntervals(
    "status",              # Metric column(s) to monitor
    state_definition="="   # Comparison operator: =, !=, <, <=, >, >=
)

intervals.show()
```

--------------------------------

### Fourier Transform with Tempo TSDF

Source: https://github.com/databrickslabs/tempo/blob/master/python/README.md

Transforms a time series to the frequency domain using the Fourier Transform method. It requires the timestep value for the frequency scale and the name of the data column to be transformed.

```python
ft_df = tsdf.fourier_transform(timestep=1, valueCol="data_col")
display(ft_df)
```

--------------------------------

### Resample TSDF with High-Frequency Data

Source: https://context7.com/databrickslabs/tempo/llms.txt

Demonstrates resampling a Time Series DataFrame (TSDF) with high-frequency data. Supports various aggregation functions like 'mean', 'floor', and 'max', and can fill missing time buckets.

```python
from tempo import TSDF
from pyspark.sql.functions import to_timestamp

data = [
    ("device_1", "2024-01-01 10:00:00", 100.0),
    ("device_1", "2024-01-01 10:00:15", 105.0),
    ("device_1", "2024-01-01 10:00:30", 102.0),
    ("device_1", "2024-01-01 10:00:45", 108.0),
    ("device_1", "2024-01-01 10:01:00", 110.0),
    ("device_1", "2024-01-01 10:01:15", 112.0),
]

df = spark.createDataFrame(data, ["device_id", "event_ts", "value"])
    .withColumn("event_ts", to_timestamp("event_ts"))

tsdf = TSDF(df, ts_col="event_ts", partition_cols=["device_id"])

# Resample to 1 minute intervals using mean aggregation
resampled = tsdf.resample(freq="1 minute", func="mean")
resampled.df.show()

# Resample using floor (earliest value in window)
resampled_floor = tsdf.resample(freq="1 min", func="floor")
resampled_floor.df.show()

# Resample using max aggregation with specific columns
resampled_max = tsdf.resample(
    freq="30 seconds",
    func="max",
    metricCols=["value"]
)
resampled_max.df.show()

# Resample with fill to create records for empty time buckets
resampled_filled = tsdf.resample(
    freq="15 sec",
    func="mean",
    fill=True
)
resampled_filled.df.show()
```

--------------------------------

### Simple Moving Average with Tempo TSDF

Source: https://github.com/databrickslabs/tempo/blob/master/python/README.md

Computes rolling statistics, specifically the mean, based on a specified time range looking back from the distinguished timestamp column. It requires the column for which to compute the mean and the lookback window in seconds.

```python
moving_avg = watch_accel_tsdf.withRangeStats("y", rangeBackWindowSecs=600)
moving_avg.select('event_ts', 'x', 'y', 'z', 'mean_y').show(10, False)
```

--------------------------------

### Perform AS OF Join on Time Series DataFrames

Source: https://context7.com/databrickslabs/tempo/llms.txt

Executes a temporal AS OF join between two TSDFs, matching records based on the closest preceding timestamp. Supports skew optimization and tolerance for flexible matching.

```python
from tempo import TSDF
from pyspark.sql.functions import to_timestamp

# Create trades data (fact table)
trades_data = [
    ("AAPL", "2024-01-01 10:00:05", 150.0, 100),
    ("AAPL", "2024-01-01 10:00:15", 151.0, 200),
    ("AAPL", "2024-01-01 10:00:25", 149.5, 150),
]
trades_df = spark.createDataFrame(
    trades_data,
    ["symbol", "event_ts", "price", "quantity"]
).withColumn("event_ts", to_timestamp("event_ts"))

trades_tsdf = TSDF(trades_df, ts_col="event_ts", partition_cols=["symbol"])

# Create quotes data (source table)
quotes_data = [
    ("AAPL", "2024-01-01 10:00:00", 149.8, 150.2),
    ("AAPL", "2024-01-01 10:00:10", 150.5, 150.8),
    ("AAPL", "2024-01-01 10:00:20", 149.2, 149.6),
]
quotes_df = spark.createDataFrame(
    quotes_data,
    ["symbol", "event_ts", "bid", "ask"]
).withColumn("event_ts", to_timestamp("event_ts"))

quotes_tsdf = TSDF(quotes_df, ts_col="event_ts", partition_cols=["symbol"])

# Perform AS OF join - get latest quote for each trade
joined = trades_tsdf.asofJoin(
    quotes_tsdf,
    right_prefix="quote"
)
jined.df.show()

# Skew-optimized AS OF join for large datasets
joined_skew = trades_tsdf.asofJoin(
    quotes_tsdf,
    right_prefix="quote",
    tsPartitionVal=10,  # Partition by 10-second brackets
    fraction=0.1        # Overlap fraction for boundary handling
)
jined_skew.df.show()

# AS OF join with tolerance (only join within 5 seconds)
joined_tolerance = trades_tsdf.asofJoin(
    quotes_tsdf,
    right_prefix="quote",
    tolerance=5  # Maximum lookback in seconds
)
jined_tolerance.df.show()
```

--------------------------------

### Create Lookback Features with Tempo TSDF

Source: https://context7.com/databrickslabs/tempo/llms.txt

Generates a 2D feature tensor from historical values of specified columns. This method is useful for preparing time series data for machine learning models. It requires the TSDF object, feature columns, a lookback window size, and an option to ensure exact window sizes. The output includes a 'features' column containing the historical data.

```python
from tempo import TSDF
from pyspark.sql.functions import to_timestamp

# Create sequential data
data = [
    ("user_1", "2024-01-01 10:00:00", 10.0, 5.0),
    ("user_1", "2024-01-01 10:01:00", 12.0, 6.0),
    ("user_1", "2024-01-01 10:02:00", 11.0, 7.0),
    ("user_1", "2024-01-01 10:03:00", 15.0, 8.0),
    ("user_1", "2024-01-01 10:04:00", 14.0, 9.0),
]

df = spark.createDataFrame(data, ["user_id", "event_ts", "metric_a", "metric_b"])
    .withColumn("event_ts", to_timestamp("event_ts"))

tsdf = TSDF(df, ts_col="event_ts", partition_cols=["user_id"])

# Create lookback feature tensor
features_df = tsdf.withLookbackFeatures(
    featureCols=["metric_a", "metric_b"],
    lookbackWindowSize=3,    # Include 3 previous observations
    exactSize=True,          # Only include rows with full lookback history
    featureColName="features"
)

features_df.show(truncate=False)
```

--------------------------------

### Create TSDF from Spark DataFrame

Source: https://context7.com/databrickslabs/tempo/llms.txt

Demonstrates how to create a Time Series DataFrame (TSDF) from a PySpark DataFrame. Requires a SparkSession, input data, a timestamp column, and optional partition columns to define individual time series.

```python
from pyspark.sql import SparkSession
from pyspark.sql.functions import col, to_timestamp
from tempo import TSDF

# Initialize Spark session
spark = SparkSession.builder.appName("tempo-example").getOrCreate()

# Create sample data
data = [
    ("sensor_1", "2024-01-01 10:00:00", 25.5, 60.2),
    ("sensor_1", "2024-01-01 10:01:00", 25.7, 60.5),
    ("sensor_1", "2024-01-01 10:02:00", 25.9, 61.0),
    ("sensor_2", "2024-01-01 10:00:00", 22.1, 55.3),
    ("sensor_2", "2024-01-01 10:01:00", 22.3, 55.8),
]

df = spark.createDataFrame(
    data,
    ["sensor_id", "event_ts", "temperature", "humidity"]
).withColumn("event_ts", to_timestamp("event_ts"))

# Create TSDF with timestamp and partition columns
tsdf = TSDF(
    df,
    ts_col="event_ts",
    partition_cols=["sensor_id"]
)

# Display the TSDF
tsdf.show()
# Output:
# +----------+-------------------+-----------+--------+
# | sensor_id|           event_ts|temperature|humidity|
# +----------+-------------------+-----------+--------+
# |  sensor_1|2024-01-01 10:00:00|       25.5|    60.2|
# |  sensor_1|2024-01-01 10:01:00|       25.7|    60.5|
# |  sensor_1|2024-01-01 10:02:00|       25.9|    61.0|
# |  sensor_2|2024-01-01 10:00:00|       22.1|    55.3|
# |  sensor_2|2024-01-01 10:01:00|       22.3|    55.8|
# +----------+-------------------+-----------+--------+

```