FireDucks

https://fireducks-dev.github.io/docs
Admin
FireDucks is a pandas-compatible dataframe library that provides high-performance data processing...
Tokens:12,894
Snippets:211
Trust Score:9.7
Update:5 months ago
Show doc for...
Tokens:
Raw
### Install FireDucks using pip

Source: https://fireducks-dev.github.io/docs/get-started

Install the FireDucks library using pip. Ensure you are using Python version 3.9 to 3.13.

```bash
pip install fireducks
```

--------------------------------

### StringMethods getitem with Numeric Index

Source: https://fireducks-dev.github.io/docs/release-note

Example of using `s.str[2]` for accessing elements in string or list-like columns with a numeric index. This fallback support enhances string manipulation capabilities.

```python
s.str[2]
```

--------------------------------

### read_csv() with encoding='utf8'

Source: https://fireducks-dev.github.io/docs/release-note

Example of `read_csv()` with `encoding='utf8'`. The removal of this fallback simplifies CSV reading with UTF-8 encoding.

```python
read_csv with encoding=utf8
```

--------------------------------

### Binary Comparison with Date Instance

Source: https://fireducks-dev.github.io/docs/release-note

Demonstrates binary comparison with a 'date' instance as a scalar value. This fallback removal enables more direct date comparisons.

```python
supported binop comparison with ‘date’ instance as for scalar value
```

--------------------------------

### Print Optimized IR with FIRE_LOG_LEVEL=3

Source: https://fireducks-dev.github.io/docs/release-note

Shows how to print the optimized Intermediate Representation (IR) by setting the environment variable `FIRE_LOG_LEVEL=3`. This is useful for debugging and understanding query optimization.

```python
print optimized IR when FIRE_LOG_LEVEL=3
```

--------------------------------

### DataFrame.groupby() with Series Key

Source: https://fireducks-dev.github.io/docs/release-note

Shows how `DataFrame.groupby()` now supports a Series as the input key. This expands the flexibility of grouping operations.

```python
DataFrame.groupby()
```

--------------------------------

### Enable FireDucks Benchmark Mode in Code

Source: https://fireducks-dev.github.io/docs/user-guide/02-exec-model

Programmatically enable benchmark mode by importing get_fireducks_options and calling set_benchmark_mode(True). This forces immediate execution of methods for performance measurement.

```python
from fireducks.core import get_fireducks_options
get_fireducks_options().set_benchmark_mode(True)
```

--------------------------------

### DataFrame.loc with Column Slicing

Source: https://fireducks-dev.github.io/docs/release-note

Demonstrates using DataFrame.loc with column slicing, such as `df.loc[:, A: C]`. This functionality was improved to support various column types.

```python
df.loc[:, A: C]
```

--------------------------------

### Optimized reset_index() followed by dropna, concat, melt, explode

Source: https://fireducks-dev.github.io/docs/release-note

Highlights performance improvements for operations following `reset_index(drop=True)`, such as `dropna`, `concat`, `melt`, and `explode`. This optimization reduces overhead on index computations.

```python
reset_index(drop=True)
```

--------------------------------

### Optimized sort_values().reset_index() Pattern

Source: https://fireducks-dev.github.io/docs/release-note

Illustrates an optimization pattern where `sort_values(...).reset_index(drop=True)` is transformed into `sort_values(..., ignore_index=True)`. This improves performance by directly setting the ignore_index.

```python
sort_values(...).reset_index(drop=True)
```

```python
sort_values(..., ignore_index=True)
```

--------------------------------

### Force Immediate Execution with _evaluate()

Source: https://fireducks-dev.github.io/docs/user-guide/02-exec-model

Use the _evaluate() method after operations to force immediate execution of the lazy computation. This is an alternative to benchmark mode for measuring individual steps.

```python
t0 = time.time()
df = pd.read_csv("data.csv")._evaluate()
t1 = time.time()
df = df.sort_values("a")._evaluate()
t2 = time.time()
df.to_csv("sorted.csv")._evaluate()
t3 = time.time()
```

--------------------------------

### Enable FireDucks Benchmark Mode via Environment Variable

Source: https://fireducks-dev.github.io/docs/user-guide/02-exec-model

Set the FIREDUCKS_FLAGS environment variable to "--benchmark-mode" to enable immediate execution of each method call. This is useful for measuring individual method performance but may disable some FireDucks optimizations.

```bash
FIREDUCKS_FLAGS="--benchmark-mode"
```

--------------------------------

### FireDucks and Pandas Versioning

Source: https://fireducks-dev.github.io/docs/release-note

Explains how to retrieve the version of FireDucks and Pandas. Use `fireducks.__version__` for the FireDucks version and `fireducks.pandas.__version__` for the underlying Pandas version.

```python
fireducks.pandas.__version__
```

```python
fireducks.__version__
```

--------------------------------

### to_csv() with column parameter

Source: https://fireducks-dev.github.io/docs/release-note

Shows the usage of the `column` parameter in `to_csv()`. This fallback removal provides more control over which columns are written to the CSV file.

```python
column parameter of to_csv()
```

--------------------------------

### Optimized drop_duplicates().reset_index() Pattern

Source: https://fireducks-dev.github.io/docs/release-note

Shows an optimization pattern where `df.drop_duplicates(...).reset_index(drop=True)` is transformed into `df.drop_duplicates(..., ignore_index=True)`. This improves performance by avoiding intermediate index resets.

```python
df.drop_duplicates(...).reset_index(drop=True)
```

```python
df.drop_duplicates(..., ignore_index=True)
```

--------------------------------

### GroupBy Aggregate Method 'last'

Source: https://fireducks-dev.github.io/docs/release-note

Demonstrates the support for the aggregate method 'last' in GroupBy operations. This fallback removal expands the available aggregation functions.

```python
aggregate method “last” for GroupBy
```

--------------------------------

### drop_duplicates() with ignore_index

Source: https://fireducks-dev.github.io/docs/release-note

Illustrates the `ignore_index` parameter for `drop_duplicates()`. This fallback removal simplifies index handling after duplicate removal.

```python
supported ignore_index parameter for drop_duplicates
```

--------------------------------

### Series.reset_index() with name parameter

Source: https://fireducks-dev.github.io/docs/release-note

Demonstrates the `name` parameter for `Series.reset_index()`. This fallback removal allows specifying a name for the index column when resetting it.

```python
Series.reset_index() with name parameter
```

--------------------------------

### Handling Range Index with Step != 1

Source: https://fireducks-dev.github.io/docs/release-note

Addresses an issue in handling range indexes where the step is not equal to 1. This ensures correct behavior for custom range indexes.

```python
handling range index with step != 1
```

--------------------------------

### read_csv() with dtype_backend='pyarrow'

Source: https://fireducks-dev.github.io/docs/release-note

Illustrates the `dtype_backend='pyarrow'` parameter for `read_csv()`. This fallback removal allows for more efficient data reading with pyarrow backend.

```python
dtype_backend=“pyarrow”
```

--------------------------------

### Series/DataFrame dropna() with ignore_index

Source: https://fireducks-dev.github.io/docs/release-note

Illustrates the `ignore_index` parameter for `Series/DataFrame.dropna()`. This fallback removal simplifies index management after dropping null values.

```python
“ignore_index” parameter for Series/DataFrame dropna()
```

--------------------------------

### DataFrame.index Setter Fallback

Source: https://fireducks-dev.github.io/docs/release-note

Shows the fix for the DataFrame index setter, `df.index = ...`, which now works correctly even with fallback mechanisms enabled.

```python
df.index = ...
```

--------------------------------

### DataFrame.sort_values() with Empty Projection

Source: https://fireducks-dev.github.io/docs/release-note

Illustrates `df.sort_values("C")[[]]`, which supports projection-pushdown when the projection target is empty. This optimization improves performance for specific sorting and selection operations.

```python
df.sort_values("C")[[]]
```

--------------------------------

### Load FireDucks Import Hook in IPython/Jupyter

Source: https://fireducks-dev.github.io/docs/get-started

Load the FireDucks import hook as a magic command in IPython or Jupyter Notebooks. This enables FireDucks for pandas usage within the notebook environment.

```python
%load_ext fireducks.pandas
import pandas as pd
```

--------------------------------

### DataFrame.iloc by Row Index

Source: https://fireducks-dev.github.io/docs/release-note

Demonstrates using `df.iloc[0]` for selecting rows by index. This fallback support enhances row-based selection capabilities.

```python
df.iloc[0]
```

--------------------------------

### astype() and read_csv() with dtype='string'

Source: https://fireducks-dev.github.io/docs/release-note

Shows the support for `dtype='string'` as an input for `astype()` and `read_csv()`. This fallback removal simplifies string type handling.

```python
dtype=“string”
```

--------------------------------

### Parquet Reading with Index Columns

Source: https://fireducks-dev.github.io/docs/release-note

Fixes a bug in reading parquet files where index columns were stored at the beginning. This ensures correct reconstruction of DataFrames from parquet files.

```python
reading the parquet file when index columns are stored at the beginningg
```

--------------------------------

### Run Python script with FireDucks Import Hook

Source: https://fireducks-dev.github.io/docs/get-started

Execute a Python script using the FireDucks import hook. This automatically replaces 'import pandas' with FireDucks, useful for existing programs.

```bash
python3 -m fireducks.pandas your_script.py
```

--------------------------------

### DataFrame Setitem with Non-String Key

Source: https://fireducks-dev.github.io/docs/release-note

Shows DataFrame `setitem` support with a non-string key, such as an integer key `df[1] = ...`. This fallback removal allows using integer keys for column assignment.

```python
DataFrame.setitem with non-string key. e.g., `df[1] = ...` (key is integer)
```

--------------------------------

### to_csv() with List or Struct-like Columns

Source: https://fireducks-dev.github.io/docs/release-note

Demonstrates the `to_csv()` function's ability to handle DataFrames or Series containing list or struct-like columns. This fix improves data serialization.

```python
to_csv()
```

--------------------------------

### Handle KeyError with Lazy Execution (Incorrect)

Source: https://fireducks-dev.github.io/docs/user-guide/02-exec-model

Avoid using try-catch blocks for KeyError with FireDucks' default lazy execution, as the key access might not be evaluated immediately, leading to unexpected behavior. The _evaluate() call can enforce immediate execution if this structure is unavoidable.

```python
def project(df, cname):
    try:
        s = df[cname]
    except KeyError:
        print(f"Column {cname} doesn't exist in input dataframe")
    else:
        return s
```

```python
def project(df, cname):
    try:
        s = df[cname]._evaluate()
    except KeyError:
        print(f"Column {cname} doesn't exist in input dataframe")
    else:
        return s
```

--------------------------------

### Datetime Properties dt.date and dt.time

Source: https://fireducks-dev.github.io/docs/release-note

Shows support for datetime properties `dt.date` and `dt.time`. This fallback removal allows direct access to date and time components from datetime columns.

```python
dt.date
```

```python
dt.time
```

--------------------------------

### DataFrame.groupby().size() on Empty Data

Source: https://fireducks-dev.github.io/docs/release-note

Fixes an issue with `groupby().size()` when applied to an empty DataFrame. This ensures correct size calculation for empty groups.

```python
groupby().size()
```

--------------------------------

### pd.read_json() with lines=True

Source: https://fireducks-dev.github.io/docs/release-note

Shows support for `pd.read_json()` with `lines=True`. This fallback removal simplifies reading JSON files where each line is a separate JSON object.

```python
pd.read_json() with lines=True
```

--------------------------------

### Avoid Loops in FireDucks

Source: https://fireducks-dev.github.io/docs/user-guide/tips

Iterating through DataFrame elements with loops is slow. Use DataFrame API methods like .sum() for better performance.

```python
s = 0
for i in range(len(df)):
    if df["A"][i] > 2:
        s += df["B"][i]
```

```python
s = df[df["A"] > 2]["B"].sum()
```

--------------------------------

### DataFrame Setitem/Getitem with Nx1 NumPy Array

Source: https://fireducks-dev.github.io/docs/release-note

Illustrates DataFrame setitem/getitem support when using a NumPy array of dimension Nx1. This enhances compatibility with NumPy operations.

```python
DataFrame setitem/getitem with numpy array of dimension Nx1
```

--------------------------------

### Casting Datetime Column Units

Source: https://fireducks-dev.github.io/docs/release-note

Fixes an issue in casting a datetime column from one unit to another, such as `datetime64[ns]` to `datetime64[ms]`. This ensures accurate unit conversions for datetime data.

```python
datetime64[ns] -> datetime64[ms]
```

--------------------------------

### DataFrame.groupby() with kwargs Aggregation

Source: https://fireducks-dev.github.io/docs/release-note

Fixes an issue with `df.groupby("a")["b"].agg(Sum="sum")`, where aggregation with keyword arguments was not handled correctly.

```python
df.groupby("a")["b"].agg(Sum="sum")
```

--------------------------------

### Join with Categorical Columns

Source: https://fireducks-dev.github.io/docs/release-note

Fixes an issue with `join` operations involving categorical columns, ensuring correct data integrity during joins.

```python
join
```

--------------------------------

### Explicitly Evaluate DataFrame Intermediate Language

Source: https://fireducks-dev.github.io/docs/user-guide/05-api

Use the `_evaluate` method to force execution of the intermediate language for a DataFrame up to that point. This is useful for measuring the processing time of specific operations.

```python
df = pd.read_csv(...)._evaluate()      # end read_csv so that it is not in the measurement range
t0 = time.time()                       # start measuring time
g = df.groupby(...).sum()._evaluate()  # immediately after generating intermediate language for groupby
t1 = time.time()                       # end of time measurement
```

--------------------------------

### Optimized DataFrame Row and Column Extraction

Source: https://fireducks-dev.github.io/docs/user-guide/03-acceleration

This optimized version first selects the relevant columns ('a', 'b') before filtering rows, which can be more efficient for DataFrames with many columns.

```python
tmp = df[["a", "b"]]
selected = tmp[df["a"] > 10]["b"]
```

--------------------------------

### melt() with Non-String Value Vars

Source: https://fireducks-dev.github.io/docs/release-note

Addresses a bug in `melt()` when used with non-string `value_vars`. This ensures correct behavior when melting DataFrames with diverse column types.

```python
melt()
```

--------------------------------

### Handle Missing Columns Safely in FireDucks

Source: https://fireducks-dev.github.io/docs/user-guide/02-exec-model

Use the 'in' operator to check for column existence before accessing it. This operator invokes a non-lazy method, ensuring the check is evaluated immediately and preventing unexpected behavior with lazy execution.

```python
def project(df, cname):
    if cname not in df:
        print(f"Column {cname} doesn't exist in input dataframe")
    else:
        s = df[cname]
        return s
```

--------------------------------

### Measure FireDucks Execution Time

Source: https://fireducks-dev.github.io/docs/user-guide/02-exec-model

To measure the total execution time of a sequence of FireDucks operations, include timers around the operations that trigger evaluation, such as saving to a file. The interval between t2 and t3 will encompass all preceding lazy operations.

```python
t0 = time.time()
df = pd.read_csv("data.csv")
t1 = time.time()
df = df.sort_values("a")
t2 = time.time()
df.to_csv("sorted.csv")
t3 = time.time()
```

--------------------------------

### dropna(axis=0) Performance Improvement

Source: https://fireducks-dev.github.io/docs/release-note

Highlights performance improvements for `dropna(axis=0)` when the input DataFrame contains no null values. This optimization speeds up operations on clean data.

```python
dropna(axis=0)
```

--------------------------------

### Using FireDucks DataFrame with pandas Libraries

Source: https://fireducks-dev.github.io/docs/user-guide/04-compatibility

When passing a FireDucks DataFrame to a library expecting a pandas DataFrame, convert it using `.to_pandas()` before passing it. This ensures compatibility with libraries that rely on pandas' internal structures.

```python
import somelib
import fireducks.pandas as pd

df = pd.read_csv(...)
somelib.process_pandas_dataframe(df.to_pandas())
```

--------------------------------

### Replace pandas import with FireDucks

Source: https://fireducks-dev.github.io/docs/get-started

Explicitly import FireDucks by replacing the pandas import statement. This is suitable for integrating FireDucks into existing pandas programs.

```python
# import pandas as pd
import fireducks.pandas as pd
```

--------------------------------

### DataFrame Row and Column Extraction

Source: https://fireducks-dev.github.io/docs/user-guide/03-acceleration

This code extracts rows where column 'a' is greater than 10, then selects column 'b'. It can be inefficient if the DataFrame has many columns.

```python
selected = df[df["a"] > 10]["b"]
```

--------------------------------

### Avoid Undefined Pandas Behavior in FireDucks

Source: https://fireducks-dev.github.io/docs/user-guide/tips

Modifying elements using chained indexing like `df["A"][1] = 2` can lead to undefined behavior in pandas and may not update the DataFrame in FireDucks. Use `.iloc` for explicit element modification.

```python
df["A"][1] = 2
```

```python
df.iloc[1, 0] = 2  # if A is the first column
```
Tokens:
Raw