### Setup Development Environment Source: https://nested-pandas.readthedocs.io/en/latest/_sources/gettingstarted/contributing.rst Use these bash commands to create a conda environment, clone the repository, install development dependencies, and run unit tests. ```bash conda create -n nested_pandas_env python=3.11 conda activate nested_pandas_env git clone https://github.com/lincc-frameworks/nested-pandas.git cd nested-pandas/ bash ./.setup_dev.sh pip install pytest pytest ``` -------------------------------- ### Setup Development Environment and Run Tests Source: https://nested-pandas.readthedocs.io/en/latest/gettingstarted/contributing.html Use these commands to create a conda environment, activate it, clone the repository, install development dependencies, and run unit tests for verification. ```bash conda create -n nested_pandas_env python=3.11 conda activate nested_pandas_env git clone https://github.com/lincc-frameworks/nested-pandas.git cd nested-pandas/ bash ./.setup_dev.sh pip install pytest pytest ``` -------------------------------- ### Import necessary libraries and generate example data Source: https://nested-pandas.readthedocs.io/en/latest/pre_executed/njit_map_rows.html Imports the required libraries and generates a sample nested frame for demonstration purposes. Ensure these libraries are installed. ```python from nested_pandas.datasets import generate_data import numpy as np from numba import njit # example frame nf = generate_data(10_000, 1000, seed=1) ``` -------------------------------- ### Install and Run Pytest Source: https://nested-pandas.readthedocs.io/en/latest/_sources/gettingstarted/installation.rst Install the pytest framework and run the unit tests to verify the integrity of your local nested-pandas installation. ```bash pip install pytest pytest ``` -------------------------------- ### Install Development Dependencies Source: https://nested-pandas.readthedocs.io/en/latest/_sources/gettingstarted/installation.rst Install optional dependencies required for running unit tests and building documentation. Note that the exact command might require single quotes around '.[dev]' depending on your shell. ```bash pip install .[dev] ``` -------------------------------- ### Generate Example Data with Nested-Pandas Source: https://nested-pandas.readthedocs.io/en/latest/_sources/tutorials/data_manipulation.ipynb Imports the nested_pandas library and generates an example dataset for demonstration purposes. ```python import nested_pandas as npd from nested_pandas.datasets import generate_data # Begin by generating an example dataset ndf = generate_data(5, 20, seed=1) ndf ``` -------------------------------- ### Install nested-pandas from Source Source: https://nested-pandas.readthedocs.io/en/latest/_sources/gettingstarted/installation.rst Clone the repository and install the latest development version of nested-pandas. This is useful for contributing to the project or testing new features. ```bash git clone https://github.com/lincc-frameworks/nested-pandas.git cd nested-pandas pip install . ``` -------------------------------- ### Install nested-pandas from source Source: https://nested-pandas.readthedocs.io/en/latest/gettingstarted/installation.html Downloads the nested-pandas source code, installs it, and any necessary dependencies within the activated conda environment. Use `pip install .[dev]` to also install development dependencies. ```bash git clone https://github.com/lincc-frameworks/nested-pandas.git cd nested-pandas pip install . ``` ```bash pip install .[dev] # it may be necessary to use `pip install .'[dev]'` (with single quotes) depending on your machine. ``` -------------------------------- ### Generate Toy Dataset with Nested Pandas Source: https://nested-pandas.readthedocs.io/en/latest/reference/api/nested_pandas.datasets.generation.generate_data.html Use generate_data to create a NestedFrame. The first example shows generating a dataset with a single nested layer. The second example demonstrates creating multiple nested columns with custom sizing using a dictionary. ```python from nested_pandas.datasets import generate_data nf1 = generate_data(10,100) ``` ```python nf2 = generate_data(10, {"nested_a": 100, "nested_b": 200}) ``` -------------------------------- ### Benchmarking Setup and Plotting Function Source: https://nested-pandas.readthedocs.io/en/latest/_sources/pre_executed/njit_map_rows.ipynb Sets up functions for timing code execution and plotting performance comparisons between Python and Numba JIT. ```python # benchmarking setup import timeit import matplotlib.pyplot as plt n_nested_list = [100, 200, 300, 500, 750, 1000, 1250, 1500] n_base_list = [7500, 10_000, 12_500, 15_000] def bench(func, nf): timings = timeit.repeat( lambda: func(nf), number=10, repeat=5, ) timings = np.array(timings) return timings.mean(), timings.std() def plot_bench(py_func, njit_func, title, n_base_list=n_base_list, n_nested_list=n_nested_list): plt.figure() for n_base in n_base_list: speedups = [] speedup_errs = [] for n_nested in n_nested_list: nf = generate_data(n_base, n_nested, seed=1) # build frame for each size mean_py, std_py = bench(py_func, nf) # python bench mean_njit, std_njit = bench(njit_func, nf) # njit bench speedup = mean_py / mean_njit # mean speedup ratio ``` -------------------------------- ### Install nested-pandas with Pip Source: https://nested-pandas.readthedocs.io/en/latest/_sources/gettingstarted/installation.rst Use this command to install the latest stable release of nested-pandas from PyPI. ```bash % pip install nested-pandas ``` -------------------------------- ### Install nested-pandas Source: https://nested-pandas.readthedocs.io/en/latest/_sources/gettingstarted/quickstart.ipynb Install the nested-pandas library and its dependencies using pip. This command should be run in a valid Python environment. ```python # % pip install nested-pandas ``` -------------------------------- ### Install nested-pandas using pip Source: https://nested-pandas.readthedocs.io/en/latest/gettingstarted/installation.html Installs the latest release version of nested-pandas from the Python Package Index (PyPI). ```bash % pip install nested-pandas ``` -------------------------------- ### Initialize NestedFrame and join nested DataFrame Source: https://nested-pandas.readthedocs.io/en/latest/reference/api/nested_pandas.NestedFrame.fillna.html Demonstrates the initialization of a NestedFrame with sample data and the subsequent joining of another DataFrame as a nested column. This setup is a prerequisite for applying fillna. ```python import nested_pandas as npd import pandas as pd import numpy as np nf = npd.NestedFrame( data={"a": [np.nan, 20, np.nan], "b": [np.nan, np.nan, 30], "c": [10, np.nan, np.nan]}, index=[0, 1, 2] ) nested = pd.DataFrame( data={"d": [np.nan, np.nan, np.nan], "e": [np.nan, 1, np.nan]}, index=[0, 1, 2] ) nf = nf.join_nested(nested, "nested") ``` -------------------------------- ### keys Source: https://nested-pandas.readthedocs.io/en/latest/reference/api/nested_pandas.NestedFrame.html Get the 'info axis' (see Indexing for more). ```APIDOC ## keys ### Description Get the 'info axis' (see Indexing for more). ### Method `keys()` ``` -------------------------------- ### Set up conda environment for nested-pandas Source: https://nested-pandas.readthedocs.io/en/latest/gettingstarted/installation.html Creates and activates a new conda environment for installing nested-pandas from source. Ensures Python 3.11 is used. ```bash conda create -n nested_pandas_env python=3.11 conda activate nested_pandas_env ``` -------------------------------- ### Example: Nesting List Columns Source: https://nested-pandas.readthedocs.io/en/latest/_modules/nested_pandas/nestedframe/core.html Illustrates using `nest_lists` to pack specified list-valued columns into a single new nested column within the NestedFrame. ```python >>> import nested_pandas as npd >>> nf = npd.NestedFrame({"c":[1,2,3], "d":[2,4,6], ... "e":[[1,2,3], [4,5,6], [7,8,9]]}, ... index=[0,1,2]) >>> nf.nest_lists(columns=["e"], name="nested") c d nested 0 1 2 [{e: 1}; …] (3 rows) 1 2 4 [{e: 4}; …] (3 rows) 2 3 6 [{e: 7}; …] (3 rows) ``` -------------------------------- ### NJIT Explode Graph Example Source: https://nested-pandas.readthedocs.io/en/latest/pre_executed/njit_map_rows.html Demonstrates using njit with map_rows for an 'explode' operation. This is typically run once for JIT compilation before benchmarking. ```python def run_scaled_max_flux_njit_explode(nf): nf["nested.a"] = nf["a"] nf.map_rows( scaled_max_flux_njit_explode, columns=["nested.flux", "nested.a"], row_container="args", output_names="scaled_max_flux", njit=True, ) run_scaled_max_flux_njit_explode(nf.copy()) # run once for jit compilation before benchmark plot_bench( run_scaled_max_flux_py, run_scaled_max_flux_njit_explode, title="njit over python execution - scaled_max_flux (explode)", ) ``` -------------------------------- ### Example: Creating NestedFrame from Flat DataFrame Source: https://nested-pandas.readthedocs.io/en/latest/_modules/nested_pandas/nestedframe/core.html Demonstrates creating a NestedFrame from a flat DataFrame, specifying base columns and allowing the remaining columns to be automatically identified as nested. ```python >>> import nested_pandas as npd >>> nf = npd.NestedFrame({"a":[1,1,1,2,2], "b":[2,2,2,4,4], ... "c":[1,2,3,4,5], "d":[2,4,6,8,10]}, ... index=[0,0,0,1,1]) >>> npd.NestedFrame.from_flat(nf, base_columns=["a","b"]) a b nested 0 1 2 [{c: 1, d: 2}; …] (3 rows) 1 2 4 [{c: 4, d: 8}; …] (2 rows) ``` -------------------------------- ### get Source: https://nested-pandas.readthedocs.io/en/latest/reference/api/nested_pandas.NestedSeries.html Retrieves an item from the NestedSeries using a key. ```APIDOC ## get(key[, default]) ### Description Get item from object for given key (ex: DataFrame column). ### Parameters - **key** - object. The key to retrieve. - **default** - object, default None. The value to return if the key is not found. ``` -------------------------------- ### Create a NestedFrame from a flat DataFrame Source: https://nested-pandas.readthedocs.io/en/latest/reference/api/nested_pandas.NestedFrame.from_flat.html This example demonstrates how to initialize a NestedFrame from a flat pandas DataFrame. It shows the creation of the initial DataFrame and then its conversion using `from_flat`, specifying the base columns. ```python import nested_pandas as npd nf = npd.NestedFrame({"a":[1,1,1,2,2], "b":[2,2,2,4,4], "c":[1,2,3,4,5], "d":[2,4,6,8,10]}, index=[0,0,0,1,1]) ``` ```python npd.NestedFrame.from_flat(nf, base_columns=["a","b"]) ``` -------------------------------- ### Utility Methods Source: https://nested-pandas.readthedocs.io/en/latest/reference/api/nested_pandas.NestedFrame.html Methods for viewing and manipulating data, including getting the tail, squeezing dimensions, and setting flags. ```APIDOC ## squeeze ### Description Squeeze 1 dimensional axis objects into scalars. ### Parameters - **axis** (int or str, optional) - Select a specific axis to squeeze. ## set_axis ### Description Assign desired index to given axis. ### Parameters - **labels** (list or Index) - The new labels. - **axis** (int or str, optional) - The axis to set. - **copy** (bool, optional) - Whether to copy the object. ## set_flags ### Description Return a new object with updated flags. ### Parameters - **copy** (bool, optional) - Whether to copy the object. - **allows_duplicate_labels** (bool, optional) - Whether to allow duplicate labels. ## stack ### Description Stack the prescribed level(s) from columns to index. ### Parameters - **level** (int or str or list, optional) - Level(s) to stack. - **dropna** (bool, optional) - Whether to drop NA/null values. - **sort** (bool, optional) - Sort the resulting index. - **future_stack** (bool, optional) - Use future stack implementation. ## tail ### Description Return the last n rows. ### Parameters - **n** (int, optional) - Number of rows to return. ## take ### Description Return the elements in the given _positional_ indices along an axis. ### Parameters - **indices** (int or array-like) - Indices to take. - **axis** (int or str, optional) - Axis to take from. ``` -------------------------------- ### get Source: https://nested-pandas.readthedocs.io/en/latest/reference/api/nested_pandas.NestedFrame.html Get item from object for given key (ex: DataFrame column). ```APIDOC ## get ### Description Get item from object for given key (ex: DataFrame column). ### Method `get(key[, default])` ### Parameters - **key**: The key to retrieve. - **default**: Value to return if key is not found. ``` -------------------------------- ### Basic njit_map_rows Usage Source: https://nested-pandas.readthedocs.io/en/latest/_sources/pre_executed/njit_map_rows.ipynb Demonstrates the fundamental application of njit_map_rows for iterating and processing rows of a NumPy array using Numba's njit decorator. Ensure Numba is installed. ```python from numba import njit import numpy as np @njit def njit_map_rows(func, arr): """Applies a function to each row of a NumPy array using Numba. Args: func: The function to apply to each row. arr: The input NumPy array. Returns: A new NumPy array with the results of applying the function to each row. """ return np.array([func(row) for row in arr]) @njit def add_one(row): return row + 1 # Example usage: my_array = np.array([[1, 2], [3, 4]]) result = njit_map_rows(add_one, my_array) print(result) ``` -------------------------------- ### Handling Different Row Lengths (Caution) Source: https://nested-pandas.readthedocs.io/en/latest/_sources/pre_executed/njit_map_rows.ipynb This example demonstrates a potential issue when applying njit_map_rows to arrays where rows might have different lengths or the function expects a fixed size. Numba might raise errors or produce unexpected results. It's generally recommended to have uniform row lengths. ```python import numba import numpy as np @numba.njit def safe_row_op(row): # Safely access elements, assuming row might be shorter than expected val = 0.0 if row.shape[0] > 0: val += row[0] if row.shape[0] > 1: val += row[1] return val # Example with potentially ragged data (Numba prefers contiguous arrays) data = np.array([ [1, 2, 3], [4, 5], [6, 7, 8, 9] ], dtype=object) # Using dtype=object to allow raggedness, but this is often not JIT-friendly # Note: njit_map_rows typically expects a homogeneous NumPy array. # Applying it directly to a dtype=object array with varying lengths might not work as expected or efficiently. # For truly ragged arrays, consider alternative approaches or padding. # Example with a homogeneous array for demonstration: data_uniform = np.arange(12.0).reshape(4, 3) result_uniform = numba.njit_map_rows(safe_row_op, data_uniform) print("Result for uniform data:", result_uniform) ``` -------------------------------- ### Import generate_data from nested_pandas.datasets Source: https://nested-pandas.readthedocs.io/en/latest/reference/api/nested_pandas.NestedFrame.get_subcolumns.html Import the necessary function to generate sample data for demonstration. ```python from nested_pandas.datasets import generate_data ``` -------------------------------- ### Numba-compiled function for NumPy array processing Source: https://nested-pandas.readthedocs.io/en/latest/_sources/pre_executed/njit_map_rows.ipynb This example shows a typical Numba-optimized function that operates on NumPy arrays, which is often the underlying data structure for pandas DataFrames. This is a more direct way to leverage njit for high-performance numerical computations. ```python from numba import njit import numpy as np @njit def process_numpy_array(arr): result = np.empty(arr.shape[0], dtype=np.float64) for i in range(arr.shape[0]): # Example computation on each row (represented as a 1D slice of the array) result[i] = arr[i, 0] * arr[i, 1] + arr[i, 2] return result # Example usage with a NumPy array data = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]]) processed_data = process_numpy_array(data) print(processed_data) ``` -------------------------------- ### Generate Sample Data Source: https://nested-pandas.readthedocs.io/en/latest/reference/api/nested_pandas.NestSeriesAccessor.query.html Generates a sample NestedSeries for demonstration purposes. This is a prerequisite for using the query method. ```python from nested_pandas.datasets.generation import generate_data nf = generate_data(5, 5, seed=1) ``` -------------------------------- ### Benchmarking njit vs Python - Weighted Mean Slope (Asymptotic) Source: https://nested-pandas.readthedocs.io/en/latest/pre_executed/njit_map_rows.html Analyzes the asymptotic behavior of njit versus Python for the weighted_mean_slope function, showing how performance changes with increasing nested column width. Useful for identifying performance bottlenecks. ```python n_base_shrink = [7500, 10_000] n_nested_list_asy = [500, 1000, 1500, 2000, 2500, 3000, 3500, 4000] plot_bench( run_weighted_mean_slope_py, run_weighted_mean_slope_njit, title="njit over python execution - weighted_mean_slope (asymptotic behavior)", n_base_list=n_base_shrink, n_nested_list=n_nested_list_asy, ) ``` -------------------------------- ### Generate Sample Nested DataFrame Source: https://nested-pandas.readthedocs.io/en/latest/reference/api/nested_pandas.utils.count_nested.html Sets up pandas display options and generates a sample nested DataFrame for demonstration purposes. This is a prerequisite for using other utility functions. ```python import pandas as pd # Show all columns pd.set_option("display.width", 200) pd.set_option("display.max_columns", None) from nested_pandas.datasets.generation import generate_data nf = generate_data(5, 10, seed=1) ``` -------------------------------- ### mode Source: https://nested-pandas.readthedocs.io/en/latest/reference/api/nested_pandas.NestedFrame.html Get the mode(s) of each element along the selected axis. ```APIDOC ## mode ### Description Get the mode(s) of each element along the selected axis. ### Parameters - `axis` ({0 or 'index', 1 or 'columns', None}): Axis to direct the mode calculation. - `numeric_only` (bool): If True, only consider numeric columns. - `dropna` (bool): Whether to exclude NA values. ### Example ```python nested_frame.mode(axis=0, numeric_only=False, dropna=True) ``` ``` -------------------------------- ### Generate Sample Data Source: https://nested-pandas.readthedocs.io/en/latest/reference/api/nested_pandas.NestedFrame.describe.html Generates a sample NestedFrame for demonstration purposes. This is a prerequisite for using the `describe()` method. ```python from nested_pandas.datasets.generation import generate_data nf = generate_data(5,5, seed=1) ``` -------------------------------- ### Get Base Columns Source: https://nested-pandas.readthedocs.io/en/latest/_modules/nested_pandas/nestedframe/core.html Returns a list of column names that are not nested. ```python @property def base_columns(self) -> list[str]: """Returns the list of base (non-nested) column names""" nested_mask = self.dtypes.apply(lambda dtype: not isinstance(dtype, NestedDtype)) return self.columns[nested_mask].tolist() ``` -------------------------------- ### Initialize Arrays for Flat Spectrum Dataframe Source: https://nested-pandas.readthedocs.io/en/latest/pre_executed/nested_spectra.html Initializes empty NumPy arrays to hold the wavelength, flux, error, and index data for building a flat spectrum dataframe. ```python import numpy as np # Build a flat spectrum dataframe # Initialize some empty arrays to hold the flat data wave = np.array([]) flux = np.array([]) err = np.array([]) index = np.array([]) ``` -------------------------------- ### rsub Source: https://nested-pandas.readthedocs.io/en/latest/reference/api/nested_pandas.NestedFrame.html Get Subtraction of dataframe and other, element-wise (binary operator rsub). ```APIDOC ## rsub ### Description Get Subtraction of dataframe and other, element-wise (binary operator rsub). ### Parameters - `other`: The object to subtract with. - `axis` ({0 or 'index', 1 or 'columns', None}): Select axis of DataFrame to broadcast over. - `level` (int or label): Broadcast across this level of the index. - `fill_value`: Fill existing NaN values, and use this value to replace all too. ### Example ```python nested_frame.rsub(5) ``` ``` -------------------------------- ### Numba JIT vs. Python: Weighted Mean Slope (Asymptotic Behavior) Source: https://nested-pandas.readthedocs.io/en/latest/_sources/pre_executed/njit_map_rows.ipynb Compares the performance of a Numba njit-compiled loop against a pure Python implementation for calculating the weighted mean slope. This snippet visualizes the asymptotic behavior as the nested column width increases, showing a slight advantage for njit even at large scales. ```python n_base_shrink = [7500, 10_000] n_nested_list_asy = [500, 1000, 1500, 2000, 2500, 3000, 3500, 4000, 4500, 5000] plot_bench( run_weighted_mean_slope_py, run_weighted_mean_slope_njit_loop, title="njit over python execution - weighted_mean_slope (asymptotic behavior for loop)", n_base_list=n_base_shrink, n_nested_list=n_nested_list_asy, ) ``` -------------------------------- ### rmul Source: https://nested-pandas.readthedocs.io/en/latest/reference/api/nested_pandas.NestedFrame.html Get Multiplication of dataframe and other, element-wise (binary operator rmul). ```APIDOC ## rmul ### Description Get Multiplication of dataframe and other, element-wise (binary operator rmul). ### Parameters - `other`: The object to multiply with. - `axis` ({0 or 'index', 1 or 'columns', None}): Select axis of DataFrame to broadcast over. - `level` (int or label): Broadcast across this level of the index. - `fill_value`: Fill existing NaN values, and use this value to replace all too. ### Example ```python nested_frame.rmul(2) ``` ``` -------------------------------- ### rmod Source: https://nested-pandas.readthedocs.io/en/latest/reference/api/nested_pandas.NestedFrame.html Get Modulo of dataframe and other, element-wise (binary operator rmod). ```APIDOC ## rmod ### Description Get Modulo of dataframe and other, element-wise (binary operator rmod). ### Parameters - `other`: The object to take the modulo with. - `axis` ({0 or 'index', 1 or 'columns', None}): Select axis of DataFrame to broadcast over. - `level` (int or label): Broadcast across this level of the index. - `fill_value`: Fill existing NaN values, and use this value to replace all too. ### Example ```python nested_frame.rmod(5) ``` ``` -------------------------------- ### .nest Accessor Source: https://nested-pandas.readthedocs.io/en/latest/reference.html Documentation for the .nest accessor, including its constructor and functions. ```APIDOC ## .nest Accessor ### Constructor Details on how to use the .nest accessor. ### Functions List and description of functions available through the .nest accessor. ``` -------------------------------- ### radd Source: https://nested-pandas.readthedocs.io/en/latest/reference/api/nested_pandas.NestedFrame.html Get Addition of dataframe and other, element-wise (binary operator radd). ```APIDOC ## radd ### Description Get Addition of dataframe and other, element-wise (binary operator radd). ### Parameters - `other`: The object to add with. - `axis` ({0 or 'index', 1 or 'columns', None}): Select axis of DataFrame to broadcast over. - `level` (int or label): Broadcast across this level of the index. - `fill_value`: Fill existing NaN values, and use this value to replace all too. ### Example ```python nested_frame.radd(5) ``` ``` -------------------------------- ### ne Source: https://nested-pandas.readthedocs.io/en/latest/reference/api/nested_pandas.NestedFrame.html Get Not equal to of dataframe and other, element-wise (binary operator ne). ```APIDOC ## ne ### Description Get Not equal to of dataframe and other, element-wise (binary operator ne). ### Parameters - `other`: The object to compare with. - `axis` ({0 or 'index', 1 or 'columns', None}): Select axis of DataFrame to broadcast over. - `level` (int or label): Broadcast across this level of the index. ### Example ```python nested_frame.ne(10) ``` ``` -------------------------------- ### Benchmarking njit vs Python - Max Slope (Asymptotic) Source: https://nested-pandas.readthedocs.io/en/latest/pre_executed/njit_map_rows.html Compares the performance of njit and Python execution for the max_slope function as nested column width increases. Useful for understanding the limits of njit optimization. ```python n_base_shrink = [7500, 10_000] n_nested_list_asy = [500, 1000, 1500, 2000, 2500, 3000, 3500, 4000] plot_bench( run_max_slope_py, run_max_slope_njit, title="njit over python execution - max_slope (asymptotic behavior)", n_base_list=n_base_shrink, n_nested_list=n_nested_list_asy, ) ``` -------------------------------- ### multiply Source: https://nested-pandas.readthedocs.io/en/latest/reference/api/nested_pandas.NestedFrame.html Get Multiplication of dataframe and other, element-wise (binary operator mul). ```APIDOC ## multiply ### Description Get Multiplication of dataframe and other, element-wise (binary operator mul). ### Parameters - `other`: The object to multiply with. - `axis` ({0 or 'index', 1 or 'columns', None}): Select axis of DataFrame to broadcast over. - `level` (int or label): Broadcast across this level of the index. - `fill_value`: Fill existing NaN values, and use this value to replace all too. ### Example ```python nested_frame.multiply(2) ``` ``` -------------------------------- ### mul Source: https://nested-pandas.readthedocs.io/en/latest/reference/api/nested_pandas.NestedFrame.html Get Multiplication of dataframe and other, element-wise (binary operator mul). ```APIDOC ## mul ### Description Get Multiplication of dataframe and other, element-wise (binary operator mul). ### Parameters - `other`: The object to multiply with. - `axis` ({0 or 'index', 1 or 'columns', None}): Select axis of DataFrame to broadcast over. - `level` (int or label): Broadcast across this level of the index. - `fill_value`: Fill existing NaN values, and use this value to replace all too. ### Example ```python nested_frame.mul(2) ``` ``` -------------------------------- ### Generate Sample Data Source: https://nested-pandas.readthedocs.io/en/latest/reference/api/nested_pandas.NestSeriesAccessor.set_filled_column.html Generates sample data using the `generate_data` function. This is a prerequisite for demonstrating the `set_filled_column` method. ```python from nested_pandas.datasets.generation import generate_data nf = generate_data(3, 2, seed=1) ``` -------------------------------- ### mod Source: https://nested-pandas.readthedocs.io/en/latest/reference/api/nested_pandas.NestedFrame.html Get Modulo of dataframe and other, element-wise (binary operator mod). ```APIDOC ## mod ### Description Get Modulo of dataframe and other, element-wise (binary operator mod). ### Parameters - `other`: The object to take the modulo with. - `axis` ({0 or 'index', 1 or 'columns', None}): Select axis of DataFrame to broadcast over. - `level` (int or label): Broadcast across this level of the index. - `fill_value`: Fill existing NaN values, and use this value to replace all too. ### Example ```python nested_frame.mod(5) ``` ``` -------------------------------- ### Asymptotic Behavior Plotting for Njit Loop Source: https://nested-pandas.readthedocs.io/en/latest/pre_executed/njit_map_rows.html Visualizes the performance scaling of an njit-compiled function with manual loops across different data sizes. ```python n_base_shrink = [7500, 10_000] nested_list_asy = [500, 1000, 1500, 2000, 2500, 3000, 3500, 4000, 4500, 5000] plot_bench( run_weighted_mean_slope_py, run_weighted_mean_slope_njit_loop, title="njit over python execution - weighted_mean_slope (asymptotic behavior for loop)", n_base_list=n_base_shrink, n_nested_list=n_nested_list_asy, ) ``` -------------------------------- ### expanding Source: https://nested-pandas.readthedocs.io/en/latest/reference/api/nested_pandas.NestedFrame.html Provide expanding window calculations. ```APIDOC ## expanding ### Description Provide expanding window calculations. ### Method `expanding(min_periods, axis, , ...)` ### Parameters - **min_periods**: Minimum number of observations in window required for a value; otherwise, NA. - **axis**: Axis to compute the expanding window over (0 or 'index', 1 or 'columns'). - **...**: Additional parameters. ``` -------------------------------- ### lt Source: https://nested-pandas.readthedocs.io/en/latest/reference/api/nested_pandas.NestedFrame.html Get Less than of dataframe and other, element-wise (binary operator lt). ```APIDOC ## lt ### Description Get Less than of dataframe and other, element-wise (binary operator lt). ### Method `lt(other[, axis, level])` ### Parameters - **other**: The other operand. - **axis**: Axis to use for the binary operation. - **level**: Broadcast across level, matching on index values on given level(s). ``` -------------------------------- ### gt Source: https://nested-pandas.readthedocs.io/en/latest/reference/api/nested_pandas.NestedFrame.html Get Greater than of dataframe and other, element-wise (binary operator gt). ```APIDOC ## gt ### Description Get Greater than of dataframe and other, element-wise (binary operator gt). ### Method `gt(other[, axis, level])` ### Parameters - **other**: The other operand. - **axis**: Axis to use for the binary operation. - **level**: Broadcast across level, matching on index values on given level(s). ``` -------------------------------- ### rtruediv Source: https://nested-pandas.readthedocs.io/en/latest/reference/api/nested_pandas.NestedFrame.html Get Floating division of dataframe and other, element-wise (binary operator rtruediv). ```APIDOC ## rtruediv ### Description Get Floating division of dataframe and other, element-wise (binary operator rtruediv). ### Parameters - `other`: The object to divide by. - `axis` ({0 or 'index', 1 or 'columns', None}): Select axis of DataFrame to broadcast over. - `level` (int or label): Broadcast across this level of the index. - `fill_value`: Fill existing NaN values, and use this value to replace all too. ### Example ```python nested_frame.rtruediv(2) ``` ``` -------------------------------- ### head Source: https://nested-pandas.readthedocs.io/en/latest/reference/api/nested_pandas.NestedFrame.html Return the first n rows. ```APIDOC ## head ### Description Return the first n rows. ### Method `head([n])` ### Parameters - **n**: Number of rows to return. Defaults to 5. ``` -------------------------------- ### rpow Source: https://nested-pandas.readthedocs.io/en/latest/reference/api/nested_pandas.NestedFrame.html Get Exponential power of dataframe and other, element-wise (binary operator rpow). ```APIDOC ## rpow ### Description Get Exponential power of dataframe and other, element-wise (binary operator rpow). ### Parameters - `other`: The object to raise the power to. - `axis` ({0 or 'index', 1 or 'columns', None}): Select axis of DataFrame to broadcast over. - `level` (int or label): Broadcast across this level of the index. - `fill_value`: Fill existing NaN values, and use this value to replace all too. ### Example ```python nested_frame.rpow(2) ``` ``` -------------------------------- ### Create and Activate Conda Environment Source: https://nested-pandas.readthedocs.io/en/latest/_sources/gettingstarted/installation.rst Set up a dedicated conda environment for nested-pandas development to avoid dependency conflicts. ```bash conda create -n nested_pandas_env python=3.11 conda activate nested_pandas_env ``` -------------------------------- ### NestedSeries.__init__() Source: https://nested-pandas.readthedocs.io/en/latest/reference/api/nested_pandas.NestedSeries.html Initializes a NestedSeries object. This is the constructor for the class. ```APIDOC ## NestedSeries.__init__() ### Description Initializes a NestedSeries object. ### Method __init__ ``` -------------------------------- ### rfloordiv Source: https://nested-pandas.readthedocs.io/en/latest/reference/api/nested_pandas.NestedFrame.html Get Integer division of dataframe and other, element-wise (binary operator rfloordiv). ```APIDOC ## rfloordiv ### Description Get Integer division of dataframe and other, element-wise (binary operator rfloordiv). ### Parameters - `other`: The object to divide by. - `axis` ({0 or 'index', 1 or 'columns', None}): Select axis of DataFrame to broadcast over. - `level` (int or label): Broadcast across this level of the index. - `fill_value`: Fill existing NaN values, and use this value to replace all too. ### Example ```python nested_frame.rfloordiv(2) ``` ``` -------------------------------- ### rdiv Source: https://nested-pandas.readthedocs.io/en/latest/reference/api/nested_pandas.NestedFrame.html Get Floating division of dataframe and other, element-wise (binary operator rtruediv). ```APIDOC ## rdiv ### Description Get Floating division of dataframe and other, element-wise (binary operator rtruediv). ### Parameters - `other`: The object to divide by. - `axis` ({0 or 'index', 1 or 'columns', None}): Select axis of DataFrame to broadcast over. - `level` (int or label): Broadcast across this level of the index. - `fill_value`: Fill existing NaN values, and use this value to replace all too. ### Example ```python nested_frame.rdiv(2) ``` ``` -------------------------------- ### pow Source: https://nested-pandas.readthedocs.io/en/latest/reference/api/nested_pandas.NestedFrame.html Get Exponential power of dataframe and other, element-wise (binary operator pow). ```APIDOC ## pow ### Description Get Exponential power of dataframe and other, element-wise (binary operator pow). ### Parameters - `other`: The object to raise the power to. - `axis` ({0 or 'index', 1 or 'columns', None}): Select axis of DataFrame to broadcast over. - `level` (int or label): Broadcast across this level of the index. - `fill_value`: Fill existing NaN values, and use this value to replace all too. ### Example ```python nested_frame.pow(2) ``` ``` -------------------------------- ### Generate Sample Data Source: https://nested-pandas.readthedocs.io/en/latest/reference/api/nested_pandas.NestSeriesAccessor.drop.html Generates sample nested data for demonstration purposes. This is a prerequisite for using the drop method. ```python from nested_pandas.datasets.generation import generate_data nf = generate_data(5, 2, seed=1) ``` -------------------------------- ### sample Source: https://nested-pandas.readthedocs.io/en/latest/reference/api/nested_pandas.NestedFrame.html Return a random sample of items from an axis of object. ```APIDOC ## sample ### Description Return a random sample of items from an axis of object. ### Parameters - `n` (int): Number of items to return. - `frac` (float): Fraction of axis items to return. - `replace` (bool): Allow sampling of the same row multiple times. - `weights` (str or ndarray or Series, optional): Default 'None' results in equal probability weighting. - `random_state` (int, array-like, BitGenerator, np.random.RandomState, np.random.Generator, optional): If int, array-like, or BitGenerator, seed for random number generator. - `axis` ({0 or 'index', 1 or 'columns', None}): Axis to sample from. - `ignore_index` (bool, default False): If True, result index will be labeled 0, 1, ..., n - 1. ### Example ```python nested_frame.sample(n=10, random_state=42) ``` ``` -------------------------------- ### le Source: https://nested-pandas.readthedocs.io/en/latest/reference/api/nested_pandas.NestedFrame.html Get Less than or equal to of dataframe and other, element-wise (binary operator le). ```APIDOC ## le ### Description Get Less than or equal to of dataframe and other, element-wise (binary operator le). ### Method `le(other[, axis, level])` ### Parameters - **other**: The other operand. - **axis**: Axis to use for the binary operation. - **level**: Broadcast across level, matching on index values on given level(s). ``` -------------------------------- ### Initialize NestedExtensionArray Source: https://nested-pandas.readthedocs.io/en/latest/_modules/nested_pandas/series/ext_array.html Constructs a NestedExtensionArray from pyarrow.Array or pyarrow.ChunkedArray. It handles both list-struct and struct-list input formats, ensuring internal storage is consistent. ```python def __init__(self, values: pa.Array | pa.ChunkedArray, *, validate: bool = True) -> None: if isinstance(values, pa.Array): values = pa.chunked_array([values]) # If list-struct if is_pa_type_a_list(values.type): list_struct_storage = ListStructStorage(values) # If struct-list else: struct_list_storage = StructListStorage(values, validate=validate) list_struct_storage = ListStructStorage.from_struct_list_storage(struct_list_storage) self._storage = list_struct_storage self._dtype = NestedDtype(self._storage.type) ``` -------------------------------- ### ge Source: https://nested-pandas.readthedocs.io/en/latest/reference/api/nested_pandas.NestedFrame.html Get Greater than or equal to of dataframe and other, element-wise (binary operator ge). ```APIDOC ## ge ### Description Get Greater than or equal to of dataframe and other, element-wise (binary operator ge). ### Method `ge(other[, axis, level])` ### Parameters - **other**: The other operand. - **axis**: Axis to use for the binary operation. - **level**: Broadcast across level, matching on index values on given level(s). ``` -------------------------------- ### floordiv Source: https://nested-pandas.readthedocs.io/en/latest/reference/api/nested_pandas.NestedFrame.html Get Integer division of dataframe and other, element-wise (binary operator floordiv). ```APIDOC ## floordiv ### Description Get Integer division of dataframe and other, element-wise (binary operator floordiv). ### Method `floordiv(other[, axis, level, fill_value])` ### Parameters - **other**: The other operand. - **axis**: Axis to use for the binary operation. - **level**: Broadcast across level, matching on index values on given level(s). - **fill_value**: Fill existing NA/null values with the given value. ``` -------------------------------- ### head Source: https://nested-pandas.readthedocs.io/en/latest/reference/api/nested_pandas.NestedSeries.html Returns the first n rows of the NestedSeries. ```APIDOC ## head([n]) ### Description Return the first n rows. ### Parameters - **n** - int, default 5. The number of rows to return. ``` -------------------------------- ### Initialize Nested Series Source: https://nested-pandas.readthedocs.io/en/latest/reference/api/nested_pandas.NestSeriesAccessor.set_list_column.html Generates sample data for use with nested-pandas. This is a prerequisite for demonstrating other methods. ```python from nested_pandas.datasets.generation import generate_data nf = generate_data(2, 2, seed=1) ``` -------------------------------- ### NestedFrame.all_columns Source: https://nested-pandas.readthedocs.io/en/latest/reference/api/nested_pandas.NestedFrame.all_columns.html Access the all_columns property to get a dictionary of columns for each base and nested dataframe. ```APIDOC ## property NestedFrame.all_columns: dict ### Description Returns a dictionary of columns for each base/nested dataframe. ### Type dict ``` -------------------------------- ### Generate Sample Data Source: https://nested-pandas.readthedocs.io/en/latest/reference/api/nested_pandas.NestSeriesAccessor.to_flatten_inner.html Generates sample data for testing nested structures. Imports `NestedFrame` and `generate_data`. ```python from nested_pandas import NestedFrame from nested_pandas.datasets import generate_data nf = generate_data(5, 2, seed=1).rename(columns={"nested": "inner"}) nf["b"] = "b" # Shorten width of example output ``` -------------------------------- ### Get Nested Columns Source: https://nested-pandas.readthedocs.io/en/latest/_modules/nested_pandas/nestedframe/core.html Returns a list of base column names that contain nested data structures. ```python @property def nested_columns(self) -> list: """retrieves the base column names for all nested dataframes""" nested_mask = self.dtypes.apply(lambda dtype: isinstance(dtype, NestedDtype)) return self.columns[nested_mask].tolist() ``` -------------------------------- ### compare(other, align_axis=1, keep_shape=False, keep_equal=False, ...) Source: https://nested-pandas.readthedocs.io/en/latest/reference/api/nested_pandas.NestedSeries.html Compares the Series with another Series and highlights the differences. ```APIDOC ## compare(other, align_axis=1, keep_shape=False, keep_equal=False, ...) ### Description Compare to another Series and show the differences. ### Method `compare(other[, align_axis, keep_shape, ...])` ``` -------------------------------- ### Get Flat Series Field Source: https://nested-pandas.readthedocs.io/en/latest/_modules/nested_pandas/series/accessor.html Extracts a specific field from the flattened nested arrays and returns it as a pandas Series. ```APIDOC ## .nest.get_flat_series(field: str) ### Description Get the flat-array field as a pd.Series. ### Parameters #### Path Parameters - **field** (str) - Required - Name of the field to get. ### Returns - **pd.Series** - The flat-array field. ### Examples ```python >>> nf["nested"].nest.get_flat_series("flux") ``` ```