### Create DataFrame for value_counts Example Source: https://github.com/modin-project/modin/blob/main/docs/flow/modin/pandas/base.md Initializes a DataFrame used in subsequent value_counts examples. This setup is necessary before calling the method. ```python df = pd.DataFrame({'num_legs': [2, 4, 4, 6], 'num_wings': [2, 0, 0, 0]}, index=['falcon', 'dog', 'cat', 'ant']) df ``` -------------------------------- ### Install IPython Source: https://github.com/modin-project/modin/blob/main/contributing/contributing.md Install the IPython interactive shell, which is useful for testing and experimenting with Modin. ```bash pip install ipython ``` -------------------------------- ### Install Modin with Experimental Solver Source: https://github.com/modin-project/modin/blob/main/docs/getting_started/installation.md Installs Modin with Ray and the experimental libmamba solver. ```bash conda install -c conda-forge modin-ray modin- --experimental-solver=libmamba ``` -------------------------------- ### Install Release Tools Source: https://github.com/modin-project/modin/blob/main/docs/release-procedure.md Install or update necessary tools for building distribution packages like wheels. ```bash pip install --upgrade build twine ``` -------------------------------- ### Install Modin Source: https://github.com/modin-project/modin/blob/main/docs/index.md Install Modin from PyPI. If you need specific compute engine support, install with the appropriate target. ```bash pip install modin ``` ```bash pip install "modin[ray]" # Install Modin dependencies and Ray to run on Ray ``` ```bash pip install "modin[dask]" # Install Modin dependencies and Dask to run on Dask ``` ```bash pip install "modin[mpi]" # Install Modin dependencies and MPI to run on MPI through unidist ``` ```bash pip install "modin[all]" # Install all of the above ``` -------------------------------- ### Verify Modin Installation in IPython Source: https://github.com/modin-project/modin/blob/main/contributing/contributing.md Start an IPython session and import Modin to check the installed version. The output should include the version number, last commit number, and last commit hash. ```python ipython import modin modin.__version__ ``` -------------------------------- ### Install AWS Dependencies Source: https://github.com/modin-project/modin/blob/main/docs/getting_started/using_modin/using_modin_cluster.md Install the necessary boto3 library for AWS authentication. This is a prerequisite for configuring AWS credentials. ```bash pip install boto3 ``` -------------------------------- ### Install XGBoost for Modin Source: https://github.com/modin-project/modin/blob/main/docs/usage_guide/advanced_usage/modin_xgboost.md Install the xgboost package using pip. Ensure Modin is installed with the Ray execution engine. ```bash pip install xgboost ``` -------------------------------- ### Install Development Requirements Source: https://github.com/modin-project/modin/blob/main/contributing/contributing.md Install all the necessary dependencies for Modin development, as listed in the `requirements-dev.txt` file. ```bash pip install -r requirements-dev.txt ``` -------------------------------- ### Install Modin with All Engines using Pip Source: https://github.com/modin-project/modin/blob/main/README.md Installs Modin with all recommended engines (Ray and Dask). This is the recommended installation method for comprehensive functionality. ```bash pip install "modin[all]" ``` -------------------------------- ### Install Modin with All Dependencies from Local Source Source: https://github.com/modin-project/modin/blob/main/docs/getting_started/installation.md Installs Modin in editable mode from a local clone, including all engine dependencies. ```bash cd modin pip install -e ".[all]" ``` -------------------------------- ### XGBoost Training with Modin Usage Example Source: https://github.com/modin-project/modin/blob/main/docs/usage_guide/advanced_usage/modin_xgboost.md A complete example demonstrating how to load data, create Modin DataFrames, initialize DMatrix, train an XGBoost model, and make predictions. ```APIDOC ## Usage example In example below we train XGBoost model using [the Iris Dataset](https://scikit-learn.org/stable/auto_examples/datasets/plot_iris_dataset.html) and get prediction on the same data. All processing will be in a single node mode. ```python from sklearn import datasets import ray # Look at the Ray documentation with respect to the Ray configuration suited to you most. ray.init() # Start the Ray runtime for single-node import modin.pandas as pd import modin.experimental.xgboost as xgb # Load iris dataset from sklearn iris = datasets.load_iris() # Create Modin DataFrames X = pd.DataFrame(iris.data) y = pd.DataFrame(iris.target) # Create DMatrix dtrain = xgb.DMatrix(X, y) dtest = xgb.DMatrix(X, y) # Set training parameters xgb_params = { "eta": 0.3, "max_depth": 3, "objective": "multi:softprob", "num_class": 3, "eval_metric": "mlogloss", } steps = 20 # Create dict for evaluation results evals_result = dict() # Run training model = xgb.train( xgb_params, dtrain, steps, evals=[(dtrain, "train")], evals_result=evals_result ) # Print evaluation results print(f'Evals results:\n{evals_result}') # Predict results prediction = model.predict(dtest) # Print prediction results print(f'Prediction results:\n{prediction}') ``` ``` -------------------------------- ### Install Development Dependencies Source: https://github.com/modin-project/modin/blob/main/docs/development/contributing.md Install all required dependencies for running tests and formatting code using either conda or pip. ```bash conda env create --file environment-dev.yml # or pip install -r requirements-dev.txt ``` -------------------------------- ### Install Conda with Libmamba Solver Source: https://github.com/modin-project/modin/blob/main/README.md Installs the libmamba solver in the base conda environment to speed up conda installations. This is recommended for faster package resolution. ```bash conda install -n base conda-libmamba-solver ``` -------------------------------- ### Build Documentation Source: https://github.com/modin-project/modin/blob/main/docs/development/contributing.md Install documentation dependencies and build the HTML documentation using Sphinx. ```bash pip install -r docs/requirements-doc.txt sphinx-build -b html docs docs/build ``` -------------------------------- ### Install Modin Consortium Standard Implementation Source: https://github.com/modin-project/modin/blob/main/docs/getting_started/installation.md Installs Modin with support for the consortium standard-compatible implementation. ```bash pip install "modin[consortium-standard]" ``` -------------------------------- ### Example Usage Source: https://github.com/modin-project/modin/blob/main/docs/flow/modin/distributed/dataframe/pandas.md Demonstrates how to use `unwrap_partitions` and `from_partitions` to extract and reconstruct Modin DataFrames. ```APIDOC ## Example ```python import modin.pandas as pd from modin.distributed.dataframe.pandas import unwrap_partitions, from_partitions import numpy as np # Create a sample Modin DataFrame data = np.random.randint(0, 100, size=(2 ** 10, 2 ** 8)) df = pd.DataFrame(data) # Unwrap partitions along axis 0 and get IP addresses partitions = unwrap_partitions(df, axis=0, get_ip=True) print("Unwrapped partitions with IP info:", partitions) # Reconstruct the DataFrame from partitions new_df = from_partitions(partitions, axis=0) print("Reconstructed DataFrame:", new_df) ``` ``` -------------------------------- ### Install Modin with Spreadsheet Support Source: https://github.com/modin-project/modin/blob/main/docs/usage_guide/advanced_usage/spreadsheets_api.md Install Modin with the spreadsheet extra to enable spreadsheet functionality. This command installs Modin and all necessary dependencies for the spreadsheet API. ```bash pip install "modin[spreadsheet]" ``` -------------------------------- ### Install Modin with All Dependencies Source: https://github.com/modin-project/modin/blob/main/contributing/contributing.md Install Modin in editable mode from your local source code, including all optional dependencies. This is useful for development. ```bash pip install -e ".[all]" ``` -------------------------------- ### Install dask_cloudprovider for AWS Source: https://github.com/modin-project/modin/blob/main/examples/tutorial/jupyter/execution/pandas_on_dask/cluster/exercise_5.ipynb Install the necessary dependencies for using dask_cloudprovider with AWS. This is a prerequisite for setting up an EC2 cluster. ```python !pip install dask_cloudprovider[aws] ``` -------------------------------- ### Install Modin Stable Version with Pip Source: https://github.com/modin-project/modin/blob/main/docs/getting_started/installation.md Installs the latest stable release of Modin. Use the -U flag to upgrade if an older version is already installed. ```bash pip install -U modin # -U for upgrade in case you have an older version ``` -------------------------------- ### Install tqdm for Progress Bar Source: https://github.com/modin-project/modin/blob/main/docs/usage_guide/advanced_usage/progress_bar.md Install the tqdm library, which is used by Modin for its progress bar functionality. This is a prerequisite for enabling the progress bar. ```bash pip install tqdm ``` -------------------------------- ### Install Modin from Local Source Source: https://github.com/modin-project/modin/blob/main/contributing/contributing.md Install Modin from the local source code in editable mode. This command should be run from the root of the cloned Modin repository. ```bash pip install -e . ``` -------------------------------- ### Test Installation with Pip Source: https://github.com/modin-project/modin/blob/main/docs/release-procedure.md Tests the uploaded wheels by installing Modin with all optional dependencies in a new environment. This should be performed on Linux, Mac, and Windows. ```bash pip install -U "modin[all]" ``` -------------------------------- ### Install Modin with Conda using Libmamba Solver Source: https://github.com/modin-project/modin/blob/main/README.md Installs Modin using the libmamba solver for faster installation. This command demonstrates using the experimental solver flag. ```bash conda install -c conda-forge modin-ray --experimental-solver=libmamba ``` -------------------------------- ### Install Modin Release Candidate with Pip Source: https://github.com/modin-project/modin/blob/main/docs/getting_started/installation.md Installs a pre-release version of Modin for testing. Use this to test your existing code against upcoming releases. ```bash pip install --pre modin ``` -------------------------------- ### Install PandasOnRay Requirements Source: https://github.com/modin-project/modin/blob/main/examples/tutorial/jupyter/README.md Use this command to install dependencies required for running Modin notebooks with the PandasOnRay execution backend. ```bash pip install -r execution/pandas_on_ray/requirements.txt ``` -------------------------------- ### from_dataframe Configuration Source: https://github.com/modin-project/modin/blob/main/examples/spreadsheet/tutorial.ipynb Example of using `from_dataframe` with configuration options. ```APIDOC ## from_dataframe(df, show_toolbar, grid_options) ### Description Creates a SpreadsheetWidget instance from a DataFrame with optional configuration for the toolbar and grid. ### Method `mss.from_dataframe(df, show_toolbar=False, grid_options={})` ### Parameters #### Path Parameters - **df** (DataFrame) - Required - The input DataFrame. - **show_toolbar** (bool) - Optional - Whether to display the toolbar. Defaults to `False`. - **grid_options** (dict) - Optional - Configuration options for the grid. Defaults to `{}`. ``` -------------------------------- ### Install Modin Release Version Source: https://github.com/modin-project/modin/blob/main/contributing/contributing.md Install a specific release version of Modin from PyPI. This is for using Modin as a library, not for development. ```bash pip install modin ``` -------------------------------- ### Install PandasOnUnidist Requirements Source: https://github.com/modin-project/modin/blob/main/examples/tutorial/jupyter/README.md Use this command to install dependencies required for running Modin notebooks with the PandasOnUnidist execution backend. ```bash pip install -r execution/pandas_on_unidist/requirements.txt ``` -------------------------------- ### Install PandasOnDask Requirements Source: https://github.com/modin-project/modin/blob/main/examples/tutorial/jupyter/README.md Use this command to install dependencies required for running Modin notebooks with the PandasOnDask execution backend. ```bash pip install -r execution/pandas_on_dask/requirements.txt ``` -------------------------------- ### Modin XGBoost Training Example Source: https://github.com/modin-project/modin/blob/main/docs/usage_guide/advanced_usage/modin_xgboost.md This example demonstrates training an XGBoost model using Modin DataFrames with the Iris dataset. It covers data loading, DMatrix creation, parameter setting, model training, and prediction. ```python from sklearn import datasets import ray # Look at the Ray documentation with respect to the Ray configuration suited to you most. ray.init() # Start the Ray runtime for single-node import modin.pandas as pd import modin.experimental.xgboost as xgb # Load iris dataset from sklearn iris = datasets.load_iris() # Create Modin DataFrames X = pd.DataFrame(iris.data) y = pd.DataFrame(iris.target) # Create DMatrix dtrain = xgb.DMatrix(X, y) dtest = xgb.DMatrix(X, y) # Set training parameters xgb_params = { "eta": 0.3, "max_depth": 3, "objective": "multi:softprob", "num_class": 3, "eval_metric": "mlogloss", } steps = 20 # Create dict for evaluation results evals_result = dict() # Run training model = xgb.train( xgb_params, dtrain, steps, evals=[(dtrain, "train")], evals_result=evals_result ) # Print evaluation results print(f'Evals results:\n{evals_result}') # Predict results prediction = model.predict(dtest) # Print prediction results print(f'Prediction results:\n{prediction}') ``` -------------------------------- ### Serve Documentation Locally Source: https://github.com/modin-project/modin/blob/main/docs/development/contributing.md Run a local HTTP server from the build folder to visualize the generated documentation in a browser. ```bash python -m http.server # python -m http.server 1234 ``` -------------------------------- ### Initialize Ray for Timing Comparisons Source: https://github.com/modin-project/modin/blob/main/examples/quickstart.ipynb This setup is for timing comparisons and requires initializing the Ray backend. Ensure your Ray configuration is appropriate for your environment. ```python import time import ray # Look at the Ray documentation with respect to the Ray configuration suited to you most. ray.init() from IPython.display import Markdown, display def printmd(string): display(Markdown(string)) ``` -------------------------------- ### dt_start_time() Source: https://github.com/modin-project/modin/blob/main/docs/flow/modin/core/storage_formats/base/query_compiler.md Get the timestamp of the start time for each period value. This method is only supported by one-column query compilers. ```APIDOC ## dt_start_time() ### Description Get the timestamp of start time for each period value. ### Returns New QueryCompiler with the same shape as self, where each element is the timestamp of start time for the corresponding period value. ### Return type [BaseQueryCompiler](#modin.core.storage_formats.base.query_compiler.BaseQueryCompiler) ### Notes Please refer to `modin.pandas.Series.dt.start_time` for more information about parameters and output format. ### WARNING This method is supported only by one-column query compilers. ``` -------------------------------- ### Display Training Input Examples Source: https://github.com/modin-project/modin/blob/main/examples/jupyter/integrations/huggingface.ipynb Prints the training InputExamples to show the structure of the prepared data. ```python train_InputExamples ``` -------------------------------- ### Install Modin on Google Colab Source: https://github.com/modin-project/modin/blob/main/docs/getting_started/installation.md Installs Modin with all dependencies on Google Colab. Requires a runtime restart after installation. ```bash !pip install "modin[all]" ``` -------------------------------- ### Initialize SpreadsheetWidget with Configuration Source: https://github.com/modin-project/modin/blob/main/examples/spreadsheet/tutorial.ipynb Example of using `from_dataframe` with configuration options to control toolbar visibility and column fitting. ```python mss.from_dataframe(df, show_toolbar=False, grid_options={'forceFitColumns': False, 'editable': False, 'highlightSelectedCell': True}) ``` -------------------------------- ### Install Modin with All Engines using Conda Source: https://github.com/modin-project/modin/blob/main/README.md Installs Modin and its associated engines (Ray, Dask, MPI) from conda-forge. This command installs Modin with 'modin-all'. ```bash conda install -c conda-forge modin-all ``` -------------------------------- ### modin.config.envvars.EnvironmentVariable.get_help Source: https://github.com/modin-project/modin/blob/main/docs/flow/modin/config.md Generates user-presentable help information for the configuration setting. ```APIDOC ## modin.config.envvars.EnvironmentVariable.get_help ### Description Generates user-presentable help for the configuration. ### Method `get_help()` ### Returns - **str**: User-presentable help string for the config. ### Return Type str ``` -------------------------------- ### Initialize Modin with Ray Source: https://github.com/modin-project/modin/blob/main/docs/getting_started/quickstart.md Initialize the Ray backend for Modin. Ensure your Ray configuration is suited to your needs. ```python import modin.pandas as pd import pandas ############################################# ### For the purpose of timing comparisons ### ############################################# import time import ray # Look at the Ray documentation with respect to the Ray configuration suited to you most. ray.init() ############################################# ``` -------------------------------- ### DataFrame Creation Example Source: https://github.com/modin-project/modin/blob/main/docs/flow/modin/pandas/dataframe.md Demonstrates how to create a Modin DataFrame from various sources, including CSV files, Python dictionaries, and existing pandas DataFrames, and how to configure the number of partitions. ```APIDOC ## Usage Guide The most efficient way to create Modin `DataFrame` is to import data from external storage using the highly efficient Modin IO methods (for example using `pd.read_csv`, see details for Modin IO methods in the [IO](../core/io/index.md) page), but even if the data does not originate from a file, any pandas supported data type or `pandas.DataFrame` can be used. Internally, the `DataFrame` data is divided into partitions, which number along an axis usually corresponds to the number of the user’s hardware CPUs. If needed, the number of partitions can be changed by setting `modin.config.NPartitions`. Let’s consider simple example of creation and interacting with Modin `DataFrame`: ```python import modin.config # This explicitly sets the number of partitions modin.config.NPartitions.put(4) import modin.pandas as pd import pandas # Create Modin DataFrame from the external file pd_dataframe = pd.read_csv("test_data.csv") # Create Modin DataFrame from the python object # data = {f'col{x}': [f'col{x}_{y}' for y in range(100, 356)] for x in range(4)} # pd_dataframe = pd.DataFrame(data) # Create Modin DataFrame from the pandas object # pd_dataframe = pd.DataFrame(pandas.DataFrame(data)) # Show created DataFrame print(pd_dataframe) # List DataFrame partitions. Note, that internal API is intended for # developers needs and was used here for presentation purposes # only. partitions = pd_dataframe._query_compiler._modin_frame._partitions print(partitions) # Show the first DataFrame partition print(partitions[0][0].get()) ``` ``` -------------------------------- ### Initialize Ray Runtime Source: https://github.com/modin-project/modin/blob/main/docs/usage_guide/advanced_usage/modin_xgboost.md Initialize the Ray runtime for a single-node setup. Ensure Ray is configured appropriately for your environment. ```python import ray # Look at the Ray documentation with respect to the Ray configuration suited to you most. ray.init() ``` -------------------------------- ### from_arrow Source: https://github.com/modin-project/modin/blob/main/docs/flow/modin/core/dataframe/pandas/partitioning/partition_manager.md Creates partitions from an Apache Arrow Table. ```APIDOC ## from_arrow(at, return_dims=False) ### Description Return the partitions from Apache Arrow (PyArrow). ### Parameters * **at** (*pyarrow.table*) – Arrow Table. * **return_dims** (*bool* *,* *default: False*) – If it’s True, return as (np.ndarray, row_lengths, col_widths), else np.ndarray. ### Returns A NumPy array with partitions (with dimensions or not). ### Return type (np.ndarray, backend) or (np.ndarray, backend, row_lengths, col_widths) ``` -------------------------------- ### Ray Initialization Source: https://github.com/modin-project/modin/blob/main/docs/usage_guide/advanced_usage/modin_xgboost.md Instructions on how to initialize the Ray runtime for single-node or cluster setups, which is necessary for Modin's distributed operations. ```APIDOC ## A Single Node / Cluster setup The XGBoost part of Modin uses a Ray resources by similar way as all Modin functions. To start the Ray runtime on a single node: ```python import ray # Look at the Ray documentation with respect to the Ray configuration suited to you most. ray.init() ``` If you already had the Ray cluster you can connect to it by next way: ```python import ray ray.init(address='auto') ``` A detailed information about initializing the Ray runtime you can find in [starting ray](https://docs.ray.io/en/master/starting-ray.html) page. ``` -------------------------------- ### Run Jupyter Notebook Server Source: https://github.com/modin-project/modin/blob/main/examples/tutorial/jupyter/README.md Launch a Jupyter Notebook server from the current directory to access and run Modin's tutorial notebooks. ```bash jupyter notebook ``` -------------------------------- ### Run Entire Test Suite Source: https://github.com/modin-project/modin/blob/main/docs/development/contributing.md Execute the complete test suite from the project root to verify code correctness. ```bash pytest modin/pandas/test ``` -------------------------------- ### Start Jupyter Notebook with MPI Backend Source: https://github.com/modin-project/modin/blob/main/examples/tutorial/jupyter/execution/pandas_on_unidist/README.md Use this command to launch a Jupyter notebook server that utilizes the MPI backend for distributed execution. ```bash mpiexec -n 1 jupyter notebook ``` -------------------------------- ### Setup Environment Variables for Benchmarking Source: https://github.com/modin-project/modin/blob/main/asv_bench/README.md These environment variables are used to configure the dataset size and the number of CPUs for Modin benchmarks. ```bash export MODIN_TEST_DATASET=Big export MODIN_CPUS=44 ``` -------------------------------- ### Install Specific Modin Version Source: https://github.com/modin-project/modin/blob/main/contributing/contributing.md Install a specific version of Modin from PyPI. Replace '0.11' with the desired version number. ```bash pip install modin==0.11 ``` -------------------------------- ### Install Modin with Multiple Engines via Conda Source: https://github.com/modin-project/modin/blob/main/docs/getting_started/installation.md Installs Modin with Dask, Ray, and MPI engines using the conda-forge channel. ```bash conda install -c conda-forge modin-ray modin-dask modin-mpi ``` -------------------------------- ### Simple Batch Pipelining with Modin Source: https://github.com/modin-project/modin/blob/main/docs/usage_guide/advanced_usage/batch.md Demonstrates building a Modin DataFrame, creating a PandasQueryPipeline, adding queries with and without output specifications, updating the DataFrame, and computing batch results. It also shows how to specify output IDs for queries. ```python from modin.experimental.batch import PandasQueryPipeline import modin.pandas as pd import numpy as np df = pd.DataFrame( np.random.randint(0, 100, (100, 100)), columns=[f"col {i}" for i in range(1, 101)], ) # Build the dataframe we will pipeline. pipeline = PandasQueryPipeline(df) # Build the pipeline. pipeline.add_query(lambda df: df + 1, is_output=True) # Add the first query and specify that # it is an output query. pipeline.add_query( lambda df: df.rename(columns={f"col {i}":f"col {i-1}" for i in range(1, 101)}) ) # Add a second query. pipeline.add_query( lambda df: df.drop(columns=['col 99']), is_output=True, ) # Add a third query and specify that it is an output query. new_df = pd.DataFrame( np.ones((100, 100)), columns=[f"col {i}" for i in range(1, 101)], ) # Build a second dataframe that we will pipeline now instead. pipeline.update_df(new_df) # Update the dataframe that we will pipeline to be `new_df` # instead of `df`. result_dfs = pipeline.compute_batch() # Begin batch processing. # Print pipeline results print(f"Result of Query 1:\n{result_dfs[0]}") print(f"Result of Query 2:\n{result_dfs[1]}") # Output IDs can also be specified pipeline = PandasQueryPipeline(df) # Build the pipeline. pipeline.add_query( lambda df: df + 1, is_output=True, output_id=1, ) # Add the first query, specify that it is an output query, as well as specify an output id. pipeline.add_query( lambda df: df.rename(columns={f"col {i}":f"col {i-1}" for i in range(1, 101)}) ) # Add a second query. pipeline.add_query( lambda df: df.drop(columns=['col 99']), is_output=True, output_id=2, ) # Add a third query, specify that it is an output query, and specify an output_id. result_dfs = pipeline.compute_batch() # Begin batch processing. # Print pipeline results - should be a dictionary mapping Output IDs to resulting dataframes: print(f"Mapping of Output ID to dataframe:\n{result_dfs}") # Print results for query_id, res_df in result_dfs.items(): print(f"Query {query_id} resulted in\n{res_df}") ``` -------------------------------- ### Install Modin from GitHub Main Branch Source: https://github.com/modin-project/modin/blob/main/docs/getting_started/installation.md Installs the latest development version of Modin directly from the GitHub main branch using pip. ```bash pip install "modin[all] @ git+https://github.com/modin-project/modin" ``` -------------------------------- ### Start Ray Cluster Source: https://github.com/modin-project/modin/blob/main/docs/getting_started/using_modin/using_modin_cluster.md Launch a Ray cluster using a configuration file. This command initiates the head and worker nodes as defined in the `modin-cluster.yaml` file. ```bash ray up modin-cluster.yaml ``` -------------------------------- ### Install Modin with Engine Dependencies via Pip Source: https://github.com/modin-project/modin/blob/main/docs/getting_started/installation.md Installs Modin along with dependencies for specific execution engines. Choose the appropriate command based on your desired engine. ```bash pip install "modin[ray]" # Install Modin dependencies and Ray to run on Ray ``` ```bash pip install "modin[dask]" # Install Modin dependencies and Dask to run on Dask ``` ```bash pip install "modin[mpi]" # Install Modin dependencies and MPI to run on MPI through unidist ``` ```bash pip install "modin[all]" # Install Ray and Dask ``` -------------------------------- ### Modin Series Creation and Usage Example Source: https://github.com/modin-project/modin/blob/main/docs/flow/modin/pandas/series.md Demonstrates how to create a Modin Series from various sources (CSV, Python objects, pandas Series) and interact with its partitions. ```APIDOC ## Usage Guide The most efficient way to create Modin `Series` is to import data from external storage using the highly efficient Modin IO methods (for example using `pd.read_csv`, see details for Modin IO methods in the [IO](../core/io/index.md) page), but even if the data does not originate from a file, any pandas supported data type or `pandas.Series` can be used. Internally, the `Series` data is divided into partitions, which number along an axis usually corresponds to the number of the user’s hardware CPUs. If needed, the number of partitions can be changed by setting `modin.config.NPartitions`. Let’s consider simple example of creation and interacting with Modin `Series`: ```python import modin.config # This explicitly sets the number of partitions modin.config.NPartitions.put(4) import modin.pandas as pd import pandas # Create Modin Series from the external file pd_series = pd.read_csv("test_data.csv", header=None).squeeze() # Create Modin Series from the python object # pd_series = pd.Series([x for x in range(256)]) # Create Modin Series from the pandas object # pd_series = pd.Series(pandas.Series([x for x in range(256)])) # Show created `Series` print(pd_series) # List `Series` partitions. Note, that internal API is intended for # developers needs and was used here for presentation purposes # only. partitions = pd_series._query_compiler._modin_frame._partitions print(partitions) # Show the first `Series` partition print(partitions[0][0].get()) Output: # created `Series` 0 100 1 101 2 102 3 103 4 104 ... 251 351 252 352 253 353 254 354 255 355 Name: 0, Length: 256, dtype: int64 # List of `Series` partitions [[] [] [] []] # The first `Series` partition 0 0 100 1 101 2 102 3 103 4 104 .. ... 60 160 61 161 62 162 63 163 64 164 [65 rows x 1 columns] ``` As we show in the example above, Modin `Series` can be easily created, and supports any input that pandas `Series` supports. Also note that tuning of the `Series` partitioning can be done by just setting a single config. ``` -------------------------------- ### Install Modin with Specific Engines using Pip Source: https://github.com/modin-project/modin/blob/main/README.md Installs Modin with a specific compute engine. Choose the command that corresponds to your desired engine (Ray, Dask, or MPI). ```bash pip install "modin[ray]" ``` ```bash pip install "modin[dask]" ``` ```bash pip install "modin[mpi]" ``` -------------------------------- ### Configure Modin Partitions and Create Series Source: https://github.com/modin-project/modin/blob/main/docs/flow/modin/pandas/series.md This snippet demonstrates how to configure the number of partitions for Modin and create a Series from a CSV file. It also shows how to inspect the internal partitions of the Series. ```python import modin.config # This explicitly sets the number of partitions modin.config.NPartitions.put(4) import modin.pandas as pd import pandas # Create Modin Series from the external file pd_series = pd.read_csv("test_data.csv", header=None).squeeze() # Create Modin Series from the python object # pd_series = pd.Series([x for x in range(256)]) # Create Modin Series from the pandas object # pd_series = pd.Series(pandas.Series([x for x in range(256)])) # Show created `Series` print(pd_series) # List `Series` partitions. Note, that internal API is intended for # developers needs and was used here for presentation purposes # only. partitions = pd_series._query_compiler._modin_frame._partitions print(partitions) # Show the first `Series` partition print(partitions[0][0].get()) ``` -------------------------------- ### Restart Colab Environment After Modin Installation Source: https://github.com/modin-project/modin/blob/main/docs/getting_started/installation.md This Python code snippet is used to automatically restart the Google Colab environment after Modin has been installed, which is necessary for Modin to function correctly. ```python # Post-install automatically kill and restart Colab environment import os os.kill(os.getpid(), 9) ``` -------------------------------- ### Install Modin with Specific Engines using Conda Source: https://github.com/modin-project/modin/blob/main/README.md Installs Modin with a specific compute engine using conda. Select the command for the desired engine (Ray, Dask, or MPI). ```bash conda install -c conda-forge modin-ray ``` ```bash conda install -c conda-forge modin-dask ``` ```bash conda install -c conda-forge modin-mpi ``` -------------------------------- ### Prepare Data for train_test_split Source: https://github.com/modin-project/modin/blob/main/examples/jupyter/integrations/sklearn.ipynb Creates sample data using NumPy and Modin DataFrames for demonstrating train_test_split. ```python # From https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html import numpy as np from sklearn.model_selection import train_test_split X, y = pd.DataFrame(np.arange(10).reshape((5, 2))), pd.Series(range(5)) X list(y) ``` -------------------------------- ### dtypes Source: https://github.com/modin-project/modin/blob/main/docs/flow/modin/core/storage_formats/base/query_compiler.md Get columns dtypes. ```APIDOC ## dtypes ### Description Get columns dtypes. ### Returns Series with dtypes of each column. ### Return type pandas.Series ``` -------------------------------- ### Install Modin with Conda using Libmamba Solver (Conda >= 22.11) Source: https://github.com/modin-project/modin/blob/main/README.md Installs Modin using the libmamba solver, compatible with conda versions 22.11 and later. This command uses the standard --solver flag. ```bash conda install -c conda-forge modin-ray --solver=libmamba ``` -------------------------------- ### Draft Release Notes Source: https://github.com/modin-project/modin/blob/main/docs/release-procedure.md Use the release script to draft release notes. This command outputs the draft to a file. Consider using the --token option if encountering GitHub API rate limiting. ```bash python scripts/release.py notes > draft.txt ``` -------------------------------- ### unique Source: https://github.com/modin-project/modin/blob/main/docs/flow/modin/core/storage_formats/pandas/query_compiler.md Gets unique rows from the QueryCompiler. ```APIDOC ## unique(keep='first', ignore_index=True, subset=None) ### Description Get unique rows of self. ### Parameters * **keep** ( *{"first"}* *,* *"last"* *,* *False}* *,* *default: "first"*) – Which duplicates to keep. * **ignore_index** (*bool* *,* *default: True*) – If `True`, the resulting axis will be labeled `0, 1, …, n - 1`. * **subset** (*list* *,* *optional*) – Only consider certain columns for identifying duplicates, if None, use all of the columns. ### Returns New QueryCompiler with unique values. ### Return type [BaseQueryCompiler](../base/query_compiler.md#modin.core.storage_formats.base.query_compiler.BaseQueryCompiler) ### Notes Please refer to `modin.pandas.DataFrame.drop_duplicates` for more information about parameters and output format. ``` -------------------------------- ### Initialize Ray Engine with Modin Source: https://github.com/modin-project/modin/blob/main/docs/usage_guide/advanced_usage/modin_engines.md Initialize the Ray engine with a specified number of CPUs for computation. Ensure Modin is configured to use the Ray engine and set the CPU count. ```python import ray import modin.config as modin_cfg ray.init(num_cpus=) modin_cfg.Engine.put("ray") # Modin will use Ray engine modin_cfg.CpuCount.put() ``` -------------------------------- ### Modin Spreadsheet API for Interactive DataFrames Source: https://context7.com/modin-project/modin/llms.txt Render a Modin DataFrame as an interactive spreadsheet widget in Jupyter. Supports sorting, filtering, cell editing, and exporting transformation history. Install with 'pip install "modin[spreadsheet]"'. ```python import modin.pandas as pd import modin.experimental.spreadsheet as mss # Install: pip install "modin[spreadsheet]" df = pd.read_csv( "https://raw.githubusercontent.com/fivethirtyeight/data/master/college-majors/all-ages.csv" ) # Render as interactive spreadsheet in Jupyter spreadsheet = mss.from_dataframe(df) spreadsheet # Display in notebook cell # Programmatic API # Sort by a column # Click column header in UI, or: # Get the transformation history as reproducible Python code history_code = spreadsheet.get_history() print(history_code) # Filter to only operations relevant to the final state filtered_code = spreadsheet.filter_relevant_history(persist=True) print(filtered_code) # Clear the history spreadsheet.reset_history() # Convert back to Modin DataFrame after UI edits updated_df = mss.to_dataframe(spreadsheet) print(updated_df.head()) ``` -------------------------------- ### Convert Examples to TensorFlow Dataset Source: https://github.com/modin-project/modin/blob/main/examples/jupyter/integrations/huggingface.ipynb Transforms InputExamples into a TensorFlow dataset, preparing it for model training. This function handles tokenization, padding, and attention masks. ```python def convert_examples_to_tf_dataset(examples, tokenizer, max_length=128): features = [] # -> will hold InputFeatures to be converted later for e in tqdm(examples): input_dict = tokenizer.encode_plus( e.text_a, add_special_tokens=True, # Add 'CLS' and 'SEP' max_length=max_length, # truncates if len(s) > max_length return_token_type_ids=True, return_attention_mask=True, pad_to_max_length=True, # pads to the right by default # CHECK THIS for pad_to_max_length truncation=True ) input_ids, token_type_ids, attention_mask = (input_dict["input_ids"],input_dict["token_type_ids"], input_dict['attention_mask']) features.append(InputFeatures( input_ids=input_ids, attention_mask=attention_mask, token_type_ids=token_type_ids, label=e.label) ) def gen(): for f in features: yield ( { "input_ids": f.input_ids, "attention_mask": f.attention_mask, "token_type_ids": f.token_type_ids, }, f.label, ) return tf.data.Dataset.from_generator( gen, ({"input_ids": tf.int32, "attention_mask": tf.int32, "token_type_ids": tf.int32}, tf.int64), ( { "input_ids": tf.TensorShape([None]), "attention_mask": tf.TensorShape([None]), "token_type_ids": tf.TensorShape([None]), }, tf.TensorShape([]), ), ) DATA_COLUMN = 'review' LABEL_COLUMN = 'sentiment' ``` -------------------------------- ### dt_days Source: https://github.com/modin-project/modin/blob/main/docs/flow/modin/core/storage_formats/base/query_compiler.md Gets the number of days for each interval value. ```APIDOC ## dt_days() ### Description Get days for each interval value. ### Returns New QueryCompiler with the same shape as self, where each element is days for the corresponding interval value. ### Return type [BaseQueryCompiler](#modin.core.storage_formats.base.query_compiler.BaseQueryCompiler) ``` -------------------------------- ### dt_day Source: https://github.com/modin-project/modin/blob/main/docs/flow/modin/core/storage_formats/base/query_compiler.md Gets the day component for each datetime value. ```APIDOC ## dt_day() ### Description Get day component for each datetime value. ### Returns New QueryCompiler with the same shape as self, where each element is day component for the corresponding datetime value. ### Return type [BaseQueryCompiler](#modin.core.storage_formats.base.query_compiler.BaseQueryCompiler) ### Notes Please refer to `modin.pandas.Series.dt.day` for more information about parameters and output format. ### WARNING This method is supported only by one-column query compilers. ``` -------------------------------- ### Create DataFrame with Pandas Source: https://github.com/modin-project/modin/blob/main/docs/flow/modin/pandas/base.md Demonstrates the creation of a Pandas DataFrame with specified data and columns. This is a foundational step for many data manipulation tasks. ```python >>> df = pd.DataFrame(data={'Animal': ['cat', 'penguin', 'dog', ... 'spider', 'snake'], ... 'Number_legs': [4, 2, 4, 8, np.nan]}) >>> df Animal Number_legs 0 cat 4.0 1 penguin 2.0 2 dog 4.0 3 spider 8.0 4 snake NaN ``` -------------------------------- ### flags Source: https://github.com/modin-project/modin/blob/main/docs/flow/modin/pandas/base.md Get the properties associated with this pandas object. ```APIDOC ## flags ### Description Get the properties associated with this pandas object. ### Properties * **Flags.allows_duplicate_labels**: bool Indicates if the object allows duplicate labels. ### SEE ALSO `Flags` Flags that apply to pandas objects. `DataFrame.attrs` Global metadata applying to this dataset. ``` -------------------------------- ### Import TensorFlow, Scikit-learn, and Tqdm Source: https://github.com/modin-project/modin/blob/main/examples/jupyter/integrations/huggingface.ipynb Import necessary libraries for machine learning tasks, including TensorFlow, scikit-learn, and Tqdm for progress bars. ```python import tensorflow as tf import sklearn from tqdm import tqdm ```