### Install Coffea with Multiple Extras

Source: https://github.com/scikit-hep/coffea/blob/master/docs/source/getting_started/installation.md

Install Coffea with support for multiple optional dependencies, such as Dask and Parsl.

```bash
pip install coffea[dask,parsl]
```

--------------------------------

### Install Coffea with Development Dependencies

Source: https://github.com/scikit-hep/coffea/blob/master/docs/source/getting_started/installation.md

Install Coffea in editable mode, including development dependencies. Use the `[dask]` extra if you need to work with Dask executors.

```bash
cd coffea
pip install --editable .[dev]
```

```bash
pip install --editable .[dev,dask]
```

--------------------------------

### Install Coffea as Local User

Source: https://github.com/scikit-hep/coffea/blob/master/docs/source/getting_started/installation.md

Install Coffea for the current user only, useful if you lack administrator permissions.

```bash
pip install --user coffea
```

--------------------------------

### Minimal Coffea Processor Example

Source: https://github.com/scikit-hep/coffea/blob/master/docs/source/getting_started/index.md

Implement a ProcessorABC to turn NanoEvents into accumulators. This example applies muon scale factors and produces a histogram. Requires `correctionlib` and `hist` libraries.

```python
import awkward as ak
import correctionlib
import hist
from coffea import processor

class MuonProcessor(processor.ProcessorABC):
    def __init__(self, sf_path: str):
        self.corrections = correctionlib.CorrectionSet.from_file(sf_path)
        self.muon_sf = self.corrections["muon_sf"]

    def process(self, events):
        dataset = events.metadata["dataset"]

        # Create histogram with category axis
        h_mass = hist.Hist.new.StrCat([], growth=True, name=”dataset”).Reg(
            60, 60, 120, name=”mass”, label=”mμμ [GeV]”
        ).Weight()

        # select OS dimuons
        muons = events.Muon[events.Muon.tightId]
        dimuons = ak.combinations(muons, 2, fields=["lead", "trail"])
        dimuons = dimuons[dimuons.lead.charge != dimuons.trail.charge]

        # correctionlib returns per-muon weights; take product per event
        sf_lead = self.muon_sf.evaluate(dimuons.lead.eta, dimuons.lead.pt)
        sf_trail = self.muon_sf.evaluate(dimuons.trail.eta, dimuons.trail.pt)
        event_weight = sf_lead * sf_trail

        mass = (dimuons.lead + dimuons.trail).mass
        h_mass.fill(
            dataset=dataset,
            mass=mass,
            weight=event_weight,
        )

        return {
            dataset: {
                "mass": h_mass,
                "events": len(events),
            }
        }

    def postprocess(self, accumulator):
        return accumulator

```

--------------------------------

### Create and Activate Virtual Environment

Source: https://github.com/scikit-hep/coffea/blob/master/docs/source/getting_started/installation.md

Steps to create a new Python virtual environment, activate it, and install Coffea within it.

```bash
python -m venv my_env
source my_env/bin/activate
pip install coffea
```

--------------------------------

### Install Coffea with Parsl Executor

Source: https://github.com/scikit-hep/coffea/blob/master/docs/source/getting_started/installation.md

Install Coffea along with the necessary dependencies for the Parsl distributed executor.

```bash
pip install coffea[parsl]
```

--------------------------------

### Install Coffea with Dask Executor

Source: https://github.com/scikit-hep/coffea/blob/master/docs/source/getting_started/installation.md

Install Coffea along with the necessary dependencies for the Dask distributed executor.

```bash
pip install coffea[dask]
```

--------------------------------

### Run Docker Container

Source: https://github.com/scikit-hep/coffea/blob/master/docs/source/getting_started/installation.md

Example command to run a pre-built Coffea Docker image on AlmaLinux 8.

```bash
docker run -it –name coffea-container coffeateam/coffea-dask-almalinux8:latest
```

--------------------------------

### Complete ROOT file specification

Source: https://github.com/scikit-hep/coffea/blob/master/binder/filespec.ipynb

Example of a complete CoffeaROOTFileSpec with all required fields set.

```python
# 1.5 Parquet specifications (similar pattern)
print("=== CoffeaParquetFileSpec ===")

# Optional Parquet specification
parquet_optional = CoffeaParquetFileSpecOptional(
    steps=[[0, 5000]],
    num_entries=5000,
    uuid="parquet-uuid-example"
)
print("Optional Parquet spec:")
rich.print(parquet_optional)

# Required Parquet specification
parquet_required = CoffeaParquetFileSpec(
    steps=[[0, 5000], [5000, 10000]],
    num_entries=10000,
    uuid="parquet-uuid-complete"
)
print("Required Parquet spec:")
rich.print(parquet_required)
```

--------------------------------

### Dask Computation Setup and Execution

Source: https://github.com/scikit-hep/coffea/blob/master/binder/advanced_numba.ipynb

Sets up and executes a Dask computation using a specified processor and dataset. Records the time taken for computation.

```python
import time

tstart = time.time()

to_compute = apply_to_fileset(
    FancyDimuonProcessor("dask"),
    dataset_runnable,
    schemaclass=BaseSchema,
)
(out,) = dask.compute(to_compute)

elapsed = time.time() - tstart
elapsed
```

--------------------------------

### Set up Dask distributed client

Source: https://github.com/scikit-hep/coffea/blob/master/binder/advanced_numba.ipynb

Initializes a Dask distributed client for parallel processing. This example shows how to set up a local cluster for testing, but can be adapted to connect to remote clusters.

```python
from dask.distributed import Client

# here we connect to the cluster of our analysis facility.
# if you are on coffea-casa for example, that should be something like
# client = Client("tls://localhost:8786")
# locally we can make our own cluster though
from dask.distributed import LocalCluster

cluster = LocalCluster(threads_per_worker=1)
client = Client(cluster)

client
```

--------------------------------

### Create Portable Virtual Environment

Source: https://github.com/scikit-hep/coffea/blob/master/docs/source/getting_started/installation.md

Use this command to create a virtual environment with copies of installed packages. This is useful for creating portable environments.

```bash
python -m venv –copies $NAME
source $NAME/bin/activate
LOCALPATH=$NAME$(python -c ‘import sys; print(f”/lib/python{sys.version_info.major}.{sys.version_info.minor}/site-packages”)’)
export PYTHONPATH=${LOCALPATH}:$PYTHONPATH
python -m pip install setuptools pip wheel –upgrade
python -m pip install coffea
sed -i ‘1s/#!.\*python$/#!/usr/bin/env python/’ $NAME/bin/\*
sed -i ’40s/.\*VIRTUAL_ENV=”$(cd “$(dirname “$(dirname “${BASH_SOURCE[0]}” )”)” && pwd)”/’ $NAME/bin/activate
sed -i “2a source ${LCG}/setup.sh” $NAME/bin/activate
sed -i “3a export PYTHONPATH=${LOCALPATH}:$PYTHONPATH” $NAME/bin/activate
tar -zcf ${NAME}.tar.gz ${NAME}
```

--------------------------------

### Install coffea using pip

Source: https://github.com/scikit-hep/coffea/blob/master/README.rst

Install coffea using pip. Use sudo, --user, virtualenv, or pip-in-conda if desired.

```bash
pip install coffea
```

--------------------------------

### ROOT File Specification Example

Source: https://github.com/scikit-hep/coffea/blob/master/binder/filespec.ipynb

Defines the structure for specifying ROOT files within a dataset, including object paths and optional parameters.

```python
Result:
{
    'files': InputFiles(
        root={
            'ttbar_part0.root': CoffeaROOTFileSpecOptional(
                object_path='Events',
                steps=None,
                num_entries=None,
                format='root',
                lfn=None,
                pfn=None,
                uuid=None,
                num_selected_entries=None
            ),
            'ttbar_part1.root': CoffeaROOTFileSpecOptional(
                object_path='Events',
                steps=None,
                num_entries=None,
                format='root',
                lfn=None,
                pfn=None,
                uuid=None,
                num_selected_entries=None
            ),
            'ttbar_part2.root': CoffeaROOTFileSpecOptional(
                object_path='Events',
                steps=None,
                num_entries=None,
                format='root',
                lfn=None,
                pfn=None,
                uuid=None,
                num_selected_entries=None
            ),
            'ttbar_part3.root': CoffeaROOTFileSpecOptional(
                object_path='Events',
                steps=None,
                num_entries=None,
                format='root',
                lfn=None,
                pfn=None,
                uuid=None,
                num_selected_entries=None
            ),
            'ttbar_part4.root': CoffeaROOTFileSpecOptional(
                object_path='Events',
                steps=None,
                num_entries=None,
                format='root',
                lfn=None,
                pfn=None,
                uuid=None,
                num_selected_entries=None
            ),
            'ttbar_part5.root': CoffeaROOTFileSpecOptional(
                object_path='Events',
                steps=None,
                num_entries=None,
                format='root',
                lfn=None,
                pfn=None,
                uuid=None,
                num_selected_entries=None
            )
        }
    ),
    'metadata': {'process': 'ttbar', 'cross_section': 831.8, 'is_signal': False},
    'format': 'root',
    'compressed_form': None,
    'did': None
}

```

--------------------------------

### Start Dataset Discovery CLI

Source: https://github.com/scikit-hep/coffea/blob/master/binder/dataset_discovery.ipynb

Launch the dataset discovery CLI by specifying the --cli flag and providing a dataset definition file using the -d option. This initiates an interactive session for dataset querying.

```bash
!python -m coffea.dataset_tools.dataset_query --cli  -d dataset_definition.json
```

--------------------------------

### Setup Dask LocalCluster

Source: https://github.com/scikit-hep/coffea/blob/master/binder/processing.ipynb

Initializes a local Dask cluster for distributed processing. Configure threads_per_worker for optimal performance.

```python
from dask.distributed import LocalCluster

cluster = LocalCluster(threads_per_worker=1)
client = Client(cluster)

client
```

--------------------------------

### Applying Weights to Data

Source: https://github.com/scikit-hep/coffea/blob/master/binder/packedselection.ipynb

This snippet shows an example of weights applied to data. Ensure weights are correctly formatted and applied during analysis.

```python
weights: [ 26331.20117188  26331.20117188  26331.20117188  25807.2109375
  26331.20117188  26331.20117188 -26331.20117188  26067.890625
  26331.20117188  26331.20117188  26331.20117188  26331.20117188
  26331.20117188 -26331.20117188  26331.20117188  26331.20117188
 -26067.890625    26331.20117188 -26331.20117188  26331.20117188
  26331.20117188 -26331.20117188  26331.20117188  26331.20117188
  26331.20117188  26331.20117188  26331.20117188  26331.20117188
  26331.20117188  26331.20117188  26331.20117188  26331.20117188
  26331.20117188 -26331.20117188  26331.20117188 -26331.20117188
  26331.20117188 -26331.20117188  26331.20117188 -26331.20117188]
```

--------------------------------

### Get Rucio Client

Source: https://github.com/scikit-hep/coffea/blob/master/binder/dataset_discovery.ipynb

Initializes and retrieves a Rucio client instance. This client is used for subsequent Rucio operations.

```python
client = rucio_utils.get_rucio_client()
client
```

--------------------------------

### Import Rucio and Dataset Query Utilities

Source: https://github.com/scikit-hep/coffea/blob/master/binder/dataset_discovery.ipynb

Imports necessary modules from Coffea for Rucio interaction and dataset querying. Ensure these libraries are installed.

```python
from coffea.dataset_tools import rucio_utils
from coffea.dataset_tools.dataset_query import print_dataset_query
from coffea.util import coffea_console
from rich.table import Table
```

--------------------------------

### Running ParticleNet with Dask Awkward

Source: https://github.com/scikit-hep/coffea/blob/master/binder/mltools.ipynb

Example of applying a ParticleNet model to Dask-processed events and computing the results.

```python
dask_events = open_events()
dask_jets = dask_events.Jet
dask_jets["MLresults"] = pn_example2(dask_events)
dask_events["Jet"] = dask_jets
print(dask_events.Jet.MLresults.compute())
```

--------------------------------

### Define a Fileset with Multiple Datasets

Source: https://github.com/scikit-hep/coffea/blob/master/docs/source/notebooks/filespec.ipynb

This snippet shows how to create a fileset containing multiple datasets, each with its own root file specification and metadata. It includes examples for simulation and data files.

```python
Fileset(\n    datasets={\n        'ttbar_simulation': CoffeaROOTFileSpecOptional(\n            object_path='Events',\n            steps=None,\n            num_entries=None,\n            format='root',\n            lfn=None,\n            pfn=None,\n            uuid=None,\n            num_selected_entries=None\n        ),\n        'single_top': CoffeaROOTFileSpecOptional(\n            object_path='Events',\n            steps=None,\n            num_entries=None,\n            format='root',\n            lfn=None,\n            pfn=None,\n            uuid=None,\n            num_selected_entries=None\n        ),\n        'data_2023B.root': CoffeaROOTFileSpecOptional(\n            object_path='Events',\n            steps=None,\n            num_entries=None,\n            format='root',\n            lfn=None,\n            pfn=None,\n            uuid=None,\n            num_selected_entries=None\n        )\n    },\n    metadata={'is_data': True, 'era': '2023'},\n    format='root',\n    compressed_form=None,\n    did=None\n)
```

--------------------------------

### Get Dataset Replicas with Site Blocklist

Source: https://github.com/scikit-hep/coffea/blob/master/binder/dataset_discovery.ipynb

Fetches dataset file replicas, excluding specified sites. Handles potential exceptions during the process.

```python
try:
    ( 
        outfiles, 
        outsites, 
        sites_counts, 
    ) = rucio_utils.get_dataset_files_replicas(
        dataset, 
        allowlist_sites=[], 
        blocklist_sites=["T2_DE_DESY", "T3_CH_PSI"], 
        regex_sites=None, 
        mode="full",  # full or first. "full"==all the available replicas
        client=client,
    )
except Exception as e:
    print(f"\n[red bold] Exception: {e}[/]")

print_replicas(sites_counts)
```

--------------------------------

### DataGroupSpec Structure Example

Source: https://github.com/scikit-hep/coffea/blob/master/binder/filespec.ipynb

Illustrates the detailed structure of a DataGroupSpec object, including nested DatasetSpec, InputFiles, and CoffeaROOTFileSpec. This is useful for understanding how files and their metadata are organized.

```python
rich.print(optimized_subset)
```

--------------------------------

### Load NanoAOD Events with Eager Mode

Source: https://github.com/scikit-hep/coffea/blob/master/binder/packedselection.ipynb

Loads NanoAOD events from a ROOT file using NanoAODSchema in eager mode. Ensure the 'coffea' library and its dependencies are installed.

```python
import awkward as ak
import numpy as np
from coffea.nanoevents import NanoEventsFactory, NanoAODSchema
from matplotlib import pyplot as plt

NanoAODSchema.warn_missing_crossrefs = False

fname = "coffea/tests/samples/nano_dy.root"
events = NanoEventsFactory.from_root(
    {fname: "Events"},
    metadata={"dataset": "nano_dy"},
    schemaclass=NanoAODSchema,
    mode="eager",
).events()

events
```

--------------------------------

### Fetch Dataset Replicas using Rucio

Source: https://github.com/scikit-hep/coffea/blob/master/docs/source/notebooks/dataset_discovery.ipynb

Fetches all available replicas for a given dataset from Rucio. Requires a Rucio client and can be filtered by regex. Defaults to 'full' mode to get all replicas.

```python
from rucio.client.client import Client
from coffea.util import coffea_console
from rich.table import Table

client = Client()
dataset = "/TTToSemiLeptonic_TuneCP5CR1_13TeV-powheg-pythia8/RunIISummer20UL18NanoAODv9-106X_upgrade2018_realistic_v16_L1v1-v2/NANOAODSIM"
regex_sites=[],
mode="full",  # full or first. "full"==all the available replicas
client=client,
    )
except Exception as e:
    print(f"\n[red bold] Exception: {e}[/]")
```

--------------------------------

### Install Coffea with Conda

Source: https://github.com/scikit-hep/coffea/blob/master/docs/source/getting_started/installation.md

Install Coffea using Conda if you are using the Conda package manager.

```bash
conda install coffea
```

--------------------------------

### Display Dataset Query CLI Help

Source: https://github.com/scikit-hep/coffea/blob/master/binder/dataset_discovery.ipynb

Execute the dataset_query module with the --help flag to view all available command-line options and their descriptions.

```bash
!python -m coffea.dataset_tools.dataset_query --help
```

--------------------------------

### Initialize InputFiles with File Specifications

Source: https://github.com/scikit-hep/coffea/blob/master/binder/filespec.ipynb

Create an `InputFiles` object by providing a dictionary of file specifications. This allows for defining multiple input files with specific parameters like object paths, steps, and UUIDs.

```python
dict_of_filespecs = {
    "file1.root": CoffeaROOTFileSpec(
        object_path="Events", steps=[[0, 10]], num_entries=10, uuid="uuid1"
    ),
    "file1.parquet": CoffeaParquetFileSpec(
        steps=[[0, 100]], num_entries=100, uuid="uuid2"
    ),
    "file2.root": CoffeaROOTFileSpecOptional(
        object_path="Events", steps=[[10, 20]], num_entries=None, uuid=None
    ),
}

filedict_from_filespecs = InputFiles(dict_of_filespecs)
```

--------------------------------

### Build Documentation Locally

Source: https://github.com/scikit-hep/coffea/blob/master/docs/source/contributing.md

Build the project's documentation locally to preview changes before submitting a pull request. This ensures documentation is up-to-date with the code.

```bash
pushd docs
make html
popd
```

--------------------------------

### Basic ROOTFileSpec

Source: https://github.com/scikit-hep/coffea/blob/master/binder/filespec.ipynb

Demonstrates the creation of a basic ROOT file specification using ROOTFileSpec. It shows how to specify the object path and optionally define steps for data processing.

```python
# 1.1 Basic ROOTFileSpec for ROOT files
print("=== Basic ROOTFileSpec ===")

# Minimal ROOT file specification
uproot_spec = ROOTFileSpec(object_path="Events")
print("Basic ROOT spec:")
rich.print(uproot_spec)
print(f"Format: {uproot_spec.format}")
print(f"Steps: {uproot_spec.steps}")

# ROOT file specification with steps
uproot_spec_with_steps = ROOTFileSpec(
    object_path="Events",
    steps=[[0, 1000], [1000, 2000], [2000, 3000]]
)
print("\nROOT spec with steps:")
rich.print(uproot_spec_with_steps)
```

--------------------------------

### Install Jupyter Kernel for Coffea Environment

Source: https://github.com/scikit-hep/coffea/blob/master/docs/source/getting_started/installation.md

After activating the Coffea virtual environment, use this command to install a Jupyter kernel that points to this environment. This makes the environment available within Jupyter notebooks.

```bash
python -m ipykernel install --user --name=coffeaenv
```

--------------------------------

### Install Coffea using pip with CERN LCG

Source: https://github.com/scikit-hep/coffea/blob/master/docs/source/getting_started/installation.md

Install Coffea using pip after sourcing a CERN LCG Python release. This method may be fragile due to potential dependency conflicts.

```bash
source /cvmfs/sft.cern.ch/lcg/views/LCG_98python3/x86_64-centos7-gcc9-opt/setup.sh  # or .csh, etc.
pip install --user coffea
```

--------------------------------

### Check Python Version

Source: https://github.com/scikit-hep/coffea/blob/master/docs/source/getting_started/installation.md

Use this command to check your installed Python version.

```bash
python --version
```

```bash
python3 --version
```

--------------------------------

### Update Coffea

Source: https://github.com/scikit-hep/coffea/blob/master/docs/source/getting_started/installation.md

Use this command to update an existing Coffea installation to the latest version.

```bash
pip install --upgrade coffea
```

--------------------------------

### Run Development Tools

Source: https://github.com/scikit-hep/coffea/blob/master/docs/source/getting_started/installation.md

Execute pre-commit hooks for linting, run tests with pytest, and build documentation using make.

```bash
pre-commit run --all-files
pytest tests
pushd docs && make html && popd
```

--------------------------------

### Initialize and Filter Sites with DataDiscoveryCLI

Source: https://github.com/scikit-hep/coffea/blob/master/binder/dataset_discovery.ipynb

Initialize the DataDiscoveryCLI and filter sites using a regular expression. Load dataset definitions with specified query and replica strategies.

```python
ddc = DataDiscoveryCLI()
ddc.do_regex_sites(r"T[123]_(CH|IT|UK|FR|DE)_")
ddc.load_dataset_definition(dataset_definition, query_results_strategy="all", replicas_strategy="round-robin")
```

--------------------------------

### Initialize and Load Dataset Definition

Source: https://github.com/scikit-hep/coffea/blob/master/binder/dataset_discovery.ipynb

Initialize DataDiscoveryCLI and load the dataset definition, specifying query and replica strategies.

```python
ddc = DataDiscoveryCLI()
ddc.load_dataset_definition(dataset_definition, query_results_strategy="all", replicas_strategy="round-robin")
```

--------------------------------

### Get Total Event Weight with Variation

Source: https://github.com/scikit-hep/coffea/blob/master/binder/analysis_tools.ipynb

Retrieve the total event weight for a specific systematic variation (e.g., 'eleSFUp').

```python
weights.weight("eleSFUp")
```

--------------------------------

### Using value_accumulator for Arbitrary Types

Source: https://github.com/scikit-hep/coffea/blob/master/docs/source/user_guide/accumulators.md

Accumulates values of arbitrary types using value_accumulator. This example shows tracking the sum of weights.

```python
from coffea.processor import value_accumulator

def process(self, events):
  dataset = events.metadata["dataset"]
  
  # Track sum of weights
  sumw = value_accumulator(float, float(ak.sum(events.genWeight)))
  
  return {
    dataset: {
      "sumw": sumw,
    }
  }

```

--------------------------------

### Basic ParquetFileSpec Creation

Source: https://github.com/scikit-hep/coffea/blob/master/docs/source/notebooks/filespec.ipynb

Shows how to create a basic ParquetFileSpec. For Parquet files, 'object_path' is always None as it refers to the entire file.

```python
# 1.2 Basic ParquetFileSpec for Parquet files
print("=== Basic ParquetFileSpec ===")

# Minimal Parquet file specification
parquet_spec = ParquetFileSpec()
print("Basic Parquet spec:")
rich.print(parquet_spec)
print(f"Format: {parquet_spec.format}")
print(f"Object path (always None): {parquet_spec.object_path}")
```

--------------------------------

### Create DatasetSpec with Complete File Specifications

Source: https://github.com/scikit-hep/coffea/blob/master/binder/filespec.ipynb

Construct a DatasetSpec with detailed specifications for each ROOT file, including object paths, processing steps, number of entries, and unique identifiers.

```python
# 3.2 Creating DatasetSpec with complete file specifications
print("=== DatasetSpec with Complete Specifications ===")

# Create individual file specifications
file1_spec = CoffeaROOTFileSpec(
    object_path="Events",
    steps=[[0, 1000], [1000, 2000]],
    num_entries=2000,
    uuid="file1-uuid"
)

file2_spec = CoffeaROOTFileSpec(
    object_path="Events", 
    steps=[[0, 1500], [1500, 3000]],
    num_entries=3000,
    uuid="file2-uuid"
)

```

--------------------------------

### Initialize NanoEventsFactory and Schema

Source: https://github.com/scikit-hep/coffea/blob/master/binder/processing.ipynb

Import necessary libraries for creating NanoEvents objects. Use `NanoEventsFactory` with `BaseSchema` to interpret ROOT files for analysis, especially when dealing with custom or non-standard NanoAOD formats.

```python
from coffea.nanoevents import NanoEventsFactory, BaseSchema
import matplotlib.pyplot as plt

```

--------------------------------

### Select a Dataset for Replica Discovery

Source: https://github.com/scikit-hep/coffea/blob/master/binder/dataset_discovery.ipynb

Selects the first dataset from a list to initiate the process of finding its file replicas.

```python
dataset = outlist[0]
dataset
```

--------------------------------

### Apply Processor to Full Fileset

Source: https://github.com/scikit-hep/coffea/blob/master/binder/processing.ipynb

Apply the Coffea processor to the entire preprocessed fileset. This is similar to the small fileset example but uses `preprocessed_available`.

```python
full_tg, rep = apply_to_fileset(
    data_manipulation=MyProcessor("dask"),
    fileset=preprocessed_available,
    schemaclass=BaseSchema,
    uproot_options={"allow_read_errors_with_report": (OSError, ValueError)},
)
```

--------------------------------

### Instantiate NanoEvents with NanoAODSchema

Source: https://github.com/scikit-hep/coffea/blob/master/binder/nanoevents.ipynb

Instantiates the NanoEvents object from a ROOT file using the NanoAODSchema. Includes options for metadata, access logging, and eager mode. Ensure NanoAODSchema.warn_missing_crossrefs is set to False to suppress warnings.

```python
import awkward as ak
from coffea.nanoevents import NanoEventsFactory, NanoAODSchema

NanoAODSchema.warn_missing_crossrefs = False

fname = "coffea/tests/samples/nano_dy.root"
access_log = []
events = NanoEventsFactory.from_root(
    {fname: "Events"},
    schemaclass=NanoAODSchema,
    metadata={"dataset": "DYJets"},
    mode="eager",
    access_log=access_log,
).events()
```

--------------------------------

### Create an InputFiles object

Source: https://github.com/scikit-hep/coffea/blob/master/binder/filespec.ipynb

Demonstrates the initialization of an InputFiles object, which acts as a dictionary-like container for CoffeaFileSpec instances.

```python
# 2.1 Create an InputFiles
print("=== InputFiles ===")
```

--------------------------------

### Get Keys from InputFiles

Source: https://github.com/scikit-hep/coffea/blob/master/binder/filespec.ipynb

Retrieve a list of all filenames (keys) present in the `InputFiles` object. This is helpful for understanding the current set of managed files.

```python
# show keys
print(f"Keys in filedict: {list(filedict_from_filespecs.keys())}")
```

--------------------------------

### Accessing Jet Properties in Virtual Mode (Tightness)

Source: https://github.com/scikit-hep/coffea/blob/master/binder/nanoevents.ipynb

Demonstrates accessing a boolean property like `isTight` for jets. In virtual mode, this also triggers branch loading if not already accessed.

```python
events.Jet.isTight
```

--------------------------------

### Compute Dask Graph and Get Results

Source: https://github.com/scikit-hep/coffea/blob/master/docs/source/notebooks/processing.ipynb

Execute the Dask computation and retrieve the processed results and metadata. This operation can be time-consuming for large datasets.

```python
small_computed, small_rep_computed = dask.compute(small_tg, small_rep)
```

--------------------------------

### Download Model and Data

Source: https://github.com/scikit-hep/coffea/blob/master/binder/mltools.ipynb

Downloads a pre-trained PyTorch model and a sample PFNano ROOT file for testing.

```python
!wget --quiet -O model.pt https://github.com/scikit-hep/coffea/raw/master/tests/samples/triton_models_test/pn_test/1/model.pt
!wget --quiet -O pfnano.root https://github.com/scikit-hep/coffea/raw/master/tests/samples/pfnano.root
```

--------------------------------

### Create and evaluate scale factor lookups

Source: https://github.com/scikit-hep/coffea/blob/master/binder/analysis_tools.ipynb

Initializes an extractor, adds weight sets from a ROOT file containing scale factors, finalizes the extractor, and creates an evaluator for accessing these scale factors.

```python
from coffea.lookup_tools import extractor

ext = extractor()
ext.add_weight_sets(["* * data/testSF2d.histo.root"])
ext.finalize()
evaluator = ext.make_evaluator()
evaluator.keys()
```

--------------------------------

### List Available Fields in NanoEvents

Source: https://github.com/scikit-hep/coffea/blob/master/binder/nanoevents.ipynb

Use `events.Generator.fields` to get a list of all available fields in the NanoEvents object. This is useful for understanding the structure of your dataset.

```python
events.Generator.fields
```

--------------------------------

### Get Total Event Weight

Source: https://github.com/scikit-hep/coffea/blob/master/binder/analysis_tools.ipynb

Calculate the total event weight by multiplying all added weights. This is the nominal weight used for filling histograms.

```python
weights.weight()
```

--------------------------------

### Plotting Scaled Muon Count Distribution

Source: https://github.com/scikit-hep/coffea/blob/master/binder/advanced_numba.ipynb

Visualizes the 'nMuons' distribution from scaled data on a logarithmic y-axis. Ensures the y-axis starts from 1.

```python
fig, ax = plt.subplots()

scaled["nMuons"].plot1d(ax=ax, overlay="dataset")
ax.set_yscale("log")
ax.set_ylim(1, None)
plt.show()
```

--------------------------------

### Initialize ParticleNet Model and Run Inference

Source: https://github.com/scikit-hep/coffea/blob/master/binder/mltools.ipynb

Initializes a ParticleNet model and runs inference on a dask_awkward array. The results are computed and printed.

```python
pn_example1 = ParticleNetExample1("model.pt")

# Running on dask_awkward array
dask_events = open_events()
dask_results = pn_example1(dask_events)
print("Dask awkward results:", dask_results.compute())  # Runs file!
```

--------------------------------

### Initialize ParticleNet Model

Source: https://github.com/scikit-hep/coffea/blob/master/docs/source/notebooks/mltools.ipynb

Initializes the ParticleNet model with a specified model file. This is the first step before running inference.

```python
pn_example1 = ParticleNetExample1("model.pt")
```

--------------------------------

### Retrieve NminusOne Results

Source: https://github.com/scikit-hep/coffea/blob/master/binder/packedselection.ipynb

Get the results of the N-1 selection, which are returned as a namedtuple containing labels, event counts, and boolean masks for each selection step.

```python
res = nminusone.result()
print(type(res), res._fields)
```

```python
labels, nev, masks = res
labels, nev, masks
```

--------------------------------

### Saving and Loading Results with coffea.util

Source: https://github.com/scikit-hep/coffea/blob/master/docs/source/user_guide/accumulators.md

Demonstrates how to save a processing result to disk and load it back later using coffea.util.save and coffea.util.load.

```python
import coffea.util

result = runner(fileset, processor_instance=my_processor)
coffea.util.save(result, "out.coffea")

# Reload later
loaded = coffea.util.load("out.coffea")

```

--------------------------------

### Import necessary libraries for Coffea and Dask

Source: https://github.com/scikit-hep/coffea/blob/master/binder/advanced_numba.ipynb

Imports essential libraries for data processing, histogramming, and distributed computing with Coffea and Dask. Ensure these are installed before running.

```python
import hist
import hist.dask
import dask
import awkward as ak
import dask_awkward as dak
import matplotlib.pyplot as plt

from coffea import processor
from coffea.nanoevents import BaseSchema
from coffea.nanoevents.methods import candidate
from coffea.dataset_tools import (
    apply_to_fileset,
    preprocess,
)
```

--------------------------------

### Display Help for plot_vars Method

Source: https://github.com/scikit-hep/coffea/blob/master/binder/packedselection.ipynb

Displays the help documentation for the `plot_vars` method, detailing its parameters for customizing histogram generation.

```python
help(nminusone.plot_vars)
```

--------------------------------

### Compute Full Dataset and Get Results

Source: https://github.com/scikit-hep/coffea/blob/master/docs/source/notebooks/processing.ipynb

Execute the Dask computation for the entire dataset and retrieve the final results and metadata. This can take a significant amount of time.

```python
out, rep = dask.compute(full_tg, rep)
```

--------------------------------

### Create DatasetSpec from Simple File Dictionary

Source: https://github.com/scikit-hep/coffea/blob/master/binder/filespec.ipynb

Define a DatasetSpec using a dictionary of ROOT files and their object paths. Metadata can be included to describe the dataset.

```python
root_dataset_simple = DatasetSpec(
    files={
        "data_file_1.root": "Events",
        "data_file_2.root": "Events",
        "data_file_3.root": "Events"
    },
    metadata={"sample_type": "data", "year": 2023}
)

print("Simple ROOT dataset:")
rich.print(root_dataset_simple)
print(f"Detected format: {root_dataset_simple.format}")
print(f"Number of files: {len(root_dataset_simple.files)}")
print(f"Metadata: {root_dataset_simple.metadata}")
```

--------------------------------

### Chunk-based Dataset Manipulations in Coffea

Source: https://github.com/scikit-hep/coffea/blob/master/binder/filespec.ipynb

Illustrates the use of chunk-based manipulation functions for datasets in Coffea. This example serves as an introduction to limiting processing to specific chunks or files.

```python
# 7.1 Chunk-based manipulations
print("=== Chunk-based Manipulations ===")
```

--------------------------------

### Create DatasetSpec with Complete Specifications

Source: https://github.com/scikit-hep/coffea/blob/master/binder/filespec.ipynb

Define a `DatasetSpec` with explicit file specifications for multiple ROOT files, including metadata. This is useful for precisely defining the structure and content of your dataset.

```python
complete_dataset = DatasetSpec(
    files=InputFiles({
        "processed_data_1.root": file1_spec,
        "processed_data_2.root": file2_spec
    }),
    metadata={"processing_version": "v2.1", "cross_section": 1.23},
)

print("Complete dataset:")
rich.print(complete_dataset)
print(f"Detected format: {complete_dataset.format}")
print(f"Number of files: {len(complete_dataset.files)}")
print(f"Ready for column-joining: {complete_dataset.joinable}")
```

--------------------------------

### ParticleNet Example Class for Awkward Arrays

Source: https://github.com/scikit-hep/coffea/blob/master/binder/mltools.ipynb

A comprehensive Python class for preparing Awkward arrays for ParticleNet and post-processing its output. Includes data transformation and feature engineering.

```python
class ParticleNetExample(torch_wrapper):
    def prepare_awkward(self, events):
        jets = ak.flatten(events.Jet)

        def pad(arr):
            return ak.fill_none(
                ak.pad_none(arr, 100, axis=1, clip=True),
                0.0,
            )

        # Human readable version of what the inputs are
        # Each array is a N jets x 100 constituent array
        imap = {
            "points": {
                "deta": pad(jets.eta - jets.constituents.pf.eta),
                "dphi": pad(jets.delta_phi(jets.constituents.pf)),
            },
            "features": {
                "dr": pad(jets.delta_r(jets.constituents.pf)),
                "lpt": pad(np.log(jets.constituents.pf.pt)),
                "lptf": pad(np.log(jets.constituents.pf.pt / jets.pt)),
                "f1": pad(np.log(np.abs(jets.constituents.pf.d0) + 1)),
                "f2": pad(np.log(np.abs(jets.constituents.pf.dz) + 1)),
            },
            "mask": {
                "mask": pad(ak.ones_like(jets.constituents.pf.pt)),
            },
        }

        # Compacting the array elements into the desired dimension using
        # ak.concatenate
        retmap = {k: ak.concatenate([x[:, np.newaxis, :] for x in imap[k].values()], axis=1) for k in imap.keys()}

        # Returning everything using a dictionary. Also take care of type
        # conversion here.
        return (), {
            "points": ak.values_astype(retmap["points"], "float32"),
            "features": ak.values_astype(retmap["features"], "float32"),
            "mask": ak.values_astype(retmap["mask"], "float16"),
        }

    def postprocess_awkward(self, return_array, events):
        softmax = np.exp(return_array)[:, 0] / ak.sum(np.exp(return_array), axis=-1)
        njets = ak.count(events.Jet.pt, axis=-1)
        return ak.unflatten(softmax, njets)


pn_example = ParticleNetExample("model.pt")
```

--------------------------------

### Complete Parquet file specification

Source: https://github.com/scikit-hep/coffea/blob/master/docs/source/notebooks/filespec.ipynb

Demonstrates the creation of a complete CoffeaParquetFileSpec with all required fields, including steps, num_entries, and uuid.

```python
parquet_required = CoffeaParquetFileSpec(
    steps=[[0, 5000], [5000, 10000]],
    num_entries=10000,
    uuid="parquet-uuid-complete"
)
```

--------------------------------

### ParticleNet Example Class

Source: https://github.com/scikit-hep/coffea/blob/master/docs/source/notebooks/mltools.ipynb

A comprehensive Python class for preparing and post-processing data for a ParticleNet model using awkward arrays. It handles feature extraction, padding, and output transformation.

```python
class ParticleNetExample(torch_wrapper):
    def prepare_awkward(self, events):
        jets = ak.flatten(events.Jet)

        def pad(arr):
            return ak.fill_none(
                ak.pad_none(arr, 100, axis=1, clip=True),
                0.0,
            )

        # Human readable version of what the inputs are
        # Each array is a N jets x 100 constituent array
        imap = {
            "points": {
                "deta": pad(jets.eta - jets.constituents.pf.eta),
                "dphi": pad(jets.delta_phi(jets.constituents.pf)),
            },
            "features": {
                "dr": pad(jets.delta_r(jets.constituents.pf)),
                "lpt": pad(np.log(jets.constituents.pf.pt)),
                "lptf": pad(np.log(jets.constituents.pf.pt / jets.pt)),
                "f1": pad(np.log(np.abs(jets.constituents.pf.d0) + 1)),
                "f2": pad(np.log(np.abs(jets.constituents.pf.dz) + 1)),
            },
            "mask": {
                "mask": pad(ak.ones_like(jets.constituents.pf.pt)),
            },
        }

        # Compacting the array elements into the desired dimension using
        # ak.concatenate
        retmap = {k: ak.concatenate([x[:, np.newaxis, :] for x in imap[k].values()], axis=1) for k in imap.keys()}

        # Returning everything using a dictionary. Also take care of type
        # conversion here.
        return (), {
            "points": ak.values_astype(retmap["points"], "float32"),
            "features": ak.values_astype(retmap["features"], "float32"),
            "mask": ak.values_astype(retmap["mask"], "float16"),
        }

    def postprocess_awkward(self, return_array, events):
        softmax = np.exp(return_array)[:, 0] / ak.sum(np.exp(return_array), axis=-1)
        njets = ak.count(events.Jet.pt, axis=-1)
        return ak.unflatten(softmax, njets)

pm_example = ParticleNetExample("model.pt")
```

--------------------------------

### Inspect InputFiles Properties

Source: https://github.com/scikit-hep/coffea/blob/master/docs/source/notebooks/filespec.ipynb

After initializing `InputFiles`, you can inspect its properties like detected formats and the total number of files. This helps in understanding the composition of your input dataset.

```python
print("InputFiles:")
rich.print(filedict_from_filespecs)

# computed property: format
print(f"Detected format(s): {filedict_from_filespecs.format}")
print(f"Number of files: {len(filedict_from_filespecs)}")
```

--------------------------------

### Initialize Weights and Add to Cutflow

Source: https://github.com/scikit-hep/coffea/blob/master/binder/packedselection.ipynb

Initialize a Weights instance with the number of events and add event weights using the 'add' method. This prepares the weights for application to a cutflow.

```python
from coffea.analysis_tools import Weights

weights = Weights(len(events))
weights.add("genweight", events.genWeight)

wgtcutflow = selection.cutflow("noMuon", "twoElectron", "leadPt20", weights=weights, weightsmodifier=None)
wgtcutflow
```

--------------------------------

### Initialize Dask Events Factory

Source: https://github.com/scikit-hep/coffea/blob/master/binder/packedselection.ipynb

Loads events from a ROOT file in dask mode using `NanoEventsFactory`. This enables delayed computation with `dask_awkward` arrays. Ensure `coffea` and `dask_awkward` are installed.

```python
import dask
import dask_awkward as dak

fname = "coffea/tests/samples/nano_dy.root"
dakevents = NanoEventsFactory.from_root(
    {fname: "Events"},
    metadata={"dataset": "nano_dy"},
    schemaclass=NanoAODSchema,
    mode="dask",
).events()

dakevents
```