### Eager Indexing Examples

Source: https://genvarloader.readthedocs.io/en/latest/_modules/genvarloader/_dataset/_impl.html

Demonstrates eager indexing of a GenVarLoader Dataset. This method allows direct access to specific regions and samples, returning data as NumPy arrays. Examples include single element access, slicing, and advanced indexing.

```python
dataset[0, 9]  # first region, 10th sample
dataset[:10]  # first 10 regions and all samples
dataset[:10, :5]  # first 10 regions and 5 samples
dataset[[2, 2], [0, 1]]  # 3rd region, 1st and 2nd samples
```

--------------------------------

### Install GenVarLoader

Source: https://genvarloader.readthedocs.io/en/latest/_sources/index.md

Install the GenVarLoader package using pip. A PyTorch dependency is not included and may require separate installation.

```bash
pip install genvarloader
```

--------------------------------

### Get Dummy Dataset

Source: https://genvarloader.readthedocs.io/en/latest/dataset.html

Creates a dummy GenVarLoader dataset for testing and demonstration purposes. This is useful for quickly setting up an example dataset.

```python
ds = gvl.get_dummy_dataset()

```

--------------------------------

### RaggedVariants.start

Source: https://genvarloader.readthedocs.io/en/latest/_modules/genvarloader/_dataset/_rag_variants.html

Property to access the 0-based start positions.

```APIDOC
## RaggedVariants.start

### Description
Provides access to the 'start' field, which contains the 0-based start positions of variants.

### Returns
* **Ragged[POS_TYPE]**: A Ragged array representing the start positions.
```

--------------------------------

### Check Phased Genotypes with bcftools

Source: https://genvarloader.readthedocs.io/en/latest/faq.html

Provides a command-line example using bcftools to count the number of phased records in a VCF file. This is useful for verifying genotype phasing.

```bash
# for VCF, -p filters for records where all samples are phased
bcftools view -Hp $vcf | wc -l
# returns number of phased records

```

--------------------------------

### Initialize DatasetWithSites

Source: https://genvarloader.readthedocs.io/en/latest/_modules/genvarloader/_variants/_sitesonly.html

Initializes the DatasetWithSites object, performing essential setup like contig normalization, region-site mapping, and creating an indexer for efficient data access. Raises errors if sites are not SNPs or if there's no overlap between regions and sites.

```python
ds_sites.dataset = ds_sites.dataset.with_tracks("read-depth")
wt_haps, mut_haps, flags, tracks = ds_sites[0, 0]
```

```python
if max_variants_per_region > 1:
    raise NotImplementedError("max_variants_per_region > 1 not yet supported")

if not isinstance(dataset, ArrayDataset):
    raise ValueError(
        'Dataset output_length must either be "variable" or a fixed length integer.'
    )

sites = _sites_table_to_bedlike(sites)

if sites.select(
    (
        (pl.col("REF").str.len_bytes() != 1) | pl.col("ALT").str.len_bytes()
        != 1
    ).any()
).item():
    raise ValueError(
        "All sites must be SNPs. Consider filtering the VCF as either a preprocessing step or via the sites_vcf_to_table function."
    )

c_norm = ContigNormalizer(dataset.contigs)
chroms: list[str] = sites["chrom"].to_list()
norm_chroms = c_norm.norm(chroms)
norm_chroms = [
    c if norm is None else norm for c, norm in zip(chroms, norm_chroms)
]
sites = sites.with_columns(chrom=pl.Series(norm_chroms))

ds_bed = dataset.regions.with_row_index("region_idx")
if isinstance(dataset.output_length, int):
    ds_bed = ds_bed.with_columns(
        chromEnd=pl.col("chromStart") + dataset.output_length
    )
ds_pyr = sp.bed.to_pyr(ds_bed)
sites_pyr = sp.bed.to_pyr(sites.with_row_index("site_idx"))
rows = pl.from_pandas(ds_pyr.join(sites_pyr, suffix="_site").df)
if rows.height == 0:
    raise RuntimeError("No overlap between dataset regions and sites.")

rows = rows.rename(
    {
        "Chromosome": "chrom",
        "Start": "chromStart",
        "End": "chromEnd",
        "Strand": "strand",
        "Start_site": "POS0",
    },
    strict=False,
).drop("End_site")

_dataset = dataset.with_seqs("annotated").with_settings(
    deterministic=True, jitter=0
)

self.sites = sites
self.dataset = _dataset
self.rows = rows
self._row_map = rows.select("region_idx", "site_idx").to_numpy()
self._idxer = DatasetIndexer.from_region_and_sample_idxs(
    np.arange(self.rows.height), np.arange(dataset.n_samples), dataset.samples
)
```

--------------------------------

### Get Sample Library Sizes

Source: https://genvarloader.readthedocs.io/en/latest/geuvadis.html

This snippet calculates and displays the first five sample library sizes from a dataset, which is a prerequisite for normalizing read depth. It uses polars for data manipulation.

```python
sample_library_sizes = (
    pl.Series(ds.samples)
    .to_frame("sample")
    .join(bigwig_table, on="sample", how="left")["read_count"]
    .to_numpy()
)
sample_library_sizes[:5]
```

--------------------------------

### RaggedVariants Constructor

Source: https://genvarloader.readthedocs.io/en/latest/_modules/genvarloader/_dataset/_rag_variants.html

Initializes a RaggedVariants object. It requires 'alt' and 'start' fields, and either 'ref' or 'ilen'. Other fields can be passed as keyword arguments.

```APIDOC
## RaggedVariants Constructor

### Description
Initializes a RaggedVariants object. It requires 'alt' and 'start' fields, and either 'ref' or 'ilen'. Other fields can be passed as keyword arguments.

### Parameters
* **alt** (ak.Array) - Required - Array of alternative alleles.
* **start** (Ragged[POS_TYPE]) - Required - 0-based start positions.
* **ref** (ak.Array | None) - Optional - Reference alleles.
* **ilen** (Ragged[np.int32] | None) - Optional - Indel lengths.
* **dosage** (Ragged[DOSAGE_TYPE] | None) - Optional - Dosage information.
* **kwargs** (Ragged[np.number]) - Optional - Additional fields.

### Raises
* **ValueError**: If neither 'ref' nor 'ilen' is provided.
```

--------------------------------

### Transform data on-the-fly

Source: https://genvarloader.readthedocs.io/en/latest/_sources/index.md

Apply custom transformations to the data loaded from the GenVarLoader dataset. This example demonstrates one-hot encoding DNA sequences and rearranging dimensions using `seqpro` and `einops`.

```python
import seqpro as sp
from einops import rearrange


def transform(haplotypes, tracks):
    ohe = sp.DNA.ohe(haplotypes)
    ohe = rearrange(ohe, "... length alphabet -> ... alphabet length")
    return ohe, tracks


transformed_dataset = dataset.with_settings(transform=transform)
```

--------------------------------

### Create a Dummy Dataset

Source: https://genvarloader.readthedocs.io/en/latest/_sources/dataset.md

Use `gvl.get_dummy_dataset()` to create a sample dataset for testing or demonstration purposes.

```python
ds = gvl.get_dummy_dataset()
```

--------------------------------

### Prepare BigWig Sample Table

Source: https://genvarloader.readthedocs.io/en/latest/geuvadis.html

Reads a CSV file containing BigWig paths and sample information, then joins it with actual download paths to update the 'path' column. This ensures correct file referencing for BigWig data.

```python
bigwig_table = (
    pl.read_csv(bw_table_path)
    .join(
        pl.Series(bw_paths).to_frame("realpath"),
        left_on="path",
        right_on=pl.col("realpath").str.split("/").list.get(-1),
    )
    .drop("path")
    .rename({"realpath": "path"})
)
bigwig_table.head()
```

--------------------------------

### Process Variant Starts and Ends

Source: https://genvarloader.readthedocs.io/en/latest/_modules/genvarloader/_dataset/_rag_variants.html

Calculates and sorts variant start and end positions, assigning local indices. This is a preparatory step for CCF calculation.

```python
        else:
            starts_ends[i] = -end
            se_local_idx[i] = v_end_sorter[end_idx]
            end_idx += 1
```

--------------------------------

### Initialize DatasetWithSites and Access Variants

Source: https://genvarloader.readthedocs.io/en/latest/api.html

Demonstrates how to create a DatasetWithSites object from an existing dataset and variant sites, and how to access wildtype and mutated haplotypes along with flags. This is useful for applying site-specific variants to haplotype data.

```python
import genvarloader as gvl
sites = gvl.sites_vcf_to_table("path/to/variants.vcf")

ds = gvl.Dataset.open("path/to/dataset.gvl", "path/to/reference.fasta")
ds_sites = gvl.DatasetWithSites(ds, sites)
wt_haps, mut_haps, flags = ds_sites[0, 0]
# flags is a np.uint8 (or an array of np.uint8 when accessing multiple rows/samples)
```

--------------------------------

### DatasetWithSites.__init__()

Source: https://genvarloader.readthedocs.io/en/latest/api.html

Constructor for the DatasetWithSites class. Specific details about its parameters and return values are not provided in the source.

```APIDOC
## DatasetWithSites.__init__()

### Description
Constructor for the DatasetWithSites class.

### Method
Not specified in source.

### Endpoint
Not specified in source.

### Parameters
Not specified in source.

### Request Example
Not specified in source.

### Response
Not specified in source.
```

--------------------------------

### intervals

Source: https://genvarloader.readthedocs.io/en/latest/_modules/genvarloader/_bigwig.html

Reads interval data (start, end, value) for specified genomic regions and samples. It returns a 2D Ragged array of interval structures. Input parameters include contig name, start and end coordinates, and optional sample names.

```APIDOC
## intervals

### Description
Read intervals corresponding to given genomic coordinates. The output data will be a 2D Ragged array of :code:`struct{start, end, value}` with shape (regions, samples).

### Method
`intervals(contig: str, starts: ArrayLike, ends: ArrayLike, sample: str | list[str] | None = None, **kwargs) -> RaggedIntervals`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Parameters
- **contig** (str) - Required - Name of the contig/chromosome.
- **starts** (ArrayLike) - Required - Start coordinates, 0-based.
- **ends** (ArrayLike) - Required - End coordinates, 0-based, exclusive.
- **sample** (str | list[str] | None) - Optional - Name of the samples to read data from.

### Returns
- **RaggedIntervals** - A 2D Ragged array of struct{start, end, value} with shape (regions, samples).
```

--------------------------------

### Accessing RaggedVariants Properties

Source: https://genvarloader.readthedocs.io/en/latest/_modules/genvarloader/_dataset/_rag_variants.html

Provides properties to access specific fields of the RaggedVariants object: 'alt' (alternative alleles), 'start' (0-based start positions), 'ilen' (indel lengths), and 'end' (0-based, exclusive end positions). The 'ilen' property calculates lengths if not directly provided.

```python
@property
    def alt(self) -> ak.Array:
        """Alternative alleles."""
        return cast(ak.Array, super().__getitem__("alt"))

    @property
    def start(self) -> Ragged[POS_TYPE]:
        """0-based start positions."""
        return cast(Ragged[POS_TYPE], super().__getitem__("start"))

    @property
    def ilen(self) -> Ragged[np.int32]:
        """Indel lengths. Infallible."""
        if "ilen" not in self.fields:
            ilen = ak.str.length(self.alt) - ak.str.length(self.ref)  # type: ignore[missing-attribute]  # ak.str submodule isn't exposed in awkward's top-level type stubs
            ilen = Ragged(ilen)
            return ilen

        return cast(Ragged[np.int32], super().__getitem__("ilen"))

    @property
    def shape(self) -> tuple[int | None, ...]:
        return self.start.shape

    @property
    def end(self) -> Ragged[POS_TYPE]:
        """0-based, exclusive end positions."""
        if hasattr(self, "ref"):
            ref = cast(Ragged[np.bytes_], self.ref)
            return self.start + ak.num(ref, -1)
        else:
            ilen = cast(Ragged[np.int32], self.ilen)
            return self.start - np.clip(ilen, None, 0) + 1
```

--------------------------------

### Open a gvl.Dataset and create a DataLoader

Source: https://genvarloader.readthedocs.io/en/latest/_sources/index.md

Open an existing GenVarLoader dataset and prepare it for PyTorch training. This includes subsetting regions and samples, and converting it to a DataLoader with specified batch size and shuffling options.

```python
import genvarloader as gvl

dataset = gvl.Dataset.open(path="cool_dataset.gvl", reference="hg38.fa")
train_samples = ["David", "Aaron"]
train_dataset = dataset.subset_to(regions="train_regions.bed", samples=train_samples)
train_dataloader = train_dataset.to_dataloader(
    batch_size=32, shuffle=True, num_workers=1
)

# use it in your training loop
for haplotypes, tracks in train_dataloader:
    ...
```

--------------------------------

### RaggedVariants.shape

Source: https://genvarloader.readthedocs.io/en/latest/_modules/genvarloader/_dataset/_rag_variants.html

Property to access the shape of the RaggedVariants object, derived from the 'start' field.

```APIDOC
## RaggedVariants.shape

### Description
Returns the shape of the RaggedVariants object, which is determined by the shape of its 'start' field.

### Returns
* **tuple[int | None, ...]**: The shape of the object.
```

--------------------------------

### BigWigs Initialization

Source: https://genvarloader.readthedocs.io/en/latest/_modules/genvarloader/_bigwig.html

Initializes the BigWigs reader with a name and a dictionary of sample paths. It also reads and validates contig information from the provided bigWig files.

```APIDOC
## BigWigs(name: str, paths: dict[str, str])

### Description
Initializes the BigWigs reader with a name and a dictionary of sample paths. It reads and validates contig information from the provided bigWig files.

### Parameters
#### Path Parameters
- **name** (string) - Required - Name of the reader, for example 'signal'.
- **paths** (dict[str, str]) - Required - Dictionary of sample names and paths to bigWig files for those samples.

### Returns
- An instance of the BigWigs reader.
```

--------------------------------

### get_dummy_dataset()

Source: https://genvarloader.readthedocs.io/en/latest/api.html

Function to get a dummy Dataset. Specific details about its parameters and return values are not provided in the source.

```APIDOC
## get_dummy_dataset()

### Description
Function to get a dummy Dataset.

### Method
Not specified in source.

### Endpoint
Not specified in source.

### Parameters
Not specified in source.

### Request Example
Not specified in source.

### Response
Not specified in source.
```

--------------------------------

### Create and Write a GenVarLoader Dataset

Source: https://genvarloader.readthedocs.io/en/latest/_sources/geuvadis.ipynb

This snippet demonstrates how to create a GenVarLoader dataset from BigWig files and write it to disk. It specifies tracks, maximum jitter, and overwrite behavior.

```python
tracks=gvl.BigWigs.from_table(name="read-depth", table=bigwig_table),
    max_jitter=128,  # allow up to 128 bp jitter
    overwrite=True,
)
```

--------------------------------

### genvarloader.get_dummy_dataset

Source: https://genvarloader.readthedocs.io/en/latest/_sources/api.md

Function to get a dummy dataset. Specific details on parameters and return values are not provided in the source.

```APIDOC
## genvarloader.get_dummy_dataset

### Description
Function to get a dummy dataset.

### Method
Not specified

### Endpoint
Not specified

### Parameters
Not specified

### Request Example
Not specified

### Response
Not specified
```

--------------------------------

### RaggedIntervals

Source: https://genvarloader.readthedocs.io/en/latest/_modules/genvarloader/_ragged.html

Represents intervals with associated values in a ragged format. It stores starts, ends, and values as Ragged arrays.

```APIDOC
## Class RaggedIntervals

Represents intervals with associated values in a ragged format. It stores starts, ends, and values as Ragged arrays.

### Attributes
*   **starts** (Ragged[np.int32]): Ragged array of interval start positions.
*   **ends** (Ragged[np.int32]): Ragged array of interval end positions.
*   **values** (Ragged[np.float32]): Ragged array of interval values.

### Methods

#### `__getitem__(self, idx)`

Returns a new `RaggedIntervals` object with intervals indexed by `idx`.

*   **Parameters**
    *   `idx`: Index or slice to apply to the intervals.

*   **Returns**
    RaggedIntervals: A new `RaggedIntervals` object containing the selected intervals.

#### `shape` (property)

Returns the shape of the haplotypes and all annotations.

*   **Returns**
    tuple: The shape of the data.

#### `to_padded(self, start: int, end: int, value: float)`

Converts `RaggedIntervals` to a tuple of rectilinear arrays by right-padding each entry with specified values. The final axis will have the maximum length across all entries.

*   **Parameters**
    *   `start` (int): The value to use for padding start positions.
    *   `end` (int): The value to use for padding end positions.
    *   `value` (float): The value to use for padding interval values.

*   **Returns**
    tuple[NDArray[np.int32], NDArray[np.int32], NDArray[np.float32]]: A tuple containing the padded start, end, and value arrays.

#### `reshape(self, shape: int | tuple[int, ...])`

Reshapes the haplotypes and all annotations to a new shape.

*   **Parameters**
    *   `shape` (int | tuple[int, ...]): The new shape for the data. The total number of elements must remain the same.

*   **Returns**
    RaggedIntervals: A new `RaggedIntervals` object with the reshaped data.

#### `squeeze(self, axis: int | tuple[int, ...] | None = None)`

Squeezes the haplotypes and all annotations along the specified axis or axes.

*   **Parameters**
    *   `axis` (int | tuple[int, ...] | None): Axis or axes to squeeze. If None, all axes of length 1 are squeezed.

*   **Returns**
    RaggedIntervals: A new `RaggedIntervals` object with squeezed dimensions.

#### `to_fixed_shape(self, shape: tuple[int, ...])`

Converts the ragged array to a rectilinear shape if all entries have the same shape.

*   **Parameters**
    *   `shape` (tuple[int, ...]): The target shape, including the length axis. The total number of elements must remain the same.

*   **Returns**
    tuple[NDArray[np.int32], NDArray[np.int32], NDArray[np.float32]]: A tuple containing the rectilinear start, end, and value arrays.

#### `to_packed(self)`

Applies `ak.to_packed` to all internal Ragged arrays.

*   **Returns**
    RaggedIntervals: A new `RaggedIntervals` object with packed arrays.

#### `to_nested_tensor_batch(self, device: str | torch.device = "cpu")`

Converts the `RaggedIntervals` to a batch of nested tensors, suitable for PyTorch.

*   **Parameters**
    *   `device` (str | torch.device, optional): The device to place the tensors on. Defaults to "cpu".

*   **Returns**
    list[RagItvBatch]: A list of nested tensor batches, one for each track.
```

--------------------------------

### Open a gvl.Dataset and create a PyTorch DataLoader

Source: https://genvarloader.readthedocs.io/en/latest/index.html

Open an existing GenVarLoader dataset and create a PyTorch DataLoader for training. You can subset the dataset to specific regions and samples, and configure batch size, shuffling, and number of workers. The DataLoader can then be used in a training loop.

```python
import genvarloader as gvl

dataset = gvl.Dataset.open(path="cool_dataset.gvl", reference="hg38.fa")
train_samples = ["David", "Aaron"]
train_dataset = dataset.subset_to(regions="train_regions.bed", samples=train_samples)
train_dataloader = train_dataset.to_dataloader(
    batch_size=32, shuffle=True, num_workers=1
)

# use it in your training loop
for haplotypes, tracks in train_dataloader:
    ...

```

--------------------------------

### Get Dataset Length

Source: https://genvarloader.readthedocs.io/en/latest/_modules/genvarloader/_dataset/_reference.html

Returns the length (number of rows) of the dataset. Similar to shape, it considers splicing information.

```python
def __len__(self) -> int:
    """Length of the dataset."""
    if self._splice_map is not None:
        return self._splice_map.n_rows
    return self.regions.height
```

--------------------------------

### genvarloader.get_splice_bed

Source: https://genvarloader.readthedocs.io/en/latest/_sources/api.md

Function to get splice BED data. Specific details on parameters and return values are not provided in the source.

```APIDOC
## genvarloader.get_splice_bed

### Description
Function to get splice BED data.

### Method
Not specified

### Endpoint
Not specified

### Parameters
Not specified

### Request Example
Not specified

### Response
Not specified
```

--------------------------------

### Dataset.with_settings()

Source: https://genvarloader.readthedocs.io/en/latest/api.html

Configures the Dataset with specified settings. Specific details about its parameters and return values are not provided in the source.

```APIDOC
## Dataset.with_settings()

### Description
Configures the Dataset with specified settings.

### Method
Not specified in source.

### Endpoint
Not specified in source.

### Parameters
Not specified in source.

### Request Example
Not specified in source.

### Response
Not specified in source.
```

--------------------------------

### Write a gvl.Dataset

Source: https://genvarloader.readthedocs.io/en/latest/_sources/index.md

Create a GenVarLoader dataset file from VCF variants, BigWig tracks, and specified regions. The `samples_to_bigwigs.csv` file should map sample names to their BigWig file paths.

```python
import genvarloader as gvl

gvl.write(
    path="cool_dataset.gvl",
    bed="interesting_regions.bed",
    variants="cool_variants.vcf",
    tracks=gvl.BigWigs.from_table("bigwig", "samples_to_bigwigs.csv"),
    max_jitter=128,
)
```

--------------------------------

### Load Data for Basenji2 Evaluation

Source: https://genvarloader.readthedocs.io/en/latest/_sources/basenji2_eval.ipynb

Loads data for Basenji2 model evaluation. Ensure the necessary libraries are installed and data paths are correctly configured.

```python
from basenji_kit.data import Dataset
from basenji_kit.lib import save_json
import numpy as np
import pandas as pd
import os

# Parameters
params = "/home/gridsan/groups/biolabs/projects/basenji2/basenji2/basenji2/params.json"
model_dir = "/home/gridsan/groups/biolabs/projects/basenji2/basenji2/models/res19/res19.h5"
output_dir = "/home/gridsan/groups/biolabs/projects/basenji2/basenji2/eval/res19/"

# Create output directory if it doesn't exist
os.makedirs(output_dir, exist_ok=True)

# Load parameters
params = save_json.load_json(params)

# Load dataset
dataset = Dataset(params, data_split='valid')

# Print dataset information
print(f"Dataset loaded: {dataset.name}")
print(f"Number of sequences: {dataset.num_sequences}")
print(f"Sequence length: {dataset.seq_length}")
print(f"Number of targets: {dataset.num_targets}")
```

--------------------------------

### RaggedVariants Initialization

Source: https://genvarloader.readthedocs.io/en/latest/_modules/genvarloader/_dataset/_rag_variants.html

Initializes a RaggedVariants object. Requires 'alt' and 'start' fields, and either 'ref' or 'ilen'. Other fields like 'dosage' can also be included.

```python
def __init__(
        self,
        alt: ak.Array,
        start: Ragged[POS_TYPE],
        ref: ak.Array | None = None,
        ilen: Ragged[np.int32] | None = None,
        dosage: Ragged[DOSAGE_TYPE] | None = None,
        **kwargs: Ragged[np.number],
    ):
        if ref is None and ilen is None:
            raise ValueError("Must provide one of refs or ilens.")

        to_zip = {"alt": alt, "start": start}
        if ref is not None:
            to_zip["ref"] = ref
        if ilen is not None:
            to_zip["ilen"] = ilen
        if dosage is not None:
            to_zip["dosage"] = dosage

        arr = ak.zip(
            to_zip | kwargs, 1, parameters={\"__record__\": RaggedVariants.__name__}
        )

        super().__init__(arr)
```

--------------------------------

### BigWigs.__init__()

Source: https://genvarloader.readthedocs.io/en/latest/api.html

Constructor for the BigWigs class. Specific details about its parameters and return values are not provided in the source.

```APIDOC
## BigWigs.__init__()

### Description
Constructor for the BigWigs class.

### Method
Not specified in source.

### Endpoint
Not specified in source.

### Parameters
Not specified in source.

### Request Example
Not specified in source.

### Response
Not specified in source.
```

--------------------------------

### Display Gene Metadata

Source: https://genvarloader.readthedocs.io/en/latest/_sources/basenji2_eval.ipynb

Displays the first few rows of gene metadata, including gene ID, chromosome, start and end positions, HGNC symbol, and strand.

```python
shape: (5, 6)

```

```text
┌─────────────────┬───────┬────────────┬─────────┬────────┬───────────┐
│ gene_id         ┆ chrom ┆ chromStart ┆ hgnc    ┆ strand ┆ chromEnd  │
│ ---             ┆ ---   ┆ ---        ┆ ---     ┆ ---    ┆ ---       │
│ str             ┆ str   ┆ i64        ┆ str     ┆ str    ┆ i64       │
╞═════════════════╪═══════╪════════════╪═════════╪════════╪═══════════╡

```

```text
│ "ENSG00000000457" ┆ "1"   ┆ 169797872  ┆ "SCYL3" ┆ "-"    ┆ 169928944 │
│ "ENSG00000001630" ┆ "7"   ┆ 91706730   ┆ "CYP51A1" ┆ "-"    ┆ 91837802  │
│ "ENSG00000002549" ┆ "4"   ┆ 17513279   ┆ "LAP3"  ┆ "+"    ┆ 17644351  │
│ "ENSG00000002745" ┆ "7"   ┆ 120899885  ┆ "WNT16" ┆ "+"    ┆ 121030957 │
│ "ENSG00000003056" ┆ "12"  ┆ 9037015    ┆ "M6PR"  ┆ "-"    ┆ 9168087   │
└─────────────────┴───────┴────────────┴─────────┴────────┴───────────┘
```

--------------------------------

### Download and prepare Geuvadis dataset files

Source: https://genvarloader.readthedocs.io/en/latest/geuvadis.html

Downloads essential files for the Geuvadis dataset, including reference genome, PLINK files, BigWig data, and BED files, using the 'pooch' library for retrieval and caching. It also prepares the reference genome by converting it to bgzip format.

```python
# GRCh38 chromosome 22 sequence
reference = pooch.retrieve(
    url="https://ftp.ensembl.org/pub/release-112/fasta/homo_sapiens/dna/Homo_sapiens.GRCh38.dna.chromosome.22.fa.gz",
    known_hash="sha256:974f97ac8ef7ffae971b63b47608feda327403be40c27e391ee4a1a78b800df5",
    progressbar=True,
)
if not Path(f"{reference[:-3]}.bgz").exists():
    !gzip -dc {reference} | bgzip > {reference[:-3]}.bgz
reference = reference[:-3] + ".bgz"

# PLINK 2 files
variants = pooch.retrieve(
    url="doi:10.5281/zenodo.13656224/1kGP.chr22.pgen",
    known_hash="md5:31aba970e35f816701b2b99118dfc2aa",
    progressbar=True,
    fname="1kGP.chr22.pgen",
)
pooch.retrieve(
    url="doi:10.5281/zenodo.13656224/1kGP.chr22.psam",
    known_hash="md5:eefa7aad5acffe62bf41df0a4600129c",
    progressbar=True,
    fname="1kGP.chr22.psam",
)
pooch.retrieve(
    url="doi:10.5281/zenodo.13656224/1kGP.chr22.pvar",
    known_hash="md5:5f922af91c1a2f6822e2f1bb4469d12b",
    progressbar=True,
    fname="1kGP.chr22.pvar",
)

# BigWigs and sample ID mapping
bw_paths = pooch.retrieve(
    url="doi:10.5281/zenodo.13656224/bw_chr22.tar.gz",
    known_hash="md5:14bf72e9e9d3e2318d07315c4a2675fb",
    progressbar=True,
    processor=pooch.Untar(),
)
bw_table_path = pooch.retrieve(
    url="doi:10.5281/zenodo.13656224/bigwig_table.csv",
    known_hash="md5:7fe7c55b61c7dfa66cfd0a49336f3b08",
    progressbar=True,
)

# BED
bed_path = pooch.retrieve(
    url="doi:10.5281/zenodo.13656224/chr22_egenes.bed",
    known_hash="md5:ccb55548e4ddd416d50dbe6638459421",
    progressbar=True,
)
```

--------------------------------

### Get Dataset Shape

Source: https://genvarloader.readthedocs.io/en/latest/_modules/genvarloader/_dataset/_reference.html

Returns the shape of the dataset. If the dataset is spliced, it returns the number of rows in the splice map; otherwise, it returns the height of the regions DataFrame.

```python
@property
def shape(self) -> tuple[int]:
    """Shape of the dataset."""
    if self._splice_map is not None:
        return (self._splice_map.n_rows,)
    return (self.regions.height,)
```

--------------------------------

### Prepare BED Data for Basenji2

Source: https://genvarloader.readthedocs.io/en/latest/basenji2_eval.html

Prepares BED data by setting chromosome end to chromosome start and casting chromosome to string. This is a preparatory step before writing the dataset.

```python
bed = gvl.with_length(
    genes.with_columns(chromEnd=pl.col("chromStart")),
    basenji2_in_len
).with_columns(pl.col("chrom").cast(pl.Utf8))
bed.head()
```

--------------------------------

### Ragged Array Indexing and Squeezing

Source: https://genvarloader.readthedocs.io/en/latest/_modules/seqpro/rag/_array.html

Explains how to access elements using `__getitem__` and how to use the `squeeze` method.

```APIDOC
## Ragged Array Indexing and Squeezing

### Description
Explains how to access elements using `__getitem__` and how to use the `squeeze` method.

### Methods

#### `__getitem__(where: Any)`

*   **Description**: Accesses elements of the Ragged array using indexing. Returns a new Ragged array or a NumPy array depending on the result of the operation.
*   **Parameters**:
    *   `where` (Any) - The index or slice to access.
*   **Returns**: `Self | NDArray[RDTYPE_co] | dict[str, NDArray[RDTYPE_co]]`

#### `squeeze(axis: int | tuple[int, ...] | None = None)`

*   **Description**: Squeeze the ragged array along the given non-ragged axis. If squeezing would result in a 1D array, return the data as a numpy array. For record layouts, dispatches per-field; if fields collapse to 1D ndarrays, returns a dict of ndarrays, otherwise returns a record Ragged.
*   **Parameters**:
    *   `axis` (int | tuple[int, ...] | None) - Optional - Axis or axes to squeeze. Must have size 1. If `None`, squeeze all size-1 axes.
*   **Returns**: `Self | NDArray[RDTYPE_co] | dict[str, NDArray[RDTYPE_co]]`
```

--------------------------------

### Create Temporary Directory and Path

Source: https://genvarloader.readthedocs.io/en/latest/_sources/geuvadis.ipynb

Creates a temporary directory and assigns its path to a variable. Useful for temporary data storage during processing.

```python
tmp_dir = TemporaryDirectory(suffix=".gvl")
        ds_path = tmp_dir.name
```

--------------------------------

### Convert to PyTorch Dataset

Source: https://genvarloader.readthedocs.io/en/latest/_modules/genvarloader/_dataset/_impl.html

Converts the genvarloader dataset into a PyTorch `Dataset` object. This requires PyTorch to be installed. Allows for optional inclusion of indices and application of custom transforms to batches.

```python
    def to_torch_dataset(
        self,
        return_indices: bool,
        transform: Callable | None
    ) -> TorchDataset:
        """Convert the dataset to a PyTorch :class:`Dataset <torch.utils.data.Dataset>`. Requires PyTorch to be installed.

        Parameters
        ----------
        return_indices
            Whether to append arrays of row and sample indices of the non-subset dataset to each batch.
        transform
            The transform to apply to each batch of data. The transform should take input matching the output of the dataset and can

```

--------------------------------

### Create Dummy RaggedDataset

Source: https://genvarloader.readthedocs.io/en/latest/_modules/genvarloader/_dummy.html

This snippet shows the construction of a RaggedDataset, a core data structure for Genvarloader. It initializes the dataset with various parameters including dummy sequences, tracks, and indexers, and handles optional spliced bed file processing.

```python
    if spliced:
        dummy_bed = dummy_bed.with_columns(chrom=pl.lit("chr1"))
        sm, sp_bed = SpliceMap.from_bed(("gene", "exon"), dummy_bed)
        dummy_spi = SpliceIndexer(map=sm, dsi=dummy_idxer)
    else:
        dummy_spi = None
        sp_bed = None

    dummy_dataset: RaggedDataset[RaggedSeqs, Ragged[np.float32]] = RaggedDataset(
        path=Path("dummy"),
        output_length="ragged",
        max_jitter=max_jitter,
        return_indices=False,
        contigs=dummy_contigs,
        jitter=0,
        deterministic=True,
        rc_neg=True,
        _full_bed=dummy_bed,
        _spliced_bed=sp_bed,
        _full_regions=dummy_regions,
        _idxer=dummy_idxer,
        _sp_idxer=dummy_spi,
        _seqs=dummy_haps,
        _tracks=dummy_tracks,
        _seqs_kind="haplotypes",
        _recon=dummy_recon,
        _rng=np.random.default_rng(),
    )

    return dummy_dataset
```

--------------------------------

### Writing Intervals to Memmap File

Source: https://genvarloader.readthedocs.io/en/latest/_modules/genvarloader/_dataset/_write.html

Writes interval start, end, and value data to a memory-mapped numpy file. Supports creating a new file or appending to an existing one.

```python
out = np.memmap(
    out_dir / "intervals.npy",
    dtype=INTERVAL_DTYPE,
    mode="w+" if interval_offset == 0 else "r+",
    shape=intervals.values.data.shape,
    offset=interval_offset,
)
out["start"] = intervals.starts.data
out["end"] = intervals.ends.data
out["value"] = intervals.values.data
out.flush()
```

--------------------------------

### Ragged Array Dtype Example for Record Layouts

Source: https://genvarloader.readthedocs.io/en/latest/api.html

Illustrates the structured NumPy dtype returned for record layouts in Ragged arrays, showing how field names and dtypes are represented.

```python
np.dtype([("seq", "S1"), ("score", "f4")])
```

--------------------------------

### Fetch Implementation for Reference Dataset

Source: https://genvarloader.readthedocs.io/en/latest/_modules/genvarloader/_dataset/_reference.html

This is a low-level implementation for fetching sequences from a reference genome. It is optimized for parallel execution using Numba.

```python
import numba as nb
import numpy as np
from numpy.typing import NDArray

from genvarloader.utils.numba_utils import padded_slice


@nb.njit(parallel=True, nogil=True, cache=True)
def _fetch_impl(
    c_idxs: NDArray[np.integer],
    starts: NDArray[np.integer],
    ends: NDArray[np.integer],
    reference: NDArray[np.integer],
    ref_offsets: NDArray[np.integer],
    pad_char: int,
    out: NDArray[np.uint8],
    out_offsets: NDArray[np.integer],
):
    for i in nb.prange(len(c_idxs)):
        r_s, r_e = ref_offsets[c_idxs[i]], ref_offsets[c_idxs[i] + 1]
        o_s, o_e = out_offsets[i], out_offsets[i + 1]
        padded_slice(reference[r_s:r_e], starts[i], ends[i], pad_char, out[o_s:o_e])
    return out
```

--------------------------------

### Initialize BigWigs reader

Source: https://genvarloader.readthedocs.io/en/latest/_modules/genvarloader/_bigwig.html

Initializes the BigWigs reader with a name and a dictionary of sample names to bigWig file paths. It also determines common contigs and their sizes across all provided files.

```python
from genvarloader._bigwig import BigWigs

reader = BigWigs(
    name="signal",
    paths={
        "sample1": "path/to/sample1.bw",
        "sample2": "path/to/sample2.bw",
    },
)
```

--------------------------------

### Get Phased Genotype Info from PLINK

Source: https://genvarloader.readthedocs.io/en/latest/_sources/faq.md

This command uses `plink2` to retrieve information about phased genotypes from a PLINK file. It's a way to inspect genotype phasing status.

```bash
# for PLINK
plink2 --pgen-info $prefix
```

--------------------------------

### Read data from BigWigs

Source: https://genvarloader.readthedocs.io/en/latest/_modules/genvarloader/_bigwig.html

Reads data for a specified contig and genomic regions (starts, ends) across one or more samples. The output shape is (samples length, total length of regions).

```python
from genvarloader._bigwig import BigWigs

reader = BigWigs(
    name="signal",
    paths={
        "sample1": "path/to/sample1.bw",
        "sample2": "path/to/sample2.bw",
    },
)

# Read data for a specific contig and regions
data = reader.read(
    contig="chr1",
    starts=[100, 500],
    ends=[200, 600],
    sample="sample1", # Optional: specify samples, defaults to all
)

# Data shape will be (1, 200) if sample="sample1" is specified
# Data shape will be (2, 200) if sample is not specified and there are 2 samples
print(data.shape)
```

--------------------------------

### Load Basenji2 Model and Move to Device

Source: https://genvarloader.readthedocs.io/en/latest/basenji2_eval.html

Loads the Basenji2 model state dictionary, determines the appropriate device (CUDA or CPU), and moves the model to the selected device. The model is then compiled for performance.

```python
basenji2 = Basenji2(basenji2_params["model"])
basenji2.load_state_dict(torch.load(basenji2_weights()))
device = "cuda" if torch.cuda.is_available() else "cpu"
basenji2 = basenji2.to(device).eval()
basenji2 = torch.compile(basenji2)
```

--------------------------------

### Initialize and Load Basenji2 Model

Source: https://genvarloader.readthedocs.io/en/latest/geuvadis.html

Initializes the Basenji2 PyTorch model and loads its pre-trained weights. This snippet is essential for setting up the model for inference.

```python
import torch
from basenji2_pytorch import Basenji2, basenji2_params, basenji2_weights

device = "cuda" if torch.cuda.is_available() else "cpu"

torch.set_float32_matmul_precision("medium")

basenji2 = Basenji2(basenji2_params["model"]).to(device)
basenji2.load_state_dict(torch.load(basenji2_weights(), weights_only=True))
basenji2.eval();
```

--------------------------------

### Iterate and Print Dataloader Batch Shapes

Source: https://genvarloader.readthedocs.io/en/latest/geuvadis.html

This snippet shows how to get the first batch from a dataloader and print the shapes of the haplotype and track tensors. This is useful for verifying the output dimensions of the dataloader.

```python
haps, tracks = next(iter(dl))
print(haps.shape, tracks.shape)
```

--------------------------------

### RaggedIntervals Initialization and Access

Source: https://genvarloader.readthedocs.io/en/latest/_modules/genvarloader/_ragged.html

Demonstrates how to create a RaggedIntervals object and access its elements using slicing. Note that slicing RaggedIntervals widens the type to Array per awkward stubs.

```python
from __future__ import annotations

from dataclasses import dataclass
from functools import partial
from typing import TYPE_CHECKING, Any, TypedDict, TypeVar, cast

import awkward as ak
import awkward.operations.str as ak_str
import numba as nb
import numpy as np
from awkward.contents import NumpyArray
from numpy.typing import NDArray
from phantom import Phantom
from seqpro.rag import Ragged, is_rag_dtype
from seqpro.rag import RDTYPE_co as RDTYPE
from seqpro.rag import reverse_complement as _sp_reverse_complement
from seqpro.rag import to_padded as _sp_to_padded

from ._torch import TORCH_AVAILABLE
from ._types import AnnotatedHaps

if TORCH_AVAILABLE or TYPE_CHECKING:
    import torch
    from torch.nested import nested_tensor_from_jagged as nt_jag

__all__ = ["Ragged", "RaggedIntervals", "RaggedTracks"]
INTERVAL_DTYPE = np.dtype(
    [("start", np.int32), ("end", np.int32), ("value", np.float32)], align=True
)




@dataclass(slots=True)
class RaggedIntervals:
    starts: Ragged[np.int32]
    ends: Ragged[np.int32]
    values: Ragged[np.float32]

    def __getitem__(self, idx) -> RaggedIntervals:
        out = RaggedIntervals(self.starts[idx], self.ends[idx], self.values[idx])  # type: ignore[bad-argument-type]  # Ragged.__getitem__ widens to Array per awkward stubs
        return out

    @property
    def shape(self):
        """Shape of the haplotypes and all annotations."""
        return self.values.shape

```

--------------------------------

### Create PyTorch Dataloader with Transform

Source: https://genvarloader.readthedocs.io/en/latest/geuvadis.html

This snippet demonstrates how to define a transformation function, open a dataset, apply transformations, subset the data, and create a PyTorch dataloader. It's useful for preparing data for machine learning models.

```python
def transform(haps, tracks):
    haps = rearrange(
        sp.DNA.ohe(haps), "... length alphabet -> ... alphabet length"
    ).astype(np.float32)
    return haps, tracks


ds = (
    gvl.Dataset.open(ds_path, reference=reference)
    .with_seqs("haplotypes")
    .with_tracks("read-depth")
    .with_len(2**17)
    .with_transform(transform)
)
n_train = round(ds.n_samples * 0.8)
gene1_train_ds = ds.subset_to(samples=slice(0, n_train))
dl = gene1_train_ds.to_dataloader(batch_size=16, num_workers=0, shuffle=True)
```

--------------------------------

### RaggedVariants Class

Source: https://genvarloader.readthedocs.io/en/latest/api.html

Represents an awkward record array for variants, with fields for alternative alleles, start positions, indel lengths, and end positions. Supports creation from Awkward arrays, reshaping, and squeezing.

```APIDOC
## class genvarloader.RaggedVariants

An awkward record array with shape `(batch, ploidy, ~variants, [~length])`. Guaranteed to at least have the field `"alt"` and `"start"` and one of `"ref"` or `"ilen"`.

### classmethod from_ak

Create a RaggedVariants object from an awkward array.

**Parameters:**
* **arr** (`Array`) – The awkward array to create a RaggedVariants object from.

**Return type:**
`RaggedVariants`

### alt

Alternative alleles.

**Type:** Array

### start

0-based start positions.

**Type:** Ragged[int32]

### ilen

Indel lengths. Infallible.

**Type:** Ragged[int32]

### end

0-based, exclusive end positions.

**Type:** Ragged[int32]

### reshape

Reshape leading, regular axes. Assumes no trailing regular axes.

**Return type:**
`Self`

**Parameters:**
* **shape** (_tuple_ _[__int_ _|__None_ _,__...__]_

### squeeze

Squeeze first axis.

**Return type:**
`Self`

**Parameters:**
* **axis** (`None`, default: `None`)
* ***kwargs**
```

--------------------------------

### fetch

Source: https://genvarloader.readthedocs.io/en/latest/_modules/genvarloader/data_registry.html

Downloads and caches specified datasets for constructing or opening a GenVarLoader dataset. The files are stored in the user's home directory under `~/.cache/genvarloader`.

```APIDOC
## fetch

### Description
Download and cache data for constructing/opening a GVL dataset. Files are cached in the user's home directory under :code:`~/.cache/genvarloader`.

### Method
`fetch(name: Literal["geuvadis_ebi", "1kgp"]) -> dict[str, Path]`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Parameters
- **name** (Literal["geuvadis_ebi", "1kgp"]) - Required - The name of the dataset to fetch. Can be one of:
    - "geuvadis_ebi": Geuvadis data for the original analyses by Lappalainen et al. 2013. Phased, normalized, and split into biallelic variants.
    - "1kgp": 1000 Genomes Project, all 3,202 individuals. Phased, normalized, and split into biallelic variants.

### Request Example
```python
fetch("geuvadis_ebi")
fetch("1kgp")
```

### Response
#### Success Response (200)
- **dict[str, Path]**: A dictionary of paths to the fetched data.

#### Response Example
```json
{
  "genes": "/path/to/gene_list.csv",
  "expr": "/path/to/GD462.GeneQuantRPKM.50FN.samplename.resk10.txt.gz",
  "pgen": "/path/to/geuvadis.pgen",
  "basenji2_targets": "/path/to/targets_human.txt",
  "samples": "/path/to/samples.txt"
}
```

### Error Handling
- Raises `ValueError` if an unknown dataset name is provided.
```