GenomicsDB

https://genomicsdb.readthedocs.io/en/latest/
Admin
GenomicsDB is a highly performant scalable data storage written in C++ for importing, querying and...
Tokens:12,252
Snippets:181
Trust Score:9.6
Update:3 months ago
Show doc for...
Tokens:
Raw
### Build and Install GenomicsDB from Source

Source: https://genomicsdb.readthedocs.io/en/latest/_sources/building-installing.rst.txt

Use this sequence to clone the repository, install prerequisites, and compile the project on a Linux system.

```bash
# Clone GenomicsDB and change to scripts directory
git clone https://github.com/GenomicsDB/GenomicsDB && cd GenomicsDB/scripts/

# Install the prerequisites to build GenomicsDB and create a genomicsdb_prereqs.sh in $HOME
sudo prereqs/install_prereqs.sh
source $HOME/genomicsdb_prereqs.sh
cmake -DCMAKE_INSTALL_PREFIX=$HOME /path/to/GenomicsDB # simplest

# Build and install the include/lib/bin files to $HOME
make && make install
```

--------------------------------

### Example query.json configuration

Source: https://genomicsdb.readthedocs.io/en/latest/_sources/cli-tools.rst.txt

Defines the workspace, array, and specific column/row ranges for a query.

```python
{
    "workspace" : "/tmp/ws/",
    "array" : "t0_1_2",
    "query_column_ranges" : [ [ [0, 100 ], 500 ] ],
    "query_row_ranges" : [ [ [0, 2 ] ] ],
    "vid_mapping_file": "tests/inputs/vid.json",
    "callset_mapping_file": "tests/inputs/callset_mapping.json",
    "query_attributes" : [ "REF", "ALT", "BaseQRankSum", "MQ", "MQ0", "ClippingRankSum", "MQRankSum", "ReadPosRankSum", "DP", "GT", "GQ", "SB", "AD", "PL", "DP_FORMAT", "MIN_DP" ]
}
```

--------------------------------

### setupGenomicsDBImporter

Source: https://genomicsdb.readthedocs.io/en/latest/doxyrst/javaimporter.html

Finalizes the importer setup after all streams are added.

```APIDOC
## setupGenomicsDBImporter()

### Description
Prepares the importer after all buffer streams have been added. No further streams can be added after this method is called.

### Throws
- **IOException** - If the modified callsets JSON cannot be written.
```

--------------------------------

### Install GenomicsDB Python package

Source: https://genomicsdb.readthedocs.io/en/latest/_sources/apis.rst.txt

Use pip to install the experimental Python bindings for GenomicsDB. Requires Python version 3.6 or higher.

```python
pip install genomicsdb
```

--------------------------------

### Check GenomicsDB Version

Source: https://genomicsdb.readthedocs.io/en/latest/_sources/examples/python.ipynb.txt

Verify the installed version of the GenomicsDB library.

```python
import genomicsdb
print(genomicsdb.version())
```

--------------------------------

### CSV Variant Call Format Example

Source: https://genomicsdb.readthedocs.io/en/latest/_sources/import-etl.rst.txt

Example of a CSV line containing fixed fields followed by a fixed-length SB field and a variable-length PL field.

```python
2,1857210,1857210,G,A|T,894.77,0,,,,,6,923,0,599,996,701,1697
```

--------------------------------

### Example of Incorrect Cell Order

Source: https://genomicsdb.readthedocs.io/en/latest/common-issues.html

This error occurs when alleles at the same position span multiple lines in the input VCF.

```text
$ vcf2genomicsdb loader.json
terminate called after throwing an instance of 'VCF2GenomicsDBException'
what():  VCF2GenomicsDBException : Incorrect cell order found - cells must be in column major order. Previous cell: [ 0, 114111 ] current cell: [ 0, 114111 ]
Aborted
```

--------------------------------

### VCF Input with Overlapping Alleles

Source: https://genomicsdb.readthedocs.io/en/latest/common-issues.html

Example of VCF input lines that cause cell order errors due to overlapping alleles at the same position.

```text
chrX    114112  .       TCT     T       999     PASS    .       GT:DP:GQ:MIN_DP:PL      0/0:0:0:0:0,0,0
chrX    114112  .       TCT     TTT     999     PASS    .       GT:DP:GQ:MIN_DP:PL      0/0:0:0:0:0,0,0
```

--------------------------------

### Initialize GenomicsDB workspace

Source: https://genomicsdb.readthedocs.io/en/latest/_sources/cli-tools.rst.txt

Creates a workspace and initializes it using configuration files.

```bash
vcf2genomicsdb_init -w /path/to/workspace -s <list-of-sample-vcfs>
```

--------------------------------

### Create a GenomicsDB workspace

Source: https://genomicsdb.readthedocs.io/en/latest/_sources/cli-tools.rst.txt

Initializes a new workspace at the specified path.

```bash
create_genomicsdb_workspace /path/to/workspace
```

--------------------------------

### vcf2genomicsdb_init

Source: https://genomicsdb.readthedocs.io/en/latest/_sources/cli-tools.rst.txt

Initializes a GenomicsDB workspace with configuration files.

```APIDOC
## vcf2genomicsdb_init

### Description
Tool for creating a GenomicsDB workspace, and initializing it with configuration json files.

### Usage
`vcf2genomicsdb_init -w /path/to/workspace -s <list-of-sample-vcfs>`
```

--------------------------------

### Configure Azure Blob Storage Environment

Source: https://genomicsdb.readthedocs.io/en/latest/_sources/examples/python.ipynb.txt

Set up environment variables and define workspace URIs for Azure Blob Storage access.

```python
# set up environment variables and configuration for query
import os
storageaccount = ""
os.environ["AZURE_STORAGE_ACCOUNT"] = storageaccount
os.environ["AZURE_STORAGE_KEY"] = ""

container = ""
workspace_prefix = ""
workspace = f"az://{container}@{storageaccount}.blob.core.windows.net/{workspace_prefix}"
callset_file = f"az://{container}@{storageaccount}.blob.core.windows.net/{workspace_prefix}/callset_mapping.json"
vid_file = f"az://{container}@{storageaccount}.blob.core.windows.net/{workspace_prefix}/vid_mapping.json"
array = ""
attributes = ["REF", "ALT", "GT"]
```

--------------------------------

### Initialize GenomicsDB Query Protobuf

Source: https://genomicsdb.readthedocs.io/en/latest/_sources/examples/python.ipynb.txt

Create and configure an ExportConfiguration protobuf object to define the query parameters for a GenomicsDB workspace.

```python
from genomicsdb.protobuf import genomicsdb_export_config_pb2 as query_pb
from genomicsdb.protobuf import genomicsdb_coordinates_pb2 as query_coords

# create the query protobuf, and point to the workspace we want to query
query = query_pb.ExportConfiguration()
query.workspace = workspace
query.array_name = array
query.attributes.extend(["REF", "ALT", "GT"])
query.callset_mapping_file = callset_file
query.vid_mapping_file = vid_file
```

--------------------------------

### Validate JSON Configuration

Source: https://genomicsdb.readthedocs.io/en/latest/_sources/common-issues.rst.txt

Use this command to ensure your JSON configuration files are syntactically correct before execution.

```none
cat <json_file> | json_verify
```

--------------------------------

### Ingest VCFs using Docker

Source: https://genomicsdb.readthedocs.io/en/latest/_sources/building-installing.rst.txt

Run the GenomicsDB Docker container to initialize a workspace and ingest VCF files from a host directory.

```bash
host> docker run -v /path/to/vcfs:/opt/vcfs -v /path/to/where/workspace/should/be/created:/opt/workspace_parent -u $( id -u $USER ):$( id -g $USER ) -it ghcr.io/genomicsdb/genomicsdb:v1.4.4
docker> vcf2genomicsdb_init -w /opt/workspace_parent/workspace -S /path/to/vcfs -n 0
docker> vcf2genomicsdb /opt/workspace_parent/loader.json
```

--------------------------------

### Import VCF files

Source: https://genomicsdb.readthedocs.io/en/latest/_sources/cli-tools.rst.txt

Imports VCF data into an existing GenomicsDB workspace.

```bash
vcf2genomicsdb <loader-json.json>
```

--------------------------------

### GenomicsDBImporter Constructors

Source: https://genomicsdb.readthedocs.io/en/latest/doxyrst/javaimporter.html

Initializes the GenomicsDBImporter using either a configuration file or a parallel import configuration object.

```APIDOC
## GenomicsDBImporter(String loaderJSONFile)
## GenomicsDBImporter(String loaderJSONFile, int rank)
## GenomicsDBImporter(ImportConfig config)

### Description
Constructs a new GenomicsDBImporter instance. The first two constructors use a loader JSON configuration file, while the third uses an ImportConfig object for parallel chromosome interval imports.

### Parameters
- **loaderJSONFile** (String) - Required - Path to the GenomicsDB loader JSON configuration file.
- **rank** (int) - Optional - Rank of the process for partitioning.
- **config** (ImportConfig) - Required - Parallel import configuration object.

### Throws
- **FileNotFoundException** - If files cannot be read or written.
```

--------------------------------

### Partitioning Configuration Error

Source: https://genomicsdb.readthedocs.io/en/latest/_sources/common-issues.rst.txt

GenomicsDB arrays cannot be partitioned by rows and columns simultaneously.

```none
Cannot have both "row_partitions" and "column_partitions" simultaneously in the JSON file
```

--------------------------------

### GenomicsDBFeatureReader Constructor

Source: https://genomicsdb.readthedocs.io/en/latest/doxyrst/javareader.html

Initializes a new reader for a GenomicsDB workspace using the provided export configuration, codec, and optional loader JSON file.

```APIDOC
## GenomicsDBFeatureReader(exportConfiguration, codec, loaderJSONFile)

### Description
Constructs a reader instance for GenomicsDB.

### Parameters
- **exportConfiguration** (GenomicsDBExportConfiguration.ExportConfiguration) - Required - Query parameters.
- **codec** (FeatureCodec<T, SOURCE>) - Required - FeatureCodec, currently only htsjdk.variant.bcf2.BCF2Codec and htsjdk.variant.vcf.VCFCodec are tested.
- **loaderJSONFile** (Optional<String>) - Optional - GenomicsDB loader JSON configuration file.

### Throws
- **IOException** - when data cannot be read from the stream.
```

--------------------------------

### write

Source: https://genomicsdb.readthedocs.io/en/latest/doxyrst/javaimporter.html

Writes data to TileDB/GenomicsDB using specified configurations.

```APIDOC
## void write(String loaderJSONFile, int rank)

### Description
Writes data to TileDB/GenomicsDB using the configuration specified in the loader file.

### Parameters
- **loaderJSONFile** (String) - Required - GenomicsDB loader JSON configuration file.
- **rank** (int) - Required - Rank of this process (TileDB/GenomicsDB partition idx).
```

--------------------------------

### Compare VCF files

Source: https://genomicsdb.readthedocs.io/en/latest/_sources/cli-tools.rst.txt

Verifies if two VCF files are identical.

```bash
vcfdiff /path/to/vcf1 /path/to/vcf2
```

--------------------------------

### Query GenomicsDB with Protobuf

Source: https://genomicsdb.readthedocs.io/en/latest/_sources/examples/python.ipynb.txt

Configures sample names and genomic intervals to perform a variant call query. Requires a pre-configured query object and connection to the database.

```python
# specify the samples we wish to query
query.query_sample_names.extend([
    '0x00922C4598840C041CB1BB19DC75231C969E9057F7E3B9CC04EE8B44E714B793_1xAFO897WL2I',
    '0x00AFE0D460A52BC28399A89E68A62D4CF2D279B56F482166B11BA7F29AFB45C9_1xGQPZ8BXWU0',
    '0x008AA255A3EF6E54E308C9B6728F9D7D60D71428A2DDD339DD6A7956DFB73B90_1x7HNHG5JNCA',
    '0x00AFE0D460A52BC28399A89E68A62D4CF2D279B56F482166B11BA7F29AFB45C9_1x6E9Z1KV502'
])

# specify the genomic intervals we wish to query
intervals = []
intervals.append(query_coords.ContigInterval(contig="1"))
intervals.append(query_coords.ContigInterval(contig="2", begin=1, end=100000000))
intervals.append(query_coords.ContigInterval(contig="5", begin=5000000, end=75000000))
query.query_contig_intervals.extend(intervals)

gdb = genomicsdb.connect_with_protobuf(query)
list = gdb.query_variant_calls()
print(*list, sep='\n')
```

--------------------------------

### Specify Absolute File Paths

Source: https://genomicsdb.readthedocs.io/en/latest/common-issues.html

Avoid using the tilde (~) character for home directories, as it is not interpreted by GenomicsDB file I/O.

```text
/home/<user>/<directory>/vid_mapping_file.json OR
../../<directory>/vid_mapping_file.json
```

--------------------------------

### Identify File Path Errors

Source: https://genomicsdb.readthedocs.io/en/latest/_sources/common-issues.rst.txt

GenomicsDB does not resolve shell-specific characters like '~'. Use absolute or relative paths instead.

```none
Could not open vid mapping file "~/ <directory>/vid_mapping_file.json OR
Could not open callsets file "~/ <directory>/callset_mapping_file.json
```

```none
/home/<user>/<directory>/vid_mapping_file.json OR
../../<directory>/vid_mapping_file.json
```

--------------------------------

### create_genomicsdb_workspace

Source: https://genomicsdb.readthedocs.io/en/latest/_sources/cli-tools.rst.txt

Tool for creating a new GenomicsDB workspace.

```APIDOC
## create_genomicsdb_workspace

### Description
Creates a new GenomicsDB workspace at the specified path.

### Usage
create_genomicsdb_workspace [workspace_path]
```

--------------------------------

### Handle File Access Errors

Source: https://genomicsdb.readthedocs.io/en/latest/_sources/common-issues.rst.txt

Error message displayed when a VCF/BCF file cannot be opened despite existence.

```none
Cannot open VCF/BCF file <path.vcf.gz>
```

--------------------------------

### doSingleImport

Source: https://genomicsdb.readthedocs.io/en/latest/doxyrst/javaimporter.html

Executes the import process.

```APIDOC
## doSingleImport()

### Description
Performs the import operation for the configured streams.
```

--------------------------------

### executeImport

Source: https://genomicsdb.readthedocs.io/en/latest/doxyrst/javaimporter.html

Imports multiple chromosome intervals into GenomicsDB.

```APIDOC
## void executeImport(int numThreads)

### Description
Imports multiple chromosome intervals. Optionally accepts the number of threads to use for parallel import.

### Parameters
- **numThreads** (int) - Optional - Number of threads used to import partitions.
```

--------------------------------

### Generate a VCF histogram

Source: https://genomicsdb.readthedocs.io/en/latest/_sources/cli-tools.rst.txt

Creates a histogram from a JSON input to assist in workspace partitioning decisions.

```bash
vcf_histogram <json>
```

--------------------------------

### Query GenomicsDB Workspace using Docker

Source: https://genomicsdb.readthedocs.io/en/latest/_sources/building-installing.rst.txt

Use the gt_mpi_gather tool within a Docker container to query a workspace and generate a VCF/gVCF output.

```bash
host> docker run -v /path/to/workspace:/opt/workspace -u $( id -u $USER ):$( id -g $USER ) -it ghcr.io/genomicsdb/genomicsdb:v1.4.4
# run with no arguments for help
docker> gt_mpi_gather
# Create a query.json file based on help
docker> gt_mpi_gather -j /path/to/query.json --produce-Broad-GVCF
```

--------------------------------

### vcfdiff

Source: https://genomicsdb.readthedocs.io/en/latest/_sources/cli-tools.rst.txt

Check whether VCFs are identical.

```APIDOC
## vcfdiff

### Description
Compares two VCF files to determine if they are identical.

### Usage
vcfdiff [vcf1_path] [vcf2_path]
```

--------------------------------

### doConsolidate

Source: https://genomicsdb.readthedocs.io/en/latest/doxyrst/javaimporter.html

Consolidates all intervals or arrays in a workspace into a single fragment.

```APIDOC
## void doConsolidate(int numThreads)

### Description
Consolidate all intervals/arrays in a given workspace into a single fragment.

### Parameters
- **numThreads** (int) - Required - Number of threads to use to parallelize consolidation.
```

--------------------------------

### Query GenomicsDB with gt_mpi_gather

Source: https://genomicsdb.readthedocs.io/en/latest/_sources/cli-tools.rst.txt

Executes a query against the database using a JSON configuration file.

```bash
gt_mpi_gather -j <query.json>
```

--------------------------------

### vcf2genomicsdb

Source: https://genomicsdb.readthedocs.io/en/latest/_sources/cli-tools.rst.txt

Imports VCF files into a GenomicsDB workspace.

```APIDOC
## vcf2genomicsdb

### Description
Tool for importing VCF files into GenomicsDB.

### Usage
`vcf2genomicsdb <loader-json.json>`
```

--------------------------------

### add

Source: https://genomicsdb.readthedocs.io/en/latest/doxyrst/javaimporter.html

Writes a VariantContext object to a specific stream.

```APIDOC
## add(VariantContext vc, int streamIdx)

### Description
Writes a VariantContext object to the specified stream. If the buffer is full, the operation may fail.

### Parameters
- **vc** (VariantContext) - Required - The VariantContext object to write.
- **streamIdx** (int) - Required - The index of the stream returned by addBufferStream().

### Returns
- **boolean** - True if the object was written successfully, false otherwise.
```

--------------------------------

### getSequenceNames

Source: https://genomicsdb.readthedocs.io/en/latest/doxyrst/javareader.html

Retrieves the list of contigs present in the combined VCF header.

```APIDOC
## List<String> getSequenceNames()

### Description
Returns the list of contigs in the combined VCF header.
```

--------------------------------

### generate_vcf

Source: https://genomicsdb.readthedocs.io/en/latest/doxyrst/cplusplus.html

Generates multi-sample VCF files from GenomicsDB in the Broad GVCF format.

```APIDOC
## generate_vcf

### Description
Generate multi-sample vcf files from GenomicsDB in the Broad GVCF format using set configuration. This method is useful with parallelism paradigms (MPI, Intel TBB).

### Signature
`void generate_vcf (const std::string &output="", const std::string &output_format="", bool overwrite=false)`

### Parameters
- **output** (std::string) - Optional - The output file path.
- **output_format** (std::string) - Optional - The format of the output file.
- **overwrite** (bool) - Optional - Whether to overwrite existing files.
```

--------------------------------

### org.genomicsdb.model.ImportConfig

Source: https://genomicsdb.readthedocs.io/en/latest/_sources/doxyrst/javaimporter.rst.txt

Configuration model used to define parameters for the import process.

```APIDOC
## org.genomicsdb.model.ImportConfig

### Description
Configuration object used to specify settings for the GenomicsDB import operation.
```

--------------------------------

### iterator

Source: https://genomicsdb.readthedocs.io/en/latest/doxyrst/javareader.html

Returns an iterator over VariantContext objects for the configured TileDB array.

```APIDOC
## CloseableTribbleIterator<T> iterator()

### Description
Returns an iterator over htsjdk.variant.variantcontext.VariantContext objects for the specified TileDB array and query configuration.
```

--------------------------------

### GenomicsDB::generate_vcf

Source: https://genomicsdb.readthedocs.io/en/latest/doxyrst/cplusplus.html

Generates multi-sample VCF files from GenomicsDB in the Broad GVCF format.

```APIDOC
## GenomicsDB::generate_vcf(const std::string &array, genomicsdb_ranges_t column_ranges, genomicsdb_ranges_t row_ranges, const std::string &reference_genome, const std::string &vcf_header, const std::string &output, const std::string &output_format, bool overwrite)

### Description
Generate multi-sample vcf files from GenomicsDB in the Broad GVCF format for given array constrained by column/row ranges.

### Parameters
- **array** (std::string) - Required - The array to generate VCF from.
- **column_ranges** (genomicsdb_ranges_t) - Required - Column ranges.
- **row_ranges** (genomicsdb_ranges_t) - Required - Row ranges.
- **reference_genome** (std::string) - Required - Path to reference genome.
- **vcf_header** (std::string) - Optional - Path to VCF header file.
- **output** (std::string) - Optional - Output file path.
- **output_format** (std::string) - Optional - Output format.
- **overwrite** (bool) - Optional - Whether to overwrite existing files.
```

--------------------------------

### getHeader

Source: https://genomicsdb.readthedocs.io/en/latest/doxyrst/javareader.html

Retrieves the VCF header associated with the combined gVCF stream.

```APIDOC
## Object getHeader()

### Description
Returns the VCF header of the combined gVCF stream.
```

--------------------------------

### vcf_histogram

Source: https://genomicsdb.readthedocs.io/en/latest/_sources/cli-tools.rst.txt

Create a histogram showing the number of variants per histogram bin.

```APIDOC
## vcf_histogram

### Description
Generates a histogram showing the number of variants per bin, which is useful for deciding how to partition the GenomicsDB workspace.

### Usage
vcf_histogram [json_input]
```

--------------------------------

### query

Source: https://genomicsdb.readthedocs.io/en/latest/doxyrst/javareader.html

Queries the GenomicsDB workspace for a specific genomic interval.

```APIDOC
## CloseableTribbleIterator<T> query(chr, start, end)

### Description
Returns an iterator over htsjdk.variant.variantcontext.VariantContext objects for the specified TileDB array and queried position.

### Parameters
- **chr** (String) - Required - Contig name.
- **start** (int) - Required - Start position (1-based).
- **end** (int) - Required - End position, inclusive (1-based).
```

--------------------------------

### org.genomicsdb.model.BatchCompletionCallbackFunctionArgument

Source: https://genomicsdb.readthedocs.io/en/latest/_sources/doxyrst/javaimporter.rst.txt

Model representing arguments passed to batch completion callbacks.

```APIDOC
## org.genomicsdb.model.BatchCompletionCallbackFunctionArgument

### Description
Defines the structure of arguments provided to callback functions triggered upon the completion of an import batch.
```

--------------------------------

### consolidate_genomicsdb_array

Source: https://genomicsdb.readthedocs.io/en/latest/_sources/cli-tools.rst.txt

Consolidate fragments from incremental imports into a single fragment.

```APIDOC
## consolidate_genomicsdb_array

### Description
Consolidates fragments from incremental imports into a single fragment for a specific array within a workspace.

### Usage
consolidate_genomicsdb_array -w [workspace_path] -a [array_name]
```

--------------------------------

### generate_plink

Source: https://genomicsdb.readthedocs.io/en/latest/doxyrst/cplusplus.html

Generates PLINK .ped and .map files from GenomicsDB.

```APIDOC
## generate_plink

### Description
Generate plink files from GenomicsDB for given array constrained by column/row ranges and given format to generate plink .ped and .map files.

### Signatures
1. `void generate_plink (const std::string &array, genomicsdb_ranges_t column_ranges, genomicsdb_ranges_t row_ranges, unsigned char format=7, int compression=1, bool one_pass=false, bool verbose=false, double progress_interval=-1, const std::string &output_prefix="output", const std::string &fam_list="")`
2. `void generate_plink (unsigned char format=7, int compression=1, bool one_pass=false, bool verbose=false, double progress_interval=-1, const std::string &output_prefix="output", const std::string &fam_list="")`

### Parameters
- **array** (std::string) - Required (in first overload) - The array name.
- **column_ranges** (genomicsdb_ranges_t) - Required (in first overload) - Column ranges to constrain.
- **row_ranges** (genomicsdb_ranges_t) - Required (in first overload) - Row ranges to constrain.
- **format** (unsigned char) - Optional - PLINK format.
- **compression** (int) - Optional - Compression level.
- **one_pass** (bool) - Optional - Enable one-pass mode.
- **verbose** (bool) - Optional - Enable verbose output.
- **progress_interval** (double) - Optional - Progress reporting interval.
- **output_prefix** (std::string) - Optional - Prefix for output files.
- **fam_list** (std::string) - Optional - Path to family list file.
```

--------------------------------

### Consolidate GenomicsDB array fragments

Source: https://genomicsdb.readthedocs.io/en/latest/_sources/cli-tools.rst.txt

Merges fragments from incremental imports into a single fragment for a specific array.

```bash
consolidate_genomicsdb_array -w /path/to/workspace -a <name-of-array-to-consolidate>
```

--------------------------------

### org.genomicsdb.importer.GenomicsDBImporter

Source: https://genomicsdb.readthedocs.io/en/latest/_sources/doxyrst/javaimporter.rst.txt

The primary class for importing VCF files into GenomicsDB. Users interact with this class to initiate and manage the import process.

```APIDOC
## org.genomicsdb.importer.GenomicsDBImporter

### Description
Main entry point for importing VCF data into GenomicsDB. This class provides the interface to execute import operations.

### Methods
- **GenomicsDBImporter(String workspace, String arrayName)**: Constructor to initialize the importer with a workspace and array name.
- **importVCF(String vcfFile)**: Imports a specified VCF file into the configured GenomicsDB array.
- **close()**: Closes the importer and releases associated resources.
```

--------------------------------

### Collapse Overlapping Variants

Source: https://genomicsdb.readthedocs.io/en/latest/common-issues.html

Use bcftools to merge multiple lines with the same position into a single line to resolve overlapping variant exceptions.

```bash
bcftools norm -m +any [-O <output_format> -o <output>] <input_file>
```

--------------------------------

### addSortedVariantContextIterator

Source: https://genomicsdb.readthedocs.io/en/latest/doxyrst/javaimporter.html

Adds a sorted VariantContext iterator as a data source for the import process.

```APIDOC
## addSortedVariantContextIterator(String streamName, VCFHeader vcfHeader, Iterator<VariantContext> vcIterator, long bufferCapacity, VariantContextWriterBuilder.OutputType streamType, Map<Integer, SampleInfo> sampleIndexToInfo)

### Description
Registers a sorted VariantContext iterator as a data source. After adding all iterators, the caller must invoke setupGenomicsDBImporter() and then doSingleImport().

### Parameters
- **streamName** (String) - Required - Unique name for the stream.
- **vcfHeader** (VCFHeader) - Required - VCF header for the stream.
- **vcIterator** (Iterator<VariantContext>) - Required - Iterator over VariantContext objects.
- **bufferCapacity** (long) - Required - Capacity of the stream buffer in bytes.
- **streamType** (OutputType) - Required - BCF_STREAM or VCF_STREAM.
- **sampleIndexToInfo** (Map<Integer, SampleInfo>) - Optional - Mapping from sample index to SampleInfo.

### Returns
- **int** - The stream index.
```

--------------------------------

### get_interval

Source: https://genomicsdb.readthedocs.io/en/latest/doxyrst/cplusplus.html

Utility function to extract interval information from a variant.

```APIDOC
## get_interval

### Description
Utility template function to extract information from Variant and VariantCall classes.

### Signature
`interval_t get_interval (const genomicsdb_variant_t *variant)`

### Parameters
- **variant** (const genomicsdb_variant_t*) - Required - Pointer to the variant object.
```

--------------------------------

### columnPartitionIterator

Source: https://genomicsdb.readthedocs.io/en/latest/doxyrst/javaimporter.html

Utility function to return a MultiChromosomeIterator for a specific column partition.

```APIDOC
## MultiChromosomeIterator columnPartitionIterator(FeatureReader<VariantContext> reader, String loaderJSONFile, int partitionIdx)

### Description
Returns a MultiChromosomeIterator that iterates over VariantContext objects in the reader belonging to the specified column partition.

### Parameters
- **reader** (FeatureReader<VariantContext>) - Required - AbstractFeatureReader over VariantContext objects.
- **loaderJSONFile** (String) - Required - Path to loader JSON file.
- **partitionIdx** (int) - Required - Rank/partition index.
```

--------------------------------

### CSV format specification

Source: https://genomicsdb.readthedocs.io/en/latest/_sources/import-etl.rst.txt

The required structure for CSV files when importing variant calls into GenomicsDB.

```python
<row>,<begin_column>,<end_column>,<REF>,<concatenated_ALT>,<QUAL>,<FILTER_field_specification>[,<other_fields_specification>]
```

--------------------------------

### Add GenomicsDB Java dependency

Source: https://genomicsdb.readthedocs.io/en/latest/_sources/apis.rst.txt

Include the GenomicsDB dependency in your Maven project configuration. Ensure the version matches the desired release.

```xml
<!-- https://mvnrepository.com/artifact/org.genomicsdb/genomicsdb -->
<dependency>
   <groupId>org.genomicsdb</groupId>
   <artifactId>genomicsdb</artifactId>
   <version>1.4.5</version>
</dependency>
```

--------------------------------

### Transform Query Results to Pandas DataFrame

Source: https://genomicsdb.readthedocs.io/en/latest/_sources/examples/python.ipynb.txt

Flattens the list of tuples returned by the query and converts the variant calls into a Pandas DataFrame for easier analysis.

```python
import pandas as pd

x,y,calls = zip(*list)
flattened = [variant for sublist in calls for variant in sublist]
print(pd.DataFrame(flattened).to_markdown())
```

--------------------------------

### gt_mpi_gather

Source: https://genomicsdb.readthedocs.io/en/latest/_sources/cli-tools.rst.txt

Queries GenomicsDB and outputs results as VariantCalls or Variants.

```APIDOC
## gt_mpi_gather

### Description
Tool for querying GenomicsDB. Outputs results as either VariantCalls or Variants.

### Usage
`gt_mpi_gather -j <query.json>`

### Query JSON Parameters
- **workspace** (string/list) - Mandatory - Path to the workspace.
- **array** (string/list) - Mandatory - Name of the array.
- **query_column_ranges** (list of lists) - Mandatory - Column ranges to be queried.
- **query_attributes** (list of strings) - Mandatory - Attributes to be fetched.
- **query_row_ranges** (list of lists) - Optional - Row ranges to be queried.
- **vid_mapping_file** (string/list) - Optional - Path to vid mapping JSON.
- **callset_mapping_file** (string/list) - Optional - Path to callset mapping JSON.
```

--------------------------------

### Debug Incorrect Cell Order

Source: https://genomicsdb.readthedocs.io/en/latest/_sources/common-issues.rst.txt

This error occurs when alleles at the same position span multiple lines. Normalize the VCF to merge alleles.

```bash
$ vcf2genomicsdb loader.json 
terminate called after throwing an instance of 'VCF2GenomicsDBException'
what():  VCF2GenomicsDBException : Incorrect cell order found - cells must be in column major order. Previous cell: [ 0, 114111 ] current cell: [ 0, 114111 ] 
Aborted
```

```none
chrX	114112	.	TCT	T	999	PASS	.	GT:DP:GQ:MIN_DP:PL	0/0:0:0:0:0,0,0
chrX	114112	.	TCT	TTT	999	PASS	.	GT:DP:GQ:MIN_DP:PL	0/0:0:0:0:0,0,0
```

```none
chrX	114112	.	TCT	T,TTT	999	PASS	.	GT:DP:GQ:MIN_DP:PL	0/2:0:0:0:0,0,0,0,0,0
```

--------------------------------

### Resolve Overlapping Variants

Source: https://genomicsdb.readthedocs.io/en/latest/_sources/common-issues.rst.txt

GenomicsDB requires non-overlapping variants. Use bcftools to normalize and collapse overlapping lines.

```none
Unhandled overlapping variants at columns <col1> and <col2> for row <row>
```

```bash
bcftools norm -m +any [-O <output_format> -o <output>] <input_file>
```

--------------------------------

### Corrected VCF Format

Source: https://genomicsdb.readthedocs.io/en/latest/common-issues.html

The result of running bcftools norm to merge overlapping alleles into a single line.

```text
chrX    114112  .       TCT     T,TTT   999     PASS    .       GT:DP:GQ:MIN_DP:PL      0/2:0:0:0:0,0,0,0,0,0
```

--------------------------------

### GenomicsDB::query_variants

Source: https://genomicsdb.readthedocs.io/en/latest/doxyrst/cplusplus.html

Queries a GenomicsDB array for variants constrained by column and row ranges.

```APIDOC
## GenomicsDB::query_variants(const std::string &array, genomicsdb_ranges_t column_ranges, genomicsdb_ranges_t row_ranges)

### Description
Query GenomicsDB array for variants constrained by column and row ranges.

### Parameters
- **array** (std::string) - Required - The name of the array to query.
- **column_ranges** (genomicsdb_ranges_t) - Optional - Column ranges to constrain the query.
- **row_ranges** (genomicsdb_ranges_t) - Optional - Row ranges to constrain the query.

### Returns
- **GenomicsDBVariants** - The resulting variants.
```

--------------------------------

### VariantCall JSON object structure

Source: https://genomicsdb.readthedocs.io/en/latest/_sources/cli-tools.rst.txt

Represents a single sample's data for a specific genomic interval.

```python
{
    "row": 2,
    "interval": [ 17384, 17386 ],
    "fields": {
        "REF": "GAT",
        "ALT": [ "GT","<NON_REF>" ],
        "PL": [ 1018,0,1116,1137,1224,2361 ],
        "BaseQRankSum": [ 1.046 ]
    }
}
```

--------------------------------

### GenomicsDB::query_variant_calls

Source: https://genomicsdb.readthedocs.io/en/latest/doxyrst/cplusplus.html

Queries a GenomicsDB array for variant calls constrained by column and row ranges.

```APIDOC
## GenomicsDB::query_variant_calls(const std::string &array, genomicsdb_ranges_t column_ranges, genomicsdb_ranges_t row_ranges)

### Description
Query the array for variant calls constrained by the column and row ranges.

### Parameters
- **array** (std::string) - Required - The name of the array to query.
- **column_ranges** (genomicsdb_ranges_t) - Optional - Column ranges to constrain the query.
- **row_ranges** (genomicsdb_ranges_t) - Optional - Row ranges to constrain the query.

### Returns
- **GenomicsDBVariantCalls** - The resulting variant calls.
```

--------------------------------

### Define a Variant JSON object

Source: https://genomicsdb.readthedocs.io/en/latest/_sources/cli-tools.rst.txt

Represents a variant with common fields and specific variant calls for different rows.

```python
{
    "interval": [ 17384, 17384 ],
        "common_fields" : {
            "REF": "G",
            "ALT": [ "A","<NON_REF>" ]
        },
        "variant_calls": [
        {
            "row": 0,
            "interval": [ 17384, 17384 ],
            "fields": {
                "REF": "G",
                "ALT": [ "A","<NON_REF>" ],
                "BaseQRankSum": [ -2.096000 ]
            }
        },
        {
            "row": 2,
            "interval": [ 17384, 17384 ],
            "fields": {
                "REF": "G",
                "ALT": [ "A","<NON_REF>" ],
                "BaseQRankSum": [ 1.046000 ]
            }
        }
    ]
}
```
Tokens:
Raw