RabbitFX

https://rabbitfx-doc.readthedocs.io/en/latest/
Admin
RabbitFX is an efficient framework for FASTA/Q file parsing on modern multi-core platforms with...
Tokens:5,812
Snippets:53
Trust Score:9.6
Update:5 months ago
Show doc for...
Tokens:
Raw
### Build and Install RabbitFX with CMake

Source: https://rabbitfx-doc.readthedocs.io/en/latest/API/tutorial.html

Build and install RabbitFX to a specified path using CMake. This is followed by a command to compile your C++ file, linking against the installed RabbitFX library.

```bash
mkdir build
cmake .. -DCMAKE_INSTALL_PREFIX=/your/install/path
make && make install
g++ -std=c++11 YOURFILE.cpp -I/your/install/path/include -L/your/install/path/lib -lrabbitfx -lz -lpthread

```

--------------------------------

### Install RabbitFX-Ktrim

Source: https://rabbitfx-doc.readthedocs.io/en/latest/API/casestudy.html

Compile RabbitFX-Ktrim from source. Ensure you are in the correct directory after cloning the repository.

```bash
git https://github.com/RabbitBio/RabbitFX-Casestudy.git
cd RabbitFX-Casestudy/RabbitFX-Ktrim
mkdir build
cd build
cmake ..
make

```

--------------------------------

### Build and Run RabbitFX Test Cases

Source: https://rabbitfx-doc.readthedocs.io/en/latest/API/tutorial.html

Compile the RabbitFX project using CMake and make, then run the provided test executable to verify the installation.

```bash
cd RabbitFX
mkdir build && cd build
cmake ..
make
#then there is an test file in build file
time ./test
time ./testcount

```

--------------------------------

### Build RabbitFX-Mash

Source: https://rabbitfx-doc.readthedocs.io/en/latest/API/casestudy.html

Standard build process for RabbitFX-Mash. Ensure dependencies like Cap'n Proto, GSL, and Zlib are installed. GCC version 5 or higher is required.

```bash
cd RabbitFX-Mash
./bootstrap.sh
./configure [--prefix=...]
./configure [--with-capnp=...]
./configure [--with-gsl=...]
./configure [--with-simd=yes/no]
make -j4
#optional
make install
#optional
make test
```

--------------------------------

### Install Build Dependencies on CentOS 8.1

Source: https://rabbitfx-doc.readthedocs.io/en/latest/API/casestudy.html

Installs necessary build dependencies for RabbitFX-Mash on CentOS 8.1 using DNF. Requires root privileges.

```bash
sudo dnf install capnproto capnproto-devel gsl gsl-devel
```

--------------------------------

### Build RabbitFX with ISA-L for Faster Gzip

Source: https://rabbitfx-doc.readthedocs.io/en/latest/API/tutorial.html

Build RabbitFX with support for ISA-L to accelerate gzip file processing. This involves specifying the ISA-L installation path during the CMake configuration.

```bash
mkdir build
cmake .. -DCMAKE_INSTALL_PREFIX=/your/install/path -DIGZIP_PREFIX=/path/to/libisal
make && make install
g++ -std=c++11 YOURFILE.cpp -I/your/install/path/include -L/your/install/path/lib -lrabbitfx -lz -lpthread

```

--------------------------------

### Define Single-End FASTQ Processing Task

Source: https://rabbitfx-doc.readthedocs.io/en/latest/API/tutorial.html

Example of defining a multithreaded task for processing single-end FASTQ files. It includes setting up a CLI parser for input file and threads, and initializing data pools and queues for producer-consumer pattern.

```cpp
int test_fastq_se(int argc, char** argv){
  std::string file = "/home/old_home/haoz/workspace/QC/out_1.fq";
  //---------------cmd parser----------------
  CLI::App app{"Wellcome to RabbitFX"};
  CLI::Option* opt;
  std::string filename ;
  int th;
  app.add_option("-f, --file", filename, "input file name")
    ->default_val(file);
  app.add_option("-t, --threads", th, "worktreads")
    ->default_val(2);
  //----------------------------------------
  CLI11_PARSE(app, argc, argv);
  if(app.count("-f"))
    std::cout << "filename: " << filename << std::endl;
  else{
    std::cout << "-f not find, use default: " << filename << std::endl;
  }
  rabbit::fq::FastqDataPool *fastqPool = new rabbit::fq::FastqDataPool(32, 1<<22);
  rabbit::core::TDataQueue<rabbit::fq::FastqDataChunk> queue1(64, 1);
  std::thread producer(producer_fastq_task, filename, fastqPool, std::ref(queue1));
  std::thread** threads = new std::thread*[th];
  for(int t = 0; t < th; t++){
    threads[t] = new std::thread(std::bind(consumer_fastq_task, fastqPool, std::ref(queue1)));
  }
  producer.join();
  for(int t = 0; t < th; t++){
    threads[t]->join();
  }
  //-----free
  delete fastqPool;
  delete[] threads;
  return 0;
}

```

--------------------------------

### FASTA Data Chunk Functions

Source: https://rabbitfx-doc.readthedocs.io/en/latest/API/api.html

Functions for processing FASTA data chunks, including getting sequences, lines, and formatting data into reference structures.

```APIDOC
## getSequence

### Description
Get Sequence from chunk start at position.

### Signature
string getSequence(FastaDataChunk *&_chunk_, uint64 &_pos_)

### Parameters
- **chunk** (FastaDataChunk *&): The data to get sequence from
- **pos** (uint64_t &): Start position at chunk data

### Return
New sequence string if pos < chunk size, else return an empty string (“”)
```

```APIDOC
## getLine

### Description
Get a new line from chunk start at position.

### Signature
string getLine(FastaDataChunk *&_chunk_, uint64_t &_pos_)

### Parameters
- **chunk** (FastaDataChunk *&): The data to get line from
- **pos** (uint64_t &): Start position at chunk data

### Return
New line string if pos < chunk size, else return an empty string (“”)
```

```APIDOC
## chunkListFormat

### Description
Format FASTA chunks (listed) into a vector of `Reference` struct.

### Signature
int chunkListFormat(FastaChunk &_fachunk_, vector<Reference> &_refs_)

### Parameters
- **fachunk** (FastaChunk &): Source FASTA chunk data to format
- **refs** (vector<Reference> &): Destination vector to store at

### Return
Total number of Reference instances in vector refs.
```

```APIDOC
## chunkFormat (FastaChunk)

### Description
Format FASTA chunks into a vector of `Reference` struct.

### Signature
int chunkFormat(FastaChunk &_fachunk_, vector<Reference> &_refs_)

### Parameters
- **fachunk** (FastaChunk &): Source FASTA chunk data to format
- **refs** (vector<Reference> &): Destination vector to store at

### Return
Total number of Reference instances in vector refs.
```

```APIDOC
## chunkFormat (FastaChunk, int)

### Description
Format FASTA chunks into a vector of `Reference` struct (filter out sequence length < kmerSize).

### Signature
int chunkFormat(FastaChunk &_fachunk_, vector<Reference> &_refs_, int _kmerSize_)

### Parameters
- **fachunk** (FastaChunk &): Source FASTA chunk data to format
- **refs** (vector<Reference> &): Destination vector to store at
- **kmerSize** (int): Formatted Reference’s sequence length < kmerSize will be dropped out

### Return
Total number of Reference instances in vector refs.
```

```APIDOC
## getNextSeq

### Description
Get next `Reference` data.

### Signature
Reference getNextSeq(FastaChunk &_fachunk_, bool &_done_, uint64_t &_pos_)

### Parameters
- **fachunk** (FastaChunk &): Source FASTA chunk data to format
- **done** (bool &): If reach the end of fachunk
- **pos** (uint64_t &): Start position in fachunk to format

### Return
A `Reference` instance starting at pos in fachunk.

### Note
Because FASTA data does not contain quality and other information except sequence information, the `Sequence` in FASTA equals to the `Reference` in FASTQ.
```

--------------------------------

### Run RabbitFX-Ktrim (Single-end)

Source: https://rabbitfx-doc.readthedocs.io/en/latest/API/casestudy.html

Use for single-end data with Illumina TruSeq kit. Specify input read 1 file and output directory.

```bash
./RabbitFX-Ktrim -U /path/to/read1.fq -o /path/to/output/dir

```

--------------------------------

### Run RabbitFX-Ktrim (Paired-end, Custom Adapters)

Source: https://rabbitfx-doc.readthedocs.io/en/latest/API/casestudy.html

Use for paired-end data with custom adapters, multiple lanes, and specific quality/size thresholds. Supports gzipped files and multi-threading.

```bash
./RabbitFX-Ktrim -1 /path/to/lane1.read1.fq.gz,/path/to/lane2.read1.fq.gz,/path/to/lane3.read1.fq \
                  -2 /path/to/lane1.read2.fq.gz,/path/to/lane2.read2.fq.gz,/path/to/lane3.read2.fq \
                  -t 4 -p 35 -q 30 -s 50 -o /path/to/output/dir \
                  -a READ1_ADAPTER_SEQUENCE -b READ2_ADAPTER_SEQUENCE

```

--------------------------------

### Run RabbitFX-fastp (Single-end)

Source: https://rabbitfx-doc.readthedocs.io/en/latest/API/casestudy.html

Perform trimming on single-end sequencing data using RabbitFX-fastp. Specify the number of threads, input, and output files.

```bash
RabbitFX-fastp -w nthreads -i in.fq -o out.fq

```

--------------------------------

### Build Dependency-Free RabbitFX-Mash Binary

Source: https://rabbitfx-doc.readthedocs.io/en/latest/API/casestudy.html

Builds a dependency-free binary of RabbitFX-Mash, allowing for static linking of GSL and C++ libraries. This is useful for deployment in environments with limited pre-installed dependencies.

```bash
cd RabbitMash
./bootstrap.sh
./configure [--prefix=...]
./configure [--with-capnp=...]
./configure [--with-gsl=...]
./configure [--with-simd=yes/no]
./configure [--enable-static-gsl=yes]
./configure [--enable-static-cpp=yes]
make -j4
#optional
make install
#optional
make test
```

--------------------------------

### Build RabbitFX-fastp

Source: https://rabbitfx-doc.readthedocs.io/en/latest/API/casestudy.html

Compile RabbitFX-fastp. This command is for the Linux version only.

```bash
cd RabbitFX-fastp && make

```

--------------------------------

### Run RabbitFX-fastp (Paired-end)

Source: https://rabbitfx-doc.readthedocs.io/en/latest/API/casestudy.html

Perform trimming on paired-end sequencing data using RabbitFX-fastp. Specify the number of threads, input files for both reads, and output files for both reads.

```bash
RabbitFX-fastp -w nthreads -i in.R1.fq -I in.R2.fq -o out.R1.fq -O out.R2.fq

```

--------------------------------

### FASTQ Producer and Consumer Tasks

Source: https://rabbitfx-doc.readthedocs.io/en/latest/API/tutorial.html

Defines the producer task for reading FASTQ files chunk by chunk and pushing them to a data queue, and the consumer task for processing these chunks from the queue.

```cpp
int producer_fastq_task(std::string file, rabbit::fq::FastqDataPool* fastqPool, rabbit::core::TDataQueue<rabbit::fq::FastqDataChunk> &dq){
  rabbit::fq::FastqFileReader *fqFileReader;
  fqFileReader = new rabbit::fq::FastqFileReader(file, *fastqPool);
  rabbit::int64 n_chunks = 0;
  while(true){
    rabbit::fq::FastqDataChunk* fqdatachunk;// = new rabbit::fq::FastqDataChunk;
    fqdatachunk = fqFileReader->readNextChunk();
    if (fqdatachunk == NULL) break;
    n_chunks++;
    //std::cout << "readed chunk: " << n_chunks << std::endl;
    dq.Push(n_chunks, fqdatachunk);
  }
  dq.SetCompleted();
  std::cout << "file " << file << " has " << n_chunks << " chunks" << std::endl;
  return 0;
}

void consumer_fastq_task(rabbit::fq::FastqDataPool* fastqPool, rabbit::core::TDataQueue<rabbit::fq::FastqDataChunk> &dq){
    long line_sum = 0;
    rabbit::int64 id = 0;
    std::vector<neoReference> data;
    rabbit::fq::FastqDataChunk* fqdatachunk;// = new rabbit::fq::FastqDataChunk;
    data.resize(10000);
    while(dq.Pop(id, fqdatachunk)){
      line_sum += rabbit::fq::chunkFormat(fqdatachunk, data);
      fastqPool->Release(fqdatachunk);
    }
    std::cout << "line_sum: " << line_sum << std::endl;
}

```

--------------------------------

### Integrate RabbitFX with CMake

Source: https://rabbitfx-doc.readthedocs.io/en/latest/API/tutorial.html

Add RabbitFX as a library to your CMake project by copying the 'io' folder and linking against it.

```cmake
AUX_SOURCE_DIRECTORY(. SOURCE_LIST)
ADD_LIBRARY(rabbitfx ${SOURCE_LIST})
TARGET_LINK_LIBRAIES(rabbitfx z)

```

--------------------------------

### Sketch Genome with RabbitFX-Mash

Source: https://rabbitfx-doc.readthedocs.io/en/latest/API/casestudy.html

Generates a sketch file for a given genome. This is a preliminary step for further analysis.

```bash
./mash sketch -i test/genome1.fna -p nthreads -o test/genome1.fna.msh
```

--------------------------------

### Process FASTA File with Single Thread

Source: https://rabbitfx-doc.readthedocs.io/en/latest/API/tutorial.html

This C++ code demonstrates how to read and process a FASTA file using a single thread. It counts the number of chunks and lines in the file.

```cpp
int proces_fasta_task(std::string file) {
  rabbit::fa::FastaDataPool *fastaPool = new rabbit::fa::FastaDataPool(256, 1 << 22);
  rabbit::fa::FastaFileReader *faFileReader;
  faFileReader = new rabbit::fa::FastaFileReader(file, *fastaPool, false);
  int n_chunks = 0;
  int line_sum = 0;
  while (true) {
    rabbit::fa::FastaChunk *fachunk = new rabbit::fa::FastaChunk;
    fachunk = faFileReader->readNextChunkList();
    if (fachunk == NULL) break;
    n_chunks++;
    //-----relaease
    rabbit::fa::FastaDataChunk *tmp = fachunk->chunk;
    do {
      fastaPool->Release(tmp);
      tmp = tmp->next;
    } while (tmp != NULL);
    // line_sum += rabbit::fa::chunkFormat(*fachunk, data);
  }
  std::cout << "file " << file << " has " << line_sum << " lines" << std::endl;
  return 0;
}

int test_fasta(int argc, char** argv){
  producer_fasta_task("/home/old_home/haoz/workspace/data/hg19/hg19.fa");
  return 0;
}

```

--------------------------------

### Buffer Extend

Source: https://rabbitfx-doc.readthedocs.io/en/latest/API/api.html

Resizes the buffer to the specified size. Optionally copies existing data.

```APIDOC
## Extend

### Description
Resize the buffer.

### Parameters
- **_size__** (uint64) - The new size for the buffer.
- **_copy__** (bool) - Optional. If true, copy existing data to the new buffer. Defaults to false.
```

--------------------------------

### Screen Genome with RabbitFX-Mash

Source: https://rabbitfx-doc.readthedocs.io/en/latest/API/casestudy.html

Screens a genome sketch against sequencing reads. This command helps in identifying potential matches or contaminants.

```bash
./mash screen test/genome1.fna.msh test/reads1.fastq -p nthreads > scr.out
```

--------------------------------

### TDataPool Acquire

Source: https://rabbitfx-doc.readthedocs.io/en/latest/API/api.html

Acquires a DataType data from the DataPool and assigns it to the provided pointer. If no data is available, the program will wait.

```APIDOC
## Acquire

### Description
Acquire an DataType data in DataPool and assign to part_.
Acquired data from availablePartsPool, if there is no available data in availablePartsPool, program will wait.

### Parameters
- **part_** (DataType *&) - The pointer to acquire the space.
```

--------------------------------

### TDataPool Constructor

Source: https://rabbitfx-doc.readthedocs.io/en/latest/API/api.html

Initializes the TDataPool with specified maximum part number and buffer part size.

```APIDOC
## TDataPool Constructor

### Description
Constructor for the DataPool class, which provides an data pool for reusing memory space.

### Parameters
- **maxPartNum_** (uint32) - The maximum number of part contained in DataPool.
- **bufferPartSize_** (uint32) - Bytes of each part in DataPool (e.g., 1<<22 means 4MB each part)
```

--------------------------------

### FASTQ Data Chunk Functions

Source: https://rabbitfx-doc.readthedocs.io/en/latest/API/api.html

Functions for processing FASTQ data chunks, including formatting data into neoReference and Reference structures.

```APIDOC
## print_read

### Description
Prints a neoReference object.

### Signature
void print_read(neoReference &_ref_)

### Parameters
- **ref** (neoReference &): The neoReference object to print.
```

```APIDOC
## chunkFormat (FastqChunk, neoReference)

### Description
Format FASTQ chunks into a vector of `neoReference` struct (no-copy format).

### Signature
int chunkFormat(FastqChunk *_fqChunk_, std::vector<neoReference> &_data_)

### Parameters
- **fqChunk** (FastqChunk *): Source FASTQ chunk data to format
- **data** (std::vector<neoReference> &): Destination vector to store at

### Return
Total number of neoReference instances in vector `data`.
```

```APIDOC
## chunkFormat (FastqDataChunk, neoReference)

### Description
Format FASTQ data chunks into a vector of `neoReference` struct.

### Signature
int chunkFormat(FastqDataChunk *_fqDataChunk_, std::vector<neoReference> &_data_)

### Parameters
- **fqDataChunk** (FastqDataChunk *): Source FASTQ data chunk to format
- **data** (std::vector<neoReference> &): Destination vector to store at

### Return
Total number of neoReference instances in vector `data`.
```

```APIDOC
## chunkFormat (FastqChunk, Reference, bool)

### Description
Format FASTQ chunks into a vector of `Reference` struct (copy format).

### Signature
int chunkFormat(FastqChunk *_fqChunk_, std::vector<Reference> &_data_, bool _mHasQuality_ = true)

### Parameters
- **fqChunk** (FastqChunk *): Source FASTQ chunk data to format
- **data** (std::vector<Reference> &): Destination vector to store at
- **mHasQuality** (bool): If the FASTQ data has quality information (default: true)

### Return
Total number of Reference instances in vector `data`.
```

```APIDOC
## chunkFormat (FastqDataChunk, Reference, bool)

### Description
Format FASTQ data chunks into a vector of `Reference` struct.

### Signature
int chunkFormat(FastqDataChunk *_fqDataChunk_, std::vector<Reference> &_data_, bool _mHasQuality_ = true)

### Parameters
- **fqDataChunk** (FastqDataChunk *): Source FASTQ data chunk to format
- **data** (std::vector<Reference> &): Destination vector to store at
- **mHasQuality** (bool): If the FASTQ data has quality information (default: true)

### Return
Total number of Reference instances in vector `data`.
```

```APIDOC
## getLine (FastqDataChunk)

### Description
Gets a line from a FASTQ data chunk.

### Signature
string getLine(FastqDataChunk *&_chunk_, int &_pos_)

### Parameters
- **chunk** (FastqDataChunk *&): The FASTQ data chunk.
- **pos** (int &): The starting position within the chunk.
```

```APIDOC
## neoGetLine

### Description
Gets a line from a FASTQ data chunk for neoReference processing.

### Signature
int neoGetLine(FastqDataChunk *&_chunk_, uint64_t &_pos_, uint64_t &_len_)

### Parameters
- **chunk** (FastqDataChunk *&): The FASTQ data chunk.
- **pos** (uint64_t &): The starting position within the chunk.
- **len** (uint64_t &): The length of the line to retrieve.
```

--------------------------------

### FastqFileReader::readNextChunk

Source: https://rabbitfx-doc.readthedocs.io/en/latest/API/api.html

Reads the next chunk of data from a FASTQ file. Returns a pointer to FastqChunk if data is available, otherwise NULL.

```APIDOC
## FastqFileReader::readNextChunk

### Description
Reads the next chunk of data from a FASTQ file.

### Method
FastqDataChunk *

### Parameters
None

### Return
FastqChunk pointer if next chunk data has data, else return NULL
```

--------------------------------

### FastqFileReader::readNextPairChunk

Source: https://rabbitfx-doc.readthedocs.io/en/latest/API/api.html

Reads the next paired chunk in parallel using two threads. Returns a pointer to FastqDataPairChunk if data is available, otherwise NULL.

```APIDOC
## FastqFileReader::readNextPairChunk

### Description
Reads the next paired chunk in parallel (two threads).

### Method
FastqDataPairChunk *

### Parameters
None

### Return
FastqDataPairChunk pointer if next chunk data has data, else return NULL
```

--------------------------------

### FastaFileReader::readNextChunk

Source: https://rabbitfx-doc.readthedocs.io/en/latest/API/api.html

Reads the next chunk of data from a FASTA file. Returns a pointer to FastaChunk if data is available, otherwise NULL.

```APIDOC
## FastaFileReader::readNextChunk

### Description
Reads the next chunk of data from a FASTA file.

### Method
FastaChunk *

### Parameters
None

### Return
FastaChunk pointer if next chunk data has data, else return NULL
```

--------------------------------

### FastqFileReader::readNextPairChunk1

Source: https://rabbitfx-doc.readthedocs.io/en/latest/API/api.html

Reads the next paired chunk in a single thread. Returns a pointer to FastqDataPairChunk if data is available, otherwise NULL.

```APIDOC
## FastqFileReader::readNextPairChunk1

### Description
Reads the next paired chunk in single thread.

### Method
FastqDataPairChunk *

### Parameters
None

### Return
FastqDataPairChunk pointer if next chunk data has data, else return NULL
```

--------------------------------

### FastaFileReader::readNextChunkList

Source: https://rabbitfx-doc.readthedocs.io/en/latest/API/api.html

Reads the next listed chunk from a FASTA file, ensuring each FastaChunk contains at least one whole sequence. Returns a pointer to FastaChunk if data is available, otherwise NULL.

```APIDOC
## FastaFileReader::readNextChunkList

### Description
Reads the next listed chunk. This function ensures one FastaChunk (dataPart) contains at least one whole sequence.

### Method
FastaChunk *

### Parameters
None

### Return
FastaChunk pointer if next chunk data has data, else return NULL
```

--------------------------------

### TDataPool Release

Source: https://rabbitfx-doc.readthedocs.io/en/latest/API/api.html

Releases data back to the DataPool, making it available for reuse. Notifies waiting processes if any.

```APIDOC
## Release

### Description
Realease data to DataPool.
Realease the data in part_ to AvailablePartsPool and notify.

### Parameters
- **part_** (_const_ DataType *) - The pointer to be realesed to DataPool.
```
Tokens:
Raw