### Install hictkpy with minimal dependencies using pip Source: https://github.com/paulsengroup/hictkpy/blob/main/docs/installation.rst Installs hictkpy with no runtime dependencies. This is the most basic installation. ```bash pip install hictkpy ``` -------------------------------- ### Install hictkpy with all dependencies using pip Source: https://github.com/paulsengroup/hictkpy/blob/main/docs/installation.rst Installs hictkpy along with all its optional third-party dependencies. Use this for a full-featured installation. ```bash pip install 'hictkpy[all]' ``` -------------------------------- ### Install Dependencies and Build Docs Source: https://github.com/paulsengroup/hictkpy/blob/main/docs/README.md Installs necessary packages for documentation building and then executes the documentation build process using make. Ensure you are in the repository root and have activated the virtual environment. ```bash venv/bin/pip install --upgrade pip # --group option requires a modern version of pip virtualenv/bin/pip install . --group docs -v # Activate venv . venv/bin/activate # Clean old build files (optional) make -C docs clean make -C docs linkcheck html latexpdf ``` -------------------------------- ### Hictkpy Logging Warning with Fork Source: https://github.com/paulsengroup/hictkpy/blob/main/docs/multithreading_and_multiprocessing.rst This example demonstrates the UserWarning raised by hictkpy when using the 'fork' start method for multiprocessing on Linux. It advises switching to 'spawn' or 'forkserver'. ```text /usr/lib64/python3.14/multiprocessing/popen_fork.py:70: UserWarning: hictkpy: detected a call to fork(): hictkpy's logger does not support multiprocessing when using fork() as start method. Please change process start method to spawn or forkserver. For more details, refer to Python's documentation: https://docs.python.org/3/library/multiprocessing.html#multiprocessing.set_start_method self.pid = os.fork() ``` -------------------------------- ### Example of missing dependency error Source: https://github.com/paulsengroup/hictkpy/blob/main/docs/installation.rst Illustrates the ModuleNotFoundError that occurs when a method requiring a specific dependency (e.g., numpy) is called without that dependency installed. The error message suggests how to install the missing dependency. ```ipythonconsole In [3] f.fetch().to_numpy() ModuleNotFoundError: No module named 'numpy' The above exception was the direct cause of the following exception: Traceback (most recent call last): File "", line 1, in ModuleNotFoundError: To enable numpy support, please install numpy with: pip install 'hictkpy[numpy]' Alternatively, you can install hictkpy with all its dependencies by running: pip install 'hictkpy[all]' ``` -------------------------------- ### Install Python Type Stubs Source: https://github.com/paulsengroup/hictkpy/blob/main/src/CMakeLists.txt Installs Python type stub files (.pyi) to the hictkpy package directory. These files are used for static type checking. ```cmake install(FILES "${CMAKE_CURRENT_BINARY_DIR}/hictkpy/__init__.pyi" DESTINATION hictkpy) install(FILES "${CMAKE_CURRENT_BINARY_DIR}/hictkpy/cooler.pyi" DESTINATION hictkpy) install(FILES "${CMAKE_CURRENT_BINARY_DIR}/hictkpy/hic.pyi" DESTINATION hictkpy) ``` -------------------------------- ### Install hictkpy from a release archive via pip Source: https://github.com/paulsengroup/hictkpy/blob/main/docs/installation.rst Installs hictkpy from a source release archive URL. This method is useful if you have a direct link to a specific version's tarball. ```bash pip install 'hictkpy[all] @ https://pypi.python.org/packages/source/h/hictkpy/hictkpy-1.4.0.tar.gz' ``` -------------------------------- ### Load and Fetch Data with hictkpy Source: https://github.com/paulsengroup/hictkpy/blob/main/README.md Demonstrates how to load a .mcool or .hic file, fetch data for a specific chromosome, and convert it into different formats. Requires hictkpy to be installed. ```python3 import hictkpy path_to_clr = "file.mcool" # "file.hic" clr = hictkpy.File(path_to_clr, 100_000) sel = clr.fetch("chr1") df = sel.to_df() # Get interactions as a pandas.DataFrame m1 = sel.to_numpy() # Get interactions as a numpy matrix m2 = sel.to_csr() # Get interactions as a scipy.sparse.csr_matrix ``` -------------------------------- ### Get Methods Source: https://github.com/paulsengroup/hictkpy/blob/main/docs/api/generic.rst Methods for retrieving data. ```APIDOC ## get ### Description Retrieves specific data. ### Method `get()` ### Endpoint N/A (Python method) ### Parameters None specified in source. ### Request Example ```python hictkpy.get() ``` ### Response Specific data. ``` ```APIDOC ## get_id ### Description Retrieves data by ID. ### Method `get_id()` ### Endpoint N/A (Python method) ### Parameters None specified in source. ### Request Example ```python hictkpy.get_id() ``` ### Response Data by ID. ``` ```APIDOC ## get_ids ### Description Retrieves multiple data entries by IDs. ### Method `get_ids()` ### Endpoint N/A (Python method) ### Parameters None specified in source. ### Request Example ```python hictkpy.get_ids() ``` ### Response Multiple data entries by IDs. ``` -------------------------------- ### Install hictkpy with specific dependencies using pip Source: https://github.com/paulsengroup/hictkpy/blob/main/docs/installation.rst Installs hictkpy with a specific set of optional dependencies like numpy, pandas, pyarrow, or scipy. Choose based on your project's needs. ```bash pip install 'hictkpy[numpy]' ``` ```bash pip install 'hictkpy[pandas]' ``` ```bash pip install 'hictkpy[pyarrow]' ``` ```bash pip install 'hictkpy[scipy]' ``` -------------------------------- ### Install latest hictkpy from main branch via pip Source: https://github.com/paulsengroup/hictkpy/blob/main/docs/installation.rst Installs the most recent version of hictkpy directly from the main branch of its GitHub repository. Use this for the absolute latest development version. ```bash pip install 'hictkpy[all] @ git+https://github.com/paulsengroup/hictkpy.git@main' ``` -------------------------------- ### Install specific hictkpy version from git tag via pip Source: https://github.com/paulsengroup/hictkpy/blob/main/docs/installation.rst Installs a specific version of hictkpy identified by a git tag from its GitHub repository. This is useful for reproducible builds. ```bash pip install 'hictkpy[all] @ git+https://github.com/paulsengroup/hictkpy.git@v1.4.0' ``` -------------------------------- ### Install hictkpy using Conda Source: https://github.com/paulsengroup/hictkpy/blob/main/docs/installation.rst Installs hictkpy from the conda-forge and bioconda channels. This is an alternative to pip installation. ```bash conda install -c conda-forge -c bioconda hictkpy ``` -------------------------------- ### Run hictkpy automated tests Source: https://github.com/paulsengroup/hictkpy/blob/main/docs/installation.rst Clones the hictkpy repository, checks out a specific version, installs pytest, and runs the test suite. It is highly recommended to run these tests after building from source to ensure integrity. ```bash git clone https://github.com/paulsengroup/hictkpy.git cd hictkpy # make sure to run tests for the same version/tag/commit used to build hictkpy git checkout v1.4.0 # if you installed hictkpy in a venv make sure to install pytest in the venv pip install pytest pytest test/ ``` -------------------------------- ### Fetch Cis Interactions using ProcessPoolExecutor Source: https://github.com/paulsengroup/hictkpy/blob/main/docs/multithreading_and_multiprocessing.rst Use ProcessPoolExecutor to fetch cis interactions in parallel. Ensure the process start method is explicitly set (e.g., 'spawn') and initialize the logger in child processes using the 'initializer' argument. ```python import logging import multiprocessing as mp import os import sys import time from concurrent.futures import ProcessPoolExecutor import hictkpy import pandas as pd def fetch_chroms(path): """ Get the list of chromosomes available in the given file """ with hictkpy.MultiResFile(path) as f: return list(f.chromosomes().keys()) def fetch_pixels(path, resolution, query): """ Fetch interactions for the given query and return them as a pandas.DataFrame """ with hictkpy.File(path, resolution) as f: logging.info("[%s; PID=%d]: fetching...", query, os.getpid()) df = f.fetch(query, join=True).to_df() logging.info("[%s; PID=%d]: fetched %d interactions!", query, os.getpid(), len(df)) return df def fetch_cis_interactions(path, resolution, nthreads): """ Fetch cis interactions from the given file and return them as a pandas.DataFrame """ chroms = fetch_chroms(path) with ProcessPoolExecutor(nthreads, initializer=setup_logger) as ppool: tasks = [] for chrom in chroms: tasks.append(ppool.submit(fetch_pixels, path, resolution, chrom)) results = (task.result() for task in tasks) return pd.concat((df for df in results if len(df) != 0)) def setup_logger(level=logging.INFO): fmt = "[%(asctime)s] %(levelname)s: %(message)s" logging.basicConfig(format=fmt) logging.getLogger().setLevel(level) # suppress log messages generated by hictkpy for level INFO or lower hictkpy.logging.setLevel(logging.WARN) def main(): mp.set_start_method("spawn") setup_logger() path = "test/data/hic_test_file.hic" resolution = 100_000 t0 = time.time() df = fetch_cis_interactions(path, resolution, nthreads=os.cpu_count()) print(df, file=sys.stderr) logging.info("fetched %d interactions in %.2fs!", len(df), time.time() - t0) if __name__ == "__main__": main() ``` -------------------------------- ### Fetch interactions as NumPy array and plot heatmap with hictkpy Source: https://github.com/paulsengroup/hictkpy/blob/main/docs/quickstart.rst Retrieves interaction data for a specified region as a NumPy array and visualizes it as a heatmap using matplotlib. Requires matplotlib to be installed. ```ipythonconsole sel = f.fetch("2L:10,000,000-20,000,000") m = sel.to_numpy() import matplotlib.pyplot as plt from matplotlib.colors import LogNorm plt.imshow(m, norm=LogNorm()) plt.show() ``` -------------------------------- ### Get file path with hictkpy Source: https://github.com/paulsengroup/hictkpy/blob/main/docs/quickstart.rst Retrieves the path of the opened Hi-C file. ```ipythonconsole f.path() ``` -------------------------------- ### Read file resolution with hictkpy Source: https://github.com/paulsengroup/hictkpy/blob/main/docs/quickstart.rst Gets the resolution (bin size) of the Hi-C file. ```ipythonconsole f.resolution() ``` -------------------------------- ### Compute Descriptive Statistics Source: https://github.com/paulsengroup/hictkpy/blob/main/docs/quickstart.rst Computes all supported descriptive statistics for Hi-C data efficiently without loading all interactions into memory. Use this to get a quick overview of your data's properties. ```python f.fetch().describe() ``` -------------------------------- ### Open HTML Documentation Source: https://github.com/paulsengroup/hictkpy/blob/main/docs/README.md Command to open the generated HTML documentation in your default web browser. This command varies slightly between Linux and macOS. ```bash # Linux xdg-open docs/_build/html/index.html # macOS open docs/_build/html/index.html ``` -------------------------------- ### Open PDF Documentation Source: https://github.com/paulsengroup/hictkpy/blob/main/docs/README.md Command to open the generated PDF documentation in your default PDF viewer. This command varies slightly between Linux and macOS. ```bash # Linux xdg-open docs/_build/latex/hictkpy.pdf # macOS open docs/_build/latex/hictkpy.pdf ``` -------------------------------- ### Ingest all interactions at once for faster file creation Source: https://github.com/paulsengroup/hictkpy/blob/main/docs/creating_cool_hic_files.rst If memory permits, bypass temporary file creation by setting a large 'chunk_size' in FileWriter to ingest all interactions simultaneously, significantly speeding up the process. ```ipythonconsole # Initialize an empty .cool file cols = ["chrom1", "start1", "end1", "chrom2", "start2", "end2", "count"] df = pd.read_table("pixels.bg2", names=cols) with htk.cooler.FileWriter("out.cool", chroms, resolution=50_000, chunk_size=len(df) + 1) as writer: writer.add_pixels(df) ``` -------------------------------- ### Add Sources and Include Directories Source: https://github.com/paulsengroup/hictkpy/blob/main/src/cpp/nanobind/CMakeLists.txt Specifies the source files and include directories for the 'hictkpy_nanobind' static library. This ensures that the C++ code and its headers are correctly compiled. ```cmake target_sources( hictkpy_nanobind PRIVATE "${CMAKE_CURRENT_SOURCE_DIR}/nanobind.cpp" "${CMAKE_CURRENT_SOURCE_DIR}/nanobind_impl.hpp" ) target_include_directories(hictkpy_nanobind PUBLIC "${CMAKE_CURRENT_SOURCE_DIR}/include") ``` -------------------------------- ### Configure Windows-Specific Compile Definitions for Portable Wheels Source: https://github.com/paulsengroup/hictkpy/blob/main/CMakeLists.txt Sets Windows-specific compile definitions when building portable wheels on Windows. Requires including sdkddkver.h. ```cmake target_compile_definitions( hictkpy_project_options INTERFACE _WIN32_WINNT=_WIN32_WINNT_WIN10 /FI # required to use _WIN32_WINNT_WIN10 ) ``` -------------------------------- ### Open Hi-C file with hictkpy Source: https://github.com/paulsengroup/hictkpy/blob/main/docs/quickstart.rst Opens a .hic or .mcool file using hictkpy.File. It's recommended to use a context manager for file handling. ```ipythonconsole import hictkpy as htk # .mcool and .cool files are also supported f = htk.File("4DNFIOTPSS3L.hic", 10_000) ``` ```python with htk.File("4DNFIOTPSS3L.hic", 10_000) as f: # use the file ``` -------------------------------- ### Find Python and Nanobind Packages Source: https://github.com/paulsengroup/hictkpy/blob/main/src/cpp/nanobind/CMakeLists.txt Configures the build to find Python 3.10 with specific components and the nanobind library. Ensures required packages are available for building Python extensions. ```cmake find_package( Python 3.10 COMPONENTS Interpreter Development.Module NumPy REQUIRED ) find_package(FMT REQUIRED QUIET) find_package(nanobind REQUIRED QUIET) ``` -------------------------------- ### Configure hictkpy_pixel_table Library Source: https://github.com/paulsengroup/hictkpy/blob/main/src/cpp/table/CMakeLists.txt Defines the static library 'hictkpy_pixel_table', its source files, include directories, and links essential libraries including Arrow, fmt, spdlog, and other hictkpy modules. ```cmake add_library(hictkpy_pixel_table STATIC) target_sources( hictkpy_pixel_table PRIVATE "${CMAKE_CURRENT_SOURCE_DIR}/pixel_table.cpp" "${CMAKE_CURRENT_SOURCE_DIR}/pixel_table_coo.cpp" "${CMAKE_CURRENT_SOURCE_DIR}/pixel_table_bg2.cpp" "${CMAKE_CURRENT_SOURCE_DIR}/pixel_table_impl.hpp" ) target_include_directories(hictkpy_pixel_table PUBLIC "${CMAKE_CURRENT_SOURCE_DIR}/include/pixel_table") target_link_libraries( hictkpy_pixel_table PRIVATE hictkpy_project_options hictkpy_project_warnings hictk::reference PUBLIC hictkpy_arrow hictkpy_arrow_compute hictkpy_common hictkpy_table hictkpy_type fmt::fmt-header-only hictk::bin_table hictk::pixel spdlog::spdlog_header_only ) ``` -------------------------------- ### Configure hictkpy_file_writer_helper Library Source: https://github.com/paulsengroup/hictkpy/blob/main/src/cpp/file_writer/CMakeLists.txt Defines a static library for file writer helper functions. It includes source files and sets up public include directories and private/public link libraries. ```cmake find_package(FMT REQUIRED QUIET) find_package(spdlog REQUIRED QUIET) find_package(Filesystem REQUIRED QUIET) add_library(hictkpy_file_writer_helper STATIC) target_sources(hictkpy_file_writer_helper PRIVATE "${CMAKE_CURRENT_SOURCE_DIR}/file_writer_helpers.cpp") target_include_directories(hictkpy_file_writer_helper PUBLIC "${CMAKE_CURRENT_SOURCE_DIR}/include/helpers") target_link_libraries( hictkpy_file_writer_helper PRIVATE hictkpy_common hictkpy_project_options hictkpy_project_warnings fmt::fmt-header-only PUBLIC hictkpy_arrow hictkpy_nanobind hictkpy_table ) ``` -------------------------------- ### Specify temporary directory for FileWriter Source: https://github.com/paulsengroup/hictkpy/blob/main/docs/creating_cool_hic_files.rst When creating large .cool or .hic files, specify a custom temporary directory using the 'tmpdir' argument to manage disk space. ```ipythonconsole f = htk.cooler.FileWriter("out.cool", chroms, resolution=50_000, tmpdir="/var/tmp/hictk") ``` -------------------------------- ### Find Python and FMT Packages Source: https://github.com/paulsengroup/hictkpy/blob/main/src/cpp/type/CMakeLists.txt Locates required Python and FMT development components. Ensures these dependencies are available for the build. ```cmake find_package(Python 3.10 COMPONENTS Development.Module REQUIRED QUIET) find_package(FMT REQUIRED QUIET) ``` -------------------------------- ### Create a .cool file from BedGraph2 data Source: https://github.com/paulsengroup/hictkpy/blob/main/docs/creating_cool_hic_files.rst Initialize a FileWriter for a .cool file and add pixels in chunks from a BedGraph2 file. This method is memory-efficient for large datasets. ```ipythonconsole import hictkpy as htk import pandas as pd # Create a dictionary mapping chromosome names to chromosome sizes chroms = pd.read_table("chrom.sizes", names=["name", "length"]) .set_index("name")["length"] .to_dict() chroms # Define the name of the columns for later use cols = ["chrom1", "start1", "end1", "chrom2", "start2", "end2", "count"] # Initialize an empty .cool file with htk.cooler.FileWriter("out.cool", chroms, resolution=50_000) as writer: # Lazily load pixels in chunks to reduce memory usage pixels = pd.read_table("pixels.bg2", names=cols, chunksize=1_000_000) # Add chunks of pixels one by one for i, df in enumerate(pixels): print(f"adding chunk #{i}...") writer.add_pixels(df) # Check that the resulting file has some interactions htk.File("out.cool").attributes()["nnz"] ``` -------------------------------- ### File Class Source: https://github.com/paulsengroup/hictkpy/blob/main/docs/api/generic.rst Represents a single-resolution Hi-C file. ```APIDOC ## File ### Description Represents a single-resolution Hi-C file, providing methods for data access and analysis. ### Methods - `__init__(self, path: str)`: Initializes the File object. - `__enter__(self)`: Enters a context manager for the file. - `__exit__(self, exc_type, exc_val, exc_tb)`: Exits the context manager. - `attributes(self)`: Returns the attributes of the file. - `avail_normalizations(self)`: Returns available normalizations. - `bins(self)`: Returns the bins of the file. - `chromosomes(self)`: Returns the list of chromosomes. - `close(self)`: Closes the file. - `fetch(self, region: str, join: bool = False)`: Fetches pixel data for a given region. - `has_normalization(self, norm: str)`: Checks if a normalization is available. - `is_cooler(self)`: Checks if the file is a cooler file. - `is_hic(self)`: Checks if the file is a HiC file. - `nbins(self)`: Returns the number of bins. - `nchroms(self)`: Returns the number of chromosomes. - `path(self)`: Returns the path to the file. - `resolution(self)`: Returns the resolution of the file. - `uri(self)`: Returns the URI of the file. - `weights(self)`: Returns the weights of the file. ``` -------------------------------- ### Configure hictkpy_file_writer_cooler Library Source: https://github.com/paulsengroup/hictkpy/blob/main/src/cpp/file_writer/CMakeLists.txt Defines a static library for the cooler file writer. It specifies source files, include directories, and links against necessary hictkpy and external libraries. ```cmake add_library(hictkpy_file_writer_cooler STATIC) target_sources(hictkpy_file_writer_cooler PRIVATE "${CMAKE_CURRENT_SOURCE_DIR}/cooler_file_writer.cpp") target_include_directories(hictkpy_file_writer_cooler PUBLIC "${CMAKE_CURRENT_SOURCE_DIR}/include/cooler") target_link_libraries( hictkpy_file_writer_cooler PRIVATE hictkpy_arrow hictkpy_common hictkpy_file_writer_helper hictkpy_locking hictkpy_pixel_table hictkpy_project_options hictkpy_project_warnings hictkpy_type fmt::fmt-header-only hictk::file spdlog::spdlog_header_only PUBLIC hictkpy_bin_table hictkpy_file hictkpy_nanobind hictkpy_reference hictk::bin_table hictk::cooler hictk::reference hictk::tmpdir std::filesystem ) ``` -------------------------------- ### Configure hictkpy_table Library Source: https://github.com/paulsengroup/hictkpy/blob/main/src/cpp/table/CMakeLists.txt Defines the static library 'hictkpy_table', its source files, include directories, and links necessary libraries including fmt and hictkpy internal modules. ```cmake find_package(FMT REQUIRED QUIET) find_package(spdlog REQUIRED QUIET) add_library(hictkpy_table STATIC) target_sources( hictkpy_table PRIVATE "${CMAKE_CURRENT_SOURCE_DIR}/table.cpp" "${CMAKE_CURRENT_SOURCE_DIR}/table_impl.hpp" ) target_include_directories(hictkpy_table PUBLIC "${CMAKE_CURRENT_SOURCE_DIR}/include/table") target_link_libraries( hictkpy_table PRIVATE hictkpy_locking hictkpy_project_options hictkpy_project_warnings hictkpy_type fmt::fmt-header-only PUBLIC hictkpy_arrow hictkpy_common hictkpy_nanobind ) ``` -------------------------------- ### Find Arrow and ArrowCompute Packages Source: https://github.com/paulsengroup/hictkpy/blob/main/src/cpp/arrow/CMakeLists.txt Finds the required Arrow and ArrowCompute packages. It checks for the existence of shared or static libraries. ```cmake find_package(Arrow REQUIRED QUIET) if(NOT TARGET ArrowCompute::arrow_compute_shared AND NOT TARGET ArrowCompute::arrow_compute_static) find_package(ArrowCompute REQUIRED QUIET) endif() ``` -------------------------------- ### Add Nanobind Module with Build Options Source: https://github.com/paulsengroup/hictkpy/blob/main/src/cpp/nanobind/CMakeLists.txt Defines a nanobind module, setting build arguments like LTO or NOMINSIZE based on the build type. This snippet handles the creation of a static nanobind module if it doesn't already exist. ```cmake if(NOT TARGET nanobind-static) set(HICTKPY_NANOBIND_DUMMY_FILE_DIR "${CMAKE_CURRENT_BINARY_DIR}/generated/nanobind/") set(HICTKPY_NANOBIND_DUMMY_FILE "${HICTKPY_NANOBIND_DUMMY_FILE_DIR}/dummy.cpp") file(MAKE_DIRECTORY "${HICTKPY_NANOBIND_DUMMY_FILE_DIR}") file(TOUCH "${HICTKPY_NANOBIND_DUMMY_FILE}") if(CMAKE_BUILD_TYPE STREQUAL Release) set( HICTKPY_MODULE_ARGS LTO NOMINSIZE ) else() set(HICTKPY_MODULE_ARGS NOSTRIP) endif() nanobind_add_module( hictkpy_nanobind_dummy NB_DOMAIN hictkpy NB_STATIC ${HICTKPY_MODULE_ARGS} NB_SUPPRESS_WARNINGS MODULE "${HICTKPY_NANOBIND_DUMMY_FILE}" ) endif() ``` -------------------------------- ### Configure Compile Definitions for Hictkpy Project Options Source: https://github.com/paulsengroup/hictkpy/blob/main/CMakeLists.txt Sets compile definitions for the hictkpy_project_options target. Includes options for fmt, spdlog, and general project settings. ```cmake target_compile_definitions( hictkpy_project_options INTERFACE # Tweak fmt FMT_HEADER_ONLY # starting with fmt v12, clang-tidy breaks when FMT_ENFORCE_COMPILE_STRING is defined # https://github.com/fmtlib/fmt/commit/619b3a5aa031a25e8d06fec1e03992e5e727c5a5#diff-bdc6f79e8e9f5b4331d66fb785636a87d29f55cf729865e13925b4209424c878R4223 $<$>:FMT_ENFORCE_COMPILE_STRING> FMT_USE_FULL_CACHE_DRAGONBOX HICTK_WITH_ARROW HICTK_WITH_EIGEN # Tweak spdlog SPDLOG_ACTIVE_LEVEL=SPDLOG_LEVEL_DEBUG SPDLOG_CLOCK_COARSE SPDLOG_FMT_EXTERNAL SPDLOG_NO_THREAD_ID # Windows-specific tweaks $<$:NOMINMAX> $<$:_CRT_SECURE_NO_WARNINGS> $<$:_DISABLE_CONSTEXPR_MUTEX_CONSTRUCTOR> # https://github.com/gabime/spdlog/issues/3212 ) ``` -------------------------------- ### Define Build Configuration Variables Source: https://github.com/paulsengroup/hictkpy/blob/main/src/cpp/arrow/CMakeLists.txt Sets CMake variables to determine if the build is on Windows and if it's a static build. ```cmake set(IS_WINDOWS "$") set(STATIC_BUILD "$>") ``` -------------------------------- ### Configure Clang-Tidy for Nanobind Source: https://github.com/paulsengroup/hictkpy/blob/main/src/CMakeLists.txt Copies a Clang-Tidy configuration file to the nanobind source directory. This is a workaround to disable clang-tidy for nanobind. ```cmake configure_file("${PROJECT_SOURCE_DIR}/cmake/.clang-tidy.in" "${nanobind_SOURCE_DIR}/.clang-tidy" COPYONLY) ``` -------------------------------- ### FileWriter Source: https://github.com/paulsengroup/hictkpy/blob/main/docs/api/cooler.rst Facilitates the creation and writing of Cooler files, including adding pixel data and managing file finalization. ```APIDOC ## FileWriter ### Description Enables writing and manipulation of Cooler files. ### Methods - `__init__`: Initializes the FileWriter. - `__enter__`: Enters a context manager. - `__exit__`: Exits a context manager. - `add_pixels`: Adds pixel data to the Cooler file. - `add_pixels_from_dict`: Adds pixel data from a dictionary. - `bins`: Accesses bin information during writing. - `chromosomes`: Accesses chromosome information during writing. - `finalize`: Finalizes the Cooler file after writing. - `path`: Gets the file path for the FileWriter. - `resolution`: Sets or gets the resolution for the FileWriter. ``` -------------------------------- ### Improve .hic file creation performance with threads Source: https://github.com/paulsengroup/hictkpy/blob/main/docs/creating_cool_hic_files.rst For faster .hic file generation, utilize multiple threads by setting the 'n_threads' argument in htk.hic.FileWriter. ```ipythonconsole f = htk.hic.FileWriter("out.hic", chroms, resolution=50_000, n_threads=8) ``` -------------------------------- ### BinTable Class Source: https://github.com/paulsengroup/hictkpy/blob/main/docs/api/generic.rst Represents a table of bins. ```APIDOC ## BinTable ### Description Represents a table containing multiple bins. ### Methods - `__init__(self)`: Initializes the BinTable object. ``` -------------------------------- ### Concurrent Data Fetching with ThreadPoolExecutor Source: https://github.com/paulsengroup/hictkpy/blob/main/docs/multithreading_and_multiprocessing.rst This Python snippet shows how to fetch cis interactions concurrently from a Cooler file using concurrent.futures.ThreadPoolExecutor. It includes helper functions for fetching chromosomes and pixels, and demonstrates setting up logging. ```python import logging import os import sys import threading import time from concurrent.futures import ThreadPoolExecutor import hictkpy import pandas as pd def fetch_chroms(path): """ Get the list of chromosomes available in the given file """ with hictkpy.MultiResFile(path) as f: return list(f.chromosomes().keys()) def fetch_pixels(path, resolution, query): """ Fetch interactions for the given query and return them as a pandas.DataFrame """ with hictkpy.File(path, resolution) as f: logging.info("[%s; TID=%d]: fetching...", query, threading.get_native_id()) df = f.fetch(query, join=True).to_df() logging.info("[%s; TID=%d]: fetched %d interactions!", query, threading.get_native_id(), len(df)) return df def fetch_cis_interactions(path, resolution, nthreads): """ Fetch cis interactions from the given file and return them as a pandas.DataFrame """ chroms = fetch_chroms(path) with ThreadPoolExecutor(nthreads) as tpool: tasks = [] for chrom in chroms: tasks.append(tpool.submit(fetch_pixels, path, resolution, chrom)) results = (task.result() for task in tasks) return pd.concat((df for df in results if len(df) != 0)) def setup_logger(level=logging.INFO): fmt = "[%(asctime)s] %(levelname)s: %(message)s" logging.basicConfig(format=fmt) logging.getLogger().setLevel(level) # suppress log messages generated by hictkpy for level INFO or lower hictkpy.logging.setLevel(logging.WARN) def main(): setup_logger() path = "test/data/hic_test_file.hic" resolution = 100_000 t0 = time.time() df = fetch_cis_interactions(path, resolution, nthreads=os.cpu_count()) print(df, file=sys.stderr) ``` -------------------------------- ### Define hictkpy_bin_table Library Source: https://github.com/paulsengroup/hictkpy/blob/main/src/cpp/bin_table/CMakeLists.txt Configures a static library named hictkpy_bin_table. It specifies the source files and include directories, and links against various hictkpy and external libraries. ```cmake find_package(spdlog REQUIRED QUIET) add_library(hictkpy_bin_table STATIC) target_sources(hictkpy_bin_table PRIVATE "${CMAKE_CURRENT_SOURCE_DIR}/bin_table.cpp") target_include_directories(hictkpy_bin_table PUBLIC "${CMAKE_CURRENT_SOURCE_DIR}/include") target_link_libraries( hictkpy_bin_table PRIVATE hictkpy_arrow hictkpy_locking hictkpy_project_options hictkpy_project_warnings hictkpy_table hictk::genomic_interval fmt::fmt-header-only PUBLIC hictkpy_nanobind hictkpy_reference hictk::bin_table hictk::reference ) ``` -------------------------------- ### Link ArrowCompute Library Interface Source: https://github.com/paulsengroup/hictkpy/blob/main/src/cpp/arrow/CMakeLists.txt Creates an INTERFACE library for hictkpy_arrow_compute and links it against the appropriate ArrowCompute library (static or shared). It also sets Windows-specific compile definitions for static builds. ```cmake add_library(hictkpy_arrow_compute INTERFACE) target_link_libraries( hictkpy_arrow_compute INTERFACE "ArrowCompute::arrow_compute_$" ) target_compile_definitions( hictkpy_arrow_compute INTERFACE "<$:ARROW_COMPUTE_STATIC>" ) ``` -------------------------------- ### Define Static Library Target Source: https://github.com/paulsengroup/hictkpy/blob/main/src/cpp/nanobind/CMakeLists.txt Creates a static library target named 'hictkpy_nanobind'. This library will contain the C++ sources and headers for the nanobind Python bindings. ```cmake add_library(hictkpy_nanobind STATIC) ``` -------------------------------- ### Enable Testing Subdirectory Source: https://github.com/paulsengroup/hictkpy/blob/main/src/CMakeLists.txt Adds the 'cpp' subdirectory for testing if the HICTKPY_ENABLE_TESTING option is enabled. This is typically used for C++ related tests. ```cmake add_subdirectory(cpp) ``` -------------------------------- ### Conversion Methods Source: https://github.com/paulsengroup/hictkpy/blob/main/docs/api/generic.rst Methods for converting data to different formats. ```APIDOC ## to_arrow ### Description Converts data to Arrow format. ### Method `to_arrow()` ### Endpoint N/A (Python method) ### Parameters None specified in source. ### Request Example ```python hictkpy.to_arrow() ``` ### Response Data in Arrow format. ``` ```APIDOC ## to_df ### Description Converts data to a DataFrame format. ### Method `to_df()` ### Endpoint N/A (Python method) ### Parameters None specified in source. ### Request Example ```python hictkpy.to_df() ``` ### Response Data in DataFrame format. ``` ```APIDOC ## to_pandas ### Description Converts data to a Pandas DataFrame. ### Method `to_pandas()` ### Endpoint N/A (Python method) ### Parameters None specified in source. ### Request Example ```python hictkpy.to_pandas() ``` ### Response Data in Pandas DataFrame format. ``` -------------------------------- ### Create a .hic file from BedGraph2 data Source: https://github.com/paulsengroup/hictkpy/blob/main/docs/creating_cool_hic_files.rst Similar to creating a .cool file, but uses hictk.hic.FileWriter for .hic files. This process involves preparing chromosome sizes and ingesting data in chunks. ```ipythonconsole import hictkpy as htk import pandas as pd # Create a dictionary mapping chromosome names to chromosome sizes chroms = pd.read_table("chrom.sizes", names=["name", "length"]) .set_index("name")["length"] .to_dict() # Define the name of the columns for later use cols = ["chrom1", "start1", "end1", "chrom2", "start2", "end2", "count"] # Initialize an empty .hic file with htk.hic.FileWriter("out.hic", chroms, resolution=50_000) as writer: # Lazily load pixels in chunks to reduce memory usage pixels = pd.read_table("pixels.bg2", names=cols, chunksize=1_000_000) # Add chunks of pixels one by one for i, df in enumerate(pixels): print(f"adding chunk #{i}...") writer.add_pixels(df) ``` -------------------------------- ### Set Include Directories for hictkpy_numpy Source: https://github.com/paulsengroup/hictkpy/blob/main/src/cpp/numpy/CMakeLists.txt Configures the include directories for the 'hictkpy_numpy' INTERFACE library. The 'include' directory within the current source directory is added, making its headers accessible to dependent targets. ```cmake target_include_directories(hictkpy_numpy INTERFACE "${CMAKE_CURRENT_SOURCE_DIR}/include") ``` -------------------------------- ### Extract chromosome sizes Source: https://github.com/paulsengroup/hictkpy/blob/main/docs/creating_cool_hic_files.rst Use 'hictk dump -t chroms' to extract chromosome names and their lengths. This is required for initializing the cooler FileWriter. ```console user@dev:/tmp$ hictk dump -t chroms 4DNFIOTPSS3L.hic > chrom.sizes user@dev:/tmp$ head chrom.sizes 2L 23513712 2R 25286936 3L 28110227 3R 32079331 4 1348131 X 23542271 Y 3667352 ``` -------------------------------- ### Bin Class Source: https://github.com/paulsengroup/hictkpy/blob/main/docs/api/generic.rst Represents a bin in a Hi-C matrix. ```APIDOC ## Bin ### Description Represents a genomic bin within a Hi-C matrix. ### Properties - `id`: The unique identifier of the bin. - `rel_id`: The relative identifier of the bin within its chromosome. - `chrom`: The chromosome the bin belongs to. - `start`: The start position of the bin. - `end`: The end position of the bin. ``` -------------------------------- ### MultiResFile Class Source: https://github.com/paulsengroup/hictkpy/blob/main/docs/api/generic.rst Represents a multi-resolution Hi-C file. ```APIDOC ## MultiResFile ### Description Represents a multi-resolution Hi-C file, allowing access to different resolutions. ### Methods - `__init__(self, path: str)`: Initializes the MultiResFile object. - `__getitem__(self, resolution: int)`: Accesses a specific resolution of the file. - `__enter__(self)`: Enters a context manager for the file. - `__exit__(self, exc_type, exc_val, exc_tb)`: Exits the context manager. - `attributes(self)`: Returns the attributes of the file. - `chromosomes(self)`: Returns the list of chromosomes. - `close(self)`: Closes the file. - `is_hic(self)`: Checks if the file is a HiC file. - `is_mcool(self)`: Checks if the file is a multi-resolution cooler file. - `path(self)`: Returns the path to the file. - `resolutions(self)`: Returns the available resolutions. ``` -------------------------------- ### Read chromosomes and sizes with hictkpy Source: https://github.com/paulsengroup/hictkpy/blob/main/docs/quickstart.rst Fetches a dictionary of chromosome names and their corresponding sizes from the Hi-C file. ```ipythonconsole f.chromosomes() ``` -------------------------------- ### Fetch Balancing Weights Source: https://github.com/paulsengroup/hictkpy/blob/main/docs/quickstart.rst Fetches balancing weights for different normalizations using pandas DataFrame. This is useful for normalizing Hi-C data. ```python import pandas as pd weights = {} for norm in f.avail_normalizations(): weights[norm] = f.weights(norm) weights = pd.DataFrame(weights) weights ``` -------------------------------- ### Configure hictkpy_file_writer_hic Library Source: https://github.com/paulsengroup/hictkpy/blob/main/src/cpp/file_writer/CMakeLists.txt Defines a static library for the hic file writer. It includes source files, sets up include directories, and links against required hictkpy and external libraries. ```cmake add_library(hictkpy_file_writer_hic STATIC) target_sources(hictkpy_file_writer_hic PRIVATE "${CMAKE_CURRENT_SOURCE_DIR}/hic_file_writer.cpp") target_include_directories(hictkpy_file_writer_hic PUBLIC "${CMAKE_CURRENT_SOURCE_DIR}/include/hic") target_link_libraries( hictkpy_file_writer_hic PRIVATE hictkpy_common hictkpy_file_writer_helper hictkpy_locking hictkpy_numpy hictkpy_pixel_table hictkpy_project_options hictkpy_project_warnings fmt::fmt-header-only hictk::bin_table hictk::file spdlog::spdlog_header_only PUBLIC hictkpy_bin_table hictkpy_file hictkpy_nanobind hictkpy_reference hictk::hic hictk::reference hictk::tmpdir ) ``` -------------------------------- ### Fetch bins as pandas DataFrame with hictkpy Source: https://github.com/paulsengroup/hictkpy/blob/main/docs/quickstart.rst Retrieves the table of bins (genomic intervals) from the Hi-C file and returns it as a pandas DataFrame. ```ipythonconsole f.bins() ``` -------------------------------- ### Fetch normalized interactions with hictkpy Source: https://github.com/paulsengroup/hictkpy/blob/main/docs/quickstart.rst Fetches interactions using a specified normalization method, such as 'KR' (K-Normalization). ```ipythonconsole sel = f.fetch(normalization="KR") ``` -------------------------------- ### Fetch interactions as CSR sparse matrix with hictkpy Source: https://github.com/paulsengroup/hictkpy/blob/main/docs/quickstart.rst Retrieves interaction data for a specified region and converts it into a scipy.sparse.csr_matrix. ```ipythonconsole sel = f.fetch("2L:10,000,000-20,000,000") sel.to_csr() ``` -------------------------------- ### Link Libraries for Nanobind Target Source: https://github.com/paulsengroup/hictkpy/blob/main/src/cpp/nanobind/CMakeLists.txt Defines the dependencies for the 'hictkpy_nanobind' library. This includes internal hictk libraries, project-specific options and warnings, and external libraries like fmt, nanobind, Python, and NumPy. ```cmake target_link_libraries( hictkpy_nanobind PRIVATE hictk::numeric hictkpy_project_options hictkpy_project_warnings PUBLIC hictkpy_locking fmt::fmt-header-only nanobind-static Python::Module Python::NumPy ) ``` -------------------------------- ### Link Arrow Library Interface Source: https://github.com/paulsengroup/hictkpy/blob/main/src/cpp/arrow/CMakeLists.txt Creates an INTERFACE library for hictkpy_arrow and links it against the appropriate Arrow library (static or shared) based on the build configuration. ```cmake add_library(hictkpy_arrow INTERFACE) target_link_libraries( hictkpy_arrow INTERFACE "Arrow::arrow_$" ) target_compile_definitions( hictkpy_arrow INTERFACE "<$:ARROW_STATIC>" ) ``` -------------------------------- ### FileWriter Class Source: https://github.com/paulsengroup/hictkpy/blob/main/docs/api/hic.rst The FileWriter class is used to create and write Hi-C data to a file. It offers methods for adding pixel data, querying metadata like bins and chromosomes, and managing the file writing process. ```APIDOC ## FileWriter ### Description Represents a writer for Hi-C data files. ### Methods - `__init__(self, path, resolution, nthreads=1, chunk_size=2000000)`: Initializes the FileWriter. - `__enter__(self)`: Enters the runtime context related to this object. - `__exit__(self, exc_type, exc_val, exc_tb)`: Exits the runtime context. - `add_pixels(self, pixels)`: Adds a list of pixels to the Hi-C file. - `add_pixels_from_dict(self, pixels_dict)`: Adds pixels from a dictionary format. - `bins(self)`: Returns information about the bins. - `chromosomes(self)`: Returns information about the chromosomes. - `finalize(self)`: Finalizes the writing process and closes the file. - `path(self)`: Returns the path to the output Hi-C file. - `resolutions(self)`: Returns the available resolutions. ``` -------------------------------- ### Resolution Method Source: https://github.com/paulsengroup/hictkpy/blob/main/docs/api/generic.rst Retrieves the resolution of the data. ```APIDOC ## resolution ### Description Retrieves the resolution of the Hi-C data. ### Method `resolution()` ### Endpoint N/A (Python method) ### Parameters None ### Request Example ```python hictkpy.resolution() ``` ### Response The resolution of the data. ``` -------------------------------- ### File Type Checking Functions Source: https://github.com/paulsengroup/hictkpy/blob/main/docs/api/generic.rst These functions allow you to check the type of a given file. ```APIDOC ## is_cooler ### Description Checks if a file is a valid cooler file. ### Method `is_cooler(path: str) -> bool` ## is_mcool_file ### Description Checks if a file is a valid multi-resolution cooler file. ### Method `is_mcool_file(path: str) -> bool` ## is_scool_file ### Description Checks if a file is a valid single-resolution cooler file. ### Method `is_scool_file(path: str) -> bool` ## is_hic ### Description Checks if a file is a valid HiC file. ### Method `is_hic(path: str) -> bool` ``` -------------------------------- ### Iterator Method Source: https://github.com/paulsengroup/hictkpy/blob/main/docs/api/generic.rst Allows iteration over data. ```APIDOC ## __iter__ ### Description Provides an iterator for the data. ### Method `__iter__()` ### Endpoint N/A (Python method) ### Parameters None ### Request Example ```python for item in hictkpy: # process item ``` ### Response An iterator object. ``` -------------------------------- ### Configure hictkpy Common CMake Library Source: https://github.com/paulsengroup/hictkpy/blob/main/src/cpp/common/CMakeLists.txt Defines hictkpy_common as an INTERFACE library, sets include directories, and links to other hictkpy targets. Use this for common interface definitions. ```cmake add_library(hictkpy_common INTERFACE) target_include_directories(hictkpy_common INTERFACE "${CMAKE_CURRENT_SOURCE_DIR}/include") target_link_libraries( hictkpy_common INTERFACE hictkpy_project_options hictkpy_project_warnings hictk::common hictk::type_traits ) ``` -------------------------------- ### SingleCellFile Source: https://github.com/paulsengroup/hictkpy/blob/main/docs/api/cooler.rst Represents a single-cell Cooler file, providing methods to access its attributes, bins, cells, chromosomes, and file path, as well as context management. ```APIDOC ## SingleCellFile ### Description Provides access to data within a single-cell Cooler file. ### Methods - `__init__`: Initializes the SingleCellFile object. - `__getitem__`: Allows accessing data using item access. - `__enter__`: Enters a context manager. - `__exit__`: Exits a context manager. - `attributes`: Retrieves the attributes of the Cooler file. - `bins`: Accesses the bin information. - `cells`: Accesses the cell information. - `chromosomes`: Retrieves the chromosome information. - `close`: Closes the Cooler file. - `path`: Gets the file path of the Cooler file. - `resolution`: Retrieves the resolution of the Cooler file. ``` -------------------------------- ### Fetch all interactions genome-wide with hictkpy Source: https://github.com/paulsengroup/hictkpy/blob/main/docs/quickstart.rst Fetches all interactions across the entire genome. The `join=True` argument returns data in bedgraph2 format. ```ipythonconsole # Fetch all interactions (genome-wide query) in COO format (row, column, count) sels = f.fetch() ``` ```ipythonconsole # Fetch all interactions (genome-wide query) in bedgraph2 format sels = f.fetch(join=True) ``` -------------------------------- ### Pixel Properties Source: https://github.com/paulsengroup/hictkpy/blob/main/docs/api/generic.rst Properties available for Pixel objects. ```APIDOC ## Pixel Properties ### Description Properties of a Pixel object. ### Properties - **bin1_id** (int) - Identifier for the first bin. - **bin2_id** (int) - Identifier for the second bin. - **count** (int) - The interaction count. ### BG2 Format Specific Properties These properties are only available when pixels are in BG2 format: - **bin1** (any) - Information about the first bin. - **bin2** (any) - Information about the second bin. - **chrom1** (str) - Chromosome of the first bin. - **start1** (int) - Start position of the first bin. - **end1** (int) - End position of the first bin. - **chrom2** (str) - Chromosome of the second bin. - **start2** (int) - Start position of the second bin. - **end2** (int) - End position of the second bin. ### Request Example ```python pixel = Pixel(...) print(pixel.bin1_id) print(pixel.count) if is_bg2_format(pixel): print(pixel.chrom1) ``` ### Response Values of the respective properties. ```