### Install nanoarrow Python Bindings Source: https://arrow.apache.org/nanoarrow Install the nanoarrow Python bindings using pip or conda. ```bash pip install nanoarrow ``` ```bash conda install nanoarrow -c conda-forge ``` -------------------------------- ### Build nanoarrow Python Bindings Source: https://arrow.apache.org/nanoarrow/latest/getting-started/python.html Clone the nanoarrow repository and install the Python bindings using pip with the `-e` flag for an editable install. Ensure build dependencies like meson and cython are installed. ```bash git clone https://github.com/apache/arrow-nanoarrow.git cd arrow-nanoarrow/python # Build dependencies: # pip install meson meson-python cython pip install -e . --no-build-isolation ``` -------------------------------- ### Run nanoarrow Python Tests Source: https://arrow.apache.org/nanoarrow/latest/getting-started/python.html Install test dependencies using `pip install -e ".[test]"` and then run the test suite using pytest with the `-vvx` flags for verbose output and stopping on the first failure. ```bash # Install dependencies pip install -e ".[test]" # Run tests pytest -vvx ``` -------------------------------- ### Install nanoarrow R Package Source: https://arrow.apache.org/nanoarrow Install the nanoarrow R package from CRAN. ```bash install.packages("nanoarrow") ``` -------------------------------- ### Install nanoarrow Development Version from GitHub Source: https://arrow.apache.org/nanoarrow/latest/_sources/getting-started/r.rst.txt Install the development version of the nanoarrow package from GitHub using the remotes package. ```r # install.packages("remotes") remotes::install_github("apache/arrow-nanoarrow/r") ``` -------------------------------- ### Install nanoarrow Python Package Source: https://arrow.apache.org/nanoarrow/latest/getting-started/python.html Install nanoarrow from PyPI or conda-forge. Development versions are available from a separate index. ```bash pip install nanoarrow ``` ```bash conda install nanoarrow -c conda-forge ``` ```bash pip install --extra-index-url https://pypi.fury.io/arrow-nightlies/ \ --prefer-binary --pre nanoarrow ``` -------------------------------- ### Setup nanoarrow Build Directory with Meson Source: https://arrow.apache.org/nanoarrow Set up the build directory for nanoarrow using Meson. This is an experimental backend. ```bash meson setup builddir cd builddir ``` -------------------------------- ### Use nanoarrow as a Meson Subproject Source: https://arrow.apache.org/nanoarrow Install nanoarrow as a subproject for use with the Meson build system. ```bash mkdir subprojects meson wrap install nanoarrow ``` -------------------------------- ### Compile nanoarrow with Meson Source: https://arrow.apache.org/nanoarrow Compile the nanoarrow project after setup and configuration using Meson. ```bash meson compile ``` -------------------------------- ### Example: Using nanoarrow with reticulate Source: https://arrow.apache.org/nanoarrow/latest/r/reference/as_nanoarrow_schema.python.builtin.object.html Demonstrates how to use nanoarrow with Python objects via reticulate. This example requires the nanoarrow Python package to be installed. It shows converting a Python array-like object to a nanoarrow array and converting an R vector to a Python object. ```R if (FALSE) { # test_reticulate_with_nanoarrow() library(reticulate) py_require("nanoarrow") na <- import("nanoarrow", convert = FALSE) python_arrayish_thing <- na$Array(1:3, na_int32()) as_nanoarrow_array(python_arrayish_thing) r_to_py(as_nanoarrow_array(1:3)) } ``` -------------------------------- ### Example IPC Stream Source: https://arrow.apache.org/nanoarrow/latest/r/reference/index.html Provides an example of Arrow IPC (Inter-Process Communication) data. ```APIDOC ## example_ipc_stream() ### Description Get an example Arrow IPC stream. ### Function Signature `example_ipc_stream()` ``` -------------------------------- ### C-style Memory Management Example Source: https://arrow.apache.org/nanoarrow/latest/_sources/getting-started/cpp.rst.txt Illustrates C-style memory management for resources requiring explicit cleanup, as is common when working directly with C interfaces. ```c // Because nanoarrow is implemented // in C and provides a C interface, the library by default uses C-style // memory management (i.e., if you allocate it, you clean it up). This is // unnecessary when you have C++ at your disposal, so nanoarrow also // provides a C++ header (``nanoarrow.hpp``) with // ``std::unique_ptr<>``-like wrappers around anything that requires // explicit clean up. Whereas in C you might have to write code like this: .. code:: c ``` -------------------------------- ### Create a string view schema Source: https://arrow.apache.org/nanoarrow/latest/reference/python/schema.html Use `na.string_view()` to create a schema for string views. This is a basic schema creation example. ```python import nanoarrow as na na.string_view() ``` -------------------------------- ### Create ArrayStream from URL Source: https://arrow.apache.org/nanoarrow/latest/reference/python/array-stream.html Create an ArrayStream from an IPC stream located at a URL. This example writes example bytes to a temporary file, converts its path to a URI, and then reads it using `from_url`. ```python >>> import pathlib >>> import tempfile >>> import os >>> import nanoarrow as na >>> from nanoarrow.ipc import InputStream >>> with tempfile.TemporaryDirectory() as td: ... path = os.path.join(td, "test.arrows") ... with open(path, "wb") as f: ... nbytes = f.write(InputStream.example_bytes()) ... ... uri = pathlib.Path(path).as_uri() ... with na.ArrayStream.from_url(uri) as stream: ... stream.read_all() nanoarrow.Array>[3] {'some_col': 1} {'some_col': 2} {'some_col': 3} ``` -------------------------------- ### Import nanoarrow Source: https://arrow.apache.org/nanoarrow/latest/getting-started/python.html Import the nanoarrow package to start using its functionalities. ```python import nanoarrow as na ``` -------------------------------- ### Create ArrayStream from file path Source: https://arrow.apache.org/nanoarrow/latest/reference/python/array-stream.html Create an ArrayStream from an IPC stream located at a local file path. This example demonstrates writing example bytes to a file and then reading them back into an ArrayStream. ```python >>> import tempfile >>> import os >>> import nanoarrow as na >>> from nanoarrow.ipc import InputStream >>> with tempfile.TemporaryDirectory() as td: ... path = os.path.join(td, "test.arrows") ... with open(path, "wb") as f: ... nbytes = f.write(InputStream.example_bytes()) ... ... with na.ArrayStream.from_path(path) as stream: ... stream.read_all() nanoarrow.Array>[3] {'some_col': 1} {'some_col': 2} {'some_col': 3} ``` -------------------------------- ### example_ipc_stream Source: https://arrow.apache.org/nanoarrow/latest/r/reference/example_ipc_stream.html Generates an example Arrow IPC stream. This function can optionally take a compression argument. ```APIDOC ## example_ipc_stream ### Description Generates an example Arrow IPC stream that can be used for testing or examples. ### Usage ``` example_ipc_stream(compression = c("none", "zstd")) ``` ### Arguments * **compression** (character) - One of "none" or "zstd". ### Value A raw vector that can be passed to `read_nanoarrow()`. ### Examples ```R as.data.frame(read_nanoarrow(example_ipc_stream())) #> some_col #> 1 0 #> 2 1 #> 3 2 ``` ``` -------------------------------- ### Generate Example IPC Stream Source: https://arrow.apache.org/nanoarrow/latest/r/reference/example_ipc_stream.html Use `example_ipc_stream()` to create a raw vector representing an Arrow IPC stream. This function can optionally apply zstd compression. ```r example_ipc_stream(compression = c("none", "zstd")) ``` -------------------------------- ### Get the device of the Array buffers Source: https://arrow.apache.org/nanoarrow/latest/reference/python/array.html Retrieve the device on which the buffers for a nanoarrow Array are allocated. This example shows the default CPU device. ```python >>> import nanoarrow as na >>> array = na.Array([1, 2, 3], na.int32()) >>> array.device - device_type: CPU <1> - device_id: -1 ``` -------------------------------- ### Initialize ArrowSchema and ArrowArray Source: https://arrow.apache.org/nanoarrow/latest/_sources/getting-started/cpp.rst.txt Demonstrates basic initialization of ArrowSchema and ArrowArray using C-style functions. Note the manual error handling and resource management required. ```cpp struct ArrowSchema schema; struct ArrowArray array; // Ok: if this returns, array was not initialized NANOARROW_RETURN_NOT_OK(ArrowSchemaInitFromType(&schema, NANOARROW_TYPE_STRING)); // Verbose: if this fails, we need to release schema before returning // or it will leak. int code = ArrowArrayInitFromSchema(&array, &schema, NULL); if (code != NANOARROW_OK) { ArrowSchemaRelease(&schema); return code; } ``` -------------------------------- ### Convert PyArrow RecordBatchReader to nanoarrow C ArrayStream Source: https://arrow.apache.org/nanoarrow/latest/reference/python/advanced.html Demonstrates converting a pyarrow.RecordBatchReader to a nanoarrow C ArrayStream for efficient processing. Ensure pyarrow is installed. ```python >>> import pyarrow as pa >>> import nanoarrow as na >>> pa_column = pa.array([1, 2, 3], pa.int32()) >>> pa_batch = pa.record_batch([pa_column], names=["col1"]) >>> pa_reader = pa.RecordBatchReader.from_batches(pa_batch.schema, [pa_batch]) >>> array_stream = na.c_array_stream(pa_reader) >>> array_stream.get_schema() - format: '+s' - name: '' - flags: 0 - metadata: NULL - dictionary: NULL - children[1]: 'col1': - format: 'i' - name: 'col1' - flags: 2 - metadata: NULL - dictionary: NULL - children[0]: >>> array_stream.get_next().length 3 >>> array_stream.get_next() is None Traceback (most recent call last): ... StopIteration ``` -------------------------------- ### ArrowIpcOutputStreamInitFile Source: https://arrow.apache.org/nanoarrow/latest/reference/ipc.html Initializes an output stream to write to a C FILE* pointer. ```APIDOC ## ArrowIpcOutputStreamInitFile ### Description Create an output stream from a C FILE* pointer. ``` -------------------------------- ### Run nanoarrow Benchmarks with Meson Source: https://arrow.apache.org/nanoarrow Execute nanoarrow benchmarks using Meson. Use the --verbose flag for more detailed output. ```bash meson test nanoarrow: --benchmark --verbose # run benchmarks ``` -------------------------------- ### ArrowLayoutInit Source: https://arrow.apache.org/nanoarrow/latest/reference/c.html Initializes a description of buffer arrangements from a storage type. ```APIDOC ## ArrowLayoutInit ### Description Initialize a description of buffer arrangements from a storage type. ### Signature NANOARROW_DLL void ArrowLayoutInit (struct ArrowLayout *layout, enum ArrowType storage_type) ``` -------------------------------- ### Build nanoarrow C Library with CMake and Tests Source: https://arrow.apache.org/nanoarrow Build the nanoarrow C library with tests enabled using CMake. ```bash mkdir build && cd build cmake .. -DNANOARROW_BUILD_TESTS=ON cmake --build . ``` -------------------------------- ### ArrowHalfFloatToFloat Source: https://arrow.apache.org/nanoarrow/latest/reference/c.html Get the float value of a half float. ```APIDOC ## ArrowHalfFloatToFloat ### Description Get the float value of a half float. ### Signature static inline float ArrowHalfFloatToFloat(uint16_t value) ``` -------------------------------- ### Initialize ArrowSchema and ArrowArray with C++ Bindings Source: https://arrow.apache.org/nanoarrow/latest/_sources/getting-started/cpp.rst.txt Shows how to initialize ArrowSchema and ArrowArray using the C++ unique pointer wrappers provided by nanoarrow. This simplifies resource management. ```cpp nanoarrow::UniqueSchema schema; nanoarrow::UniqueArray array; NANOARROW_RETURN_NOT_OK(ArrowSchemaInitFromType(schema.get(), NANOARROW_TYPE_STRING)); NANOARROW_RETURN_NOT_OK(ArrowArrayInitFromSchema(array.get(), schema.get(), NULL)); ``` -------------------------------- ### ArrowFloatToHalfFloat Source: https://arrow.apache.org/nanoarrow/latest/reference/c.html Get the half float value of a float. ```APIDOC ## ArrowFloatToHalfFloat ### Description Get the half float value of a float. ### Signature static inline uint16_t ArrowFloatToHalfFloat(float value) ``` -------------------------------- ### ArrowDecimalAppendStringToBuffer Source: https://arrow.apache.org/nanoarrow/latest/reference/c.html Get the decimal value of an ArrowDecimal as a string. ```APIDOC ## ArrowDecimalAppendStringToBuffer ### Description Get the decimal value of an ArrowDecimal as a string. ### Signature NANOARROW_DLL ArrowErrorCode ArrowDecimalAppendStringToBuffer (const struct ArrowDecimal *decimal, struct ArrowBuffer *buffer) ``` -------------------------------- ### ArrowIpcWriterInit Source: https://arrow.apache.org/nanoarrow/latest/reference/ipc.html Initializes an IPC writer with an output stream. The writer takes ownership of the output stream upon successful initialization. ```APIDOC ## ArrowIpcWriterInit ### Description Initialize an output stream of bytes from an ArrowArrayStream. Returns NANOARROW_OK on success. If NANOARROW_OK is returned the writer takes ownership of the output byte stream, and the caller is responsible for releasing the writer by calling ArrowIpcWriterReset(). ### Signature NANOARROW_DLL ArrowErrorCode ArrowIpcWriterInit (struct ArrowIpcWriter *writer, struct ArrowIpcOutputStream *output_stream) ``` -------------------------------- ### ArrowDecimalAppendDigitsToBuffer Source: https://arrow.apache.org/nanoarrow/latest/reference/c.html Get the integer value of an ArrowDecimal as string. ```APIDOC ## ArrowDecimalAppendDigitsToBuffer ### Description Get the integer value of an ArrowDecimal as string. ### Signature NANOARROW_DLL ArrowErrorCode ArrowDecimalAppendDigitsToBuffer (const struct ArrowDecimal *decimal, struct ArrowBuffer *buffer) ``` -------------------------------- ### ArrowDevice.buffer_init Source: https://arrow.apache.org/nanoarrow/latest/reference/device.html Initializes an owning buffer on a target device by copying existing content from a source device. Supports asynchronous operations via a stream. Implementations must check source and destination devices and return ENOTSUP if the operation is not supported. ```APIDOC ## ArrowDevice.buffer_init ### Description Initialize an owning buffer from existing content. Creates a new buffer whose data member can be accessed by the GPU by copying existing content. Implementations must use the provided stream if non-null; implementations may error if they require a stream to be provided. Implementations must check device_src and device_dst and return ENOTSUP if not prepared to handle this operation. ### Method `ArrowErrorCode (*buffer_init)(struct ArrowDevice *device_src, struct ArrowBufferView src, struct ArrowDevice *device_dst, struct ArrowBuffer *dst, void *stream)` ``` -------------------------------- ### ArrowDeviceBufferInit Source: https://arrow.apache.org/nanoarrow/latest/reference/device.html Allocates a device buffer and copies existing content without using a stream. This is a convenience wrapper for asynchronous initialization when no stream is required. ```APIDOC ## ArrowDeviceBufferInit ### Description Allocate a device buffer and copying existing content without a stream. Convenience wrapper for the case where no stream is provided. ### Method `ArrowErrorCode ArrowDeviceBufferInit(struct ArrowDevice *device_src, struct ArrowBufferView src, struct ArrowDevice *device_dst, struct ArrowBuffer *dst)` ``` -------------------------------- ### Scalar.schema Source: https://arrow.apache.org/nanoarrow/latest/reference/python/array.html Gets the schema (data type) of this Scalar. ```APIDOC ## Scalar.schema ### Description Get the schema (data type) of this scalar ### Property `schema` (Schema) ### Examples ```python >>> import nanoarrow as na >>> array = na.Array([1, 2, 3], na.int32()) >>> array[0].schema int32 ``` ``` -------------------------------- ### Array.schema Source: https://arrow.apache.org/nanoarrow/latest/reference/python/array.html Gets the schema (data type) of this Array. ```APIDOC ## Array.schema ### Description Get the schema (data type) of this Array ### Property `schema` (Schema) ``` -------------------------------- ### Array.n_children Source: https://arrow.apache.org/nanoarrow/latest/reference/python/array.html Gets the number of children for an Array of this type. ```APIDOC ## Array.n_children ### Description Get the number of children for an Array of this type. ### Property `n_children` (int) ### Examples ```python >>> import nanoarrow as na >>> import pyarrow as pa >>> batch = pa.record_batch( ... [pa.array([1, 2, 3]), pa.array(["a", "b", "c"])], ... names=["col1", "col2"] ... ) >>> array = na.Array(batch) >>> array.n_children 2 ``` ``` -------------------------------- ### Array.n_buffers Source: https://arrow.apache.org/nanoarrow/latest/reference/python/array.html Gets the number of buffers in each chunk of this Array. ```APIDOC ## Array.n_buffers ### Description Get the number of buffers in each chunk of this Array ### Property `n_buffers` (int) ### Examples ```python >>> import nanoarrow as na >>> array = na.Array([1, 2, 3], na.int32()) >>> array.n_buffers 2 ``` ``` -------------------------------- ### ArrowDeviceArrayInit Source: https://arrow.apache.org/nanoarrow/latest/reference/device.html A convenience wrapper to initialize an ArrowDeviceArray without involving a stream, simplifying synchronous initialization. ```APIDOC ## ArrowDeviceArrayInit ### Description Initialize an ArrowDeviceArray without a stream. Convenience wrapper to initialize an ArrowDeviceArray without a stream. ### Signature ```c static inline ArrowErrorCode ArrowDeviceArrayInit(struct ArrowDevice *device, struct ArrowDeviceArray *device_array, struct ArrowArray *array, void *sync_event) ``` ### Parameters * `device` (struct ArrowDevice *) - The device to initialize the array on. * `device_array` (struct ArrowDeviceArray *) - The ArrowDeviceArray to initialize. * `array` (struct ArrowArray *) - The ArrowArray to initialize from. * `sync_event` (void *) - An optional synchronization event. Ownership is transferred if non-null. ### Returns NANOARROW_OK on success, or an error code on failure. Ownership of `array` and `sync_event` is transferred to `device_array` on success. ``` -------------------------------- ### Build nanoarrow C Library with CMake, Tests, and Arrow C++ Source: https://arrow.apache.org/nanoarrow Build the nanoarrow C library with tests enabled, including tests that depend on Arrow C++. ```bash mkdir build && cd build cmake .. -DNANOARROW_BUILD_TESTS=ON -DNANOARROW_BUILD_TESTS_WITH_ARROW=ON cmake --build . ``` -------------------------------- ### device Source: https://arrow.apache.org/nanoarrow/latest/reference/python/array.html Get the device on which the buffers for this array are allocated. ```APIDOC ## device ### Description Get the device on which the buffers for this array are allocated. ### Example ```python >>> import nanoarrow as na >>> array = na.Array([1, 2, 3], na.int32()) >>> array.device - device_type: CPU <1> - device_id: -1 ``` ``` -------------------------------- ### Array.n_chunks Source: https://arrow.apache.org/nanoarrow/latest/reference/python/array.html Gets the number of chunks in the underlying representation of this Array. ```APIDOC ## Array.n_chunks ### Description Get the number of chunks in the underlying representation of this Array. ### Property `n_chunks` (int) ### Examples ```python >>> import nanoarrow as na >>> array = na.Array([1, 2, 3], na.int32()) >>> array.n_chunks 1 ``` ``` -------------------------------- ### ArrowIpcInputStreamInitFile Source: https://arrow.apache.org/nanoarrow/latest/reference/ipc.html Initializes an input stream to read from a C FILE* pointer. Allows specifying whether the stream should close the file on release. ```APIDOC ## ArrowIpcInputStreamInitFile ### Description Create an input stream from a C FILE* pointer. Note that the ArrowIpcInputStream has no mechanism to communicate an error if file_ptr fails to close. If this behaviour is needed, pass false to close_on_release and handle closing the file independently from stream. ``` -------------------------------- ### Get Timestamp TimeUnit Source: https://arrow.apache.org/nanoarrow/latest/reference/python/schema.html Retrieve the TimeUnit for a nanoarrow timestamp type. ```python >>> import nanoarrow as na >>> na.timestamp(na.TimeUnit.SECOND).unit ``` -------------------------------- ### Get decimal scale Source: https://arrow.apache.org/nanoarrow/latest/reference/python/schema.html Retrieves the scale value for decimal types. ```python import nanoarrow as na na.decimal128(10, 3).scale ``` -------------------------------- ### ArrowBufferInit Source: https://arrow.apache.org/nanoarrow/latest/reference/c.html Initializes an ArrowBuffer with a NULL, zero-size buffer using the default buffer allocator. ```APIDOC ## ArrowBufferInit ### Description Initialize an ArrowBuffer. Initialize a buffer with a NULL, zero-size buffer using the default buffer allocator. ### Signature `static inline void ArrowBufferInit(struct ArrowBuffer *buffer)` ``` -------------------------------- ### Get decimal precision Source: https://arrow.apache.org/nanoarrow/latest/reference/python/schema.html Retrieves the precision value for decimal types. ```python import nanoarrow as na na.decimal128(10, 3).precision ``` -------------------------------- ### Configure nanoarrow Build Options with Meson Source: https://arrow.apache.org/nanoarrow Configure build options such as tests and benchmarks for nanoarrow using Meson. ```meson meson configure -Dtests=enabled -Dbenchmarks=enabled ``` -------------------------------- ### Create NanoArrow C Arrays from various sources Source: https://arrow.apache.org/nanoarrow/latest/reference/python/advanced.html Demonstrates creating nanoarrow C arrays from Python iterables, NumPy arrays, and PyArrow arrays. Also shows how to access basic array properties like length and null count. ```python >>> import nanoarrow as na >>> # Create from iterable >>> array = na.c_array([1, 2, 3], na.int32()) >>> # Create from Python buffer (e.g., numpy array) >>> import numpy as np >>> array = na.c_array(np.array([1, 2, 3])) >>> # Create from Arrow PyCapsule (e.g., pyarrow array) >>> import pyarrow as pa >>> array = na.c_array(pa.array([1, 2, 3])) >>> # Access array fields >>> array.length 3 >>> array.null_count 0 ``` -------------------------------- ### Get Integer Type Source: https://arrow.apache.org/nanoarrow/latest/reference/python/schema.html Retrieve the type enumerator for a nanoarrow integer type. ```python >>> import nanoarrow as na >>> na.int32().type ``` -------------------------------- ### ArrowDeviceCpu Source: https://arrow.apache.org/nanoarrow/latest/reference/device.html Provides a pointer to a statically-allocated singleton CPU device. This device is always available. ```APIDOC ## ArrowDeviceCpu ### Description Pointer to a statically-allocated CPU device singleton. ### Signature ```c NANOARROW_DLL struct ArrowDevice * ArrowDeviceCpu (void) ``` ### Returns A pointer to the CPU device singleton. ``` -------------------------------- ### Get Timestamp Timezone Source: https://arrow.apache.org/nanoarrow/latest/reference/python/schema.html Access the timezone property of a nanoarrow timestamp type. ```python >>> import nanoarrow as na >>> na.timestamp(na.TimeUnit.SECOND, timezone="America/Halifax").timezone 'America/Halifax' ``` -------------------------------- ### Initialize and append to a nanoarrow buffer Source: https://arrow.apache.org/nanoarrow/latest/r/reference/nanoarrow_buffer_init.html Use `nanoarrow_buffer_init()` to create an empty buffer and `nanoarrow_buffer_append()` to add data to it. Note that `nanoarrow_buffer_append` modifies the buffer in place. ```R buffer <- nanoarrow_buffer_init() nanoarrow_buffer_append(buffer, 1:5) array <- nanoarrow_array_modify( nanoarrow_array_init(na_int32()), list(length = 5, buffers = list(NULL, buffer)) ) as.vector(array) ``` -------------------------------- ### Get Dictionary Value Type Source: https://arrow.apache.org/nanoarrow/latest/reference/python/schema.html Access the value type of a nanoarrow dictionary schema. ```python >>> import nanoarrow as na >>> na.dictionary(na.int32(), na.string()).value_type string ``` -------------------------------- ### Use nanoarrow with CMake FetchContent Source: https://arrow.apache.org/nanoarrow Integrate nanoarrow into a CMake project using FetchContent to download and make the library available. ```cmake fetchcontent_declare(nanoarrow URL "https://www.apache.org/dyn/closer.lua?action=download&filename=arrow/apache-arrow-nanoarrow-0.8.0/apache-arrow-nanoarrow-0.8.0.tar.gz") fetchcontent_makeavailable(nanoarrow) ``` -------------------------------- ### Get Map Value Type Source: https://arrow.apache.org/nanoarrow/latest/reference/python/schema.html Retrieve the value type of a nanoarrow map schema. ```python >>> import nanoarrow as na >>> na.map_(na.int32(), na.string()).value_type 'value': string ``` -------------------------------- ### Run nanoarrow Tests with Meson Source: https://arrow.apache.org/nanoarrow Execute the nanoarrow test suite using Meson. Tests can also be run under Valgrind. ```bash meson test nanoarrow: # default test run ``` ```bash meson test nanoarrow: --wrap valgrind # run tests under valgrind ``` -------------------------------- ### Get List Value Type Source: https://arrow.apache.org/nanoarrow/latest/reference/python/schema.html Access the value type of a nanoarrow list schema. ```python >>> import nanoarrow as na >>> na.list_(na.int32()).value_type 'item': int32 ``` -------------------------------- ### ArrowIpcWriterStartFile Source: https://arrow.apache.org/nanoarrow/latest/reference/ipc.html Initiates writing an IPC file by writing the magic bytes and preparing to track blocks. ```APIDOC ## ArrowIpcWriterStartFile ### Description Start writing an IPC file. Writes the Arrow IPC magic and sets the writer up to track written blocks. ### Signature ArrowErrorCode ArrowIpcWriterStartFile(struct ArrowIpcWriter *writer, struct ArrowError *error) ``` -------------------------------- ### Get Pointer Address Source: https://arrow.apache.org/nanoarrow/latest/r/reference/nanoarrow_pointer_is_valid.html Retrieves different representations of the memory address of a nanoarrow pointer. ```APIDOC ## nanoarrow_pointer_addr_dbl(ptr) ### Description Returns a double representation of the pointer's address. ### Arguments * `ptr` - An external pointer to a `struct ArrowSchema`, `struct ArrowArray`, or `struct ArrowArrayStream`. ### Value A double representing the pointer's address. ## nanoarrow_pointer_addr_chr(ptr) ### Description Returns a character representation of the pointer's address. ### Arguments * `ptr` - An external pointer to a `struct ArrowSchema`, `struct ArrowArray`, or `struct ArrowArrayStream`. ### Value A character string representing the pointer's address. ## nanoarrow_pointer_addr_pretty(ptr) ### Description Returns a human-readable representation of the pointer's address. ### Arguments * `ptr` - An external pointer to a `struct ArrowSchema`, `struct ArrowArray`, or `struct ArrowArrayStream`. ### Value A character string representing the pointer's address in a pretty format. ``` -------------------------------- ### ArrowDeviceArrayViewInit Source: https://arrow.apache.org/nanoarrow/latest/reference/device.html Initializes an ArrowDeviceArrayView by zeroing its memory. Further initialization of the array_view member is required. ```APIDOC ## ArrowDeviceArrayViewInit ### Description Initialize an ArrowDeviceArrayView. Zeroes memory for the device array view struct. Callers must initialize the array_view member using nanoarrow core functions that can initialize from a type identifier or schema. ### Signature ```c NANOARROW_DLL void ArrowDeviceArrayViewInit (struct ArrowDeviceArrayView *device_array_view) ``` ### Parameters * `device_array_view` (struct ArrowDeviceArrayView *) - The ArrowDeviceArrayView to initialize. ``` -------------------------------- ### CMakeLists.txt for nanoarrow Project Source: https://arrow.apache.org/nanoarrow/latest/_sources/getting-started/cpp.rst.txt A minimal CMakeLists.txt file to build a C++ library using nanoarrow. It includes fetching nanoarrow via FetchContent and linking against the static library. ```cmake project(linesplitter) set(CMAKE_CXX_STANDARD 11) include(FetchContent) FetchContent_Declare( nanoarrow URL https://github.com/apache/arrow-nanoarrow/releases/download/apache-arrow-nanoarrow-0.2.0/apache-arrow-nanoarrow-0.2.0.tar.gz URL_HASH SHA512=38a100ae5c36a33aa330010eb27b051cff98671e9c82fff22b1692bb77ae61bd6dc2a52ac6922c6c8657bd4c79a059ab26e8413de8169eeed3c9b7fdb216c817) FetchContent_MakeAvailable(nanoarrow) add_library(linesplitter linesplitter.cc) target_link_libraries(linesplitter PRIVATE nanoarrow_static) ``` -------------------------------- ### Configure nanoarrow Build with Meson and Arrow Dependency Source: https://arrow.apache.org/nanoarrow Configure nanoarrow build with Meson, enabling tests that depend on Apache Arrow. May require specifying pkg-config path. ```meson meson configure -Dtest_with_arrow=enabled --pkg-config-path ``` -------------------------------- ### Get type parameters Source: https://arrow.apache.org/nanoarrow/latest/reference/python/schema.html Returns a dictionary of parameters used to construct the type, useful for reconstruction. ```python import nanoarrow as na na.fixed_size_binary(123).params ``` -------------------------------- ### Create Binary View Schema Source: https://arrow.apache.org/nanoarrow/latest/reference/python/schema.html Create a nanoarrow binary view schema. Use `False` for nullable to mark as non-nullable. ```python >>> import nanoarrow as na >>> na.binary_view() binary_view ``` -------------------------------- ### ArrowIpcOutputStreamInitBuffer Source: https://arrow.apache.org/nanoarrow/latest/reference/ipc.html Initializes an output stream to write to an ArrowBuffer. Bytes written are appended to the buffer, and the stream does not take ownership. ```APIDOC ## ArrowIpcOutputStreamInitBuffer ### Description Create an output stream from an ArrowBuffer. All bytes witten to the stream will be appended to the buffer. The stream does not take ownership of the buffer. ``` -------------------------------- ### Get map key type Source: https://arrow.apache.org/nanoarrow/latest/reference/python/schema.html Retrieves the data type used for the keys in a map type. ```python import nanoarrow as na na.map_(na.int32(), na.string()).key_type ``` -------------------------------- ### void (*release)(struct ArrowDevice *device) Source: https://arrow.apache.org/nanoarrow/latest/reference/device.html Releases the device and any resources it holds. ```APIDOC ## release ### Description Release this device and any resources it holds. ### Signature void (*release)(struct ArrowDevice *device) ``` -------------------------------- ### Get dictionary index type Source: https://arrow.apache.org/nanoarrow/latest/reference/python/schema.html Retrieves the data type used for the indices in a dictionary-encoded type. ```python import nanoarrow as na na.dictionary(na.int32(), na.string()).index_type ``` -------------------------------- ### Allocate and Export C Array Source: https://arrow.apache.org/nanoarrow/latest/reference/python/advanced.html Demonstrates allocating an uninitialized ArrowArray and exporting a pyarrow array into it. ```python >>> import pyarrow as pa >>> from nanoarrow.c_array import allocate_c_array >>> array = allocate_c_array() >>> pa.array([1, 2, 3])._export_to_c(array._addr()) ``` -------------------------------- ### Bitmap Utilities Source: https://arrow.apache.org/nanoarrow/latest/reference/c.html Functions for manipulating bitmaps, including getting and setting bits, and unpacking bit sequences. ```APIDOC ## Bitmap Utilities ### Description Functions for manipulating bitmaps, including getting and setting bits, and unpacking bit sequences. ### Functions - `ArrowBitGet()`: Gets the value of a bit at a specific index. - `ArrowBitSet()`: Sets a bit at a specific index. - `ArrowBitClear()`: Clears a bit at a specific index. - `ArrowBitSetTo()`: Sets a range of bits to a specific value. - `ArrowBitsSetTo()`: Sets a range of bits to a specific value (plural). - `ArrowBitCountSet()`: Counts the number of set bits in a range. - `ArrowBitsUnpackInt8()`: Unpacks int8_t values from a bitmap. - `ArrowBitsUnpackInt32()`: Unpacks int32_t values from a bitmap. - `ArrowBitmapInit()`: Initializes an ArrowBitmap. - `ArrowBitmapMove()`: Moves an ArrowBitmap. - `ArrowBitmapReserve()`: Reserves capacity for an ArrowBitmap. - `ArrowBitmapResize()`: Resizes an ArrowBitmap. - `ArrowBitmapAppend()`: Appends bits to the bitmap. - `ArrowBitmapAppendUnsafe()`: Appends bits to the bitmap without checking capacity (unsafe). - `ArrowBitmapAppendInt8Unsafe()`: Appends int8_t values to the bitmap (unsafe). - `ArrowBitmapAppendInt32Unsafe()`: Appends int32_t values to the bitmap (unsafe). - `ArrowBitmapReset()`: Resets an ArrowBitmap. ### Structs - `ArrowBitmap`: - `buffer`: Pointer to the underlying buffer. - `size_bits`: The number of bits in the bitmap. ``` -------------------------------- ### Create an Int64 Schema Source: https://arrow.apache.org/nanoarrow/latest/reference/python/schema.html Use `na.int64` to create a schema for signed 64-bit integers. ```python import nanoarrow as na na.int64() ``` -------------------------------- ### ArrowTypeString Function Source: https://arrow.apache.org/nanoarrow/latest/reference/c.html Get a string representation of an enum ArrowType value. Returns NULL for invalid types. ```APIDOC static inline const char *ArrowTypeString(enum ArrowType type)# Get a string value of an enum ArrowType value. Returns NULL for invalid values for type ``` -------------------------------- ### Create Binary Schema Source: https://arrow.apache.org/nanoarrow/latest/reference/python/schema.html Create a nanoarrow binary schema. Use `False` for nullable to mark as non-nullable. ```python >>> import nanoarrow as na >>> na.binary() binary ``` -------------------------------- ### Get Dense Union Type Codes Source: https://arrow.apache.org/nanoarrow/latest/reference/python/schema.html Access the type codes for a nanoarrow dense union type. ```python >>> import nanoarrow as na >>> na.dense_union([na.int32(), na.string()]).type_codes [0, 1] ``` -------------------------------- ### Create a Fixed-Size Binary Schema Source: https://arrow.apache.org/nanoarrow/latest/reference/python/schema.html Use `na.fixed_size_binary` to create a schema for fixed-width binary data. Specify the byte width for each element. ```python import nanoarrow as na na.fixed_size_binary(123) ``` -------------------------------- ### Get field name Source: https://arrow.apache.org/nanoarrow/latest/reference/python/schema.html Retrieves the name of a specific field within a struct type using its index. ```python import nanoarrow as na schema = na.struct({"col1": na.int32()}) schema.field(0).name ``` -------------------------------- ### ArrowTimeUnitString Function Source: https://arrow.apache.org/nanoarrow/latest/reference/c.html Get a string representation of an enum ArrowTimeUnit value. Returns NULL for invalid time units. ```APIDOC static inline const char *ArrowTimeUnitString(enum ArrowTimeUnit time_unit)# Get a string value of an enum ArrowTimeUnit value. Returns NULL for invalid values for time_unit ``` -------------------------------- ### Create Arrow Array Directly from Buffers Source: https://arrow.apache.org/nanoarrow/latest/getting-started/python.html Use `na.c_array_from_buffers()` for advanced users to construct CArrays directly from buffer views and a schema. ```python na.c_array_from_buffers( na.string(), 2, [None, na.c_buffer([0, 3, 6], na.int32()), b"abcdef"] ) ``` -------------------------------- ### Get number of child Schemas Source: https://arrow.apache.org/nanoarrow/latest/reference/python/schema.html Returns the total count of child schemas (fields) within a struct type. ```python import nanoarrow as na schema = na.struct({"col1": na.int32()}) schema.n_fields ``` -------------------------------- ### C: Verbose Error Handling for Schema Initialization Source: https://arrow.apache.org/nanoarrow/latest/getting-started/cpp.html Illustrates manual error checking after calling ArrowSchemaInitFromType. This approach requires explicit checks for NANOARROW_OK after each potentially failing operation. ```c int init_string_non_null(struct ArrowSchema* schema) { int code = ArrowSchemaInitFromType(&schema, NANOARROW_TYPE_STRING); if (code != NANOARROW_OK) { return code; } schema->flags &= ~ARROW_FLAG_NULLABLE; return NANOARROW_OK; } ``` -------------------------------- ### Library Build Information Source: https://arrow.apache.org/nanoarrow/latest/r/reference/index.html Information about the underlying 'nanoarrow' C library build. ```APIDOC ## nanoarrow_version() ### Description Get the version of the underlying 'nanoarrow' C library. ### Function Signature `nanoarrow_version()` ``` ```APIDOC ## nanoarrow_with_zstd() ### Description Check if the underlying 'nanoarrow' C library was built with Zstandard support. ### Function Signature `nanoarrow_with_zstd()` ``` -------------------------------- ### Get fixed-size list element size Source: https://arrow.apache.org/nanoarrow/latest/reference/python/schema.html Retrieves the fixed number of elements allowed in a fixed-size list type. ```python import nanoarrow as na na.fixed_size_list(na.int32(), 123).list_size ``` -------------------------------- ### Infer nanoarrow Schema Source: https://arrow.apache.org/nanoarrow/latest/r Use `infer_nanoarrow_schema()` to get the ArrowSchema object that corresponds to a given R vector type. ```APIDOC ## infer_nanoarrow_schema() ### Description Infers the ArrowSchema for a given R vector. ### Usage infer_nanoarrow_schema(x) ### Arguments * `x` - An R vector. ### Returns A `` object. ### Example ```R infer_nanoarrow_schema(1:5) ``` ``` -------------------------------- ### ArrowIpcInputStreamInitBuffer Source: https://arrow.apache.org/nanoarrow/latest/reference/ipc.html Initializes an input stream to read from an ArrowBuffer, taking ownership of the buffer. ```APIDOC ## ArrowIpcInputStreamInitBuffer ### Description Create an input stream from an ArrowBuffer. The stream takes ownership of the buffer and reads bytes from it. ``` -------------------------------- ### ArrowDeviceArrayStream Source: https://arrow.apache.org/nanoarrow/latest/reference/device.html Represents a stream of ArrowDeviceArrays on a specific device, providing methods to get schema, next batch, and last error. ```APIDOC ## ArrowDeviceArrayStream ### Description Represents a stream of ArrowDeviceArrays on a specific device. ### Public Members - **device_type** (*ArrowDeviceType*): The type of the device. - **get_schema** (*int (*)(struct ArrowDeviceArrayStream*, struct ArrowSchema*)*): Function pointer to get the schema of the stream. - **get_next** (*int (*)(struct ArrowDeviceArrayStream*, struct ArrowDeviceArray*)*): Function pointer to get the next ArrowDeviceArray from the stream. - **get_last_error** (*const char *(*)(struct ArrowDeviceArrayStream*)*): Function pointer to get the last error message from the stream. - **release** (*void (*)(struct ArrowDeviceArrayStream*)*): Function pointer to release the stream resources. - **private_data** (*void **): Private data pointer for stream implementation. ``` -------------------------------- ### uint64() Source: https://arrow.apache.org/nanoarrow/latest/reference/python/schema.html Create an instance of an unsigned 64-bit integer type. ```APIDOC ## uint64(_nullable : bool = True_) -> Schema Create an instance of an unsigned 32-bit integer type. ## Parameters# nullablebool, optional Use `False` to mark this field as non-nullable. ## Examples# ``` >>> import nanoarrow as na >>> na.uint64() uint64 ``` ``` -------------------------------- ### Get the schema of an Array Source: https://arrow.apache.org/nanoarrow/latest/reference/python/array.html The `schema` property returns the nanoarrow Schema object associated with the array, defining its data type. ```python import nanoarrow as na array = na.Array([1, 2, 3], na.int32()) array.schema ``` -------------------------------- ### Bundle nanoarrow C Library with Python Source: https://arrow.apache.org/nanoarrow Generate bundled versions of the nanoarrow C library components using a Python script. This is useful for creating self-contained distributions. ```bash python ci/scripts/bundle.py \ --source-output-dir=dist \ --include-output-dir=dist \ --header-namespace= \ --with-device \ --with-ipc \ --with-testing \ --with-flatcc ``` -------------------------------- ### Apache License 2.0 Boilerplate Source: https://arrow.apache.org/nanoarrow/latest/r/LICENSE.html Use this text as a notice when applying the Apache License to your project. Replace bracketed information with your own details and enclose in appropriate comment syntax. ```text Copyright [yyyy] [name of copyright owner] Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. ``` -------------------------------- ### Convert nanoarrow Array back to R Source: https://arrow.apache.org/nanoarrow/latest/r Use `as.vector()` or `as.data.frame()` to get the R representation of the ArrowArray object back. ```APIDOC ## as.vector() and as.data.frame() for nanoarrow Arrays ### Description Converts a nanoarrow array back to its R representation. ### Usage as.vector(x) as.data.frame(x) ### Arguments * `x` - A `` object. ### Returns An R vector or data frame. ### Example ```R array <- as_nanoarrow_array(data.frame(col1 = c(1.1, 2.2))) as.data.frame(array) ``` ``` -------------------------------- ### Infer nanoarrow Schema from R Vector Source: https://arrow.apache.org/nanoarrow/latest/_sources/getting-started/r.rst.txt Use infer_nanoarrow_schema() to get the ArrowSchema object corresponding to an R vector type. ```r infer_nanoarrow_schema(1:5) ``` -------------------------------- ### nanoarrow_buffer_init Source: https://arrow.apache.org/nanoarrow/latest/r/reference/nanoarrow_buffer_init.html Initializes an empty nanoarrow buffer. ```APIDOC ## Function: nanoarrow_buffer_init ### Description Initializes an empty nanoarrow buffer. ### Usage ``` nanoarrow_buffer_init() ``` ### Value An object of class 'nanoarrow_buffer'. ``` -------------------------------- ### Get byte width for fixed-size binary Source: https://arrow.apache.org/nanoarrow/latest/reference/python/schema.html Retrieves the byte width for a fixed-size binary type. Returns None for types where this is not applicable. ```python import nanoarrow as na na.fixed_size_binary(123).byte_width ``` -------------------------------- ### Create an Int32 Schema Source: https://arrow.apache.org/nanoarrow/latest/reference/python/schema.html Use `na.int32` to create a schema for signed 32-bit integers. ```python import nanoarrow as na na.int32() ```