### Install Dependencies Source: https://idl.uw.edu/arquero Run this command after cloning the repository to install all necessary project dependencies. ```bash npm i ``` -------------------------------- ### Arquero Data Transformation Example Source: https://idl.uw.edu/arquero Demonstrates deriving new columns, selecting, sorting, and printing data transformations using Arquero. Requires importing 'all', 'desc', 'op', and 'table'. ```javascript import { all, desc, op, table } from 'arquero'; // Average hours of sunshine per month, from https://usclimatedata.com/. const dt = table({ 'Seattle': [69, 108, 178, 207, 253, 268, 312, 281, 221, 142, 72, 52], 'Chicago': [135, 136, 187, 215, 281, 311, 318, 283, 226, 193, 113, 106], 'San Francisco': [165, 182, 251, 281, 314, 330, 300, 272, 267, 243, 189, 156] }); // Sorted differences between Seattle and Chicago. // Table expressions use arrow function syntax. dt.derive({ month: d => op.row_number(), diff: d => d.Seattle - d.Chicago }) .select('month', 'diff') .orderby(desc('diff')) .print(); // Is Seattle more correlated with San Francisco or Chicago? // Operations accept column name strings outside a function context. dt.rollup({ corr_sf: op.corr('Seattle', 'San Francisco'), corr_chi: op.corr('Seattle', 'Chicago') }) .print(); // Aggregate statistics per city, as output objects. // Reshape (fold) the data to a two column layout: city, sun. dt.fold(all(), { as: ['city', 'sun'] }) .groupby('city') .rollup({ min: d => op.min(d.sun), // functional form of op.min('sun') max: d => op.max(d.sun), avg: d => op.average(d.sun), med: d => op.median(d.sun), // functional forms permit flexible table expressions skew: ({sun: s}) => (op.mean(s) - op.median(s)) / op.stdev(s) || 0 }) .objects() ``` -------------------------------- ### String Splitting and Searching Functions Source: https://idl.uw.edu/arquero/api/op Functions to split strings into arrays and check for starting substrings. ```APIDOC ## _op_.**split**(_value_ , _separator_[, _limit_]) ### Description Divides a string _value_ into an ordered list of substrings based on a _separator_ pattern, puts these substrings into an array, and returns the array. ### Parameters #### Path Parameters - **value** (string) - Required - The input string value. - **separator** (string | RegExp) - Required - A string or regular expression pattern describing where each split should occur. - **limit** (number) - Optional - An integer specifying a limit on the number of substrings to be included in the array. ## _op_.**startswith**(_value_ , _search_[, _position_]) ### Description Determines whether a string _value_ starts with the characters of a specified _search_ string, returning `true` or `false` as appropriate. ### Parameters #### Path Parameters - **value** (string) - Required - The input string value. - **search** (string) - Required - The search string to test for. - **position** (number) - Optional - The position in the _value_ string at which to begin searching (default `0`). ``` -------------------------------- ### String Padding Functions Source: https://idl.uw.edu/arquero/api/op Functions to pad strings to a specified length from the end or start. ```APIDOC ## _op_.**padend**(_value_ , _length_[, _fill_]) ### Description Pad a string _value_ with a given _fill_ string (applied from the end of _value_ and repeated, if needed) so that the resulting string reaches a given _length_. ### Parameters #### Path Parameters - **value** (string) - Required - The input string to pad. - **length** (number) - Required - The length of the resulting string once the _value_ string has been padded. If the length is lower than `value.length`, the _value_ string will be returned as-is. - **fill** (string) - Optional - The string to pad the _value_ string with (default `''`). If _fill_ is too long to stay within the target _length_ , it will be truncated. ## _op_.**padstart**(_value_ , _length_[, _fill_]) ### Description Pad a string _value_ with a given _fill_ string (applied from the start of _value_ and repeated, if needed) so that the resulting string reaches a given _length_. ### Parameters #### Path Parameters - **value** (string) - Required - The input string to pad. - **length** (number) - Required - The length of the resulting string once the _value_ string has been padded. If the length is lower than `value.length`, the _value_ string will be returned as-is. - **fill** (string) - Optional - The string to pad the _value_ string with (default `''`). If _fill_ is too long to stay within the target _length_ , it will be truncated. ``` -------------------------------- ### Select Columns by Prefix with Arquero Source: https://idl.uw.edu/arquero/api/index Select columns whose names start with a specified string. Returns a function-valued selection compatible with other transformation verbs. ```javascript aq.startswith('prefix_') ``` -------------------------------- ### Two-Table Column Shorthands for Join Source: https://idl.uw.edu/arquero/api/expressions Provides examples of using two-element arrays as shorthands for specifying columns in join operations between two tables. ```javascript table.join(other, ['x', 'u'], [['x', 'y'], 'v']) ``` ```javascript table.join(other, [['x'], ['u']], [['x', 'y'], ['v']]) ``` ```javascript table.join(other, ['x', 'u'], [aq.all(), aq.not('u')]) ``` -------------------------------- ### Parse Fixed-Width Stream with Arquero Source: https://idl.uw.edu/arquero/api/index Use `aq.fromFixedStream` to parse a ReadableStream of fixed-width text. Options include column positions or widths, names, and custom parsing. This example demonstrates parsing from a compressed stream. ```javascript // parse fixed width text from a compressed input stream // (these stream transforms are performed internally by loadFixed) const stream = (await fetch(url).then(res => res.body)) .pipeThrough(new DecompressionStream('gzip')) // decompress bytes .pipeThrough(new TextDecoderStream()); // map bytes to strings await aq.fromFixedStream(stream); ``` -------------------------------- ### Rename Columns by Index Source: https://idl.uw.edu/arquero/api/verbs This example shows how to rename columns based on their integer index using the `aq.names()` helper function. This is useful when column names are not known or when operating on columns by position. ```javascript table.rename(aq.names('colA2', 'colB2')) ``` -------------------------------- ### Encode Table to Arrow IPC and Decode Back Source: https://idl.uw.edu/arquero/api/index This example demonstrates how to encode an Arquero table into Arrow IPC bytes using toArrowIPC() and then decode those bytes back into an Arquero table using fromArrow(). ```javascript // encode input table as Arrow IPC bytes const arrowBytes = aq.table({ x: [1, 2, 3, 4, 5], y: [3.4, 1.6, 5.4, 7.1, 2.9] }) .toArrowIPC(); // access the Arrow-encoded data as an Arquero table const dt = aq.fromArrow(arrowBytes); ``` -------------------------------- ### Build Project Source: https://idl.uw.edu/arquero Run this command to build the output files for the Arquero project. ```bash npm run build ``` -------------------------------- ### Get Column Instance by Index Source: https://idl.uw.edu/arquero/api/table Use columnAt() to get a column instance by its zero-based index. Similar to column(), the returned object provides 'length' and 'get(row)' but is table-agnostic regarding filters or order. ```javascript const dt = aq.table({ a: [1, 2, 3], b: [4, 5, 6] }) dt.columnAt(1).get(1) // 5 ``` -------------------------------- ### Get Number of Active Rows Source: https://idl.uw.edu/arquero/api/table Use numRows() to get the count of non-filtered rows. This count may be less than the total rows if the table has been filtered. ```javascript aq.table({ a: [1, 2, 3], b: [4, 5, 6] }).numRows() // 3 ``` ```javascript aq.table({ a: [1, 2, 3], b: [4, 5, 6] }).filter(d => d.a > 2).numRows() // 1 ``` -------------------------------- ### Get Column Instance by Name Source: https://idl.uw.edu/arquero/api/table The column() method retrieves a column instance by its name. The returned object has 'length' and 'get(row)' properties but does not track table filters or order. ```javascript const dt = aq.table({ a: [1, 2, 3], b: [4, 5, 6] }) dt.column('b').get(1) // 5 ``` -------------------------------- ### Run Tests Source: https://idl.uw.edu/arquero Execute this command to run the project's test suite. ```bash npm test ``` -------------------------------- ### Set and Get Table Parameters Source: https://idl.uw.edu/arquero/api/table Use params() to get or set table expression parameters. When called with an object, it adds/updates parameters and returns the table. Without arguments, it returns the current parameters. ```javascript table.params({ hi: 5 }).filter((d, $) => abs(d.value) < $.hi) ``` -------------------------------- ### Run Performance Benchmarks Source: https://idl.uw.edu/arquero Use this command to execute the performance benchmarks for the project. ```bash npm run perf ``` -------------------------------- ### Get Column Values Iterator Source: https://idl.uw.edu/arquero/api/table Returns an iterator over the values in a specified column. The iterator respects any active table filters or ordering. This is a slightly less efficient way to get all column values compared to `table.array()`. ```javascript for (const value of table.values('colA')) { // do something with ordered values from column A } ``` ```javascript // slightly less efficient version of table.array('colA') const colValues = Array.from(table.values('colA')); ``` -------------------------------- ### Create Table from Object Source: https://idl.uw.edu/arquero/api/index Use `aq.from()` with an object to create a two-column table with 'key' and 'value' columns. ```javascript aq.from({ a: 1, b: 2, c: 3 }) ``` -------------------------------- ### Load Arquero from CDN in Browser Source: https://idl.uw.edu/arquero Load the Arquero library from a content delivery network for use in a web browser. Arquero will be available as the global `aq` object. ```html ``` -------------------------------- ### Get Absolute Value Source: https://idl.uw.edu/arquero/api/op Returns the absolute value of the input number. Equivalent to Math.abs. ```javascript op.abs(value) ``` -------------------------------- ### Create Table from Map Source: https://idl.uw.edu/arquero/api/index Use `aq.from()` with a Map to create a two-column table with 'key' and 'value' columns. ```javascript aq.from(new Map([ ['d', 4], ['e', 5], ['f', 6] ]) ``` -------------------------------- ### Column Shorthands for Groupby Source: https://idl.uw.edu/arquero/api/expressions Demonstrates various ways to specify columns for the `groupby()` operation, including by name, index, and range. ```javascript table.groupby('colA', 'colB') ``` ```javascript table.groupby(['colA', 'colB']) ``` ```javascript table.groupby(0, 1) ``` ```javascript table.groupby(aq.range(0, 1)) ``` ```javascript table.groupby({ colA: d => d.colA, colB: d => d.colB }) ``` -------------------------------- ### Get UTC Millisecond Source: https://idl.uw.edu/arquero/api/op Extracts the milliseconds of the second from a date according to Coordinated Universal Time (UTC). ```javascript op.utcmilliseconds(date) ``` -------------------------------- ### Table Constructors: aq.table() Source: https://idl.uw.edu/arquero/api/index Creates a new table instance from columns, optionally specifying column names and order. ```APIDOC ## aq.table(columns[, names]) ### Description Create a new table for a set of named _columns_, optionally including an array of ordered column _names_. The _columns_ input can be an object or Map with names for keys and columns for values, or an entry array of `[name, values]` tuples. ### Method `table` ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body * **columns** (Object | Map | Array) - An object or Map providing a named set of column arrays, or an entries array of the form `[[name, values], ...]`. Keys are column name strings; the enumeration order of the keys determines the column indices if the _names_ argument is not provided. Column values should be arrays (or array-like values) of identical length. * **names** (Array) - An array of column names, specifying the index order of columns in the table. ### Request Example ```javascript aq.table({ colA: ['a', 'b', 'c'], colB: [3, 4, 5] }) ``` ### Response #### Success Response (200) * **Table** - A new Arquero table instance. #### Response Example ```json { "example": "[Table instance]" } ``` ``` -------------------------------- ### Get UTC Second Source: https://idl.uw.edu/arquero/api/op Extracts the seconds of the minute from a date according to Coordinated Universal Time (UTC). ```javascript op.utcseconds(date) ``` -------------------------------- ### Using aq.names with table.rename() Source: https://idl.uw.edu/arquero/api/index Demonstrates how to use the `aq.names` helper function with the `table.rename()` method to rename specific columns in a table. ```APIDOC ## table.rename(aq.names()) ### Description Renames the first specified number of columns in a table using the names provided by `aq.names()`. All other columns remain as-is. ### Method N/A (Usage within table.rename) ### Endpoint N/A ### Parameters N/A (Parameters are passed to aq.names) ### Request Example ```javascript // rename the first three columns, all other columns remain as-is table.rename(aq.names(['a', 'b', 'c'])) ``` ### Response #### Success Response (200) Returns a new table with the specified columns renamed. #### Response Example N/A (Output is a table) ``` -------------------------------- ### Get UTC Minute Source: https://idl.uw.edu/arquero/api/op Extracts the minute of the hour from a date according to Coordinated Universal Time (UTC). ```javascript op.utcminutes(date) ``` -------------------------------- ### Uniform Binning with aq.bin() Source: https://idl.uw.edu/arquero/api/index Generates a table expression for uniform binning of numeric values. Options include maxbins, minstep, nice, offset, and step to control binning behavior. ```javascript aq.bin('colA', { maxbins: 20 }) ``` -------------------------------- ### fromArrow - Load Data from Apache Arrow Source: https://idl.uw.edu/arquero/api/index Returns a new table backed by Apache Arrow binary data. The input can be a byte array in the Arrow IPC format, or an instantiated Flechette or Apache Arrow JS table instance. ```APIDOC ## _aq_.**fromArrow**(_input_[, _options_]) ### Description Returns a new table backed by Apache Arrow binary data. The _input_ can be a byte array in the Arrow IPC format, or an instantiated Flechette or Apache Arrow JS table instance. Binary inputs are decoded using Flechette. For many data types, Arquero uses binary-encoded Arrow columns as-is with zero data copying. For dictionary columns, Arquero unpacks columns with `null` entries or containing multiple record batches to optimize query performance. Both the Arrow IPC `stream` and `file` formats are supported; the format type is determined automatically. This method performs parsing only. To specify a URL or file to load, use loadArrow. ### Method N/A (This is a function call, not a direct HTTP endpoint) ### Endpoint N/A ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Parameters * _input_ : A byte array (e.g., ArrayBuffer or Uint8Array) in the Arrow IPC format, or a Flechette or Apache Arrow JS table instance. * _options_ : An Arrow import options object. * _columns_ (`Select`): An ordered set of columns to import. The input may consist of: column name strings, column integer indices, objects with current column names as keys and new column names as values (for renaming), or a selection helper function such as all, not, or range. * _useBigInt_ (`boolean`): Boolean flag (default `false`) to extract 64-bit integer types as JavaScript `BigInt` values. For Flechette tables, the default is to coerce 64-bit integers to JavaScript numbers and raise an error if the number is out of range. This option is only applied when parsing IPC binary data, otherwise the settings of the provided table instance are used. * _useDate_ (`boolean`): Boolean flag (default `true`) to convert Arrow date and timestamp values to JavaScript Date objects. Otherwise, numeric timestamps are used. This option is only applied when parsing IPC binary data, otherwise the settings of the provided table instance are used. * _useDecimalBigInt_ (`boolean`): Boolean flag (default `false`) to extract Arrow decimal-type data as BigInt values, where fractional digits are scaled to integers. Otherwise, decimals are (sometimes lossily) converted to floating-point numbers (default). This option is only applied when parsing IPC binary data, otherwise the settings of the provided table instance are used. * _useMap_ (`boolean`): Boolean flag (default `false`) to represent Arrow Map data as JavaScript `Map` values. For Flechette tables, the default is to produce an array of `[key, value]` arrays. This option is only applied when parsing IPC binary data, otherwise the settings of the provided table instance are used. * _useProxy_ (`boolean`): Boolean flag (default `false`) to extract Arrow Struct values and table row objects using zero-copy proxy objects that extract data from underlying Arrow batches. The proxy objects can improve performance and reduce memory usage, but do not support property enumeration (`Object.keys`, `Object.values`, `Object.entries`) or spreading (`{ ...object }`). This option is only applied when parsing IPC binary data, otherwise the settings of the provided table instance are used. ### Request Example ```javascript // encode input table as Arrow IPC bytes const arrowBytes = aq.table({ x: [1, 2, 3, 4, 5], y: [3.4, 1.6, 5.4, 7.1, 2.9] }) .toArrowIPC(); // access the Arrow-encoded data as an Arquero table const dt = aq.fromArrow(arrowBytes); ``` ### Response #### Success Response (200) An Arquero table object. #### Response Example (The response is the table object itself, not a JSON representation in this context.) ``` -------------------------------- ### Get UTC Hour Source: https://idl.uw.edu/arquero/api/op Extracts the hour of the day from a date according to Coordinated Universal Time (UTC). ```javascript op.utchours(date) ``` -------------------------------- ### Get Number of Columns Source: https://idl.uw.edu/arquero/api/table Use numCols() to retrieve the number of columns in a table. This method returns an integer. ```javascript aq.table({ a: [1, 2, 3], b: [4, 5, 6] }).numCols() // 2 ``` -------------------------------- ### Using aq.names with table.select() Source: https://idl.uw.edu/arquero/api/index Demonstrates how to use the `aq.names` helper function with the `table.select()` method to select and rename specific columns, dropping all others. ```APIDOC ## table.select(aq.names()) ### Description Selects and renames the first specified number of columns in a table using the names provided by `aq.names()`. All other columns are dropped. ### Method N/A (Usage within table.select) ### Endpoint N/A ### Parameters N/A (Parameters are passed to aq.names) ### Request Example ```javascript // select and rename the first three columns, all other columns are dropped table.select(aq.names(['a', 'b', 'c'])) ``` ### Response #### Success Response (200) Returns a new table with only the selected and renamed columns. #### Response Example N/A (Output is a table) ``` -------------------------------- ### Get UTC Date Component Source: https://idl.uw.edu/arquero/api/op Extracts the day of the month from a date according to Coordinated Universal Time (UTC). ```javascript op.utcdate(date) ``` -------------------------------- ### Table Constructors: aq.from() Source: https://idl.uw.edu/arquero/api/index Creates a new table from existing data structures like arrays of objects or key-value pairs. ```APIDOC ## aq.from(values[, names]) ### Description Create a new table from an existing object, such as an array of objects or a set of key-value pairs. For varied JSON formats, see the `fromJSON` method. ### Method `from` ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body * **values** (Array | Object | Map | Iterable) - Data values to populate the table. If array-valued or iterable, imports rows for each non-null value, using the provided column names as keys for each row object. If no _names_ are provided, the first non-null object’s own keys are used. If an object or a Map, create a two-column table with columns for the input keys and values. * **names** (Array) - Column names to include. For object or Map inputs, specifies the key and value column names. Otherwise, specifies the keys to look up on each row object. ### Request Example ```javascript aq.from([ { colA: 1, colB: 2 }, { colA: 3, colB: 4 } ]) ``` ### Response #### Success Response (200) * **Table** - A new Arquero table instance. #### Response Example ```json { "example": "[Table instance]" } ``` ``` -------------------------------- ### Load Arrow File Source: https://idl.uw.edu/arquero/api/index Loads a file in the Apache Arrow IPC binary format from a URL or local file path and returns a Promise for a table. Supports both stream and file formats, automatically detected. ```APIDOC ## POST /websites/idl_uw_edu_arquero/loadArrow ### Description Loads a file in the Apache Arrow IPC binary format from a URL or local file path and returns a Promise for a table. Supports both stream and file formats, automatically detected. ### Method POST ### Endpoint /websites/idl_uw_edu_arquero/loadArrow ### Parameters #### Query Parameters - **url** (string) - Required - The url or local file path (node.js only) to load. - **options** (object) - Optional - File loading and Arrow formatting options. - **fetch** (RequestInit) - Optional - Options to pass to the HTTP fetch method when loading a URL. - **decompress** ('gzip' | 'deflate' | null) - Optional - A decompression format to apply. If unspecified, the decompression type is inferred from the file extension (.gz for 'gzip', .zz for 'deflate'). If no matching extension is found, no decompression is performed. - **columns** (Select) - Optional - An ordered set of columns to import. The input may consist of: column name strings, column integer indices, objects with current column names as keys and new column names as values (for renaming), or a selection helper function such as all, not, or range. - **useBigInt** (boolean) - Optional - Boolean flag (default false) to extract 64-bit integer types as JavaScript BigInt values. For Flechette tables, the default is to coerce 64-bit integers to JavaScript numbers and raise an error if the number is out of range. This option is only applied when parsing IPC binary data, otherwise the settings of the provided table instance are used. - **useDate** (boolean) - Optional - Boolean flag (default true) to convert Arrow date and timestamp values to JavaScript Date objects. Otherwise, numeric timestamps are used. This option is only applied when parsing IPC binary data, otherwise the settings of the provided table instance are used. - **useDecimalBigInt** (boolean) - Optional - Boolean flag (default false) to extract Arrow decimal-type data as BigInt values, where fractional digits are scaled to integers. Otherwise, decimals are (sometimes lossily) converted to floating-point numbers (default). This option is only applied when parsing IPC binary data, otherwise the settings of the provided table instance are used. - **useMap** (boolean) - Optional - Boolean flag (default false) to represent Arrow Map data as JavaScript Map values. For Flechette tables, the default is to produce an array of [key, value] arrays. This option is only applied when parsing IPC binary data, otherwise the settings of the provided table instance are used. - **useProxy** (boolean) - Optional - Boolean flag (default false) to extract Arrow Struct values and table row objects using zero-copy proxy objects that extract data from underlying Arrow batches. The proxy objects can improve performance and reduce memory usage, but do not support property enumeration (Object.keys, Object.values, Object.entries) or spreading ({ ...object }). This option is only applied when parsing IPC binary data, otherwise the settings of the provided table instance are used. ### Request Example { "url": "data/table.arrow", "options": { "columns": ["col1", "col2"], "useBigInt": true } } ### Response #### Success Response (200) - **table** (Table) - The loaded Apache Arrow table. #### Response Example { "table": { // Arrow table data structure } } ``` -------------------------------- ### Get UTC Day of Year Source: https://idl.uw.edu/arquero/api/op Extracts the day of the year (1-366) from a date according to Coordinated Universal Time (UTC). ```javascript op.utcdayofyear(date) ``` -------------------------------- ### Table Constructors: aq.from() Source: https://idl.uw.edu/arquero/api Creates a new table from existing data structures like arrays of objects or Maps. ```APIDOC ## aq.from(values[, names]) ### Description Create a new table from an existing object, such as an array of objects or a set of key-value pairs. For varied JSON formats, see the `fromJSON` method. ### Parameters #### Arguments - **values** (Array | Object | Map) - Data values to populate the table. If array-valued or iterable, imports rows for each non-null value, using the provided column names as keys for each row object. If no _names_ are provided, the first non-null object’s own keys are used. If an object or a Map, create a two-column table with columns for the input keys and values. - **names** (Array) - Column names to include. For object or Map inputs, specifies the key and value column names. Otherwise, specifies the keys to look up on each row object. ### Request Example ```javascript // from an array, create a new table with two columns and two rows // akin to table({ colA: [1, 3], colB: [2, 4] }) aq.from([ { colA: 1, colB: 2 }, { colA: 3, colB: 4 } ]) ``` ```javascript // from an object, create a new table with 'key' and 'value columns // akin to table({ key: ['a', 'b', 'c'], value: [1, 2, 3] }) aq.from({ a: 1, b: 2, c: 3 }) ``` ```javascript // from a Map, create a new table with 'key' and 'value' columns // akin to table({ key: ['d', 'e', 'f'], value: [4, 5, 6] }) aq.from(new Map([ ['d', 4], ['e', 5], ['f', 6] ])) ``` ``` -------------------------------- ### Get Column Name by Index Source: https://idl.uw.edu/arquero/api/table Retrieve the name of a column at a specific zero-based index. Returns undefined if the index is out of range. ```javascript aq.table({ a: [1, 2, 3], b: [4, 5, 6] }).columnName(1); // 'b' ``` -------------------------------- ### Get Table Size (Active Rows) Source: https://idl.uw.edu/arquero/api/table The 'size' property returns the number of active (non-filtered) rows, equivalent to numRows(). ```javascript aq.table({ a: [1, 2, 3], b: [4, 5, 6] }).size // 3 ``` ```javascript aq.table({ a: [1, 2, 3], b: [4, 5, 6] }).filter(d => d.a > 2).size // 1 ``` -------------------------------- ### Create Table from Array of Objects Source: https://idl.uw.edu/arquero/api/index Use `aq.from()` with an array of objects to create a new table. Column names are derived from the object keys. ```javascript aq.from([ { colA: 1, colB: 2 }, { colA: 3, colB: 4 } ]) ``` -------------------------------- ### Get UTC Day of Week Source: https://idl.uw.edu/arquero/api/op Extracts the Sunday-based day of the week (0-6) from a date according to Coordinated Universal Time (UTC). ```javascript op.utcdayofweek(date) ``` -------------------------------- ### Create Table from Columns Source: https://idl.uw.edu/arquero/api/index Use `aq.table()` to create a new table from named columns. Column order is determined by object key enumeration unless the `names` argument is provided. ```javascript aq.table({ colA: ['a', 'b', 'c'], colB: [3, 4, 5] }) ``` ```javascript aq.table({ key: ['a', 'b'], 1: [9, 8], 2: [7, 6] }, ['key', '1', '2']) ``` ```javascript const map = new Map() .set('colA', ['a', 'b', 'c']) .set('colB', [3, 4, 5]); aq.table(map) ``` -------------------------------- ### fromArrow - Load Data from Apache Arrow Source: https://idl.uw.edu/arquero/api Creates a new Arquero table from Apache Arrow binary data. Supports Arrow IPC stream and file formats, automatically detecting the format type. This method focuses on parsing the data. ```APIDOC ## _aq_.fromArrow(_input_[, _options_]) ### Description Returns a new table backed by Apache Arrow binary data. The _input_ can be a byte array in the Arrow IPC format, or an instantiated Flechette or Apache Arrow JS table instance. Binary inputs are decoded using Flechette. ### Method N/A (This is a function call, not a REST endpoint) ### Endpoint N/A ### Parameters #### Path Parameters N/A #### Query Parameters N/A #### Request Body N/A ### Parameters * **_input_** (ByteArray | FlechetteTable | ArrowTable) - Required - A byte array (e.g., ArrayBuffer or Uint8Array) in the Arrow IPC format, or a Flechette or Apache Arrow JS table instance. * **_options_** (Object) - Optional - An Arrow import options object. * **columns** (`Select`) - An ordered set of columns to import. Accepts column name strings, column integer indices, objects with current column names as keys and new column names as values (for renaming), or a selection helper function (e.g., `all`, `not`, `range`). * **useBigInt** (`boolean`) - Optional (default `false`) - If true, extracts 64-bit integer types as JavaScript `BigInt` values. For Flechette tables, the default is to coerce 64-bit integers to JavaScript numbers and raise an error if the number is out of range. This option is only applied when parsing IPC binary data. * **useDate** (`boolean`) - Optional (default `true`) - If true, converts Arrow date and timestamp values to JavaScript Date objects. Otherwise, numeric timestamps are used. This option is only applied when parsing IPC binary data. * **useDecimalBigInt** (`boolean`) - Optional (default `false`) - If true, extracts Arrow decimal-type data as BigInt values, where fractional digits are scaled to integers. Otherwise, decimals are (sometimes lossily) converted to floating-point numbers. This option is only applied when parsing IPC binary data. * **useMap** (`boolean`) - Optional (default `false`) - If true, represents Arrow Map data as JavaScript `Map` values. For Flechette tables, the default is to produce an array of `[key, value]` arrays. This option is only applied when parsing IPC binary data. * **useProxy** (`boolean`) - Optional (default `false`) - If true, extracts Arrow Struct values and table row objects using zero-copy proxy objects that extract data from underlying Arrow batches. This can improve performance and reduce memory usage but limits property enumeration and spreading. This option is only applied when parsing IPC binary data. ### Request Example ```javascript // encode input table as Arrow IPC bytes const arrowBytes = aq.table({ x: [1, 2, 3, 4, 5], y: [3.4, 1.6, 5.4, 7.1, 2.9] }) .toArrowIPC(); // access the Arrow-encoded data as an Arquero table const dt = aq.fromArrow(arrowBytes); ``` ### Response #### Success Response (200) * **Table** (ArqueroTable) - A new Arquero table instance backed by the Arrow data. #### Response Example ```javascript // Assuming dt is the result of aq.fromArrow(arrowBytes) console.log(dt.print()); ``` ### Error Handling Errors may occur if the input data is not valid Arrow IPC format or if there are issues during parsing. ``` -------------------------------- ### Get Column Index by Name Source: https://idl.uw.edu/arquero/api/table Retrieve the zero-based index of a column given its name. Returns -1 if the column name is not found. ```javascript aq.table({ a: [1, 2, 3], b: [4, 5, 6] }).columnIndex('b'); // 1 ``` -------------------------------- ### Print Table to Console Source: https://idl.uw.edu/arquero/api/table Display the table contents in the console using `console.table()`. Options can limit the number of rows, set an offset, and specify which columns to print. ```javascript aq.table({ a: [1, 2, 3], b: [4, 5, 6] }).print() // ┌─────────┬───┬───┐ // │ (index) │ a │ b │ // ├─────────┼───┼───┤ // │ 0 │ 1 │ 4 │ // │ 1 │ 2 │ 5 │ // │ 2 │ 3 │ 6 │ // └─────────┴───┴───┘ ``` -------------------------------- ### Get All Column Names Source: https://idl.uw.edu/arquero/api/table Returns an array of all column names in the table. An optional filter callback can be provided to return only names that satisfy the condition. ```javascript aq.table({ a: [1, 2, 3], b: [4, 5, 6] }).columnNames(); // [ 'a', 'b' ] ``` -------------------------------- ### Basic Math Operations Source: https://idl.uw.edu/arquero/api/op Provides access to fundamental mathematical operations such as cube root, ceiling, floor, and rounding. ```APIDOC ## _op_.cbrt(_value_) ### Description Returns the cube root value of the input _value_; equivalent to Math.cbrt. ### Method N/A (Function Call) ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Request Example ``` _op_.cbrt(27) ``` ### Response #### Success Response (200) - **result** (number) - The cube root of the input value. #### Response Example ```json { "result": 3 } ``` ## _op_.ceil(_value_) ### Description Returns the ceiling of the input _value_, the nearest integer equal to or greater than the input; equivalent to Math.ceil. ### Method N/A (Function Call) ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Request Example ``` _op_.ceil(4.2) ``` ### Response #### Success Response (200) - **result** (number) - The ceiling of the input value. #### Response Example ```json { "result": 5 } ``` ## _op_.clz32(_value_) ### Description Returns the number of leading zero bits in the 32-bit binary representation of a number _value_; equivalent to Math.clz32. ### Method N/A (Function Call) ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Request Example ``` _op_.clz32(1) ``` ### Response #### Success Response (200) - **result** (number) - The number of leading zero bits. #### Response Example ```json { "result": 31 } ``` ## _op_.floor(_value_) ### Description Returns the floor of the input _value_, the nearest integer equal to or less than the input; equivalent to Math.floor. ### Method N/A (Function Call) ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Request Example ``` _op_.floor(4.7) ``` ### Response #### Success Response (200) - **result** (number) - The floor of the input value. #### Response Example ```json { "result": 4 } ``` ## _op_.fround(_value_) ### Description Returns the nearest 32-bit single precision float representation of the input number _value_; equivalent to Math.fround. Useful for translating between 64-bit `Number` values and values from a `Float32Array`. ### Method N/A (Function Call) ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Request Example ``` _op_.fround(4.333333333333333) ``` ### Response #### Success Response (200) - **result** (number) - The nearest 32-bit single precision float representation. #### Response Example ```json { "result": 4.333333373069763 } ``` ## _op_.round(_value_) ### Description Returns the value of a number rounded to the nearest integer; equivalent to Math.round. ### Method N/A (Function Call) ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Request Example ``` _op_.round(4.7) ``` ### Response #### Success Response (200) - **result** (number) - The rounded integer value. #### Response Example ```json { "result": 5 } ``` ## _op_.trunc(_value_) ### Description Returns the integer part of a number by removing any fractional digits; equivalent to Math.trunc. ### Method N/A (Function Call) ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Request Example ``` _op_.trunc(4.7) ``` ### Response #### Success Response (200) - **result** (number) - The integer part of the number. #### Response Example ```json { "result": 4 } ``` ``` -------------------------------- ### aq.fromArrowStream(stream, options) Source: https://idl.uw.edu/arquero/api/index Returns a Promise to a new table backed by Apache Arrow binary data. The stream must be a ReadableStream of bytes, which is then decoded using Flechette. Both the Arrow IPC `stream` and `file` formats are supported; the format type is determined automatically. This method performs parsing only. ```APIDOC ## aq.fromArrowStream(stream, options) ### Description Returns a Promise to a new table backed by Apache Arrow binary data. The _stream_ must be a ReadableStream of bytes, which is then decoded using Flechette. For many data types, Arquero uses binary-encoded Arrow columns as-is with zero data copying. For dictionary columns, Arquero unpacks columns with `null` entries or containing multiple record batches to optimize query performance. Both the Arrow IPC `stream` and `file` formats are supported; the format type is determined automatically. This method performs parsing only. To specify a URL or file to load, use loadArrow. ### Method `aq.fromArrowStream` ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Parameters * _stream_ : A ReadableStream of bytes. * _options_ : An Arrow import options object. * _columns_ (`Select`): An ordered set of columns to import. The input may consist of: column name strings, column integer indices, objects with current column names as keys and new column names as values (for renaming), or a selection helper function such as all, not, or range. * _useBigInt_ (`boolean`): Boolean flag (default `false`) to extract 64-bit integer types as JavaScript `BigInt` values. For Flechette tables, the default is to coerce 64-bit integers to JavaScript numbers and raise an error if the number is out of range. This option is only applied when parsing IPC binary data, otherwise the settings of the provided table instance are used. * _useDate_ (`boolean`): Boolean flag (default `true`) to convert Arrow date and timestamp values to JavaScript Date objects. Otherwise, numeric timestamps are used. This option is only applied when parsing IPC binary data, otherwise the settings of the provided table instance are used. * _useDecimalBigInt_ (`boolean`): Boolean flag (default `false`) to extract Arrow decimal-type data as BigInt values, where fractional digits are scaled to integers. Otherwise, decimals are (sometimes lossily) converted to floating-point numbers (default). This option is only applied when parsing IPC binary data, otherwise the settings of the provided table instance are used. * _useMap_ (`boolean`): Boolean flag (default `false`) to represent Arrow Map data as JavaScript `Map` values. For Flechette tables, the default is to produce an array of `[key, value]` arrays. This option is only applied when parsing IPC binary data, otherwise the settings of the provided table instance are used. * _useProxy_ (`boolean`): Boolean flag (default `false`) to extract Arrow Struct values and table row objects using zero-copy proxy objects that extract data from underlying Arrow batches. The proxy objects can improve performance and reduce memory usage, but do not support property enumeration (`Object.keys`, `Object.values`, `Object.entries`) or spreading (`{ ...object }`). This option is only applied when parsing IPC binary data, otherwise the settings of the provided table instance are used. ### Request Example ```javascript // load table from an Apache Arrow file const dt = await aq.fromArrowStream(byteStream); ``` ### Response #### Success Response (200) A Promise that resolves to a new Arquero table. #### Response Example ```json { "example": "[Arquero table object]" } ``` ``` -------------------------------- ### Get Object Values Source: https://idl.uw.edu/arquero/api/op Returns an array of an object's own enumerable property values. Invokes Map/Set `values` method directly or uses `Object.values`. ```javascript op.values(object) ``` -------------------------------- ### aq.bin() Source: https://idl.uw.edu/arquero/api/index Generates a table expression for uniform binning of numeric values. Useful for creating bins for sampling or analysis. ```APIDOC ## aq.bin(_name_[, _options_]) ### Description Generate a table expression that performs uniform binning of number values. The resulting string can be used as part of the input to table transformation verbs. ### Parameters #### Path Parameters - **name** (string) - Required - The name of the column to bin. - **options** (object) - Optional - A binning scheme options object: - **maxbins** (number) - The maximum number of bins. - **minstep** (number) - The minimum step size between bins. - **nice** (boolean) - Default `true`. Indicates if bins should snap to “nice” human-friendly values such as multiples of ten. - **offset** (number) - Step offset for bin boundaries. The default (`0`) floors to the lower bin boundary. A value of `1` snaps one step higher to the upper bin boundary, and so on. - **step** (number) - The exact step size to use between bins. If specified, the _maxbins_ and _minstep_ options are ignored. ### Request Example ```javascript aq.bin('colA', { maxbins: 20 }) ``` ``` -------------------------------- ### Get Object Keys Source: https://idl.uw.edu/arquero/api/op Returns an array of an object's own enumerable property names. Invokes Map `keys` method directly or uses `Object.keys`. ```javascript op.keys(object) ``` -------------------------------- ### Select and Rename Columns with _aq_.names() Source: https://idl.uw.edu/arquero/api/index Use the rename map from _aq_.names() with the .select() method to select and rename specific columns. Columns not included in the map are dropped. ```javascript // select and rename the first three columns, all other columns are dropped table.select(aq.names(['a', 'b', 'c'])) ``` -------------------------------- ### Get Total Number of Rows Source: https://idl.uw.edu/arquero/api/table Use totalRows() to retrieve the total number of rows, including both filtered and unfiltered rows. This value is unaffected by filtering operations. ```javascript aq.table({ a: [1, 2, 3], b: [4, 5, 6] }).totalRows() // 3 ``` ```javascript aq.table({ a: [1, 2, 3], b: [4, 5, 6] }).filter(d => d.a > 2).totalRows() // 3 ``` -------------------------------- ### Aggregate & Window Shorthands Source: https://idl.uw.edu/arquero/api/expressions Compares standard table expressions with shorthand generators for aggregate and window functions like `op.mean`. ```javascript d => op.mean(d.value) ``` ```javascript op.mean('value') ``` -------------------------------- ### Rename Columns Using _aq_.names() Helper Source: https://idl.uw.edu/arquero/api/index Apply the rename map generated by _aq_.names() to a table using the .rename() method. This renames the first three columns, leaving others unchanged. ```javascript // rename the first three columns, all other columns remain as-is table.rename(aq.names(['a', 'b', 'c'])) ``` -------------------------------- ### Get Object Entries Source: https://idl.uw.edu/arquero/api/op Returns an array of an object's own enumerable string-keyed `[key, value]` pairs. Invokes Map/Set `entries` method directly or uses `Object.entries`. ```javascript op.entries(object) ```