Redivis

https://docs.redivis.com
Admin
Redivis is an online data platform for research that enables data owners to securely host and...
Tokens:280,533
Snippets:3,029
Trust Score:8.4
Update:6 months ago
Show doc for...
Tokens:
Raw
### Example: Run a notebook

Source: https://docs.redivis.com/api/client-libraries/redivis-r/reference/notebook

Demonstrates how to get a notebook reference and run it.

```APIDOC
## Example: Run a notebook

```r
username <- "imathews"
workflow_name <- "example_workflow_climate_analysis:x7kh"
notebook_name <- "visualize_precipitation_trends:2f2v"

notebook <- redivis$notebook(f"{username}.{workflow_name}.{notebook_name}")
notebook$run()
```
```

--------------------------------

### Redivis R Client Library - Getting Started

Source: https://docs.redivis.com/api/client-libraries

Instructions for getting started with the Redivis R client library.

```APIDOC
## Redivis R Client Library - Getting Started

This section provides instructions on how to install and begin using the Redivis R client library.
```

--------------------------------

### Example Starting Data for Order Step

Source: https://docs.redivis.com/reference/workflows/transforms/step-order

This is the initial data structure used in the examples for the Order step.

```text
/*---------+--------*
 | student | score  |
 +---------+--------+
 | jane    | 83     |
 | neal    | 35     |
 | sam     | 74     |
 | pat     | 62     |
 *---------+--------*/
```

--------------------------------

### Limit Step Example

Source: https://docs.redivis.com/reference/workflows/transforms/step-limit

Demonstrates the Limit step with more complex starting data. This is useful for quick iteration during exploratory work. The output shows a subset of the input rows.

```text
/*---------+-------+---------+------------*
 | test    | score | student | date       |
 +---------+-------+---------+------------+
 | quiz    | 83    | jane    | 2020-04-01 |
 | quiz    | 35    | pat     | 2020-04-01 |
 | quiz    | 89    | sam     | 2020-04-01 |
 | midterm | 74    | jane    | 2020-05-01 |
 | midterm | 62    | pat     | 2020-05-01 |
 | midterm | 93    | sam     | 2020-05-01 |
 | final   | 77    | jane    | 2020-06-01 |
 | final   | 59    | pat     | 2020-06-01 |
 | final   | 92    | sam     | 2020-06-01 |
 *---------+-------+---------+------------*/
```

```text
/*---------+-------+---------+------------*
 | test    | score | student | date       |
 +---------+-------+---------+------------+
 | quiz    | 89    | sam     | 2020-04-01 |
 | midterm | 62    | pat     | 2020-05-01 |
 | midterm | 93    | sam     | 2020-05-01 |
 | final   | 92    | sam     | 2020-06-01 |
 *---------+-------+---------+------------*/
```

--------------------------------

### Starting Data for Basic Order Example

Source: https://docs.redivis.com/reference/workflows/transforms/step-order

The initial dataset for demonstrating a basic ordering of a table by the 'score' variable.

```text
/*---------+-------+---------+------------*
 | test    | score | student | date       |
 +---------+-------+---------+------------+
 | quiz    | 83    | jane    | 2020-04-01 |
 | quiz    | 35    | pat     | 2020-04-01 |
 | quiz    | 89    | sam     | 2020-04-01 |
 | midterm | 74    | jane    | 2020-05-01 |
 | midterm | 62    | pat     | 2020-05-01 |
 | midterm | 100   | sam     | 2020-05-01 |
 | final   | 77    | jane    | 2020-06-01 |
 | final   | 59    | pat     | 2020-06-01 |
 | final   | 100   | sam     | 2020-06-01 |
 *---------+-------+---------+------------*/
```

--------------------------------

### Launch SAS Deployment Wizard

Source: https://docs.redivis.com/reference/organizations/settings/advanced-stata-and-sas-setup

Start the SAS installation wizard in the terminal using the console flag.

```bash
./setup.sh -console
```

--------------------------------

### R Example: Accessing and Running a Notebook

Source: https://docs.redivis.com/api/client-libraries/redivis-r/reference/workflow/workflowusdnotebook

Demonstrates how to get a workflow object, create a reference to a notebook within that workflow, and then execute the notebook using the Redivis R client library.

```APIDOC
## R Example: Workflow Notebook Interaction

### Description
This example shows how to use the Redivis R client library to interact with notebooks within a workflow. It covers obtaining a workflow object, referencing a specific notebook by name, and then running that notebook.

### Method
N/A (Client Library Usage)

### Endpoint
N/A (Client Library Usage)

### Code Example
```r
# Assuming 'redivis' library is loaded and authenticated

# Get a user object
user <- redivis$user("imathews")

# Get a workflow object by its name
workflow <- user$workflow("example_workflow_climate_analysis")

# Create a reference to a specific notebook within the workflow
# The 'name' parameter is a string representing the notebook's scoped reference.
notebook <- workflow$notebook('compute_annual_precipitation')

# Execute the referenced notebook
notebook$run()
```

### Response
(Output depends on the notebook's execution results and the `run()` method's return value.)
```

--------------------------------

### Curl Example to Get Transform

Source: https://docs.redivis.com/api/rest-api/transforms/get

This example demonstrates how to use curl to fetch a transform. Replace $REDIVIS_ACCESS_TOKEN with your actual access token and adjust the URL with the correct transform reference.

```bash
# Get the transform at https://redivis.com/workflows/x7kh-5pvd4mbf1/transforms/r74d-byr3bf3h3
curl -H "Authorization: Bearer $REDIVIS_ACCESS_TOKEN" \
"https://redivis.com/api/v1/transforms/imathews.example_worfklow_climate_analysis:x7kh.calculate_annual_precipitation:r74d"
```

--------------------------------

### Start SAS Studio

Source: https://docs.redivis.com/reference/organizations/settings/advanced-stata-and-sas-setup

Execute this command to start SAS Studio and verify its functionality. Ensure both the Spawner and SAS Web Application Server start successfully.

```bash
/usr/local/SASHome/sas/studioconfig/sasstudio.sh start
```

--------------------------------

### Workflow Examples

Source: https://docs.redivis.com/api/client-libraries/redivis-python/reference/workflow

Examples demonstrating common workflow operations.

```APIDOC
## Workflow Examples

### Basic Workflow Operations

**Description**: This example shows how to get a workflow, list its tables, and check for table existence.

```python
workflow = redivis.user("imathews").workflow("example_workflow_climate_analysis")
tables = workflow.list_tables()

for table in tables:
    # Properties will be populated with the table.list resource representation
    print(table.properties)

    table.get()
    # Properties will now be populated with the table.get resource representation
    print(table.properties)

workflow.table("join_lat_lon_output").exists() # -> TRUE
```

### Scoped Queries

**Description**: This example demonstrates how to run a query within the context of a specific workflow and convert the results to a pandas DataFrame.

```python
workflow = redivis.user("imathews").workflow("example_workflow_climate_analysis")

query = workflow.query("""
    SELECT
        id,
        EXTRACT(YEAR FROM date) AS year,
        SUM(value) AS annual_precip
    FROM daily_observations
    WHERE (element = 'PRCP')
    GROUP BY id, year
""")

query.to_pandas_dataframe()
# 	id		year		annual_precip
# 0	CA0023026HN	2003		2925.0
# ...
```
```

--------------------------------

### Python Example: Basic Variable Usage

Source: https://docs.redivis.com/api/client-libraries/redivis-python/reference/variable

Demonstrates how to get a table, access a specific variable, retrieve its properties and statistics.

```APIDOC
## Example: Basic Variable Usage

```python
table = redivis.organization("Demo").dataset("iris_species").table("iris")

variable = table.variable("sepalLengthCm")

print(variable.exists()) # -> True

# Get variable properties (populates the 'properties' attribute)
variable.get(wait_for_statistics=True)
print(variable.properties)
# Expected output structure:
# {
#   "kind": "variable",
#   ...
# }

# Get variable statistics
print(variable.get_statistics())
# Expected output structure:
# {
#   "kind": "variableStatistics",
#   "status": "completed",
#   "count": 150,
#   "numDistinct": 35,
#   "min": 4.3,
#   "max": 7.9,
#   "mean": 5.8433333333333355,
#   "approxMedian": 0.8280661279778625,
#   ...
# }
```
```

--------------------------------

### Prepare SAS installation from zip

Source: https://docs.redivis.com/reference/organizations/settings/advanced-stata-and-sas-setup

Commands to extract, set permissions, and prepare the installation directory for a SAS zip file.

```bash
unzip SAS_94_TS1M7.zip
chmod -R 0775 ./SAS_94_TS1M7/
mkdir /usr/local/SASHome
sudo chown -R $(whoami) /usr/local/
cd SAS_94_TS1M7
```

--------------------------------

### Install system dependencies

Source: https://docs.redivis.com/reference/organizations/settings/advanced-stata-and-sas-setup

Install the required Java Runtime Environment for SAS.

```bash
apt install default-jre
```

--------------------------------

### Install and Use Redivis R Package

Source: https://docs.redivis.com/api/client-libraries/redivis-r/reference/redivis

Install the package from GitHub and initialize a connection to organization, dataset, and query resources.

```r
devtools::install_github("redivis/redivis-r")
library("redivis")

organization <- redivis$organization("Demo")
# organization$list_datasets(), organization$dataset("Iris species"), etc...

dataset <- organization$dataset("iris_species")
table <- dataset$table("iris")  
# table$to_tibble(), etc...
# In a notebook, can do redivis$table("_source_") to reference the source table

query <- redivis$query("
    SELECT * 
    FROM demo.iris_species.iris 
    WHERE SepalLengthCm > 5
")
# query$to_tibble(), etc...
```

--------------------------------

### CSV response format example

Source: https://docs.redivis.com/api/rest-api/tables/listrows

Example output when the format parameter is set to csv.

```csv
// e.g.: table_ref/rows?format=csv&selectedVariables=id,date,decimal,text
id,date,decimal,text
1,"2012-01-01",4.2,"Some text"
2,"2012-02-02",3.14,"Some other text"
3, ...
```

--------------------------------

### Redivis Python Client Library - Examples

Source: https://docs.redivis.com/api/client-libraries

Examples demonstrating common use cases for the Redivis Python client library.

```APIDOC
## Python Client Library Examples

### Listing resources

Demonstrates how to list various resources.

### Querying data

Shows how to query data from Redivis.

### Reading tabular data

Provides examples for reading tabular data.

### Uploading data

Illustrates how to upload data to Redivis.

### Working with non-tabular files

Explains how to handle non-tabular files.
```

--------------------------------

### Get DataSource using curl

Source: https://docs.redivis.com/api/rest-api/datasources/get

Example of how to fetch a DataSource using curl. Replace placeholders with your actual token and resource references. Requires workflow read access.

```bash
# Get the dataSource at https://redivis.com/workflows/x7kh-5pvd4mbf1/dataSources/f3st-1yxmt4nsf
curl -H "Authorization: Bearer $REDIVIS_ACCESS_TOKEN" \
"https://redivis.com/api/v1/workflows/imathews.example_worfklow_climate_analysis:x7kh/dataSources/f3st-1yxmt4nsf"
```

--------------------------------

### Curl Example for Listing Tables

Source: https://docs.redivis.com/api/rest-api/tables/list

This example shows how to list tables in a specific dataset using curl, including the necessary authorization header.

```bash
# List the tables on the EPA Air Quality dataset from the StanfordPHS organization
curl -H "Authorization: Bearer $REDIVIS_ACCESS_TOKEN" \
"https://redivis.com/api/v1/datasets/stanfordphs.epa_air_quality/tables"
```

--------------------------------

### Qualified reference example

Source: https://docs.redivis.com/api/client-libraries/redivis-python/reference/notebook

An example of a fully qualified reference string for a notebook.

```http
imathews.example_workflow_climate_analysis:x7kh.visualize_precipitation_trends:2f2v
```

--------------------------------

### Install redivis-r from GitHub

Source: https://docs.redivis.com/api/client-libraries/redivis-r/getting-started

Use devtools to install the latest version of the library from the main branch.

```r
devtools::install_github("redivis/redivis-r", ref="main")
```

--------------------------------

### Qualified Parameter Reference Example

Source: https://docs.redivis.com/api/client-libraries/redivis-python/reference/parameter

An example of a fully qualified reference string for a parameter.

```http
imathews.example_workflow_climate_analysis:x7kh.my_param:27sa
```

--------------------------------

### Install Redivis Python Library

Source: https://docs.redivis.com/api/client-libraries/redivis-python/getting-started

Install the redivis library using pip. This command upgrades to the latest version.

```bash
pip install --upgrade redivis
```

--------------------------------

### Install Python Dependencies for Redivis

Source: https://docs.redivis.com/guides/export-and-publish-your-work/build-your-own-site-with-observable

Commands to upgrade pip and install required packages from a requirements.txt file within a virtual environment.

```bash
python3 -m pip install --upgrade pip
pip install -r requirements.txt
```

--------------------------------

### Construct and Get User Workflow

Source: https://docs.redivis.com/api/client-libraries/redivis-r/reference/user/userusdworkflow

Instantiate a workflow object by referencing a workflow owned by a user. Call $get() to populate workflow properties.

```r
user <- redivis$user("imathews")
workflow <- user$workflow("example_workflow_climate_analysis")

workflow$get()
print(workflow$properties) # Properties will be fully populated after calling $get()
```

--------------------------------

### Example: Create a notebook output table

Source: https://docs.redivis.com/api/client-libraries/redivis-r/reference/notebook

Shows how to create an output table from data within a running notebook.

```APIDOC
## Example: Create a notebook output table

```r
df <- get_dataframe_somehow()

notebook <- redivis$current_notebook()

# Create an output table on the current notebook
# Optional parameter, append (default FALSE), determines whether the output appends to
#  or replaces the output table.
notebook$create_output_table(df, append=FALSE)

# Can also pass a path to a parquet file or csv
notebook$create_output_table("path/to/file.parquet")
```
```

--------------------------------

### GET Notebook

Source: https://docs.redivis.com/api/client-libraries/redivis-r/reference/notebook/notebookusdget

Fetches the notebook details and populates the notebook.properties dictionary.

```APIDOC
## GET Notebook

### Description
Fetches the notebook, after which notebook.properties will contain a dict with entries corresponding to the properties on the notebook resource definition. Will raise an error if the notebook does not exist.

### Method
GET

### Returns
- **self** (Notebook) - The current Notebook instance.
```

--------------------------------

### Get Variable Statistics in R

Source: https://docs.redivis.com/api/client-libraries/redivis-r/reference/variable/variableusdget_statistics

Fetches statistics for a variable. Waits for computation if statistics are not yet available. Assumes the Redivis library is installed and authenticated.

```r
table <- redivis$organization("Demo")$dataset("iris_species")$table("iris")

statistics <- table$variable("sepalLengthCm")$get_statistics()

print(statistics)
```

```json
{
	"kind": "variableStatistics",
	"status": "completed",
	"count": 150,
	"numDistinct": 35,
	"min": 4.3,
	"max": 7.9,
	"mean": 5.8433333333333355,
	"approxMedian": 0.8280661279778625,
	...
}
```

--------------------------------

### Curl Request to Get Raw File

Source: https://docs.redivis.com/api/rest-api/files/get

Example using curl to fetch a raw file. Ensure your access token has the 'data.data' scope and appropriate data access.

```bash
curl -H "Authorization: Bearer $REDIVIS_ACCESS_TOKEN" \
"https://redivis.com/api/v1/rawFiles/93qh-6wdzwrz84.bc574P4o8ypzp8yKCnrgIw"
```

--------------------------------

### Download Model via `post_install.sh`

Source: https://docs.redivis.com/guides/analyze-data-in-a-workflow/running-ml-workloads

Execute a Python command within the `post_install.sh` script to download Hugging Face models. This ensures models are available before the notebook starts and internet access is disabled.

```shell
python -c '
from huggingface_hub import snapshot_download
snapshot_download(repo_id="sentence-transformers/all-MiniLM-L6-v2")
'
```

--------------------------------

### Get Current User and List Datasets (R)

Source: https://docs.redivis.com/api/client-libraries/redivis-r/reference/redivis/redivisusdcurrent_user

Retrieves the current user object and then lists all datasets associated with that user. Ensure the Redivis library is installed and authenticated.

```r
user <- redivis$current_user()

print(user$list_datasets())
```

--------------------------------

### Create and List Directory Contents

Source: https://docs.redivis.com/api/client-libraries/redivis-r/reference/directory

Demonstrates how to load a table as a directory and list its contents. Use `recursive=True` to list all files within subdirectories.

```r
dir <- redivis$table("table_ref")$to_directory()
print(dir$list()) # list files + directories within this directory
all_files <- dir$list(mode="files", recursive=True)

file <- dir$get("path/to/file.txt") # Will return None if doesn't exist
subdir <- dir$get("path/to/subdir")
```

--------------------------------

### Get fully qualified table reference

Source: https://docs.redivis.com/api/client-libraries/redivis-python/reference/table

Provides the fully qualified reference for a table, which can be used in contexts like SQL queries. Example format: demo.reddit:prpw:v1_0.posts:7q4m

```http
demo.reddit:prpw:v1_0.posts:7q4m
```

--------------------------------

### Access Dataset Version Metadata in R

Source: https://docs.redivis.com/api/client-libraries/redivis-r/reference/dataset/datasetusdversion

Use this snippet to get a reference to a specific dataset version by its tag. You can then access its properties. Ensure you have the redivis library installed and loaded.

```r
dataset <- redivis$organization("demo")$dataset("ghcn_daily_weather_data")

version <- dataset$version("1.0")
print(version$get()$properties)
```

--------------------------------

### List and Access Dataset Versions

Source: https://docs.redivis.com/api/client-libraries/redivis-python/reference/version

Demonstrates how to list all versions of a dataset and access their properties. Requires organization and dataset objects to be initialized.

```python
my_org = redivis.organization("some_organization")
dataset = my_org.dataset("some_dataset")
current_version = dataset.version("1.0")

current_version.exists() # True

# Get the full API representation of the version and print its properties
print(current_version.get().properties)

for version in dataset.list_versions():
    print(version.properties)
```

--------------------------------

### Accessing Redivis Resources

Source: https://docs.redivis.com/api/client-libraries/redivis-python/reference/redivis

Demonstrates how to initialize the redivis module and access organizations, datasets, tables, and execute SQL queries.

```python
import redivis

organization = redivis.organization("Demo")
# organization.list_datasets(), organization.dataset("Iris species"), etc...

dataset = organization.dataset("iris_species")
table = dataset.table("iris")  
# table.to_pandas_dataframe(), etc...
# In a notebook, can do redivis.table("_source_") to reference the source table

query = redivis.query("""
    SELECT * 
    FROM demo.iris_species.iris 
    WHERE SepalLengthCm > 5
""")
# query.to_pandas_dataframe(), etc...
```

--------------------------------

### Install Python Packages for Redivis Notebook

Source: https://docs.redivis.com/guides/analyze-data-in-a-workflow/example-workflows/create-an-image-classification-model

Installs essential Python libraries for machine learning tasks in Redivis notebooks. Ensure packages are installed when the notebook is stopped for restricted data.

```python
import keras
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
from keras.preprocessing.image import ImageDataGenerator
from keras.models import Sequential, Model
from tensorflow.keras.optimizers import RMSprop
from keras.layers import Activation, Dropout, Flatten, Dense, GlobalMaxPooling2D, Conv2D, MaxPooling2D
from keras.callbacks import CSVLogger
from livelossplot.keras import PlotLossesCallback
import efficientnet.keras as efn
import redivis
import os
```

--------------------------------

### Qualified Reference Example

Source: https://docs.redivis.com/api/client-libraries/redivis-r/reference/transform

An example of a fully qualified reference string for a transform.

```http
imathews.example_workflow_climate_analysis:x7kh.join_lat_lon:m9tz
```

--------------------------------

### Reading and Downloading Files in Python

Source: https://docs.redivis.com/api/client-libraries/redivis-python/reference/table

Illustrates how to interface with file index tables, read file contents, and download files to the local filesystem.

```python
import redivis
from io import TextIOWrapper
from PIL import Image

# See https://redivis.com/datasets/yz1s-d09009dbb/files for example data
table = redivis.table("demo.example_data_files:yz1s:v1_3.example_file_types:4c10")
text_file = table.file("pandas_core.py")
image_file = table.file("bogota.tiff"")

## Read file contents
str = text_file.read(as_text=True)
bytes = image_file.read()

## Open the file, as if it was on the filesystem
with file.open() as f:
  f.read(100) # read 100 bytes

with TextIOWrapper(file.open()) as f:
  f.readline() # read first line
  
Image.open(table.file("bogota.tiff")) # PIL will automatically call open() on the file
  
## Download the file  
image_file.download("./path") # will be downloaded as ./path/bogota.tiff
text_file.download("./path/renamed.txt") # will be downloaded as ./path/renamed.txt
```

--------------------------------

### Create Directory from Table and List Contents

Source: https://docs.redivis.com/api/client-libraries/redivis-python/reference/directory

Demonstrates how to create a directory from a Redivis table and list its contents, including recursive file listing. Ensure the 'table_ref' is a valid Redivis table reference.

```python
import redivis

dir = redivis.table("table_ref").to_directory()
print(dir.list()) # list files + directories within this directory
all_files = dir.list(mode="files", recursive=True)

file = dir.get("path/to/file.txt") # Will return None if doesn't exist
subdir = dir.get("path/to/subdir")
```

--------------------------------

### List Workflows Notebooks with Curl

Source: https://docs.redivis.com/api/rest-api/notebooks/list

This example demonstrates how to list notebooks for a specific workflow using curl. Replace '$REDIVIS_ACCESS_TOKEN' with your actual access token and update the workflow URL.

```bash
# List the notebooks at https://redivis.com/workflows/x7kh-5pvd4mbf1
curl -H "Authorization: Bearer $REDIVIS_ACCESS_TOKEN" \
"https://redivis.com/api/v1/workflows/imathews.example_worfklow_climate_analysis:x7kh/notebooks"
```

--------------------------------

### Preload package data with post-install scripts

Source: https://docs.redivis.com/reference/workflows/notebooks/notebook-concepts

Use a shell command to preload data for packages like tidycensus when internet access is restricted.

```sh
R -e '
  library(tidycensus)
  library(tidyverse)

  census_api_key("YOUR API KEY GOES HERE")
  get_decennial(geography = "state", 
                 variables = "P13_001N", 
                 year = 2020,
                 sumfile = "dhc")
'
```

--------------------------------

### Variable Get Definition

Source: https://docs.redivis.com/api/resource-definitions/variable

Extends the list definition with properties specific to get requests.

```APIDOC
## Variable Get Definition

### Description
Extends the list definition with properties specific to get requests.

### Method
N/A (Schema Definition)

### Endpoint
N/A (Schema Definition)

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
None

### Response
#### Success Response (200)
- **kind** (string) - The resource type. Will always be "variable".
- **uri** (string) - The fully qualified reference to this variable.
- **url** (string) - The variable's discoverable url.
- **name** (string) - The variable's name.
- **type** (string) - The variable's type.
- **isFileId** (boolean) - Whether the variable contains file_ids.
- **index** (integer) - A zero based counter for the variable's ordinality for its table.
- **label** (string) - A user-provided label for the variable. May be null.
- **description** (string) - A user-provided description for the variable. May be null.
- **valueLabels** (array[object]) - An array of value labels used to provide mappings between data content and human-readable meaning.
  - **value** (string) - The value associated with a given valueLabel.
  - **label** (string) - The human-readable label for the associated value.

#### Response Example
```json
{
    "kind": "variable",
    "uri": "string",
    "url": "string",
    "name": "string",
    "type": "string",
    "isFileId": true,
    "index": 0,
    "label": "string",
    "description": "string",
    "valueLabels": [
        { "value": "string", "label": "string" }
    ]
}
```
```

--------------------------------

### Create and Update Datasource Reference

Source: https://docs.redivis.com/api/client-libraries/redivis-r/reference/datasource

This example demonstrates how to create a reference to a specific datasource within a workflow and then update it to a new version. Ensure a default workflow is set or use the Workflow$datasource method for specific workflow contexts.

```r
username <- "imathews"
workflow_name <- "example_workflow_climate_analysis:x7kh"
dataset_name <- "ghcn_daily_weather_data"

workflow <- redivis$workflow(f"{username}.{workflow_name}")
datasource <- workflow$datasource(dataset_name)

datasource$update(version="v2.0")
```

--------------------------------

### Multiple Aggregation Variables Example

Source: https://docs.redivis.com/reference/workflows/transforms/step-aggregate

This example demonstrates adding a count aggregation to an existing dataset.

```text
/*---------+-------+---------+------------*
 | test    | score | student | date       |
 +---------+-------+---------+------------+
 | quiz    | 83    | jane    | 2020-04-01 |
 | quiz    | 35    | pat     | 2020-04-01 |
 | quiz    | 89    | sam     | 2020-04-01 |
 | midterm | 74    | jane    | 2020-05-01 |
 | midterm | 62    | pat     | 2020-05-01 |
 | midterm | 93    | sam     | 2020-05-01 |
 | final   | 77    | jane    | 2020-06-01 |
 | final   | 59    | pat     | 2020-06-01 |
 | final   | 92    | sam     | 2020-06-01 |
 *---------+-------+---------+------------*/
```

```text
/*---------+-------------+---------------+-------------*
 | test    | date        | average_score | test_count  |
 +---------+-------------+---------------+-------------+
 | quiz    | 2020-04-01  | 69            | 3           |
 | midterm | 2020-05-01  | 76.3333       | 3           |
 | final   | 2020-06-01  | 76            | 3           |
 *---------+-------------+---------------+-------------*/
```

--------------------------------

### Interact with Redivis files

Source: https://docs.redivis.com/api/client-libraries/redivis-python/reference/file

Demonstrates referencing files from a table, reading contents, using file-like operations, and downloading files locally.

```python
import redivis
from io import TextIOWrapper
from PIL import Image

# See https://redivis.com/datasets/yz1s-d09009dbb/files for example data
table = redivis.table("demo.example_data_files:yz1s:v1_3.example_file_types:4c10")
text_file = table.file("pandas_core.py")
image_file = table.file("bogota.tiff")

## Read file contents
str = text_file.read(as_text=True)
bytes = image_file.read()

## Open the file, as if it was on the filesystem
with file.open("rb") as f:
  f.read(100) # read 100 bytes

with file.open() as f:
  f.readline() # read first line
  
# Tools that integrate with fsspec can open Redivis URIs:
pystac.Catalog.from_file("redivis://table_ref/stac/catalog.json")
  
Image.open(table.file("bogota.tiff")) # PIL will automatically call open() on the file
  
## Download the file  
image_file.download("./path") # will be downloaded as ./path/bogota.tiff
text_file.download("./path/renamed.txt") # will be downloaded as ./path/renamed.txt
```

--------------------------------

### Workflow Get Definition

Source: https://docs.redivis.com/api/resource-definitions/workflow

Extends the base and list definitions with properties relevant to get requests.

```APIDOC
## Workflow Get Definition

### Description
This definition includes additional properties returned when retrieving a specific workflow resource, building upon the base and list definitions.

### Additional Properties for Get Requests
- **lastActive** (integer) - The timestamp of the last activity on the workflow.
- **tableCount** (integer) - The number of tables associated with the workflow.
- **transformCount** (integer) - The number of transformations associated with the workflow.
- **notebookCount** (integer) - The number of notebooks associated with the workflow.
- **description** (string) - A description of the workflow.
```

--------------------------------

### CSV Response Format Example

Source: https://docs.redivis.com/api/rest-api/readsessions/get

Example of the comma-separated values format returned when requesting data in CSV.

```csv
// e.g.: table_ref/rows?format=csv&selectedVariables=id,date,decimal,text
"1","2012-01-01","4.2","Some text"
"2","2012-02-02","3.14","Some other text"
"3", ...
```

--------------------------------

### Load Tabular Data into Python Notebooks

Source: https://docs.redivis.com/reference/workflows/notebooks/python-notebooks

Demonstrates how to initialize a table reference and load it into various data frame formats. Ensure the redivis-python library is available in your environment.

```python
table = redivis.table("_source_")

pandas_df = table.to_pandas_dataframe(
  # max_results,      -> optional, max records to load
  # variables=list(), -> optional, a list of variables
  # ... consult the redivis-python docs for additional args
)

# other methods accept the same arguments, other than dtype_backend
dask_df = table.to_dask_dataframe()
polars_lf = table.to_polars_lazyframe()
arrow_table = table.to_arrow_table()
arrow_dataset = table.to_arrow_dataset()
```

--------------------------------

### Build Docker image for ARM Macs

Source: https://docs.redivis.com/reference/organizations/settings/advanced-stata-and-sas-setup

Use buildx to build a Linux/amd64 image when working on M-series (ARM) Macs.

```bash
docker buildx build --platform linux/amd64 -t <IMAGE-NAME>:<TAG>
```

--------------------------------

### JSONL Response Format Example

Source: https://docs.redivis.com/api/rest-api/readsessions/get

Example of the JSON Lines format returned when requesting data in JSONL.

```javascript
// e.g.: ...table_ref/rows?selectedVariables=id,date,decimal,text
{"id":1, "date":"2012-01-01", "decimal":4.2, "text":"Some text"]
{"id":2, "date":"2012-01-01", "decimal":3.14, "text":"More text"]
...
```

--------------------------------

### Using R's Native `open()`

Source: https://docs.redivis.com/api/client-libraries/redivis-r/reference/file

Illustrates how to use R's native `open()` function with a Redivis file reference.

```APIDOC
## Using R's Native `open()`

You can also use R's built-in `open()` function by passing the Redivis file object directly.

```r
# Open a file using R's native open()
con <- open(redivis$table("table_ref")$file("filename"), "rb")
```
```

--------------------------------

### List installed R packages

Source: https://docs.redivis.com/reference/workflows/notebooks/r-notebooks

Displays a tibble containing all currently installed R packages and their versions.

```r
tibble::tibble(
  Package = names(installed.packages()[,3]),
  Version = unname(installed.packages()[,3])
)
```

--------------------------------

### Create New Dataset

Source: https://docs.redivis.com/api/client-libraries/redivis-python/reference/dataset

Construct a dataset instance for a user and create it with a specified public access level. Prints the properties of the newly created dataset.

```python
dataset = redivis.user("my_username").dataset("My dataset")

dataset.create(public_access_level="overview")

print(dataset.properties)
```

--------------------------------

### Get Table Resource Properties

Source: https://docs.redivis.com/api/resource-definitions/table

Extended properties available when performing a specific get request for a table resource.

```javascript
{
  "canDownload": boolean,
  "entity": {
    "name": string,
  },
  "temporalRange": [integer, integer], 
  "temporalPrecision": string,
  "geoBBox": {
    "westLongitude": number,
    "eastLongitude": number,
    "northLatitude": number,
    "southLatitude": number
  },
  "publicAccessLevel": string("none"|"overview"|"metadata"|"data"),
  "accessLevel": string("overview"|"metadata"|"data"),
  "container": {
    ...(dataset.base | workflow.base)
  }
}
```

--------------------------------

### Example Output Data: Order by Score Descending

Source: https://docs.redivis.com/reference/workflows/transforms/step-order

The result of ordering the example data by the 'score' variable in descending order.

```text
/*---------+--------*
 | student | score  |
 +---------+--------+
 | jane    | 83     |
 | sam     | 74     |
 | pat     | 62     |
 | neal    | 35     |
 *---------+--------*/
```

--------------------------------

### Execute Basic SQL Queries in Redivis

Source: https://docs.redivis.com/api/client-libraries/redivis-r/reference/query

Demonstrates executing simple SQL queries and retrieving results as a tibble. Includes examples of querying specific tables and lists available data retrieval methods.

```R
library("redivis")

# Execute any SQL query and read the results
query <- redivis$query("SELECT 1 + 1 AS two, 'foo' AS bar")
query$to_tibble()
# 	two	bar
# 0	2	foo

# The query can reference any table on Redivis 
query <- redivis$query("
    SELECT * 
    FROM demo.iris_species.iris 
    WHERE SepalLengthCm > 5
")
query$to_tibble()
# 	Id	SepalLengthCm	SepalWidthCm	PetalLengthCm	PetalWidthCm	Species
# 0	33	5.2	        4.1	        1.5	        0.1	        Iris-setosa
# ...

# Other methods to read data:
# query$to_arrow_batch_reader()
# query$to_arrow_dataset()
# query$to_arrow_dataset()
# query$to_data_frame()
# query$to_data_table()
# query$to_sf_tibble()
```

--------------------------------

### Create an output table in a notebook

Source: https://docs.redivis.com/api/client-libraries/redivis-r/reference/notebook/notebookusdcreate_output_table

Demonstrates creating an output table from a dataframe and from a parquet file path.

```r
df <- get_dataframe_somehow()

notebook <- redivis$current_notebook()

# Create an output table on the current notebook
# Optional parameter, append (default False), determines whether the output appends to
#  or replaces the output table.
notebook$create_output_table(df, append=FALSE) 

# Can also pass a path to a parquet file
notebook$create_output_table("path/to/file.parquet")
```

--------------------------------

### Create a subsequent version on an existing dataset

Source: https://docs.redivis.com/api/client-libraries/redivis-python/examples/uploading-data

Initializes a new version for an existing dataset.

```python
import redivis

dataset = redivis.user("your-username").dataset("some dataset")

# dataset.create_next_version will throw an error if a "next" version already exists,
```

--------------------------------

### Opening and Reading Files

Source: https://docs.redivis.com/api/client-libraries/redivis-r/reference/file

Demonstrates how to open files in different modes (text or binary) and read their contents, either by getting a connection or reading all content directly into memory.

```APIDOC
## Opening and Reading Files

### Open a file connection

Use the `open()` method to get a connection to the file. Specify the mode (e.g., `'r'` for text, `'rb'` for binary).

```r
# Open a text file
text_file <- t$file("pandas_core.py")
con <- text_file$open()
readLines(con)

# Open a binary file
binary_file <- t$file("bogota.tiff")
con <- binary_file$open("rb")
readBin(con)
```

### Read all file contents

Use the `read()` method to read the entire file content directly into memory. The `as_text` argument controls whether the content is returned as text or raw bytes.

```r
# Read text file contents
file_contents <- text_file$read(as_text=TRUE)
```
```

--------------------------------

### Get DataSource HTTP Request

Source: https://docs.redivis.com/api/rest-api/datasources/get

Use this HTTP GET request to retrieve a DataSource. Ensure you have read access to the workflow.

```http
GET /workflows/:workflowReference/dataSources/:dataSourceReference
```

--------------------------------

### Extend Parameter Resource for Get Requests

Source: https://docs.redivis.com/api/resource-definitions/parameter

Includes additional properties returned when performing a get request on a parameter resource.

```javascript
{
    ...parameter.base,
    "values": str[],
}
```

--------------------------------

### Construct User and Reference Resources

Source: https://docs.redivis.com/api/client-libraries/redivis-r/reference/user

Construct a User object using their username and then reference specific datasets owned by that user. Ensure the 'redivis' library is loaded.

```python
library("redivis")

user <- redivis$user("imathews")

# List all datasets
datasets = user$list_datasets()

# Create a reference to a specific dataset
dataset = user$dataset("test_dataset")
```

--------------------------------

### Reference a User and List Datasets

Source: https://docs.redivis.com/api/client-libraries/redivis-python/reference/user

Construct a reference to a specific user by their username and then list all datasets owned by that user. Ensure the user exists and you have access to their resources.

```python
user = redivis.user("imathews")

# List all datasets
datasets = user.list_datasets()

# Create a reference to a specific dataset
dataset = user.dataset("test_dataset")
```

--------------------------------

### Redivis Transform Get Definition

Source: https://docs.redivis.com/api/resource-definitions/transform

Extends the list definition with detailed source, referenced, and output table information for get requests.

```json
{
  ...transform.list,
  "sourceTable": { ...tables.list },
  "referencedTables": [
      { ...tables.list },
      ...
  ],
  "outputTable": { ...tables.list }

}
```

--------------------------------

### Download and Mount Files from Directory

Source: https://docs.redivis.com/api/client-libraries/redivis-r/reference/directory

Shows how to download all files from a Redivis directory to a local path or mount the directory for lazy file loading.

```r
dir <- redivis$table("table_ref")$to_directory()

# Download all files to the provided path
dir$download("/download/path")

# dir$mount() behaves the same as dir$download() was called (all files appear on disk)
# However, this doesn't actually download the files until they're needed, 
#    and as such is much faster, particularly when all files may not need to be read.
dir$mount("/download/path")
```

--------------------------------

### HTTP GET Request to List Datasets

Source: https://docs.redivis.com/api/rest-api/versions/listversions

Use this HTTP GET request to retrieve a list of datasets. Ensure you include the datasetReference in the URL.

```http
GET /api/v1/datasets/:datasetReference/versions
```

--------------------------------

### Create and Update Datasource Reference

Source: https://docs.redivis.com/api/client-libraries/redivis-python/reference/datasource

Demonstrates how to create a reference to a datasource within a specific workflow and then update it to a new version. Ensure a default workflow is set or use the Workflow.datasource constructor.

```python
username = "imathews"
workflow_name = "example_workflow_climate_analysis:x7kh"
dataset_name = "ghcn_daily_weather_data"

workflow = redivis.workflow(f"{username}.{workflow_name}")
datasource = workflow.datasource(dataset_name)

datasource.update(version="v2.0")
```

--------------------------------

### Join Input Tables

Source: https://docs.redivis.com/reference/workflows/transforms/step-join

Example of two source tables containing student data to be joined.

```text
Table A:                     Table B:
/*---------+----------*      /*---------+-------*
 | student | absences |       | student | score |
 +---------+----------+       +---------+-------+
 | jane    | 0        |       | jane    | 85    |
 | sam    | 6       |       | sam    | 64   |
 | pat    | 1       |       | pat    | 88   |
*----------+----------*/      *---------+-------*/
```

--------------------------------

### Version Get Definition

Source: https://docs.redivis.com/api/resource-definitions/version

Includes additional properties for version resources when retrieved via a get request, such as release notes and version history.

```APIDOC
## Version Get Definition

### Description
Extends the base and list definitions with details specific to individual version retrieval.

### Fields
- **releaseNotes** (string) - Notes associated with the release of this version.
- **canRelease** (boolean) - Indicates if the current user can release this version.
- **previousVersion** (version.base) - Reference to the previous version.
- **nextVersion** (version.base) - Reference to the next version.
```
Tokens:
Raw