TokSearch

https://ga-fdp.github.io/toksearch/latest/
Admin
TokSearch is a Python package for parallel retrieving, processing, and filtering of...
Tokens:47,420
Snippets:346
Trust Score:9.7
Update:1 month ago
Show doc for...
Tokens:
Raw
### Install TokSearch in a fresh environment

Source: https://ga-fdp.github.io/toksearch

Install TokSearch into a new environment using mamba.

```bash
mamba create -n toksearch -c ga-fdp -c conda-forge toksearch
```

--------------------------------

### Install FDP environment via fdp-installer

Source: https://ga-fdp.github.io/toksearch

Recommended installation method using the fdp-installer metapackage to set up the full Fusion Data Platform environment.

```bash
conda create -n fdp-installer -c ga-fdp -c conda-forge fdp-installer
conda activate fdp-installer
fdp-install -d /path/to/project
cd /path/to/project
pixi shell
```

--------------------------------

### Build TokSearch from source

Source: https://ga-fdp.github.io/toksearch

Clone the repository and use pixi to install, build, and test the package.

```bash
git clone https://github.com/GA-FDP/toksearch.git
cd toksearch
pixi install
pixi run build          # editable install into the pixi environment
pixi run test           # run the test suite
```

--------------------------------

### Install Fusion Data Platform with conda and pixi

Source: https://ga-fdp.github.io/toksearch/latest/LLM_Tutorial

Creates a conda environment with the Fusion Data Platform installer, activates it, installs the platform into a project directory, and opens a Pixi shell. This environment includes toksearch, toksearch_d3d, the fdp CLI, and DIII-D transport. Requires conda and pixi installed.

```bash
conda create -n fdp-installer -c ga-fdp -c conda-forge fdp-installer
conda activate fdp-installer
fdp-install -d /path/to/project
cd /path/to/project && pixi shell

```

--------------------------------

### LLM Configuration in ~/.fdp/config.toml

Source: https://ga-fdp.github.io/toksearch/latest/llm

Example configuration file defining backend, model, and iteration settings, along with a custom preset for specific site requirements.

```toml
[llm]
backend = "anthropic"          # or openai, claude-max, amsc, or a user preset name
model = "claude-sonnet-4-6"    # overrides the backend's default
max_iterations = 20
anthropic_api_key = "sk-..."   # only if not using env var

[llm.presets.mysite]
backend = "anthropic"          # the underlying backend class
base_url = "https://llm.mysite.gov"
api_key_env = "MYSITE_KEY"
model = "claude-sonnet-4-6"
```

--------------------------------

### Install skills and add MCP server for Claude Code

Source: https://ga-fdp.github.io/toksearch/latest/LLM_Tutorial

Installs TokSearch skills into Claude Code, Cursor, or Codex. The first command installs skills to ~/.claude/skills; the second registers the MCP server so the agent can use them.

```bash
fdp skills install                                  # → ~/.claude/skills
claude mcp add toksearch-skills -- fdp run python -m toksearch.llm.mcp

```

--------------------------------

### Python API Session

Source: https://ga-fdp.github.io/toksearch/latest/llm

Example of initializing a session and sending a query using the Python API.

```python
from toksearch.llm import Session
from toksearch.llm.backends.anthropic import AnthropicBackend

sess = Session(backend=AnthropicBackend(api_key="sk-..."))
result = sess.send(
    "Use run_python to compute 2 + 2.",
    on_tool_call=lambda c: print(f"[{c.name}] {c.thought}"),
    on_tool_result=lambda r: print(r.output),
)
print(result.final_text)
```

--------------------------------

### LLM Conversational Interface Example

Source: https://ga-fdp.github.io/toksearch/latest/llm

Demonstrates a multi-turn interaction where the agent fetches data and performs calculations using a persistent namespace.

```text
you> Fetch ip for shot 200000.
agent> [run_python]
         ip = PtDataSignal('ip').fetch(200000)
       [output] (no output)
       Done. The result is in `ip`.

you> What's the peak value in MA?
agent> [run_python]
         print(np.nanmax(np.abs(ip['data'])) / 1e6)
       [output] 1.4193  ← did NOT re-fetch
       Peak |Ip| is 1.42 MA.
```

--------------------------------

### fdp skills commands and MCP server setup

Source: https://ga-fdp.github.io/toksearch

Commands to list and install skills for external agents, and to add the MCP server to Claude Code.

```bash
fdp skills list                  # what's available, and what's installed
fdp skills install               # → ~/.claude/skills
fdp skills install --backend cursor    # or codex, or all; -f to overwrite

claude mcp add toksearch-skills -- fdp run python -m toksearch.llm.mcp
```

--------------------------------

### Environment Setup and Timer Utility

Source: https://ga-fdp.github.io/toksearch/latest/Parallelization

Configures environment variables to disable numpy multithreading and defines a Timer class for benchmarking execution time.

```python
import time
import os
os.environ["MKL_NUM_THREADS"] = "1"
os.environ["NUMEXPR_NUM_THREADS"] = "1"
os.environ["OMP_NUM_THREADS"] = "1"

import numpy as np

# Context manager that we can use to time execution
class Timer(object):
    def __init__(self):
        self.start = None
        
    def __enter__(self):
        self.start = time.time()
        
    def __exit__(self, *args):
        elapsed = time.time() - self.start
        print('Ran in {0:.2f} s'.format(elapsed))
```

--------------------------------

### Run with Ray backend: toksearch_submit -N 3 python toksearch_example.py -- ray

Source: https://ga-fdp.github.io/toksearch/latest/distributed_computing

Runs the example script with the Ray backend using toksearch_submit -N 3. The output shows the SLURM allocation, Ray cluster startup on saga03-05, and the final result count.

```bash
toksearch_submit -N 3 python toksearch_example.py -- ray

```

```text
salloc -N 3 --exclusive --gres=gpu:volta:1 --x11=all srun -u --pty -N 1 --exclusive --propagate --gres=gpu:volta:0 --overlap python toksearch_example.py ray
salloc: Granted job allocation 16902
salloc: Waiting for resource configuration
salloc: Nodes saga[03-05] are ready for job
Initializing ray with _temp_dir = /tmp/tmp1ei083q3
['10.0.0.43', '10.0.0.44', '10.0.0.45']
STARTING CLUSTER
--nodes=1 --ntasks=1 -w saga03 --gres=gpu:volta:1 --overlap --propagate ray start --block --node-ip-address 10.0.0.43 --temp-dir /tmp/tmp1ei083q3 --port 6543 --head --object-store-memory=95000000000 --memory=80000000000
Ok, started head node
dict_keys(['saga04', 'saga05'])
Starting saga04...
--nodes=1 --ntasks=1 -w saga04 --gres=gpu:volta:1 --overlap --propagate ray start --block --node-ip-address 10.0.0.44 --temp-dir /tmp/tmp1ei083q3 --address=10.0.0.43:6543 --object-store-memory=95000000000 --memory=80000000000
Starting saga05...
--nodes=1 --ntasks=1 -w saga05 --gres=gpu:volta:1 --overlap --propagate ray start --block --node-ip-address 10.0.0.45 --temp-dir /tmp/tmp1ei083q3 --address=10.0.0.43:6543 --object-store-memory=95000000000 --memory=80000000000
********************************************************************************
BATCH 1/1
NUM CPUS: 144
NUM PARTITIONS: 10000
MEDIAN PARTITION SIZE: 1
Got 10000 results using ray
salloc: Relinquishing job allocation 16902

```

--------------------------------

### Start interactive REPL with toksearch chat --backend anthropic

Source: https://ga-fdp.github.io/toksearch/latest/LLM_Tutorial

Starts the interactive REPL with the anthropic backend. The session shows the agent writing and executing a one-shot MdsSignal fetch, printing both code and output, and keeping variables in scope for follow-up turns.

```bash
toksearch chat --backend anthropic

```

```text
toksearch chat (backend: anthropic, model: claude-sonnet-4-6)
Type /help for commands. Ctrl-D to exit.

you> What does TokSearch do?

TokSearch is a Python library for fetching fusion-experiment diagnostic
signals (MDSplus, PTData, IMAS, Zarr) across many shots in parallel and
composing analysis pipelines over them. The core abstraction is `Pipeline`,
which lets you chain fetch / map / filter operations and run them serially,
on multiprocessing pools, or distributed via Ray or Spark.

you> Show me a one-shot example using MdsSignal.

[run_python] Demonstrate fetching ipmhd from efit01 for one shot.
  pipeline = toksearch.Pipeline([165920])
  pipeline.fetch('ip', toksearch.MdsSignal(r'\ipmhd', 'efit01'))
  records = pipeline.compute_serial()
  rec = list(records)[0]
  print('shot:', rec.shot)
  print('peak |Ip|:', np.nanmax(np.abs(rec['ip']['data'])) / 1e6, 'MA')
[output] shot: 165920
         peak |Ip|: 1.1325 MA

you> /quit

```

--------------------------------

### Run with Spark backend: toksearch_submit -N 3 python toksearch_example.py -- spark

Source: https://ga-fdp.github.io/toksearch/latest/distributed_computing

Runs the example script with the Spark backend using toksearch_submit -N 3. The output shows the SLURM allocation, Spark master/worker startup on saga04-06, and the final result count.

```bash
toksearch_submit -N 3 python toksearch_example.py -- spark

```

```text
salloc -N 3 --exclusive --gres=gpu:volta:1 --x11=all srun -u --pty -N 1 --exclusive --propagate --gres=gpu:volta:0 --overlap python toksearch_example.py spark
salloc: Granted job allocation 16908
salloc: Waiting for resource configuration
salloc: Nodes saga[04-06] are ready for job
['10.0.0.44', '10.0.0.45', '10.0.0.46']
STARTING CLUSTER
MASTER IP 10.0.0.44
['--nodes=1', '--ntasks=1', '-w', 'saga04', '--gres=gpu:volta:1', '--overlap', 'spark-class', 'org.apache.spark.deploy.master.Master', '--host', '10.0.0.44', '--port', 7077]
Ok, started head node
dict_keys(['saga04', 'saga05', 'saga06'])
Starting saga04...
['--nodes=1', '--ntasks=1', '-w', 'saga04', '--gres=gpu:volta:1', '--overlap', 'spark-class', 'org.apache.spark.deploy.worker.Worker', '-i', '10.0.0.44', '-c', 48, '-m', '149G', 'spark://10.0.0.44:7077']
Starting saga05...
['--nodes=1', '--ntasks=1', '-w', 'saga05', '--gres=gpu:volta:1', '--overlap', 'spark-class', 'org.apache.spark.deploy.worker.Worker', '-i', '10.0.0.45', '-c', 48, '-m', '149G', 'spark://10.0.0.44:7077']
Starting saga06...
['--nodes=1', '--ntasks=1', '-w', 'saga06', '--gres=gpu:volta:1', '--overlap', 'spark-class', 'org.apache.spark.deploy.worker.Worker', '-i', '10.0.0.46', '-c', 48, '-m', '149G', 'spark://10.0.0.44:7077']
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
24/04/15 09:51:48 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Got 10000 results using spark                                                   
salloc: Relinquishing job allocation 16908

```

--------------------------------

### Registering toksearch.llm entry points in pyproject.toml

Source: https://ga-fdp.github.io/toksearch/latest/llm

Example configuration for a package named toksearch_d3d to register its namespace, skills directory, and presets via entry points.

```toml
[project.entry-points."toksearch.llm.namespace"]
toksearch_d3d = "toksearch_d3d"

[project.entry-points."toksearch.llm.skills"]
toksearch_d3d = "toksearch_d3d.llm:skills_path"

[project.entry-points."toksearch.llm.presets"]
amsc = "toksearch_d3d.llm:AMSC_PRESET"
```

--------------------------------

### Install TokSearch standalone

Source: https://ga-fdp.github.io/toksearch

Install the TokSearch package without the full FDP data access suite.

```bash
conda install -c ga-fdp -c conda-forge toksearch
```

--------------------------------

### Initialize Pipeline from SQL query with Pipeline.from_sql

Source: https://ga-fdp.github.io/toksearch/latest/pipeline

Shows how to initialize a Pipeline from a SQL query using Pipeline.from_sql. The example connects to the d3drdb database, runs a query without parameters, then runs a parameterized query with a threshold value. The query must produce a 'shot' column and cannot have 'key' or 'errors' columns.

```python
from toksearch import Pipeline
from toksearch.sql.mssql import connect_d3drdb

# See documentation for connect_d3drdb for more details
conn = connect_d3drdb()

# Query without parameters
query = "select shot from shots_type where shot_type = 'plasma'"
pipe = Pipeline.from_sql(conn, query)

# Query with parameters, limiting to shot numbers greater than a
# threshold
threshold = 180000
query = """
    select shot
    from shots_type
    where shot_type = 'plasma' and shot > %d
    """
pipe = Pipeline.from_sql(conn, query, threshold)
```

--------------------------------

### Install TokSearch with LLM Extras via Pip

Source: https://ga-fdp.github.io/toksearch/latest/llm

Command to install TokSearch from a source checkout including the necessary backend SDKs and matplotlib.

```bash
pip install -e '.[llm]'
```

--------------------------------

### fdp chat and query commands

Source: https://ga-fdp.github.io/toksearch

Preferred way to run the conversational CLI through the fdp wrapper, which configures data access before the session starts.

```bash
fdp chat                        # interactive, FDP environment configured
fdp query "Fetch ip for shot 200000 and report the peak in MA."
```

--------------------------------

### Swap LLM backends from the CLI with toksearch query --backend

Source: https://ga-fdp.github.io/toksearch/latest/LLM_Tutorial

Shows the CLI equivalents for selecting a backend with the `--backend` flag. The `amsc` backend requires the `toksearch_d3d` package to be installed.

```bash
toksearch query --backend anthropic  "..."
toksearch query --backend openai     "..."
toksearch query --backend claude-max "..."
toksearch query --backend amsc       "..."     # requires toksearch_d3d installed
```

--------------------------------

### Toksearch Backends Registry Output

Source: https://ga-fdp.github.io/toksearch/latest/llm

Example output of the toksearch backends command showing available backends and models.

```text
name        source      backend     model
----------  ----------  ----------  -----------------
amsc        discovered  anthropic   claude-sonnet-4-6
anthropic   built-in    anthropic   claude-sonnet-4-6  (default)
claude-max  built-in    claude-max  -
openai      built-in    openai      gpt-4o
```

--------------------------------

### Example conversation with toksearch chat

Source: https://ga-fdp.github.io/toksearch

Shows a sample interaction with the conversational CLI: fetching ipmhd for shot 165920 and querying the peak current.

```text
you> Use run_python to fetch ipmhd for shot 165920 from efit01.
[run_python] Fetch ipmhd via MdsSignal.
  pipeline = toksearch.Pipeline([165920])
  pipeline.fetch('ip', toksearch.MdsSignal(r'\ipmhd', 'efit01'))
  rec = list(pipeline.compute_serial())[0]
[output] (no output)

you> What's the peak |Ip| in MA?
[run_python] ...
[output] 1.1325
```

--------------------------------

### Output of compute_spark execution

Source: https://ga-fdp.github.io/toksearch/latest/Parallelization

Displayed output from running the compute_spark example. Shows the banner, Spark master configuration, and the run time of 9.04 s.

```text
********************************************************************************
RUNNING WITH compute_spark
MASTER: None
[('spark.master', 'None'), ('spark.app.submitTime', '1712249973365'), ('spark.submit.pyFiles', ''), ('spark.submit.deployMode', 'client'), ('spark.ui.showConsoleProgress', 'true'), ('spark.app.name', 'pyspark-shell')]
```

```text
                                                                                
```

```text
Ran in 9.04 s
```

--------------------------------

### Verify LLM setup with toksearch query --backend anthropic

Source: https://ga-fdp.github.io/toksearch/latest/LLM_Tutorial

Runs a one-off query using the Anthropic backend to compute 2 + 2 with the run_python tool. The expected output shows the tool call, the printed result, and a natural-language summary. If you see an error like `error: Could not find the 'claude' CLI`, check the configuration resolution order in the reference.

```bash
toksearch query --backend anthropic "Use run_python to compute 2 + 2."

```

--------------------------------

### Ray master process startup command

Source: https://ga-fdp.github.io/toksearch/latest/distributed_computing

Command used to start the Ray master process when compute_ray is invoked on a Pipeline object inside a SLURM job. Sections in braces are replaced by configuration options, and options with underscore + CAPS (e.g. _IP) are filled automatically by toksearch_submit.

```bash
srun --nodes=1 --ntasks=1 -w _HOST ${ray.master.srun}  \
  ray start 
  --head \
  --node-ip-address _IP \
  --port _PORT \
  --temp-dir _TEMP_DIR \
   ${ray.master.start} 

```

--------------------------------

### Run TokSearch pipeline with compute_spark

Source: https://ga-fdp.github.io/toksearch/latest/Parallelization

Initializes a Spark-backed pipeline and materializes results with list(). Spark results are lazy, so compute_spark only initializes Spark; timing starts after that. Requires a create_pipeline function and a shots variable. Output shows Spark configuration and a run time of 9.04 s.

```python
print('*'*80)
print('RUNNING WITH compute_spark')

pipeline = create_pipeline(shots)

# Spark results are generated lazily, so calling compute_spark
# just initializes spark. We won't count that in our timing.
spark_result = pipeline.compute_spark(numparts=8)

with Timer():
    list(spark_result) # Converting to a list materializes the result
spark_result.cleanup()
```

--------------------------------

### Spark master process startup command

Source: https://ga-fdp.github.io/toksearch/latest/distributed_computing

Command used to start the Spark master process when compute_spark is invoked on a Pipeline object inside a SLURM job. Sections in braces are replaced by configuration options, and options with underscore + CAPS (e.g. _PORT) are filled automatically by toksearch_submit.

```bash
 srun --nodes=1 --ntasks=1 -w _HOST ${spark.master.srun}  \
   spark-class org.apache.spark.deploy.master.Master \
     --host _HOST \
     --port _PORT \
     ${spark.master.start}

```

--------------------------------

### Build documentation locally

Source: https://ga-fdp.github.io/toksearch

Command to serve the documentation site locally using pixi.

```bash
pixi run -e docs docs-serve
```

--------------------------------

### from_sql(conn, query, *query_params)

Source: https://ga-fdp.github.io/toksearch/latest/pipeline

Initializes a Pipeline using the results of an SQL query. Accepts a database connection, a query string, and optional query parameters.

```APIDOC
## from_sql(conn, query, *query_params)

### Description
Initialize a Pipeline using the results of an sql query.

### Method
classmethod

### Parameters
#### Parameters
- **conn** ('Connection') - Required - A Connection object, compatible with the Python DB API. This can be, for example, from pyodbc, pymssql, sqlite, etc.
- **query** (str) - Required - A query string. At a minimum, the query must produce rows with the column "shot". The query cannot have columns "key" or "errors" as those are reserved words in a Pipeline. Additionally, if the query has any unnamed column a MissingColumnName exception will be raised.
- **query_params** (arbitrary type) - Optional - Used to pass parameters into a query. The exact query syntax is db-dependent. For SQL server (used for the d3drdb), use either %d or %s as placeholders (it doesn't matter which). For sqlite, ? is used as the placeholder.

### Request Example
```
from toksearch import Pipeline
from toksearch.sql.mssql import connect_d3drdb

# See documentation for connect_d3drdb for more details
conn = connect_d3drdb()

# Query without parameters
query = "select shot from shots_type where shot_type = 'plasma'"
pipe = Pipeline.from_sql(conn, query)

# Query with parameters, limiting to shot numbers greater than a
# threshold
threshold = 180000
query = """
    select shot
    from shots_type
    where shot_type = 'plasma' and shot > %d
    """
pipe = Pipeline.from_sql(conn, query, threshold)
```

### Response
#### Success Response
Returns a Pipeline object initialized with the query results.
```

--------------------------------

### compute_ray(numparts=None, batch_size=None, verbose=True, placement_group_func=None, memory_per_shot=None, provenance=None, **ray_init_kwargs)

Source: https://ga-fdp.github.io/toksearch/latest/pipeline

Applies the pipeline using Ray. Returns a RayRecordSet.

```APIDOC
## compute_ray(numparts=None, batch_size=None, verbose=True, placement_group_func=None, memory_per_shot=None, provenance=None, **ray_init_kwargs)

### Description
Apply the pipeline using Ray.

### Method
compute_ray

### Parameters
- **numparts** (Optional[int]) - Optional - The number of partitions to use when mapping. If not provided, the number of partitions will equal the number of records in the pipeline.
- **batch_size** (Optional[int]) - Optional - The number of elements to process in each batch. Defaults to the number of records in the pipeline.
- **verbose** (bool) - Optional - Whether to print verbose output. Default is True.
- **placement_group_func** (Optional[Callable]) - Optional - A function that returns a placement group. See the ray docs for more information on placement groups.
- **memory_per_shot** (Optional[int]) - Optional - Memory to allocate to each shot in bytes. If not provided, there is no limit.
- **provenance** (Optional[object]) - Optional - Optional Provenance backend. See Pipeline.compute.
- **ray_init_kwargs** (**kwargs) - Optional - Keyword arguments to pass to ray.init

### Returns
- **RayRecordSet** (RayRecordSet) - The record set
```

--------------------------------

### Compute Pipeline Serially

Source: https://ga-fdp.github.io/toksearch/latest/overview

Execute the pipeline using compute_serial to process shots sequentially. This example includes a helper function to print the resulting records.

```python
import numpy as np
try:
    from collections import Mapping
except ImportError:
    from collections.abc import Mapping

import collections
import pprint


np.set_printoptions(threshold=3, precision=1)

records = pipeline.compute_serial()
print('Number of records: {}. Should be 2.'.format(len(records)))

# Helper function for printing results
def pretty_print(record):
    
    for key in record.keys():
        val = record[key]
        if isinstance(val, Mapping):
            print('{}:'.format(key))
            for subkey, subval in val.items():
                print('\t{}: {}'.format(subkey, subval))
        else:
           print('{}: {}'.format(key, val))

# Note the list-like behavior of the records result
for record in records:
    print('*'*80)
    pretty_print(record)
```

--------------------------------

### Create a simple TokSearch pipeline

Source: https://ga-fdp.github.io/toksearch/latest/Aggregating_Data

Initializes a pipeline with specific shots and fetches flux-on-the-grid data from MDSplus using MdsSignal.

```python
import numpy as np
import xarray as xr
from toksearch import MdsSignal, Pipeline

pipeline = Pipeline([165920, 165921, 173000])

psirz_sig = MdsSignal(
    r'\psirz',                  
    'efit01',
    dims=('r', 'z', 'times'),
    data_order=('times', 'r', 'z'),
)

pipeline.fetch_dataset('ds', {'psirz': psirz_sig})
```

--------------------------------

### Swap LLM backends in Python with Session

Source: https://ga-fdp.github.io/toksearch/latest/LLM_Tutorial

Shows how to swap backends in Python by passing different backend instances to the same Session. Requires the corresponding API key environment variables, except ClaudeSDKBackend which uses your `claude` CLI login.

```python
from toksearch.llm import Session
from toksearch.llm.backends.anthropic import AnthropicBackend
from toksearch.llm.backends.openai import OpenAIBackend
from toksearch.llm.backends.claude_sdk import ClaudeSDKBackend

import os

PROMPT = "Use run_python to fetch ipmhd from efit01 for shot 165920 and report peak in MA."

# Anthropic Messages API
Session(backend=AnthropicBackend(api_key=os.environ['ANTHROPIC_API_KEY'])).send(PROMPT)

# OpenAI Chat Completions
Session(backend=OpenAIBackend(api_key=os.environ['OPENAI_API_KEY'])).send(PROMPT)

# Claude Max plan (uses your `claude` CLI login -- no API key)
Session(backend=ClaudeSDKBackend()).send(PROMPT)
```

--------------------------------

### Manage agent skills with fdp skills CLI

Source: https://ga-fdp.github.io/toksearch/latest/llm

Commands to list, install, and manage skill directories for external coding agents like Claude Code or Cursor.

```bash
fdp skills list                            # available + install status
fdp skills install                         # → ~/.claude/skills (Claude Code)
fdp skills install --backend cursor        # or codex, or all
fdp skills install --force                 # overwrite already-installed copies
```

--------------------------------

### Read per-shot NetCDF files without dask

Source: https://ga-fdp.github.io/toksearch/latest/pipeline

Reads back the per-shot NetCDF files written by `write` when `dask` is not installed. Concatenates the files along the 'shot' dimension after sorting them by filename.

```python
import glob, xarray as xr
ds = xr.concat(
    [xr.open_dataset(f) for f in sorted(glob.glob('out/*.nc'))],
    dim='shot',
)

```

--------------------------------

### Toksearch CLI Commands

Source: https://ga-fdp.github.io/toksearch/latest/llm

Basic commands for one-shot queries, interactive chat, and launching the Gradio GUI.

```bash
# One-shot
toksearch query --backend anthropic "Use run_python to compute 2 + 2."

# Interactive REPL
toksearch chat --backend anthropic

# Local Gradio GUI in a browser tab (--no-browser to skip opening it)
toksearch chat --gui

# What can --backend be, in this environment?
toksearch backends
```

--------------------------------

### Initialize MdsSignal imports

Source: https://ga-fdp.github.io/toksearch/latest/Working_with_Signals

Import necessary libraries and configure numpy print options for MdsSignal usage.

```python
import numpy as np
import pprint
from toksearch import MdsSignal
np.set_printoptions(threshold=3, precision=1)
```

--------------------------------

### Decorator write with function returning field

Source: https://ga-fdp.github.io/toksearch/latest/pipeline

Decorator form of `write`: pass neither `field` nor `fields` to get a decorator. Applying it to a function appends the operation and returns the function unchanged. The function returns the record field to write.

```python
@pipeline.write('out/peaks', fmt='netcdf')
def shot_file(rec):
    return rec['ds']

```

--------------------------------

### Set LLM backend credentials (ANTHROPIC_API_KEY, OPENAI_API_KEY, claude login, AmSC)

Source: https://ga-fdp.github.io/toksearch/latest/LLM_Tutorial

Sets the API key for the chosen LLM backend. For Anthropic set ANTHROPIC_API_KEY, for OpenAI set OPENAI_API_KEY, for Claude Max run `claude login` or set CLAUDE_CODE_OAUTH_TOKEN, and for AmSC write the key to ~/amsc_api_key with restricted permissions. Only the key for the backend you plan to use is required.

```bash
# Anthropic
export ANTHROPIC_API_KEY=sk-ant-...

# OpenAI
export OPENAI_API_KEY=sk-...

# Claude Max plan (uses your existing Claude Code login)
claude login                                # one-time; or set CLAUDE_CODE_OAUTH_TOKEN

# AmSC (available to American Science Cloud users; via toksearch_d3d's `amsc` preset)
echo 'sk-...' > ~/amsc_api_key && chmod 600 ~/amsc_api_key

```

--------------------------------

### Spark configuration section in toksearch_submit YAML

Source: https://ga-fdp.github.io/toksearch/latest/distributed_computing

Configuration section for Spark master and worker processes. The master section defines srun options, while the worker section defines srun options and start options (e.g., -c 10, -m 24G).

```yaml
spark:
    # Options for the Spark master
    master:
        srun:
            - --overlap

    worker:
        srun:
            - --overlap
        start:
            - -c
            - 10
            - -m
            - 24G

```

--------------------------------

### Drive Session with AnthropicBackend and run_python

Source: https://ga-fdp.github.io/toksearch/latest/LLM_Tutorial

Creates a Session with an AnthropicBackend and sends a prompt that uses the run_python tool. The on_tool_call and on_tool_result callbacks print tool activity, and the final result and stop_reason are printed. The sample output shows the expected console output for this exact prompt.

```python
import os
from toksearch.llm import Session
from toksearch.llm.backends.anthropic import AnthropicBackend

sess = Session(
    backend=AnthropicBackend(api_key=os.environ["ANTHROPIC_API_KEY"]),
    max_iterations=10,
)

result = sess.send(
    "Use run_python to compute 2 + 2.",
    on_tool_call=lambda c: print(f"[{c.name}] {c.thought}"),
    on_tool_result=lambda r: print(f"  -> {r.output.strip()}"),
)

print()
print("final_text:", result.final_text)
print("stop_reason:", result.stop_reason)

```

```text
[run_python] Compute 2 + 2 and print the result.
  -> 4

final_text: The result is 4.
stop_reason: end_turn

```

--------------------------------

### Ray worker process startup command

Source: https://ga-fdp.github.io/toksearch/latest/distributed_computing

Command used to start the Ray worker process when compute_ray is invoked on a Pipeline object inside a SLURM job. Sections in braces are replaced by configuration options, and options with underscore + CAPS (e.g. _IP) are filled automatically by toksearch_submit.

```bash
 srun --nodes=1 --ntasks=1 -w _HOST ${ray.worker.srun}  \
  ray start
  --node-ip-address _IP \
  --address _MASTER_ADDRESS:_MASTER_PORT \
  --temp-dir _TEMP_DIR \
  ${ray.worker.start}

```

--------------------------------

### Spark worker process startup command

Source: https://ga-fdp.github.io/toksearch/latest/distributed_computing

Command used to start the Spark worker process when compute_spark is invoked on a Pipeline object inside a SLURM job. Sections in braces are replaced by configuration options, and options with underscore + CAPS (e.g. _HOST) are filled automatically by toksearch_submit.

```bash
 srun --nodes=1 --ntasks=1 -w _HOST ${spark.worker.srun}  \
   spark-class org.apache.spark.deploy.worker.Worker \
     -i _HOST \
     ${spark.worker.start} \
     _MASTER_URL

```

--------------------------------

### Launch GUI with toksearch chat --gui and fdp chat --gui

Source: https://ga-fdp.github.io/toksearch/latest/LLM_Tutorial

Launches the same session as a local Gradio app that renders matplotlib figures inline. The --no-browser flag skips opening a browser tab; the fdp variant uses the FDP environment.

```bash
toksearch chat --gui             # opens a browser tab; --no-browser to skip
fdp chat --gui                   # same, with the FDP environment configured

```
Tokens:
Raw