OrangeGrid Examples

https://github.com/syracuseuniversity/orangegridexamples
Admin
OrangeGrid Examples provides code examples for commonly used applications and techniques on the...
Tokens:32,287
Snippets:288
Trust Score:5.6
Update:7 months ago
Show doc for...
Tokens:
Raw
### Setup Grobid Auxiliary Files and Directories

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/Grobid/readme.md

Downloads the Grobid client configuration file and creates necessary directories for input and output PDFs.

```bash
wget https://raw.githubusercontent.com/kermitt2/grobid_client_python/refs/heads/master/config.json
mkdir -p pdfin pdfout
```

--------------------------------

### Install Grobid Python Client using Conda

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/Grobid/readme.md

Installs the Grobid Python client. This involves setting up a Conda environment, activating it, and then installing the client package using pip.

```bash
eval "$(${HOME}/miniconda3/bin/conda shell.bash hook)"
conda create --name grobid
conda activate grobid
conda install pip
python3 -m pip install grobid-client-python
```

--------------------------------

### Install Conda using Miniforge Installer

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/R/readme.md

This bash script downloads and installs the Miniforge distribution of Conda, which is recommended for managing R environments. It sets up the Conda environment and initializes it for the current shell session.

```bash
wget https://github.com/conda-forge/miniforge/releases/download/24.7.1-0/Miniforge-pypy3-24.7.1-0-Linux-x86_64.sh

bash Miniforge-pypy3-24.7.1-0-Linux-x86_64.sh  -b -p $HOME/miniconda3
eval "$(${HOME}/miniconda3/bin/conda shell.bash hook)"
conda init
```

--------------------------------

### Set up HDF5 development environment with uv

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/FileManagement/hdf5.md

This bash script sets up a development environment for HDF5 examples using the 'uv' package manager. It installs uv, initializes a new environment named 'hdf5examples', and adds the 'h5py' and 'numpy' libraries.

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
source ~/.bashrc
uv init hdf5examples
cd hdf5examples
uv add h5py
uv add numpy
```

--------------------------------

### Python Checkpointing with State Loading

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/Checkpointing/README.md

This Python script enhances the previous example by checking for an existing checkpoint file on startup. If found, it loads the saved state (start iteration and total) to resume computation; otherwise, it starts from the beginning.

```python
#!/usr/bin/env python

from pathlib import Path

checkpoint = Path("checkpoint")
if checkpoint.is_file():
    with checkpoint.open() as f_in:
        line = f_in.readline()
        start_in, total_in = line.split()
        start = int(start_in)
        total = int(total_in)
else:
    start = 1
    total = 0

for i in range(start,101):
    total += i

    if i % 10 == 0:
        with open('checkpoint.dat','w') as checkpoint:
            checkpoint.write(f"{i} {total}")

print(total)

```

--------------------------------

### Install Python Packages with Conda

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/python/readme.md

Demonstrates how to install additional Python packages into the Conda environment. This example shows installing the 'numpy' package. Useful commands like 'conda list', 'conda search', and 'conda update' are also mentioned.

```bash
conda install numpy
```

--------------------------------

### Install and Use uv Package Manager

Source: https://context7.com/syracuseuniversity/orangegridexamples/llms.txt

Demonstrates how to install the uv package manager, initialize a new project, add dependencies (including CUDA-aware ones like PyTorch), and run Python scripts directly. uv simplifies Python environment and dependency management.

```bash
# Install uv
curl -LsSf https://astral.sh/uv/install.sh | sh

# Create a new project
uv init example
cd example

# Add dependencies (automatically handles CUDA libraries for PyTorch)
uv add torch numpy scipy

# Run Python scripts directly
uv run example.py
```

--------------------------------

### Install uv Package Manager

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/uv/README.md

Installs the 'uv' package manager using a script downloaded from the official source. It includes a security note about running scripts from the internet and provides an alternative method to download, inspect, and then run the installation script.

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
```

```bash
curl -LsSf https://astral.sh/uv/install.sh > install.sh
less install.sh
sh install.sh
```

--------------------------------

### Install Miniforge for Conda Environment Management (Bash)

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/octave/readme.md

Installs Miniforge, a minimal installer for Conda, to manage Octave and its dependencies. This script downloads and executes the Miniforge installer, sets up the Conda environment, and initializes Conda for automatic activation.

```bash
wget https://github.com/conda-forge/miniforge/releases/download/24.7.1-0/Miniforge-pypy3-24.7.1-0-Linux-x86_64.sh

bash Miniforge-pypy3-24.7.1-0-Linux-x86_64.sh  -b -p $HOME/miniforge3
eval "$(${HOME}/miniforge3/bin/conda shell.bash hook)"
conda init
```

--------------------------------

### Create Python Project with Conda

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/uv/README.md

Illustrates how to set up a Python project environment using 'conda'. This involves creating a new conda environment, activating it, and then installing packages such as 'numpy' using 'conda install'.

```bash
conda create --name example
conda activate example
conda install numpy
```

--------------------------------

### Download Grobid Server Container

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/Grobid/readme.md

Downloads the Grobid server Docker image using Singularity. Note that this may not work on OrangeGrid headnodes due to space limitations for temporary files.

```bash
singularity pull docker://grobid/grobid:0.8.2
```

--------------------------------

### Install PyTorch and CUDA Toolkit

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/PyTorch/readme.md

This code block demonstrates how to create and activate a Conda environment named 'pytorch', and then install PyTorch, torchvision, torchaudio, and the necessary CUDA toolkit version. It specifies channels for installation to ensure compatibility.

```bash
conda create -n pytorch
conda activate pytorch
conda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia
```

--------------------------------

### Copy Grobid Server Container Locally

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/Grobid/readme.md

Copies a pre-downloaded Grobid server Singularity container to the user's home directory. This is a workaround if direct download fails on the headnodes.

```bash
cp /home/lppekows/grobid_0.8.2.sif ~
```

--------------------------------

### Create Python Project with venv and pip

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/uv/README.md

Demonstrates the traditional method of creating a Python project environment using the built-in 'venv' module and managing packages with 'pip'. This involves creating a virtual environment, activating it, and installing a package like 'numpy'.

```bash
python -m venv example
source example/bin/activate
pip install numpy
```

--------------------------------

### Initialize Project Environment with uv

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/uv/README.md

Initializes a new project environment named 'example' using the 'uv init' command. This command sets up the necessary project structure, including 'pyproject.toml', and automatically places the user within the new environment without explicit activation.

```bash
uv init example
```

--------------------------------

### Run PyTorch Example with UV

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/uv/README.md

Executes a Python script using PyTorch for tensor calculations via the 'uv run' command. This demonstrates the basic execution of the example script within the uv environment.

```bash
uv run example.py
```

--------------------------------

### Install Conda Package Manager

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/PyTorch/readme.md

This snippet installs the Miniforge distribution of Conda, which is recommended for managing PyTorch and its dependencies. It downloads the installer, runs it silently, and initializes Conda for the current shell session. It also provides instructions for making Conda available automatically on login.

```bash
wget https://github.com/conda-forge/miniforge/releases/download/24.7.1-0/Miniforge-pypy3-24.7.1-0-Linux-x86_64.sh

bash Miniforge-pypy3-24.7.1-0-Linux-x86_64.sh  -b -p $HOME/miniconda3
eval "$(${HOME}/miniconda3/bin/conda shell.bash hook)"
conda init

if [ -e ${HOME}/.bashrc ]
then
    source ${HOME}/.bashrc
fi
```

--------------------------------

### Install Additional Conda Packages

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/PyTorch/readme.md

After setting up the PyTorch environment, this command shows how to install additional Python packages, such as SciPy, using Conda. This allows for customization of the environment based on specific project needs.

```bash
conda install scipy
```

--------------------------------

### Submit Grobid Job

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/Grobid/readme.md

Submits a Grobid processing job using a Condor submission file. Assumes PDFs have been placed in the 'pdfin' directory.

```bash
condor_submit run_grobid.sub
```

--------------------------------

### Clone OrangeGridExamples Repository using Git

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/README.md

This command allows users to download the OrangeGridExamples repository from GitHub to their home directory on the cluster. This enables direct execution of the provided code examples. No external dependencies are required beyond having git installed on the cluster.

```bash
git clone http://github.com/SyracuseUniversity/OrangeGridExamples
```

--------------------------------

### Create and Activate Octave Conda Environment (Bash)

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/octave/readme.md

Creates a new Conda environment named 'octave' with the 'octave' package installed from the 'conda-forge' channel, and then activates this environment. This ensures that Octave and its dependencies are isolated and available for use.

```bash
conda create -n octave conda-forge::octave
conda activate octave
```

--------------------------------

### Install R Base Package using Conda

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/R/readme.md

This command uses Conda to install the base R package from the conda-forge channel. Conda is used here to manage R and its dependencies, ensuring a consistent environment.

```bash
conda install conda-forge::r-base
```

--------------------------------

### Python Checkpointing with Time-Based Intervals

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/Checkpointing/README.md

This Python code snippet illustrates how to implement time-based checkpointing. It records the start time and checks if 100 seconds have elapsed before performing a checkpoint, offering an alternative to iteration-based saving.

```python
import time
start_time = int(time.time())

...

current_time = int(time.time())

if (current_time - start_time) % 100 == 0:
    # Checkpoint

```

--------------------------------

### Run Blender Rendering Jobs using Singularity and Condor

Source: https://context7.com/syracuseuniversity/orangegridexamples/llms.txt

Details the setup process for using a containerized Blender application for 3D rendering on a cluster. It includes pulling the Blender Singularity image, downloading a sample .blend file, and submitting a rendering job.

```bash
# Setup (one-time)
singularity pull --dir $PWD/sif docker://lscr.io/linuxserver/blender:latest
wget https://mirror.clarkson.edu/blender/demo/geometry-nodes/fields/ball-in-grass.blend -O $PWD/input/ball-in-grass.blend

# Submit rendering job
condor_submit blender.sub

```

--------------------------------

### Install Miniforge and Initialize Conda

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/julia/readme.md

Installs Miniforge, a minimal installer for conda, and initializes Conda for environment management. This is a prerequisite for managing Julia environments.

```bash
wget "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh"
bash Miniforge3-$(uname)-$(uname -m).sh -b
~/miniforge3/bin/conda init
```

--------------------------------

### Run LLM Inference with Ollama using Singularity and Condor

Source: https://context7.com/syracuseuniversity/orangegridexamples/llms.txt

Shows the command-line steps to download an Ollama Docker container using Singularity, submit an LLM inference job via Condor, and check the output.

```bash
# Download Ollama container (one-time)
singularity pull docker://ollama/ollama

# Submit LLM job
condor_submit ollama_app.sub

# Check results
cat output/ollama_app.out

```

--------------------------------

### Submit PyTorch Job to OrangeGrid Cluster

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/PyTorch/readme.md

This command submits a PyTorch job to the OrangeGrid cluster using a Condor submit file. It assumes a 'pytorch_demo.sub' file exists, which is configured to run the PyTorch example script within the appropriate Conda environment.

```bash
condor_submit pytorch_demo.sub
```

--------------------------------

### Basic Python Program for Summation

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/Checkpointing/README.md

A simple Python script to calculate the sum of numbers from 1 to 100. This serves as a baseline before introducing checkpointing.

```python
#!/usr/bin/env python

total = 0

for i in range(1,101):
    total += i

print(total)

```

--------------------------------

### Create and Read HDF5 Files with Python

Source: https://context7.com/syracuseuniversity/orangegridexamples/llms.txt

Demonstrates how to create HDF5 files, write datasets with compression, and read data back using the h5py library. It covers creating datasets, storing numpy arrays, and accessing specific values.

```python
import h5py
import numpy as np

with h5py.File("mytestfile.hdf5", "w") as f:
    # Create a 1D array of 10 integers
    dset = f.create_dataset("/demo/examples/example01", (10,), dtype='i')
    dset[0] = 5

    # Store random values with compression
    values = np.random.rand(10000)
    f.create_dataset('/dataset', data=values, compression="gzip")
```

```python
# Read HDF5 data
import h5py

with h5py.File("mytestfile.hdf5", "r") as f:
    # Access specific value
    print(f['/demo/examples/example01'][0])  # Output: 5

    # Iterate over groups
    for name in f.keys():
        print(name)

    # Compute statistics on compressed data
    print(np.sum(f["/dataset"]))

```

--------------------------------

### View Output Files from HTCondor Jobs

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/multipleJobs/readme.md

Displays the content of all output files generated by the HTCondor jobs. The wildcard `demo_*.out` matches all output files from the 'demo' job family.

```bash
cat output/demo_*.out
```

--------------------------------

### Add PyTorch Package with uv

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/uv/README.md

Shows how to add the PyTorch library to a 'uv' project. It highlights a common scenario where the package name might differ (e.g., 'pytorch' vs. 'torch') and how 'uv' provides helpful error messages to correct this, automatically including necessary CUDA libraries.

```bash
uv add torch
```

--------------------------------

### Run hostname command manually

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/hostname/readme.md

This snippet shows the manual execution of the 'hostname' command in a bash shell. It directly outputs the name of the current machine.

```bash
hostname
```

--------------------------------

### Download Ollama Container with Singularity

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/Ollama/readme.md

This command downloads the Ollama Docker container using Singularity. Singularity containers encapsulate a working environment into a single file, simplifying deployment. This command only needs to be run once.

```bash
singularity pull docker://ollama/ollama
```

--------------------------------

### Run PyTorch with GPU Support

Source: https://context7.com/syracuseuniversity/orangegridexamples/llms.txt

Executes PyTorch programs on GPU-enabled nodes within the OrangeGrid cluster. This example specifies CUDA requirements and requests GPU resources for accelerated computation.

```python
# pytorch_demo.py - PyTorch GPU example
#!/usr/bin/env python3
import torch
import math

dtype = torch.float

if torch.cuda.is_available():
    print("Using GPU")
    print(torch.cuda.device_count())
    device = torch.device("cuda")
else:
    print("Using CPU")
    device = torch.device("cpu")

# Create random input and output data
x = torch.linspace(-math.pi, math.pi, 2000, device=device, dtype=dtype)
y = torch.sin(x)

# Randomly initialize weights for polynomial regression
a = torch.randn((), device=device, dtype=dtype)
b = torch.randn((), device=device, dtype=dtype)
c = torch.randn((), device=device, dtype=dtype)
d = torch.randn((), device=device, dtype=dtype)

learning_rate = 1e-6
for t in range(2000):
    y_pred = a + b * x + c * x ** 2 + d * x ** 3
    loss = (y_pred - y).pow(2).sum().item()
    if t % 100 == 99:
        print(t, loss)

    grad_y_pred = 2.0 * (y_pred - y)
    a -= learning_rate * grad_y_pred.sum()
    b -= learning_rate * (grad_y_pred * x).sum()
    c -= learning_rate * (grad_y_pred * x ** 2).sum()
    d -= learning_rate * (grad_y_pred * x ** 3).sum()

print(f'Result: y = {a.item()} + {b.item()} x + {c.item()} x^2 + {d.item()} x^3')

```

```bash
# pytorch_demo.sub - GPU submit file
executable   = pytorch_wrapper.sh

output       = output/pytorch_demo.out
error        = output/pytorch_demo.err
log          = logs/pytorch_demo.log

Requirements = CUDADriverVersion >= 12.0

+request_gpus = 1

queue 1

```

--------------------------------

### Submit hostname job with HTCondor

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/hostname/readme.md

This snippet demonstrates submitting an HTCondor job using a submit file named 'hostname.sub'. This is the standard method for initiating jobs on the HTCondor pool.

```bash
condor_submit hostname.sub
```

--------------------------------

### Install Conda Package Manager

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/tensorflow/readme.md

Downloads and installs the Miniconda package manager on a Linux system. It then initializes Conda for the current shell session and configures it to be available automatically upon login.

```bash
wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh

bash Miniconda3-latest-Linux-x86_64.sh -b -p $HOME/miniconda3
eval "$(${HOME}/miniconda3/bin/conda shell.bash hook)"
conda init
```

--------------------------------

### Install Julia Packages via Command Line

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/julia/readme.md

Demonstrates how to add Julia packages like LinearAlgebra and Statistics to the current Julia environment using the Pkg manager. Also shows how to add the CUDA package for GPU support.

```bash
julia -e 'using Pkg; Pkg.add(["LinearAlgebra", "Statistics"])

# For GPU support, you would add the CUDA package:

julia -e 'using Pkg; Pkg.add("CUDA")'
```

--------------------------------

### View PyTorch Job Output

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/PyTorch/readme.md

After a Condor job completes, this command displays the standard output file generated by the PyTorch job. It allows users to review the results and any messages produced by the script.

```bash
cat output/pytorch_demo.out
```

--------------------------------

### Store Repetitive Numbers as Text and HDF5

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/FileManagement/hdf5.md

This example shows the significant space savings when storing an array of identical numbers in HDF5 format compared to plain text. It also implicitly uses HDF5's native storage, demonstrating the effectiveness of compression on non-random data.

```python
values = np.full(10000, 1.234567891234567)
```

--------------------------------

### Enable Gzip Compression in HDF5 Dataset

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/FileManagement/hdf5.md

This Python code snippet demonstrates how to enable gzip compression when creating a dataset in an HDF5 file. Compression can lead to further space savings, especially for data with repetitive patterns.

```python
f_out.create_dataset('/dataset', data=values, compression="gzip")
```

--------------------------------

### Add attributes to HDF5 dataset with git info in Python

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/FileManagement/hdf5.md

This Python code creates an HDF5 dataset and adds attributes to it, including the current timestamp, git commit SHA, and active branch name. It requires the 'gitpython' library to be installed and added to the project.

```python
import h5py
import git
import datetime

with h5py.File("mytestfile.hdf5", "w") as f:
    dset      = f.create_dataset("/demo/examples/example01", (10,), dtype='i')
    values    = dset['/demo/examples/example01']
    values[0] = 5

    now = datetime.datetime.now()
    values.attrs['run_at'] = str(now)

    repo   = git.Repo('.')
    values.attrs['gitsha'] = repo.head.object.hexsha
    values.attrs['branch'] = repo.active_branch.name
```

--------------------------------

### Submit Ollama Job to HTCondor

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/Ollama/readme.md

Submits a sample Ollama application job to the HTCondor cluster using a provided submission file. After submission, job progress can be monitored using `condor_q`.

```bash
condor_submit ollama_sub.sh
```

--------------------------------

### Naive HDF5 Write Attempt (Python)

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/FileManagement/hdf5.md

This Python snippet demonstrates a naive approach to writing data from multiple jobs to the same HDF5 file by creating separate datasets for each job's output. It highlights HDF5's limitation of not supporting multiple simultaneous writers, even to different datasets.

```python
import sys
import h5py

my_id = sys.argv[1]

with h5py.File("mytestfile.hdf5", "a") as f:
    results_data = f.create_dataset(f"/{my_id}/results", (10,) , dtype='f')
    ...
```

--------------------------------

### Command to Generate DAG File and Submit

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/Parallelism/README.md

Example commands to execute a Python script that generates a DAG file for summing squares, and then submit this DAG file to Condor.

```bash
python3 generate_dag.py 1 2 3 4 5 10 11 > sum_squares.dag
condor_submit_dat sum_squares.dag
```

--------------------------------

### Submit Basic HTCondor Job

Source: https://context7.com/syracuseuniversity/orangegridexamples/llms.txt

Submits a simple command-line job to the HTCondor cluster and captures its output. This example demonstrates the fundamental structure of an HTCondor submit file.

```bash
# hostname.sub - Basic HTCondor submit file
executable   = /bin/hostname

output       = output/hostname.out
error        = output/hostname.err
log          = logs/hostname.log

queue 1
```

```bash
# Submit the job
condor_submit hostname.sub

# Check job status
condor_q netid

# Monitor progress (updates every 5 seconds)
watch -n 5 condor_q netid

# View results after completion
cat output/hostname.out
# Output: its-og-compute123.ad.syr.edu
```

--------------------------------

### Create and write data to HDF5 file in Python

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/FileManagement/hdf5.md

This Python code demonstrates how to create a new HDF5 file ('mytestfile.hdf5') and store a one-dimensional integer dataset of shape (10,). It shows how to create the dataset, set its initial values, and how to access and modify elements.

```python
import h5py
import numpy as np

with h5py.File("mytestfile.hdf5", "w") as f:
    dset    = f.create_dataset("/demo/examples/example01", (10,), dtype='i')
    dset[0] = 5
```

```python
dset2    = f['/demo/examples/example01']
dset2[0] = 5
```

--------------------------------

### Read data from HDF5 file in Python

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/FileManagement/hdf5.md

This Python code snippet shows how to open an existing HDF5 file in read mode ('r') and print the first element of the dataset located at '/demo/examples/example01'.

```python
import h5py

with h5py.File("mytestfile.hdf5", "r") as f:
    print(f['/demo/examples/example01'][0])
```

--------------------------------

### View R Job Output

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/R/readme.md

This command displays the content of the output file generated by the R job. The output is typically stored in the `output/` directory.

```bash
cat output/r_demo.out
```

--------------------------------

### HTCondor Submit File Configuration

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/uv/README.md

Configuration snippet for an HTCondor submit file. It specifies the executable to be run, which in this case is the 'uv' tool, requiring its full path to be set.

```HTCondor
executable = PATH_TO_UV
```

--------------------------------

### Create and Query SQLite Databases with Python

Source: https://context7.com/syracuseuniversity/orangegridexamples/llms.txt

Provides Python code for creating a SQLite database, defining tables with relationships, inserting records, and querying data across multiple tables using SQL.

```python
# Create database and insert data
import sqlite3

with sqlite3.connect("publications.db") as conn:
    cursor = conn.cursor()

    # Create normalized tables
    cursor.execute("""
    CREATE TABLE authors (
      author_id    INTEGER PRIMARY KEY,
      first_name   TEXT,
      last_name    TEXT,
      birth_year   INTEGER
    )""")

    cursor.execute("""
    CREATE TABLE journals (
      journal_id   INTEGER PRIMARY KEY,
      journal_name TEXT
    )""")

    cursor.execute("""
    CREATE TABLE publications (
      author_id    INTEGER,
      journal_id   INTEGER,
      pub_month    INTEGER,
      pub_year     INTEGER,
      FOREIGN KEY(author_id)  REFERENCES authors(author_id),
      FOREIGN KEY(journal_id) REFERENCES journals(journal_id)
    )""")

    # Insert records
    cursor.execute("INSERT INTO authors(first_name,last_name,birth_year) VALUES('Jane','Doe',1995)")
    cursor.execute("INSERT INTO journals(journal_name) VALUES('Phys. Rev. D')")
    cursor.execute("INSERT INTO publications(author_id,journal_id,pub_month,pub_year) VALUES(1,1,12,2020)")

```

```python
# Query data across tables
cursor.execute("""
SELECT journals.journal_name, authors.last_name, publications.pub_year
  FROM journals, authors, publications
  WHERE publications.author_id  = authors.author_id
   AND publications.journal_id  = journals.journal_id
""")

for row in cursor.fetchall():
    print(row)  # Output: ('Phys. Rev. D', 'Doe', 2020)

```

--------------------------------

### Install SQLite using Conda

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/FileManagement/sqlite.md

This command installs the SQLite package from the conda-forge channel, which is a common way to manage packages in scientific computing environments. It ensures that the `sqlite3` command-line utility is available for interacting with SQLite databases.

```bash
conda install conda-forge::sqlite
```

--------------------------------

### View Ollama Application Output

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/Ollama/readme.md

Displays the output generated by the Ollama application after the HTCondor job has completed. The output is typically stored in a file within the 'output' directory.

```bash
cat output/ollama_app.out
```

--------------------------------

### Submit Multiple Jobs with HTCondor

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/multipleJobs/readme.md

Submits a job to the HTCondor cluster using a specified submit file. This command initiates the execution of jobs defined in the `.sub` file.

```bash
condor_submit demo.sub
```

--------------------------------

### View Octave Job Output File (Bash)

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/octave/readme.md

Displays the content of the output file generated by the Octave job. This command is used to inspect the results and any messages produced by the Octave script after its completion.

```bash
cat output/octave_demo.out
```

--------------------------------

### Update Project Dependencies with uv

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/uv/README.md

Demonstrates updating project dependencies in 'uv'. This involves manually editing the 'pyproject.toml' file to add or modify dependencies, then using 'uv lock' to resolve and save the exact versions, followed by 'uv sync' to apply these changes to the environment.

```bash
uv lock
```

```bash
uv sync
```

--------------------------------

### Configure Bash Profile for Automatic Conda Loading

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/R/readme.md

This snippet adds commands to your `~/.bash_profile` to ensure that your Conda environment is automatically sourced when you log into the cluster. This makes Conda commands available in your shell session.

```bash
if [ -e ${HOME}/.bashrc ]
then
    source ${HOME}/.bashrc
fi
```

--------------------------------

### Checkpointing Program State with Python Pickle

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/Checkpointing/README.md

This Python code snippet demonstrates how to save and load program state using the `pickle` library. It defines a `State` class to hold variables and uses `pickle.dump` to save the state to a file and `pickle.load` to restore it. This is useful for resuming interrupted computations.

```python
#!/usr/bin/env python

import pickle
from pathlib import Path

class State:
    def __init__(self):
        self.counter = 0
        self.total   = 0
	
	# Other state variables...

checkpoint = Path("checkpoint")
if checkpoint.is_file():
    with checkpoint.open('rb') as f_in:
        state = pickle.load(f_in)
else:
    state = State()

# start doing work...
# ...
# Save a new checkpoint

checkpoint = Path("checkpoint")
with checkpoint.open('wb') as f_out:
    pickle.dump(state, f_out)

```

--------------------------------

### Implement Pickle Checkpointing in Python

Source: https://context7.com/syracuseuniversity/orangegridexamples/llms.txt

Provides Python code for implementing checkpointing using the `pickle` module to save and load program state. This allows long-running jobs to recover from failures by resuming from the last saved state.

```python
# Checkpointing with Pickle for complex state
#!/usr/bin/env python

import pickle
from pathlib import Path

class State:
    def __init__(self):
        self.counter = 0
        self.total   = 0
        # Other state variables...

# Load existing checkpoint or initialize new state
checkpoint = Path("checkpoint")
if checkpoint.is_file():
    with checkpoint.open('rb') as f_in:
        state = pickle.load(f_in)
else:
    state = State()

# Continue computation from saved state
for i in range(state.counter, 101):
    state.total += i
    state.counter = i + 1

    # Checkpoint every 10 iterations
    if i % 10 == 0:
        with checkpoint.open('wb') as f_out:
            pickle.dump(state, f_out)

print(state.total)
```

```python
# Time-based checkpointing for variable iteration times
import time

start_time = int(time.time())

while not analysis_finished:
    # Do work...

    current_time = int(time.time())
    if (current_time - start_time) % 100 == 0:
        # Checkpoint every 100 seconds
        with open('checkpoint.dat', 'wb') as f_out:
            pickle.dump(state, f_out)
```

--------------------------------

### Python Checkpointing by Iteration Count

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/Checkpointing/README.md

This Python code demonstrates checkpointing by saving the program's state (current iteration and total sum) to a file every 10 iterations. It's designed to be robust against node failures.

```python
#!/usr/bin/env python

total = 0

for i in range(1,101):
    total += i

    if i % 10 == 0:
        with open('checkpoint.dat','w') as checkpoint:
            checkpoint.write(f"{i} {total}")

print(total)

```

--------------------------------

### Submit Octave Job to HTCondor (Bash)

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/octave/readme.md

Submits an Octave job to the HTCondor cluster using a provided submit file. This command initiates the job execution on the cluster, allowing for distributed computation.

```bash
condor_submit octave_demo.sub
```

--------------------------------

### Python Script for File Output

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/FileManagement/readme.md

This Python script demonstrates how to write to standard output, standard error, and a named output file. It also shows how to dynamically set the output filename using command-line arguments.

```python
import sys

print("This will be written to standard out")
print("This will be written to standard error", file=sys.stderr)

outname = len(sys.argv) > 1 and sys.argv[1] or "default.out"

with open(outname,"w") as f_out:
    print("This will be written to the named file",file=f_out)
```

--------------------------------

### DAGMan File for Map Operation (Squaring)

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/Parallelism/README.md

An example DAGMan file demonstrating a map operation where multiple jobs execute the same 'square.sub' submit file with different 'value' parameters.

```dagman
JOB A square.sub
JOB B square.sub
JOB C square.sub

VARS A value=6
VARS B value=10
VARS C value=23
```

--------------------------------

### Parallel Map Operations with HTCondor DAGMan

Source: https://context7.com/syracuseuniversity/orangegridexamples/llms.txt

Illustrates how to perform parallel map operations using HTCondor's DAGMan. It includes Python scripts for a map function (`square.py`) and a reduce function (`add.py`), along with a bash example for generating and submitting the DAG workflow.

```python
# square.py - Map function: squares a number
#!/usr/bin/env python3
import sys

my_id  = sys.argv[1]
number = int(sys.argv[2])

with open(my_id + ".out",'w') as out_f:
    print(number * number, file=out_f)
```

```bash
# square.sub - Submit file with parameters
executable = square.py
arguments  = $(id) $(value)

queue
```

```python
# add.py - Reduce function: sums results from parent jobs
#!/usr/bin/env python3
import sys

my_id      = sys.argv[1]
parent_ids = sys.argv[2].split(',')
filenames  = [n + ".out" for n in parent_ids]

result = sum([int(open(f).readline().strip()) for f in filenames])

with open(my_id + ".out",'w') as out_f:
    print(result, file=out_f)
```

```bash
# Generate DAG file for sum of squares
python3 generate_dag.py 1 2 3 4 5 10 11 > sum_squares.dag

# Submit the DAG workflow
condor_submit_dag sum_squares.dag

# Visualize workflow dependencies
dot -Tpng:cairo:gd sum_squares.dot -o sum_squares.png

# Result will be in FINAL.out after completion
```

--------------------------------

### View Tensorflow Demo Job Output

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/tensorflow/readme.md

Displays the standard output file generated by the 'tensorflow_demo.py' script after the Condor job has completed. This file contains information about available devices and tensor calculations.

```bash
cat output/tensorflow_demo.out
```

--------------------------------

### Submit HTCondor Job with UV Executable

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/uv/README.md

Submits a job to HTCondor using a provided submit file. The key step is ensuring the 'executable' line in the submit file correctly points to the 'uv' executable's full path.

```bash
condor_submit example.sub
```

--------------------------------

### Work with Zip Archives Programmatically using Python

Source: https://context7.com/syracuseuniversity/orangegridexamples/llms.txt

Demonstrates how to read specific files from a zip archive, iterate through its contents, and add new entries without fully extracting the archive, using Python's zipfile module.

```python
from zipfile import ZipFile

# Read a specific file from archive
with ZipFile('all_results.zip') as myzip:
    with myzip.open('results20260102/test5.txt') as myfile:
        print(myfile.read())

# Count total words across all files in archive
total = 0
with ZipFile('all_results.zip') as myzip:
    for name in myzip.namelist():
        with myzip.open(name) as myfile:
            for line in myfile:
                total += len([word for word in line.split(b' ')])

print(total)

# Add new entries to archive
with ZipFile('all_results.zip', 'a') as myzip:
    myzip.write('/new/entry/data.txt', 'some new data')

```

--------------------------------

### View HTCondor job output

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/hostname/readme.md

This snippet illustrates how to view the output of a completed HTCondor job using the 'cat' command. The output file, typically named 'hostname.out', contains the hostname of the machine where the job was executed.

```bash
cat output/hostname.out
```

--------------------------------

### Create and Activate Julia Conda Environment

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/julia/readme.md

Creates a new Conda environment named 'julia' and installs Julia from the conda-forge channel. It then activates this environment for use.

```bash
conda create -y -n julia conda-forge::julia
conda activate julia
```

--------------------------------

### View Python Job Output

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/python/readme.md

Displays the standard output of a completed HTCondor job. The output file is typically located in the 'output' directory and named after the submission file.

```bash
cat output/python_demo.out
```

--------------------------------

### Add Package to uv Project

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/uv/README.md

Adds the 'numpy' package to the current 'uv' project environment. After execution, the 'pyproject.toml' file is updated to include 'numpy' in the project's dependencies, optionally with a specified version.

```bash
uv add numpy
```

```bash
uv add "numpy==2.3.4"
```

--------------------------------

### Iterate through HDF5 file keys in Python

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/FileManagement/hdf5.md

This Python code demonstrates how to iterate through the keys (names of datasets or groups) within an HDF5 file object. It shows how to list keys at the root level and within a specific group ('demo').

```python
for name in f.keys():
    print(name)

for name in f['demo'].keys():
    print(name)
```

--------------------------------

### Matrix Multiplication: Functional Approach with Context in Python

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/Parallelism/README.md

This Python example demonstrates a functional approach to matrix multiplication where the `handleOneEntry` function computes a single element of the result matrix. It uses `map` to apply this function to all index pairs, passing the context of matrices `A` and `B`.

```python
def handleOneEntry(A, B, index):
    i,j      = index
    pairs    = [(A[i][k], B[k][j]) for k in range(len(B))]
    products = map(lambda v:v[0]*v[1], pairs)
    
    return reduce(__add__, products, 0)

indicies = [(i,j) for i in range(len(A)) for j in range(len(B[0]))]
entries  = map(lambda index: handleOneEntry(A, B, index), indicies)
```

--------------------------------

### Submit Tensorflow Demo Job with Condor

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/tensorflow/readme.md

Submits a Tensorflow demonstration job to the OrangeGrid cluster using Condor. The job requires a CUDA driver version of 12.0 or higher, ensuring compatibility with modern Tensorflow and GPU hardware.

```bash
condor_submit tensorflow_demo.sub
```

--------------------------------

### Submit R Job to HTCondor

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/R/readme.md

This command submits an R job to the HTCondor cluster using a provided submit file (`r_demo.sub`). HTCondor is a workload management system used for distributed computing.

```bash
condor_submit r_demo.sub
```

--------------------------------

### Consolidating Job Outputs into HDF5 (Python)

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/FileManagement/hdf5.md

This Python script demonstrates the consolidation process, where a dedicated job iterates through individual `.dat` files generated by other jobs. It reads the data, writes it into dynamically growing HDF5 datasets (using 'unlimited' or 'resizable'), and then cleans up the original `.dat` files. This avoids HDF5's concurrent write limitations.

```python
import sys
import os
import h5py

base_directory = sys.argv[1]
dir_list       = [f for f in os.listdir(base_directory) if f.endswith('.dat')]

with h5py.File("mytestfile.hdf5", "a") as f_out:
    x_values = f_out.create_dataset("unlimited", f"/data/x", (10,) , dtype='f')
    y_values = f_out.create_dataset("unlimited", f"/data/y", (10,) , dtype='f')
    z_values = f_out.create_dataset("unlimited", f"/data/z", (10,) , dtype='f')
    results  = f_out.create_dataset("unlimited", f"/data/results", (10,) , dtype='f')

    count  = 0

    for f in dir_list:
        x,y,z  = [float(v) for v in f[:-4].split('_')]
        result = float(open(f).readline().strip())

        x_values[count] = x
        y_values[count] = y
        z_values[count] = z
        result_values[count] = result

        count += 1

        try:
            os.remove(f)
        except:
            print("Unable to delete file", f)
```

--------------------------------

### HTCondor Job Output Configuration

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/FileManagement/readme.md

This snippet shows the basic HTCondor configuration for directing standard output and error streams to specific files. It's a fundamental step for capturing job results.

```HTCondor
output = my_job.out
error  = my_job.err
```

--------------------------------

### Read Parameter and Result Data from HDF5 (Python)

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/FileManagement/hdf5.md

This Python snippet shows how to read specific data points (a result and its corresponding parameters) from an HDF5 file where data is organized into separate arrays. It also demonstrates how to efficiently compute aggregate statistics like the sum of all results.

```python
value = f["/data/results"][2,3,4]
x     = f["/data/x"][2,3,4]
y     = f["/data/y"][2,3,4]
z     = f["/data/z"][2,3,4]
```

```python
np.sum(f["/data/results"])
```

--------------------------------

### Monitor HTCondor Job Queue in Real-time

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/R/readme.md

This command provides a real-time, updating view of the HTCondor job queue, refreshing every 5 seconds. Replace 'netid' with your actual SU Net ID to monitor your jobs.

```bash
watch -n 5 condor_q netid
```

--------------------------------

### Write Individual Files for Each Calculation Result (Python)

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/FileManagement/hdf5.md

This Python script iterates through parameter combinations (x, y, z), performs a calculation, and writes the result to a new file named after the parameters. It's simple for locating individual results but generates many files.

```python
import numpy as np

for x in np.arange(0.0,1.0,0.1):
    for y in np.arange(0,11,1):
        for z in np.arange(20,21,1):
            value    = some_calculation(x,y,z) 
            filename = f"{x:.3f}_{y:03d}_{z:03d}.dat"
            with open(filename,"w") as f_out:
                print(value,file=f_out)
```

--------------------------------

### Run Python with Conda Environment

Source: https://context7.com/syracuseuniversity/orangegridexamples/llms.txt

Executes Python scripts within a Conda environment using a wrapper script. This approach ensures that necessary Python packages are available for the script's execution on the cluster.

```bash
# Install Conda (one-time setup)
wget https://github.com/conda-forge/miniforge/releases/download/24.7.1-0/Miniforge-pypy3-24.7.1-0-Linux-x86_64.sh
bash Miniforge-pypy3-24.7.1-0-Linux-x86_64.sh -b -p $HOME/miniconda3
eval "$(${HOME}/miniconda3/bin/conda shell.bash hook))"
conda init

# Install packages
conda install numpy scipy
```

```bash
# conda_wrapper.sh - Activates Conda environment before running Python
#!/bin/bash
eval "$(/home/$(whoami)/miniconda3/bin/conda shell.bash hook))"
python3 python_demo.py
```

```python
# python_demo.py - Sample Python program
#!/usr/bin/env python

numbers = range(1,101)
total   = sum(numbers)

print(total)  # Output: 5050
```

```bash
# python_demo.sub - Submit file using wrapper
executable   = conda_wrapper.sh

output       = output/python_demo.out
error        = output/python_demo.err
log          = logs/python_demo.log

queue 1
```

--------------------------------

### Add Provenance Metadata to HDF5 Files with Python

Source: https://context7.com/syracuseuniversity/orangegridexamples/llms.txt

Shows how to add provenance information, such as timestamps and Git commit details, to HDF5 datasets using attributes. This is useful for tracking data origins and modifications.

```python
# Add provenance metadata with attributes
import h5py
import git
import datetime

with h5py.File("mytestfile.hdf5", "w") as f:
    dset = f.create_dataset("/demo/examples/example01", (10,), dtype='i')

    # Add metadata
    dset.attrs['run_at'] = str(datetime.datetime.now())

    repo = git.Repo('.')
    dset.attrs['gitsha'] = repo.head.object.hexsha
    dset.attrs['branch'] = repo.active_branch.name

```

--------------------------------

### Compare Python Object and String Sizes

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/FileManagement/hdf5.md

This snippet shows the difference in memory usage between a Python float object and its string representation. It highlights that storing numbers in their native format (like in HDF5) is more space-efficient than storing them as text.

```python
>>> import sys
>>> a = 1.23456789123456891234567891234
>>> a
1.234567891234569
>>> sys.getsizeof(a)
24
>>> sys.getsizeof(str(a))
58
```

--------------------------------

### Check HTCondor job status

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/hostname/readme.md

This snippet shows how to check the status of an HTCondor job using the 'condor_q' command. Users need to replace 'netid' with their specific SU netID. The command helps monitor job progression from Idle to Run and completion.

```bash
condor_q netid
```

--------------------------------

### HTCondor Submit File for Parallel Job Execution

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/Parallelism/README.md

Provides an example of an HTCondor submit file configuration for running a parallel job. It specifies the executable and requests a certain number of CPUs, essential for distributing the workload.

```htcondor
executable = squares.py

request_cpus = 5

queue
```

--------------------------------

### Individual Job Data Output (Python)

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/FileManagement/hdf5.md

This Python code snippet shows how each independent HTCondor job can calculate a value and write it to its own unique `.dat` file. This approach is part of a pattern where individual jobs produce their own output files before a consolidation step.

```python
import sys

x,y,z = [float(v) for v in sys.argv[1:]]

value    = some_calculation(x,y,z) 
filename = f"{x:.3f}_{y:03d}_{z:03d}.dat"
with open(filename,"w") as f_out:
    print(value,file=f_out)
```

--------------------------------

### HTCondor Job History JSON Example

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/FileManagement/objects.md

A sample JSON output from the `condor_history --json` command, illustrating job details like user, exit status, allocated CPUs, and submission date.

```json
[
{
  "User": "jdoe@its-condor-submit.syr.edu",
  "ToE": {
    "ExitCode": 1,
    "When": 1771426163,
  },
  "AllocatedCPUs": [0,1,4,5], 
  "QDate": 1771011291
}
,
{
  "User": "jsmith@its-condor-submit.syr.edu",
  "ToE": {
    "ExitCode": 4,
    "When": 1771426379,
  },
  "AllocatedCPUs": [15,22], 
  "QDate": 1771010021
}
]
```

--------------------------------

### Store Random Numbers as Text and HDF5

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/FileManagement/hdf5.md

This Python script generates 10,000 random numbers and saves them to two files: one as plain text and another in HDF5 format. It demonstrates the space savings achieved by using HDF5's native numerical storage.

```python
#!/usr/bin/env python

import h5py
import numpy as np

values = np.random.rand(10000)

with open('size_test.txt','w') as f_out:
    for v in values:
        print(v,file=f_out)

with h5py.File("size_test.hdf5", "w") as f_out:
    f_out.create_dataset('/dataset', data=values)
```

--------------------------------

### Conda Wrapper Script for Python Jobs

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/python/readme.md

A bash script used to set up the Conda environment before executing a Python script. This wrapper ensures that the Python job runs within the correct Conda environment, which is not enabled by default in HTCondor jobs.

```bash
# Example content of conda_wrapper.sh (actual code not provided in source)
# This script would activate the conda environment and then run the python script.
```

--------------------------------

### Create an Index for Faster Data Access in SQL

Source: https://github.com/syracuseuniversity/orangegridexamples/blob/main/FileManagement/sqlite.md

Shows how to create an index on the 'last_name' column of the 'authors' table in SQLite. This significantly speeds up queries that filter or search by last name by pre-computing search data.

```sql
CREATE INDEX authors_last_name
ON authors(last_name);
```
Tokens:
Raw