### GET /api/posts/ids request example

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/_autodocs/03-endpoints.md

Example request for fetching multiple posts using comma-separated IDs.

```http
GET https://arctic-shift.photon-reddit.com/api/posts/ids?ids=ei30r4,eitwb3
```

--------------------------------

### Install zstandard Library

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/README.md

Install the zstandard library using pip. This is a required dependency for processing compressed files.

```bash
pip install zstandard
```

--------------------------------

### Install Arctic Shift dependencies

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/_autodocs/06-usage-examples.md

Commands to clone the repository and install required Python packages.

```bash
# Clone repository with submodules
git clone --recursive https://github.com/ArthurHeitmann/arctic_shift.git
cd arctic_shift

# Install Python dependencies (Python 3.10+)
pip install zstandard
pip install orjson  # optional, for faster JSON parsing
```

--------------------------------

### Example Request for GET /api/time_series

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/_autodocs/03-endpoints.md

A sample request URL to retrieve global post counts aggregated by year.

```http
GET https://arctic-shift.photon-reddit.com/api/time_series?key=global/posts/count&precision=year
```

--------------------------------

### GET /api/comments/ids request example

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/_autodocs/03-endpoints.md

Example request for fetching multiple comments using comma-separated IDs.

```http
GET https://arctic-shift.photon-reddit.com/api/comments/ids?ids=dppum98,dppun99
```

--------------------------------

### Search Posts via GET Request

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/_autodocs/03-endpoints.md

Example request for searching posts within a specific subreddit by title and date range.

```http
GET https://arctic-shift.photon-reddit.com/api/posts/search?subreddit=worldnews&title=wuhan&after=2019-12-30&sort=asc&limit=10
```

--------------------------------

### Verify environment installation

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/_autodocs/06-usage-examples.md

Script to check Python version and availability of optional dependencies.

```python
# test_setup.py
import sys
print(f"Python version: {sys.version}")

try:
    import zstandard
    print("✓ zstandard installed")
except ImportError:
    print("✗ zstandard not installed")

try:
    import orjson
    print("✓ orjson installed (optional)")
except ImportError:
    print("○ orjson not installed (using standard json)")
```

--------------------------------

### Retrieve Tree Comments via GET Request

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/_autodocs/03-endpoints.md

Example request to fetch a specific comment tree with HTML conversion enabled.

```http
GET https://arctic-shift.photon-reddit.com/api/comments/tree?link_id=t3_7cff0b&parent_id=t1_dppum98&md2html=true
```

--------------------------------

### Search Comments via GET Request

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/_autodocs/03-endpoints.md

Example request for searching comments by a specific author and linked post ID.

```http
GET https://arctic-shift.photon-reddit.com/api/comments/search?author=PresidentObama&link_id=z1c9z&limit=100
```

--------------------------------

### Retrieve Wiki Pages Request

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/_autodocs/03-endpoints.md

Example request to fetch specific wiki pages by path.

```http
GET https://arctic-shift.photon-reddit.com/api/subreddits/wikis?paths=/r/reddit.com/wiki/faq,/r/travel/wiki/faq
```

--------------------------------

### List Wiki Pages Request

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/_autodocs/03-endpoints.md

Example request to list all wiki page paths for a specific subreddit.

```http
GET https://arctic-shift.photon-reddit.com/api/subreddits/wikis/list?subreddit=askreddit
```

--------------------------------

### Implement Custom File Processing Logic

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/_autodocs/04-python-utilities.md

Example implementation of processFile showing how to distinguish between post and comment data.

```python
def processFile(path: str):
    print(f"Processing file {path}")
    with open(path, "rb") as f:
        jsonStream = getFileJsonStream(path, f)
        if jsonStream is None:
            print(f"Skipping unknown file {path}")
            return
        progressLog = FileProgressLog(path, f)
        for row in jsonStream:
            progressLog.onRow()
            
            # Extract common fields
            author = row["author"]
            subreddit = row["subreddit"]
            id = row["id"]
            created = row["created_utc"]
            score = row["score"]
            
            # Post-specific fields
            if "title" in row:
                title = row["title"]
                body = row["selftext"]
                url = row["url"]
            
            # Comment-specific fields
            elif "body" in row:
                body = row["body"]
                parent = row["parent_id"]
                link_id = row["link_id"]
            
            # Your processing logic here
            
        progressLog.logProgress("\n")
```

--------------------------------

### JSON Lines file structure

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/_autodocs/05-data-formats.md

Example of the expected format where each line contains a complete JSON object.

```json
{"id":"abc123","author":"user1","score":100}
{"id":"abc124","author":"user2","score":50}
{"id":"abc125","author":"user3","score":75}
```

--------------------------------

### Retrieve Subreddit Rules Request

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/_autodocs/03-endpoints.md

Example request to fetch rules for multiple subreddits.

```http
GET https://arctic-shift.photon-reddit.com/api/subreddits/rules?subreddits=askreddit,politics
```

--------------------------------

### Decompress .zst Files via Command-line

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/_autodocs/05-data-formats.md

Commands for installing and using the zstd utility on various operating systems.

```bash
# Linux/macOS (install zstd)
apt install zstandard  # Debian/Ubuntu
brew install zstandard  # macOS
zstd -d file.zst

# Windows
# Download from https://github.com/facebook/zstd/releases
zstd.exe -d file.zst
```

--------------------------------

### Search Subreddits Request

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/_autodocs/03-endpoints.md

Example request to search for subreddits by prefix, sorted by creation date.

```http
GET https://arctic-shift.photon-reddit.com/api/subreddits/search?subreddit_prefix=ask&sort_type=created_utc&sort=asc
```

--------------------------------

### GET /api/subreddits/wikis/list

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/_autodocs/03-endpoints.md

Retrieve all wiki page paths for a subreddit.

```APIDOC
## GET /api/subreddits/wikis/list

### Description
Retrieve all wiki page paths for a subreddit.

### Method
GET

### Endpoint
/api/subreddits/wikis/list

### Parameters
#### Query Parameters
- **subreddit** (string) - Required - Subreddit name (without r/ prefix)
```

--------------------------------

### Clone Repository and Submodules

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/README.md

Use this command to clone the Arctic Shift repository and its submodules. Ensure you have Git installed.

```bash
git clone --recursive https://github.com/ArthurHeitmann/arctic_shift.git
cd arctic_shift
```

--------------------------------

### GET /api/users/interactions/users/list

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/_autodocs/03-endpoints.md

List all individual interactions between a user and others.

```APIDOC
## GET /api/users/interactions/users/list

### Description
List all individual interactions between a user and others.

### Method
GET

### Endpoint
/api/users/interactions/users/list

### Parameters
#### Query Parameters
- **author** (string) - Required - Primary user
- **subreddit** (string) - Optional - Filter to specific subreddit
- **after** (date) - Optional - Interactions on or after date
- **before** (date) - Optional - Interactions before date
- **min_count** (int) - Optional - Minimum interaction count
- **limit** (int) - Optional - Maximum users to return

### Response
#### Success Response (200)
- **Array** (object) - List of individual interaction details.

#### Response Example
[
  {
    "author": "spez",
    "other_user": "username",
    "type": "comment_on_post",
    "date": 1234567890,
    "item_id": "comment_id"
  }
]
```

--------------------------------

### Aggregate Posts Request

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/_autodocs/03-endpoints.md

Example request to aggregate posts by author within a specific subreddit.

```http
GET https://arctic-shift.photon-reddit.com/api/posts/search/aggregate?aggregate=author&subreddit=announcements&sort=desc
```

--------------------------------

### Get Subreddit Wikis

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/api/README.md

Retrieve wiki pages for a subreddit or specific wiki paths.

```APIDOC
## GET /api/subreddits/wikis

### Description
Retrieves wiki pages for a subreddit or specific wiki paths.

### Method
GET

### Endpoint
/api/subreddits/wikis

### Parameters
#### Query Parameters
- **paths** (string[]) - Optional - Comma-separated list of wiki page paths. Limit: 100.
- **subreddit** (string) - Optional - Return all wiki pages of a subreddit.
- **limit** (int) - Optional - Number of wiki pages to return. Max: 100. Default: 100.

### Request Example
```json
{
  "example": "/api/subreddits/wikis?subreddit=askreddit"
}
```

### Response
#### Success Response (200)
- **path** (string) - The path of the wiki page.
- **content** (string) - The content of the wiki page.
- **author** (string) - The author of the wiki page.
- **created_utc** (Date) - The creation date of the wiki page.
- **ups** (int) - The number of upvotes for the wiki page.

#### Response Example
```json
{
  "example": [
    {
      "path": "/r/askreddit/wiki/index",
      "content": "Welcome to the AskReddit wiki!",
      "author": "reddit",
      "created_utc": "2010-01-01T00:00:00Z",
      "ups": 1000
    }
  ]
}
```
```

--------------------------------

### Define JSON array structure

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/_autodocs/05-data-formats.md

Example of a standard JSON array containing multiple objects.

```json
[
  {"id":"abc123","author":"user1","score":100},
  {"id":"abc124","author":"user2","score":50},
  {"id":"abc125","author":"user3","score":75}
]
```

--------------------------------

### Initialize and use FileProgressLog

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/_autodocs/04-python-utilities.md

Instantiate the tracker with an open binary file and call onRow during iteration to track progress.

```python
with open(path, "rb") as f:
    progressLog = FileProgressLog(path, f)
    for row in jsonStream:
        progressLog.onRow()
        # process row
```

--------------------------------

### Display project file structure

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/_autodocs/MANIFEST.md

Visual representation of the generated documentation file hierarchy.

```text
output/
├── 00-index.md              # Master index and navigation
├── 01-project-overview.md   # High-level introduction
├── 02-types.md              # Type definitions reference
├── 03-endpoints.md          # API endpoints reference
├── 04-python-utilities.md   # Python modules reference
├── 05-data-formats.md       # File formats guide
├── 06-usage-examples.md     # Working code examples
└── MANIFEST.md              # This file
```

--------------------------------

### GET /api/users/interactions/subreddits

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/_autodocs/03-endpoints.md

Show subreddits where a user has been active.

```APIDOC
## GET /api/users/interactions/subreddits

### Description
Show subreddits where a user has been active (combined posts + comments).

### Method
GET

### Endpoint
/api/users/interactions/subreddits

### Parameters
#### Query Parameters
- **author** (string) - Required - Username
- **weight_posts** (float) - Optional - Weight multiplier for post activity
- **weight_comments** (float) - Optional - Weight multiplier for comment activity
- **after** (date) - Optional - Activity on or after date
- **before** (date) - Optional - Activity before date
- **min_count** (int) - Optional - Minimum activity count
- **limit** (int) - Optional - Maximum subreddits to return

### Response
#### Success Response (200)
- **Array** (object) - List of subreddits and activity counts.

#### Response Example
[
  {
    "subreddit": "AskReddit",
    "count": 150
  }
]
```

--------------------------------

### GET /api/subreddits/search

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/_autodocs/03-endpoints.md

Search and filter subreddits by properties.

```APIDOC
## GET /api/subreddits/search

### Description
Search and filter subreddits by properties.

### Method
GET

### Endpoint
/api/subreddits/search

### Parameters
#### Query Parameters
- **subreddit** (string) - Optional - Exact subreddit name match
- **subreddit_prefix** (string) - Optional - Subreddit name prefix
- **after** (date) - Optional - Subreddit creation date (on or after)
- **before** (date) - Optional - Subreddit creation date (before)
- **min_subscribers** (int) - Optional - Minimum subscriber count
- **max_subscribers** (int) - Optional - Maximum subscriber count
- **over18** (boolean) - Optional - Filter NSFW subreddits
- **limit** (int) - Optional - Results per page (1-1000)
- **sort** ("asc" | "desc") - Optional - Sort direction
- **sort_type** (enum) - Optional - "created_utc" | "subscribers" | "subreddit"
- **fields** (string) - Optional - Comma-separated field selection
```

--------------------------------

### Create JSONL from compressed formats

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/_autodocs/05-data-formats.md

Commands to convert compressed files into JSONL format using scripts or command-line tools.

```bash
# From .zst using Arctic Shift utilities
python scripts/processFiles.py  # decompress and output as .jsonl

# Using zstd command-line
zstd -d posts_2023.zst -c | (while read -r line; do echo "$line" >> posts_2023.jsonl; done)
```

--------------------------------

### GET /api/users/ids

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/_autodocs/03-endpoints.md

Retrieve user metadata by their IDs.

```APIDOC
## GET /api/users/ids

### Description
Retrieve user metadata by their IDs. Returns aggregate data only, not full user objects.

### Method
GET

### Endpoint
/api/users/ids

### Parameters
#### Query Parameters
- **ids** (string) - Required - Comma-separated list of user IDs (max 500, format: t2_xxxxx).
```

--------------------------------

### GET /api/time_series

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/_autodocs/00-index.md

Retrieve subreddit statistics over time.

```APIDOC
## GET /api/time_series

### Description
Retrieve subreddit statistics over time.

### Method
GET

### Endpoint
/api/time_series

### Parameters
#### Query Parameters
- **key** (string) - Required - The metric key (e.g., r/subreddit/subscribers).
- **precision** (string) - Required - The time precision (e.g., month).
```

--------------------------------

### GET /api/users/interactions/users

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/_autodocs/03-endpoints.md

Aggregate interactions between a specific user and other users.

```APIDOC
## GET /api/users/interactions/users

### Description
Aggregate interactions between users (author and other users).

### Method
GET

### Endpoint
/api/users/interactions/users

### Parameters
#### Query Parameters
- **author** (string) - Required - Primary user
- **subreddit** (string) - Optional - Filter to specific subreddit
- **after** (date) - Optional - Interactions on or after date
- **before** (date) - Optional - Interactions before date
- **min_count** (int) - Optional - Minimum interaction count
- **limit** (int) - Optional - Maximum users to return

### Response
#### Success Response (200)
- **Array** (object) - List of interaction objects containing author and count.

#### Response Example
[
  {
    "author": "other_user",
    "count": 25
  }
]
```

--------------------------------

### Managing resources with context managers

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/_autodocs/06-usage-examples.md

Use context managers to ensure file handles are closed correctly even if exceptions occur.

```python
# ✓ Good: Use context managers
with open(path, "rb") as f:
    jsonStream = getFileJsonStream(path, f)
    for row in jsonStream:
        process(row)

# ✗ Bad: Manual resource management
f = open(path, "rb")
jsonStream = getFileJsonStream(path, f)
for row in jsonStream:
    process(row)
f.close()  # Might not be called on exception
```

--------------------------------

### GET /api/comments/search/aggregate

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/_autodocs/03-endpoints.md

Aggregate comments by date, author, or subreddit.

```APIDOC
## GET /api/comments/search/aggregate

### Description
Aggregate comments by date, author, or subreddit.

### Method
GET

### Endpoint
/api/comments/search/aggregate

### Parameters
#### Query Parameters
- **aggregate** (enum) - Required - "created_utc" | "author" | "subreddit"
- **frequency** (string) - Optional - Time interval (required with aggregate=created_utc)
- **limit** (int) - Optional - Maximum buckets/groups to return
- **min_count** (int) - Optional - Minimum count per group
- **sort** ("asc" | "desc") - Optional - Sort direction
```

--------------------------------

### Decompress .zst Files with Python

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/_autodocs/05-data-formats.md

Uses the zstandard library to decompress a .zst file into a .jsonl file.

```python
import zstandard

# Decompress .zst file
with open('input.zst', 'rb') as ifh:
    with open('output.jsonl', 'wb') as ofh:
        dctx = zstandard.ZstdDecompressor()
        dctx.copy_stream(ifh, ofh)
```

--------------------------------

### GET /api/posts/search/aggregate

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/_autodocs/03-endpoints.md

Aggregate posts by date, author, or subreddit.

```APIDOC
## GET /api/posts/search/aggregate

### Description
Aggregate posts by date, author, or subreddit.

### Method
GET

### Endpoint
/api/posts/search/aggregate

### Parameters
#### Query Parameters
- **aggregate** (enum) - Required - "created_utc" | "author" | "subreddit"
- **frequency** (string) - Optional - Time interval (required with aggregate=created_utc). Examples: "year", "month", "day", "hour"
- **limit** (int) - Optional - Maximum buckets/groups to return
- **min_count** (int) - Optional - Minimum count per group
- **sort** ("asc" | "desc") - Optional - Sort direction

### Response
#### Success Response (200)
- **buckets** (array) - List of aggregated data objects containing key and count.
```

--------------------------------

### GET /api/comments/tree

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/_autodocs/00-index.md

Retrieve a comment thread for a specific link.

```APIDOC
## GET /api/comments/tree

### Description
Retrieve a comment thread for a specific link.

### Method
GET

### Endpoint
/api/comments/tree

### Parameters
#### Query Parameters
- **link_id** (string) - Required - The ID of the link.
- **limit** (integer) - Optional - The maximum number of comments to return.
```

--------------------------------

### GET /api/comments/search

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/_autodocs/00-index.md

Find comments authored by a specific user.

```APIDOC
## GET /api/comments/search

### Description
Find comments authored by a specific user.

### Method
GET

### Endpoint
/api/comments/search

### Parameters
#### Query Parameters
- **author** (string) - Required - The username of the author.
- **limit** (integer) - Optional - The maximum number of results to return.
```

--------------------------------

### Configure Module-Level Settings

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/_autodocs/04-python-utilities.md

Define the target path and recursion behavior for the processing script.

```python
fileOrFolderPath = r"<path to file or folder>"
recursive = False
```

--------------------------------

### GET /api/short_links

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/_autodocs/03-endpoints.md

Resolves short Reddit links to their full paths.

```APIDOC
## GET /api/short_links

### Description
Resolve short Reddit links to full paths.

### Method
GET

### Endpoint
/api/short_links

### Parameters
#### Query Parameters
- **paths** (string) - Required - Comma-separated short link paths (max 1000, case-sensitive). Format: /r/subreddit/s/xxxxx or /u/user/s/xxxxx

### Response
#### Success Response (200)
- **short_path** (string) - The original short path.
- **full_path** (string) - The resolved full path.

#### Response Example
[
  {
    "short_path": "/r/running/s/3TzXiyxaMD",
    "full_path": "/r/running/comments/abc123/post_title/"
  }
]
```

--------------------------------

### GET /api/users/aggregate_flairs

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/_autodocs/03-endpoints.md

Retrieve user flair distribution across subreddits.

```APIDOC
## GET /api/users/aggregate_flairs

### Description
User flair distribution across subreddits.

### Method
GET

### Endpoint
/api/users/aggregate_flairs

### Parameters
#### Query Parameters
- **author** (string) - Required - Username

### Response
#### Success Response (200)
- **Object** (object) - Map of subreddits to flair counts.

#### Response Example
{
  "AskReddit": {
    "Serious": 5,
    "Original Poster": 3
  },
  "Python": {
    "Intermediate": 2,
    "Helper": 1
  }
}
```

--------------------------------

### GET /api/posts/search

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/_autodocs/00-index.md

Search for posts by keyword within a specific subreddit.

```APIDOC
## GET /api/posts/search

### Description
Search for posts by keyword within a specific subreddit.

### Method
GET

### Endpoint
/api/posts/search

### Parameters
#### Query Parameters
- **title** (string) - Required - The keyword to search for in post titles.
- **subreddit** (string) - Required - The subreddit to search within.
- **limit** (integer) - Optional - The maximum number of results to return.
```

--------------------------------

### Available Post and Comment Fields

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/_autodocs/03-endpoints.md

List of fields available for both post and comment objects.

```text
author, author_fullname, author_flair_text, created_utc, distinguished,
id, retrieved_on, subreddit, subreddit_id, score
```

--------------------------------

### Get Subreddit Rules

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/api/README.md

Retrieve the rules for one or more specified subreddits.

```APIDOC
## GET /api/subreddits/rules

### Description
Retrieves the rules for one or more subreddits.

### Method
GET

### Endpoint
/api/subreddits/rules

### Parameters
#### Query Parameters
- **subreddits** (string[]) - Required - Comma-separated list of subreddit names. Limit: 1000.

### Request Example
```json
{
  "example": "/api/subreddits/rules?subreddits=askreddit,politics"
}
```

### Response
#### Success Response (200)
- **subreddit** (string) - The name of the subreddit.
- **rules** (object[]) - An array of rules for the subreddit.
  - **description** (string) - The description of the rule.
  - **kind** (string) - The type of rule.
  - **short_name** (string) - A short name for the rule.
  - **created_utc** (Date) - The creation date of the rule.
  - **edited_utc** (Date) - The last edited date of the rule.

#### Response Example
```json
{
  "example": {
    "askreddit": [
      {
        "description": "No personal questions.",
        "kind": "moderator_rule",
        "short_name": "NoPersonalQuestions",
        "created_utc": "2015-01-01T00:00:00Z",
        "edited_utc": null
      }
    ]
  }
}
```
```

--------------------------------

### Available Subreddit Fields

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/_autodocs/03-endpoints.md

List of fields available for subreddit objects.

```text
created_utc, description, public_description, display_name, id,
over18, retrieved_on, subscribers, title, (and _meta fields)
```

--------------------------------

### Process Zstandard JSON files

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/_autodocs/04-python-utilities.md

Demonstrates streaming JSON data from a file or directory using getFileJsonStream and FileProgressLog. Ensure the input path points to a valid .zst file or directory.

```python
from fileStreams import getFileJsonStream
from utils import FileProgressLog
import json

fileOrFolderPath = r"C:\data\reddit_posts.zst"
recursive = False

def processFile(path: str):
    print(f"Processing file {path}")
    with open(path, "rb") as f:
        jsonStream = getFileJsonStream(path, f)
        if jsonStream is None:
            print(f"Skipping unknown file {path}")
            return
        
        progressLog = FileProgressLog(path, f)
        results = {"posts": 0, "comments": 0, "total_score": 0}
        
        for row in jsonStream:
            progressLog.onRow()
            
            # Count different types and aggregate scores
            if "title" in row:  # Post
                results["posts"] += 1
                results["total_score"] += row.get("score", 0)
            elif "body" in row:  # Comment
                results["comments"] += 1
                results["total_score"] += row.get("score", 0)
        
        progressLog.logProgress("\n")
        print(f"Results: {json.dumps(results, indent=2)}")

def processFolder(path: str):
    import os
    fileIterator = os.listdir(path) if not recursive else None
    if not fileIterator:
        fileIterator = (os.path.join(path, file) for file in os.listdir(path))
    
    for i, file in enumerate(fileIterator):
        print(f"Processing file {i+1: 3} {file}")
        processFile(file)

def main():
    import os
    if os.path.isdir(fileOrFolderPath):
        processFolder(fileOrFolderPath)
    else:
        processFile(fileOrFolderPath)
    
    print("Done :>")

if __name__ == "__main__":
    main()
```

--------------------------------

### Implement logProgress method

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/_autodocs/04-python-utilities.md

Calculates and prints processing statistics including elapsed time, remaining time, and throughput.

```python
def logProgress(self, end=""):
    progress = self.file.tell() / self.fileSize if not self.file.closed else 1
    elapsed = time.time() - self.startTime
    remaining = (elapsed / progress - elapsed) if progress > 0 else 0
    timePerRow = elapsed / self.i
    printStr = f"{self.i:,} - {progress:.2%} - elapsed: {formatTime(elapsed)} - remaining: {formatTime(remaining)} - {formatTime(timePerRow)}/row"
    self.maxLineLength = max(self.maxLineLength, len(printStr))
    printStr = printStr.ljust(self.maxLineLength)
    print(f"\r{printStr}", end=end)
```

--------------------------------

### GET /api/users/aggregate_flairs Response Schema

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/_autodocs/03-endpoints.md

Represents the structure of the user flair distribution response.

```json
{
  "AskReddit": {
    "Serious": 5,
    "Original Poster": 3
  },
  "Python": {
    "Intermediate": 2,
    "Helper": 1
  }
}
```

--------------------------------

### Tracking progress for long-running tasks

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/_autodocs/06-usage-examples.md

Implement progress logging to monitor task status during iteration.

```python
# ✓ Good: Track progress
with open(path, "rb") as f:
    jsonStream = getFileJsonStream(path, f)
    progressLog = FileProgressLog(path, f)
    
    for row in jsonStream:
        progressLog.onRow()
        process(row)
    
    progressLog.logProgress("\n")

# ✗ Bad: No feedback on long-running tasks
for row in jsonStream:
    process(row)  # No idea how long it will take
```

--------------------------------

### GET /api/users/interactions/subreddits Response Schema

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/_autodocs/03-endpoints.md

Represents the structure of the subreddit activity aggregation response.

```json
[
  {
    "subreddit": "AskReddit",
    "count": 150
  }
]
```

--------------------------------

### Process JSON files with standard library

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/_autodocs/05-data-formats.md

Load entire JSON files into memory using the standard json module.

```python
import json

with open("posts_sample.json", 'r') as f:
    data = json.loads(f.read())
    for row in data:
        # Process each row
        pass
```

--------------------------------

### GET /api/users/interactions/users/list Response Schema

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/_autodocs/03-endpoints.md

Represents the structure of the individual interaction list response.

```json
[
  {
    "author": "spez",
    "other_user": "username",
    "type": "comment_on_post" | "comment_on_comment" | "reply_to_post" | "reply_to_comment",
    "date": 1234567890,
    "item_id": "comment_id"
  }
]
```

--------------------------------

### GET /api/users/interactions/users Response Schema

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/_autodocs/03-endpoints.md

Represents the structure of the user interaction aggregation response.

```json
[
  {
    "author": "other_user",
    "count": 25
  }
]
```

--------------------------------

### Project Directory Structure

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/_autodocs/00-index.md

Visual representation of the project file hierarchy, highlighting the separation between TypeScript schemas and Python utility scripts.

```text
arctic_shift/
├── schemas/                 # TypeScript type definitions
│   ├── RC.ts               # Reddit Comment type
│   ├── RS.ts               # Reddit Submission (Post) type
│   ├── subreddits.ts       # Subreddit type
│   ├── subreddit_rules.ts  # Rules type
│   └── subreddit_wikis.ts  # Wiki page type
├── scripts/                 # Python utilities
│   ├── processFiles.py     # Main entry point
│   ├── fileStreams.py      # File I/O
│   └── utils.py            # Progress tracking
├── api/                     # API documentation
│   └── README.md           # API endpoint reference
├── README.md               # Project README
└── download_links.md       # Links to data dumps
```

--------------------------------

### Process JSONL with Arctic Shift utilities

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/_autodocs/05-data-formats.md

Utilizes the getFileJsonStream utility to process files line-by-line.

```python
from fileStreams import getFileJsonStream
with open("posts_2023.jsonl", "rb") as f:
    for row in getFileJsonStream("posts_2023.jsonl", f):
        # Process each row
        pass
```

--------------------------------

### List Subreddit Wiki Paths

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/api/README.md

Retrieve the paths of all wiki pages for a given subreddit.

```APIDOC
## GET /api/subreddits/wikis/list

### Description
Retrieves the paths of all wiki pages for a given subreddit.

### Method
GET

### Endpoint
/api/subreddits/wikis/list

### Parameters
#### Query Parameters
- **subreddit** (string) - Required - The subreddit to retrieve wiki page paths from.

### Request Example
```json
{
  "example": "/api/subreddits/wikis/list?subreddit=askreddit"
}
```

### Response
#### Success Response (200)
- **paths** (string[]) - An array of wiki page paths.

#### Response Example
```json
{
  "example": [
    "/r/askreddit/wiki/index",
    "/r/askreddit/wiki/faq"
  ]
}
```
```

--------------------------------

### Aggregate Comments Request

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/_autodocs/03-endpoints.md

Example request to aggregate comments by date frequency for a specific author.

```http
GET https://arctic-shift.photon-reddit.com/api/comments/search/aggregate?aggregate=created_utc&frequency=year&author=spez&after=2006-01-01
```

--------------------------------

### Decompress Zstandard files via command line

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/_autodocs/05-data-formats.md

Decompresses .zst files using either the zstd CLI tool or the Python zstandard module.

```bash
# Using zstd CLI
zstd -d posts_2023.zst -o posts_2023.jsonl

# Using Python
python -m zstandard --decompress posts_2023.zst posts_2023.jsonl
```

--------------------------------

### Search for Keyword in Posts

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/_autodocs/06-usage-examples.md

Requires Python 3.10 or higher. Uses regex for pattern matching and processes files via a streaming approach to handle large datasets.

```python
# search_keyword.py
import sys
import json
import re

if sys.version_info < (3, 10):
    raise RuntimeError("Python 3.10+ required")

from fileStreams import getFileJsonStream
from utils import FileProgressLog

fileOrFolderPath = r"RS_2023-01.zst"
search_term = "python"
case_insensitive = True

def processFile(path: str):
    print(f"Processing {path}")
    print(f"Searching for: '{search_term}' (case-insensitive: {case_insensitive})")
    
    matching_posts = []
    
    with open(path, "rb") as f:
        jsonStream = getFileJsonStream(path, f)
        if jsonStream is None:
            print(f"Skipping unknown file {path}")
            return
        
        progressLog = FileProgressLog(path, f)
        
        flags = re.IGNORECASE if case_insensitive else 0
        pattern = re.compile(search_term, flags)
        
        for row in jsonStream:
            progressLog.onRow()
            
            title = row.get("title", "")
            selftext = row.get("selftext", "")
            
            if pattern.search(title) or pattern.search(selftext):
                matching_posts.append({
                    "id": row["id"],
                    "title": title,
                    "author": row.get("author"),
                    "subreddit": row.get("subreddit"),
                    "score": row.get("score"),
                    "num_comments": row.get("num_comments"),
                    "created_utc": row.get("created_utc")
                })
        
        progressLog.logProgress("\n")
    
    # Save results
    output_file = f"search_{search_term}.jsonl"
    with open(output_file, "w") as f:
        for post in matching_posts:
            f.write(json.dumps(post) + "\n")
    
    print(f"\nFound {len(matching_posts)} posts matching '{search_term}'")
    print(f"Saved to {output_file}")
    
    # Print top results
    if matching_posts:
        print("\nTop 5 by score:")
        for post in sorted(matching_posts, key=lambda x: x["score"], reverse=True)[:5]:
            print(f"  [{post['score']}] {post['title'][:60]}...")

if __name__ == "__main__":
    processFile(fileOrFolderPath)
```

--------------------------------

### Process JSONL with standard library

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/_autodocs/05-data-formats.md

Standard approach using the json module to parse lines iteratively.

```python
import json

with open("posts_2023.jsonl", 'r') as f:
    for line in f:
        data = json.loads(line)
        # Process each row
        pass
```

--------------------------------

### Batch process multiple .zst files in Python

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/_autodocs/06-usage-examples.md

Requires Python 3.10+ and specific utility modules for file streaming and progress logging. The script iterates through a directory, processes each file, and outputs a summary of posts, comments, and author counts.

```python
# batch_process.py
import os
import sys
import json
from pathlib import Path
from collections import defaultdict

if sys.version_info < (3, 10):
    raise RuntimeError("Python 3.10+ required")

from fileStreams import getFileJsonStream
from utils import FileProgressLog

def analyze_folder(folder_path: str):
    """Analyze all .zst files in a folder"""
    
    stats = defaultdict(int)
    authors = defaultdict(int)
    
    zst_files = list(Path(folder_path).glob("*.zst"))
    print(f"Found {len(zst_files)} .zst files to process\n")
    
    for i, file_path in enumerate(sorted(zst_files), 1):
        print(f"[{i}/{len(zst_files)}] Processing {file_path.name}")
        
        with open(file_path, "rb") as f:
            jsonStream = getFileJsonStream(str(file_path), f)
            if jsonStream is None:
                print(f"  Skipping unknown file format\n")
                continue
            
            progressLog = FileProgressLog(str(file_path), f)
            
            for row in jsonStream:
                progressLog.onRow()
                
                # Count by type
                if "title" in row:
                    stats["posts"] += 1
                else:
                    stats["comments"] += 1
                
                # Count by author
                author = row.get("author", "[deleted]")
                authors[author] += 1
            
            progressLog.logProgress("\n")
        
        print()
    
    # Print summary
    print("\n" + "=" * 50)
    print("SUMMARY")
    print("=" * 50)
    print(f"Total posts: {stats['posts']:,}")
    print(f"Total comments: {stats['comments']:,}")
    print(f"Unique authors: {len(authors):,}")
    print()
    
    print("Top 20 authors:")
    for author, count in sorted(authors.items(), key=lambda x: x[1], reverse=True)[:20]:
        print(f"  u/{author}: {count:,}")

if __name__ == "__main__":
    folder = "reddit_data/"  # Adjust path
    analyze_folder(folder)
```

--------------------------------

### Get comment thread via API

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/_autodocs/00-index.md

Fetch a comment tree for a specific link ID.

```http
GET /api/comments/tree?link_id=t3_xxxxx&limit=50
```

--------------------------------

### Decompress and stream .zst files

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/_autodocs/04-python-utilities.md

Reads and decompresses Zstandard files in chunks, yielding parsed JSON objects line by line.

```python
def getZstFileJsonStream(f: BinaryIO, chunk_size=1024*1024*10) -> Iterator[dict]:
    decompressor = zstandard.ZstdDecompressor(max_window_size=2**31)
    currentString = ""
    def yieldLinesJson():
        nonlocal currentString
        lines = currentString.split("\n")
        currentString = lines[-1]
        for line in lines[:-1]:
            try:
                yield json.loads(line)
            except json.JSONDecodeError:
                print("Error parsing line: " + line)
                traceback.print_exc()
                continue
    zstReader = decompressor.stream_reader(f)
    while True:
        try:
            chunk = zstReader.read(chunk_size)
        except zstandard.ZstdError:
            print("Error reading zst chunk")
            traceback.print_exc()
            break
        if not chunk:
            break
        currentString += chunk.decode("utf-8", "replace")
        
        yield from yieldLinesJson()
    
    yield from yieldLinesJson()
    
    if len(currentString) > 0:
        try:
            yield json.loads(currentString)
        except json.JSONDecodeError:
            print("Error parsing line: " + currentString)
            print(traceback.format_exc())
            pass
```

--------------------------------

### processFile(path: str)

Source: https://github.com/arthurheitmann/arctic_shift/blob/master/_autodocs/04-python-utilities.md

Processes a single data file by opening it, detecting the format, and iterating through each row of data to execute custom processing logic.

```APIDOC
## processFile(path: str)

### Description
Processes a single file, executing custom logic for each row. The function detects the file format, creates a JSON stream, and tracks progress while iterating through the data.

### Parameters
- **path** (str) - Required - File path to process

### Return Value
- **None**
```