### Start Docker Compose Services

Source: https://github.com/muckrock/documentcloud/blob/master/README.md

Starts the Docker containers defined in the Docker Compose configuration. This brings up the necessary services for the DocumentCloud development environment.

```bash
docker compose up
```

--------------------------------

### Manage Document Sections (Bash)

Source: https://context7.com/muckrock/documentcloud/llms.txt

Creates document outlines or bookmarks for navigation. Allows setting a title and the starting page number for each section. Requires an authorization token.

```bash
# List sections
curl -X GET https://api.www.documentcloud.org/api/documents/12345/sections/ \
  -H "Authorization: Bearer <token>"

# Create section
curl -X POST https://api.www.documentcloud.org/api/documents/12345/sections/ \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{"title": "Chapter 1: Introduction", "page_number": 1}'
```

--------------------------------

### Set Page Text (JSON)

Source: https://github.com/muckrock/documentcloud/blob/master/docs/api/api.md

This JSON example demonstrates how to set the text for specific pages. It includes the page number and the new text content for each page. This format is used when only updating the text is required.

```json
[
    {"page_number": 0, "text": "Page 1 text"},
    {"page_number": 1, "text": "Page 2 text"}
]
```

--------------------------------

### Document Modification: No Changes

Source: https://github.com/muckrock/documentcloud/blob/master/docs/api/api.md

This example shows how to specify that a document should remain unchanged. It takes a list of specifications, where each specification includes a 'page' range. In this case, the entire document (pages 0-447) is specified to be left as is.

```json
[
  {
    "page": "0-447"
  }
]
```

--------------------------------

### Set Page Text and Word Positions (JSON)

Source: https://github.com/muckrock/documentcloud/blob/master/docs/api/api.md

This JSON example shows how to set page text along with detailed word positions and optional OCR engine information. Each word can have its bounding box coordinates (x1, x2, y1, y2) and metadata specified. This is useful for precise text placement and OCR data management.

```json
[
    {
        "page_number": 0,
        "text": "Page 1 text",
        "ocr": "my-ocr-engine",
        "positions": [
            {
                "text": "Page",
                "x1": 0.1,
                "x2": 0.2,
                "y1": 0.1,
                "y2": 0.2,
                "metadata": {"type": "word"}
            },
            {
                "text": "1",
                "x1": 0.3,
                "x2": 0.4,
                "y1": 0.1,
                "y2": 0.2,
                "metadata": {"type": "word"}
            },
            {
                "text": "text",
                "x1": 0.5,
                "x2": 0.6,
                "y1": 0.1,
                "y2": 0.2,
                "metadata": {"type": "word"}
            }
        ]
    }
]
```

--------------------------------

### Generate Translation Messages with Django

Source: https://github.com/muckrock/documentcloud/blob/master/locale/README.rst

This snippet demonstrates how to generate initial translation message files for the DocumentCloud project using Django's management utility. The `makemessages` command scans the project for translatable strings and creates or updates `.po` files in the specified locale directories. Ensure you have Django installed and configured for your project.

```python
python manage.py makemessages
```

--------------------------------

### Initiate Document Processing (API)

Source: https://github.com/muckrock/documentcloud/blob/master/docs/api/api.md

Starts the processing of an uploaded document. An optional 'force_ocr' parameter can be included to enable OCR even for documents with embedded text. Supports single document processing or bulk processing for multiple documents.

```HTTP
POST /api/documents/<id>/process/
{
  "force_ocr": true
}
```

```HTTP
POST /api/document/process/
{
  "documents": [
    {"id": 1, "force_ocr": true},
    {"id": 2}
  ]
}
```

--------------------------------

### Monitor Storage Usage with Time Delay (Bash)

Source: https://github.com/muckrock/documentcloud/blob/master/documentcloud/documents/processing/ocr/README.md

Continuously monitors the size of the 'gs://documentcloud-upload/100muellers' bucket, printing the size every 30 seconds. It uses `gsutil du` and `sed` to extract and sort unique file sizes.

```bash
time=0; while :; do echo $(echo "scale=1;$time/2"|bc -l); time=$((time+1)); sleep 30s & gsutil du gs://documentcloud-upload/100muellers | sed 's/.*\. //' | sort | uniq -c; wait; done
```

--------------------------------

### Deploy Tesseract OCR Cloud Functions (Bash)

Source: https://github.com/muckrock/documentcloud/blob/master/documentcloud/documents/processing/ocr/README.md

Deploys multiple instances of the tesseract OCR Google Cloud Function. Each function is configured with a specific runtime, trigger topic, memory, and timeout.

```bash
gcloud functions deploy run_tesseract_001 --runtime python37 --trigger-topic page-image-extracted --memory=2048MB --timeout 540
gcloud functions deploy run_tesseract2 --runtime python37 --trigger-topic page-image-extracted --memory=2048MB --timeout 540
gcloud functions deploy run_tesseract3 --runtime python37 --trigger-topic page-image-extracted --memory=2048MB --timeout 540
gcloud functions deploy run_tesseract4 --runtime python37 --trigger-topic page-image-extracted --memory=2048MB --timeout 540
```

--------------------------------

### Run Tesseract OCR Cloud Functions with File (Bash)

Source: https://github.com/muckrock/documentcloud/blob/master/documentcloud/documents/processing/ocr/README.md

Executes deployed tesseract OCR Cloud Functions with a specified file path. It first encodes the file path into a JSON payload, then calls the function using `gcloud functions call`.

```bash
FN=$(python -c "import json; import base64; print(base64.b64encode(json.dumps({'paths': ['mueller/275.gif']}).encode('utf-8')).decode('utf-8'))"); time gcloud functions call run_tesseract --data "{\"data\": \"$FN\"}"
FN=$(python -c "import json; import base64; print(base64.b64encode(json.dumps({'paths': ['mueller/275.gif']}).encode('utf-8')).decode('utf-8'))"); time gcloud functions call run_tesseract2 --data "{\"data\": \"$FN\"}"
FN=$(python -c "import json; import base64; print(base64.b64encode(json.dumps({'paths': ['mueller/275.gif']}).encode('utf-8')).decode('utf-8'))"); time gcloud functions call run_tesseract3 --data "{\"data\": \"$FN\"}"
FN=$(python -c "import json; import base64; print(base64.b64encode(json.dumps({'paths': ['mueller/275.gif']}).encode('utf-8')).decode('utf-8'))"); time gcloud functions call run_tesseract4 --data "{\"data\": \"$FN\"}"
```

--------------------------------

### Document Modification: Import Pages

Source: https://github.com/muckrock/documentcloud/blob/master/docs/api/api.md

This example illustrates importing pages from another document into the current document. It specifies the source document using an 'id' and the pages to import ('0-49'). These imported pages are then appended to the end of the target document ('0-447').

```json
[
  { "page": "0-447" },
  {
    "page": "0-49",
    "id": "2000000"
  }
]
```

--------------------------------

### Configure Hosts File for Local Development

Source: https://github.com/muckrock/documentcloud/blob/master/README.md

Appends entries to the system's hosts file to map development domain names to the local loopback address. This ensures that API and Minio services resolve to your local machine.

```bash
echo "127.0.0.1 api.dev.documentcloud.org minio.documentcloud.org" | sudo tee -a /etc/hosts
```

--------------------------------

### Get Organization by ID

Source: https://github.com/muckrock/documentcloud/blob/master/docs/api/api.md

Retrieves detailed information for a specific organization using its unique ID.

```APIDOC
## GET /api/organizations/<id>/

### Description
Retrieves details for a specific organization.

### Method
GET

### Endpoint
/api/organizations/<id>/

### Parameters
#### Path Parameters
- **id** (Integer) - Required - The unique identifier for the organization.

#### Query Parameters
None

### Request Example
None

### Response
#### Success Response (200)
- **id** (Integer) - The ID for the organization
- **name** (String) - The name of the organization
- **slug** (String) - The slug is a URL safe version of the name
- **avatar_url** (URL) - A URL pointing to an avatar for the organization
- **individual** (Bool) - Is this organization for the sole use of an individual
- **uuid** (UUID) - UUID which links this organization to the corresponding organization on the MuckRock Accounts Site
- **monthly_credits** (Integer) - Number of monthly premium credits this organization has left.
- **purchased_credits** (Integer) - Number of purchased premium credits.
- **credit_reset_date** (Date) - The date that `monthly_credits` reset.
- **monthly_credit_allowance** (Integer) - The amount of credits that `monthly_credits` will reset to.
- **plan** (String) - The name of the plan this organization is subscribed to.

#### Response Example
{
  "example": "{
    "id": 1,
    "name": "Example Org",
    "slug": "example-org",
    "avatar_url": "http://example.com/avatar.png",
    "individual": false,
    "uuid": "a1b2c3d4-e5f6-7890-1234-567890abcdef",
    "monthly_credits": 100,
    "purchased_credits": 500,
    "credit_reset_date": "2023-12-31",
    "monthly_credit_allowance": 200,
    "plan": "premium"
  }"
}
```

--------------------------------

### Initialize Environment Variables with Python

Source: https://github.com/muckrock/documentcloud/blob/master/README.md

Runs a Python script to create necessary environment variable files for the development environment. This script is crucial for configuring the application's settings.

```python
python initialize_dotenvs.py
```

--------------------------------

### Navigate to DocumentCloud Directory

Source: https://github.com/muckrock/documentcloud/blob/master/README.md

Changes the current directory to the cloned DocumentCloud repository. This is necessary to run subsequent commands within the project.

```bash
cd documentcloud
```

--------------------------------

### Get Document Embed Code (oEmbed API)

Source: https://context7.com/muckrock/documentcloud/llms.txt

Generates an HTML iframe embed code for a given DocumentCloud document URL. Supports specifying maximum width.

```bash
curl -X GET "https://api.www.documentcloud.org/api/oembed/?url=https://www.documentcloud.org/documents/12345-sample-document/&maxwidth=800"
```

--------------------------------

### Upload File using Presigned URL (HTTP)

Source: https://github.com/muckrock/documentcloud/blob/master/docs/api/api.md

Uploads the binary data of a file to the provided presigned URL. This URL is typically obtained after creating a document record and is valid for 5 minutes. Supports individual file uploads for bulk operations.

```HTTP
PUT <presigned_url>
<binary file data>
```

--------------------------------

### Document Modification: Remove Pages

Source: https://github.com/muckrock/documentcloud/blob/master/docs/api/api.md

This example demonstrates how to remove specific sections of a document. The 'page' parameter specifies the ranges to be removed. Here, pages 0-23 and 423-447 are targeted for removal, effectively removing the middle portion of the document.

```json
[
  {
    "page": "0-23,423-447"
  }
]
```

--------------------------------

### Create Flat Page for Tip of Day

Source: https://github.com/muckrock/documentcloud/blob/master/README.md

Creates a static flat page with the URL '/tipofday/' in the DocumentCloud Django admin. This page is required for certain functionalities and can be left blank.

```bash
# Access via browser: https://api.dev.documentcloud.org/admin/flatpages/flatpage/
# Create a new flat page with URL: /tipofday/
```

--------------------------------

### POST /api/documents/

Source: https://github.com/muckrock/documentcloud/blob/master/docs/api/api.md

Initiates a document upload by creating a document record. The response includes a presigned URL for file upload and the document ID.

```APIDOC
## POST /api/documents/

### Description
Initiates a document upload by creating a document record. You can specify writable document fields (excluding `file_url`). The response will contain all document fields, including `presigned_url` and `id` for the direct upload flow.

Supports bulk uploads by POSTing a list of JSON objects to the same endpoint. The response will be a list of document objects.

### Method
POST

### Endpoint
/api/documents/

### Parameters
#### Request Body
- **title** (String) - Required - The document's title
- **projects** (List:Integer) - Create Only - The IDs of the projects this document belongs to.
- **publish_at** (Date Time) - Not Required - A timestamp when to automatically make this document public.
- **related_article** (URL) - Not Required - The URL for the article about this document.
- **revision_control** (Boolean) - Not Required - Turns revision control on for this document.
- **source** (String) - Not Required - The source who produced the document.
- **original_extension** (String) - Not Required - The original file extension for non-PDF uploads.

### Request Example
```json
{
  "title": "My Document Title",
  "projects": [1, 2],
  "publish_at": "2023-12-31T23:59:59Z",
  "source": "Example Source"
}
```

### Response
#### Success Response (200 or 201)
- **id** (Integer) - The unique identifier for the document.
- **presigned_url** (URL) - The pre-signed URL to directly PUT the PDF file to.
- **title** (String) - The document's title.
- **projects** (List:Integer) - The IDs of the projects this document belongs to.
- **publish_at** (Date Time) - A timestamp when to automatically make this document public.
- **published_url** (URL) - The URL where this document is embedded.
- **related_article** (URL) - The URL for the article about this document.
- **revision_control** (Boolean) - Indicates if revision control is enabled.
- **remaining** (JSON) - The number of pages left for text and image processing (if requested).
- **slug** (String) - A URL safe version of the title.
- **source** (String) - The source who produced the document.
- **status** (String) - The status for the document.
- **title** (String) - The document's title.
- **updated_at** (Date Time) - Time stamp when the document was last updated.
- **user** (Integer) - The ID for the user this document belongs to.

#### Response Example
```json
{
  "id": 123,
  "presigned_url": "https://example.com/upload-url?signature=...",
  "title": "My Document Title",
  "projects": [1, 2],
  "publish_at": "2023-12-31T23:59:59Z",
  "published_url": null,
  "related_article": null,
  "revision_control": false,
  "remaining": null,
  "slug": "my-document-title",
  "source": "Example Source",
  "status": "pending_upload",
  "updated_at": "2023-10-27T10:00:00Z",
  "user": 456
}
```
```

--------------------------------

### Create Document for Direct Upload (API)

Source: https://github.com/muckrock/documentcloud/blob/master/docs/api/api.md

Initiates a document upload by creating a document record. Accepts writable document fields and returns the document's ID and a presigned URL for file upload. Supports single or bulk document creation.

```HTTP
POST /api/documents/
{
  "title": "My Document",
  "project_id": 123,
  "original_extension": "pdf"
}
```

--------------------------------

### Document Modification: Rotate All Pages

Source: https://github.com/muckrock/documentcloud/blob/master/docs/api/api.md

This example shows how to apply a rotation modification to all pages of a document. The 'modifications' array contains an object with 'type' set to 'rotate' and 'angle' set to 'ccw' for counter-clockwise rotation. This applies to the entire document range '0-447'.

```json
[
  {
    "page": "0-447",
    "modifications": [
      {
        "type": "rotate",
        "angle": "ccw"
      }
    ]
  }
]
```

--------------------------------

### Clone DocumentCloud Repository

Source: https://github.com/muckrock/documentcloud/blob/master/README.md

Clones the DocumentCloud git repository to your local machine. This is the first step in setting up the development environment.

```bash
git clone git@github.com:MuckRock/documentcloud.git
```

--------------------------------

### Retrieve Cloud Function Logs (Bash)

Source: https://github.com/muckrock/documentcloud/blob/master/documentcloud/documents/processing/ocr/README.md

Fetches logs for Google Cloud Functions in `us-central1` region, specifically looking for entries containing 'OVERALL_TIME' and timestamps after a certain point. The logs are limited and formatted as JSON.

```bash
gcloud logging read 'resource.type="cloud_function" resource.labels.region="us-central1" "OVERALL_TIME" timestamp>="2019-04-26T17:55:54.907Z"' --limit 280000 --format json > logs_100m_2.json
```

--------------------------------

### Run Subset of DocumentCloud Tests

Source: https://github.com/muckrock/documentcloud/blob/master/README.md

Executes a specific subset of tests within the DocumentCloud project by specifying a path. This is useful for targeted testing during development.

```bash
inv test --path documentcloud/documents
```

--------------------------------

### List Add-on Runs (API)

Source: https://context7.com/muckrock/documentcloud/llms.txt

Lists all add-on runs associated with the authenticated user, with an option to filter by dismissal status. Requires an authorization token.

```bash
curl -X GET "https://api.www.documentcloud.org/api/addon_runs/?dismissed=false" \
  -H "Authorization: Bearer <token>"
```

--------------------------------

### Run Tesseract OCR Function with Queue (Bash)

Source: https://github.com/muckrock/documentcloud/blob/master/documentcloud/documents/processing/ocr/README.md

Invokes a specific tesseract OCR Cloud Function (`run_tesseract_001`) with a file path and an OCR queue identifier. The input is base64 encoded JSON, passed via the `--data` flag.

```bash
FN=$(python -c "import json; import base64; print(base64.b64encode(json.dumps({'paths': ['mueller-test4/97.gif'], 'queue': 'ocr-queue-001'}).encode('utf-8')).decode('utf-8'))"); time gcloud functions call run_tesseract_001 --data "{\"data\": \"$FN\"}"
```

--------------------------------

### Document Modification: Rotate Specific Pages

Source: https://github.com/muckrock/documentcloud/blob/master/docs/api/api.md

This example demonstrates rotating a specific range of pages within a document while leaving the rest untouched. The first 50 pages ('0-49') are rotated 180 degrees ('hw' likely represents 180 degrees), and the remaining pages ('50-447') are left as they are.

```json
[
  {
    "page": "0-49",
    "modifications": [
      {
        "type": "rotate",
        "angle": "hw"
      }
    ]
  },
  { "page": "50-447" }
]
```

--------------------------------

### Expandable Fields Query Parameter (HTTP)

Source: https://github.com/muckrock/documentcloud/blob/master/docs/api/api.md

This demonstrates how to use the 'expand' query parameter in an HTTP request to retrieve nested related resources. It shows how to expand single fields, multiple fields, nested fields, and all expandable fields.

```http
?expand=user
?expand=user,organization
?expand=user.organization
?expand=~all
```

--------------------------------

### Access DocumentCloud API Root

Source: https://github.com/muckrock/documentcloud/blob/master/README.md

Navigates to the DocumentCloud API root URL in a web browser to verify the Django API is running. Note the 'api' prefix in the URL.

```bash
api.dev.documentcloud.org/
```

--------------------------------

### Configure Django Environment Variables

Source: https://github.com/muckrock/documentcloud/blob/master/README.md

Sets specific environment variables within the `.envs/.local/.django` file required for DocumentCloud's Django application. These include Squarelet API credentials and JWT verification keys.

```bash
# Example content for .envs/.local/.django
SQUARELET_KEY=your_squarelet_client_id
SQUARELET_SECRET=your_squarelet_client_secret
JWT_VERIFYING_KEY=your_jwt_verifying_key_here
```

--------------------------------

### Use Page-Offset Pagination

Source: https://github.com/muckrock/documentcloud/blob/master/docs/api/api.md

Enable the older page-offset based pagination system by adding `version=1.0` to API queries. This system includes a top-level `count` key with the total number of objects. Note that this method is less performant than cursor-based pagination and will be disabled in the future.

```http
GET /api/list/?page=2&per_page=50&version=1.0
```

--------------------------------

### Begin Entity Extraction

Source: https://github.com/muckrock/documentcloud/blob/master/docs/api/api.md

Initiates the process of extracting entities from a specified document. This is an asynchronous operation.

```APIDOC
## POST /api/documents/<document_id>/entities/

### Description
Begins the process of extracting entities for a given document.

### Method
POST

### Endpoint
/api/documents/<document_id>/entities/

### Parameters
#### Request Body
(empty)

### Response
#### Success Response (200)
- **message** (String) - Confirmation message indicating entity extraction has started.

#### Response Example
{
  "message": "Entity extraction initiated for document <document_id>."
}
```

--------------------------------

### Manage Projects (DocumentCloud API)

Source: https://context7.com/muckrock/documentcloud/llms.txt

Organize documents and collaborate with users through projects. Supports listing projects (optionally pinned), creating new projects with title, description, and privacy settings, and updating existing projects. Requires an authorization token.

```bash
# List projects
curl -X GET "https://api.www.documentcloud.org/api/projects/?pinned=true" \
  -H "Authorization: Bearer <token>"

# Create project
curl -X POST https://api.www.documentcloud.org/api/projects/ \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d \
    '{ "title": "Investigation 2024", "description": "Documents related to the 2024 investigation", "private": true }'

# Update project
curl -X PATCH https://api.www.documentcloud.org/api/projects/15757/ \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{"title": "Updated Investigation Title", "pinned": false}'
```

--------------------------------

### Initialize Report List and Styles

Source: https://github.com/muckrock/documentcloud/blob/master/documentcloud/documents/processing/tests/reports.html

Defines an array of report file paths and applies CSS styles to HTML elements for layout and appearance. This sets up the visual structure and the list of reports to be loaded.

```javascript
// Add in new reports here.
const subReports = [
  'reports/document_processor.html',
  'reports/png_processor.html',
  'reports/pdf_processor.html',
  'reports/imagediff.html',
];

html, body {
  padding: 0;
  margin: 0;
}

button {
  outline: none;
  padding: 6px 10px;
  margin: 5px;
}

a {
  margin-right: 15px;
}

.container {
  position: absolute;
  top: 0;
  bottom: 0;
  left: 0;
  right: 0;
  margin-top: 50px;
}

.selected {
  background: gray;
  color: white;
}

.reports {
  position: absolute;
  top: -50px;
  width: 100%;
  height: 50px;
  padding: 15px;
  overflow: scroll-y;
}

iframe {
  position: absolute;
  width: 100%;
  height: 100%;
  border: 0px;
}
```

--------------------------------

### Run DocumentCloud Tests

Source: https://github.com/muckrock/documentcloud/blob/master/README.md

Executes all tests for the DocumentCloud project using the Invoke task runner. This command verifies the integrity and functionality of the codebase.

```bash
inv test
```

--------------------------------

### List Add-on Events (API)

Source: https://context7.com/muckrock/documentcloud/llms.txt

Retrieves a list of scheduled add-on events. Requires an authorization token.

```bash
curl -X GET https://api.www.documentcloud.org/api/addon_events/ \
  -H "Authorization: Bearer <token>"
```

--------------------------------

### POST /api/documents/<id>/process/

Source: https://github.com/muckrock/documentcloud/blob/master/docs/api/api.md

Initiates the processing of an uploaded document. Supports an optional 'force_ocr' parameter.

```APIDOC
## POST /api/documents/<id>/process/

### Description
Begins the processing of the document after the file has been uploaded. This endpoint accepts one optional parameter, `force_ocr`, which, if set to true, will perform OCR on the document even if it contains embedded text.

For bulk uploads, a single POST to `/api/document/process/` can be used to initiate processing for multiple documents. This bulk endpoint accepts a list of objects, each containing a document ID and an optional `force_ocr` flag.

### Method
POST

### Endpoint
/api/documents/<id>/process/

### Parameters
#### Path Parameters
- **id** (Integer) - Required - The ID of the document to process.

#### Query Parameters
- **force_ocr** (Boolean) - Optional - If set to true, forces OCR even if the document contains embedded text.

### Request Example
```
POST /api/documents/123/process/?force_ocr=true
```

### Response
#### Success Response (200)
- **status** (String) - The updated status of the document (e.g., "processing").

#### Response Example
```json
{
  "status": "processing"
}
```
```

--------------------------------

### Upload File to Presigned URL (Bash)

Source: https://context7.com/muckrock/documentcloud/llms.txt

Uploads a file to a presigned S3 URL obtained from the API. The URL is valid for 5 minutes. Requires the file path and Content-Type.

```bash
curl -X PUT "https://s3.amazonaws.com/documents/..." \
  -H "Content-Type: application/pdf" \
  --data-binary @/path/to/document.pdf
```