### Start Rails Server

Source: https://github.com/openaustralia/morph/wiki/Windows-install-instructions

Starts the Rails development server, typically using Puma or another application server. This command makes the application accessible for local testing and development, usually at http://localhost:3000.

```bash
rails server
```

--------------------------------

### Install mysql2 Ruby Gem with Custom Path

Source: https://github.com/openaustralia/morph/wiki/Windows-install-instructions

Installs the 'mysql2' Ruby gem, specifying a version and providing a custom path to the MySQL C connector. This is often necessary on Windows when the gem cannot be found automatically.

```bash
gem install mysql2 --version '0.3.18' --platform=ruby -- --with-mysql-dir=C:/path/to/mysql/c-connector
```

--------------------------------

### Install Ruby Version (rbenv)

Source: https://github.com/openaustralia/morph/blob/main/provisioning/README.md

Installs the required Ruby version using rbenv. Assumes rbenv and ruby-build are already installed and configured.

```bash
rbenv install
```

--------------------------------

### Install Python using Mise

Source: https://github.com/openaustralia/morph/blob/main/provisioning/README.md

Installs a specific version of Python using the 'mise' tool version manager. This ensures consistency in the Python environment for development.

```bash
mise install python
```

--------------------------------

### Create and Migrate Database

Source: https://github.com/openaustralia/morph/wiki/Windows-install-instructions

These rake tasks are used to create the application's database and apply all pending migrations to set up the database schema. This is a standard step in Rails application setup.

```bash
rake db:create
rake db:migrate
```

--------------------------------

### Install Vagrant Plugins

Source: https://github.com/openaustralia/morph/blob/main/provisioning/README.md

Installs necessary Vagrant plugins: vagrant-hostsupdater and vagrant-disksize. These plugins extend Vagrant's functionality for managing host entries and disk sizes.

```bash
vagrant plugin install vagrant-hostsupdater vagrant-disksize
```

--------------------------------

### Puma Server Log Output

Source: https://github.com/openaustralia/morph/wiki/Windows-install-instructions

Example log output from the Puma web server during a Rails application startup in the 'test' environment. It indicates the number of threads, the listening address and port, and the initiation of OmniAuth request and callback phases for GitHub authentication.

```log
Puma 2.11.1 starting...
* Min threads: 0, max threads: 16
* Environment: test
* Listening on tcp://0.0.0.0:3000
I, [2016-05-06T14:48:22.981322 #1544]  INFO -- omniauth: (github) Request phase initiated.
I, [2016-05-06T14:48:23.593462 #1544]  INFO -- omniauth: (github) Callback phase initiated.
I, [2016-05-06T14:48:23.663484 #1544]  INFO -- omniauth: (github) Callback phase initiated.
I, [2016-05-06T14:48:23.730505 #1544]  INFO -- omniauth: (github) Callback phase initiated.
I, [2016-05-06T14:48:24.053480 #1544]  INFO -- omniauth: (github) Callback phase initiated.
I, [2016-05-06T14:48:29.218677 #1544]  INFO -- omniauth: (github) Callback phase initiated.
I, [2016-05-06T14:48:59.337696 #1544]  INFO -- omniauth: (github) Callback phase initiated.
I, [2016-05-06T14:49:59.473620 #1544]  INFO -- omniauth: (github) Callback phase initiated.
```

--------------------------------

### Start Local Development Environment with Vagrant

Source: https://github.com/openaustralia/morph/blob/main/provisioning/README.md

Initiates the Vagrant environment for local development. This command builds and provisions a virtual machine that mimics the production environment, accessible at 'dev.morph.io'.

```bash
vagrant up local
```

--------------------------------

### Install Vagrant Plugins

Source: https://github.com/openaustralia/morph/blob/main/provisioning/README.md

Installs essential Vagrant plugins for managing host entries, disk size, and guest additions within the virtual environment. These plugins enhance the functionality of Vagrant for development purposes.

```bash
vagrant plugin install vagrant-hostsupdater vagrant-disksize vagrant-vbguest
```

--------------------------------

### Install Ansible Roles with Make

Source: https://github.com/openaustralia/morph/blob/main/provisioning/README.md

Installs required Ansible roles using a 'make' command. This is a convenient shortcut for installing dependencies defined in the project's Makefile.

```bash
make roles
```

--------------------------------

### Start Full Docker Development Environment (BETA)

Source: https://github.com/openaustralia/morph/blob/main/README.md

A Make target to launch a full Docker-based development environment, including Ruby containers. This is a BETA feature, indicating it may be experimental or subject to change.

```bash
make docker-up
```

--------------------------------

### Run Quick and Full Test Suites

Source: https://github.com/openaustralia/morph/blob/main/TESTING.md

This command provides a convenient way to run the quick tests first to quickly verify basic functionality. If the quick tests pass, you can then proceed to run the full test suite for more comprehensive validation. This two-step approach helps in rapid feedback cycles during development.

```shell
make test
```

--------------------------------

### Install Ruby Gem Dependencies

Source: https://github.com/openaustralia/morph/blob/main/README.md

Command to install the necessary Ruby gem dependencies for the project within the web container. This is a crucial step after initial configuration to ensure the application has all its required libraries.

```bash
bundle install
```

--------------------------------

### Manage Development Services with Docker Compose

Source: https://github.com/openaustralia/morph/blob/main/README.md

Commands to start, stop, and view logs for services like Redis, Elasticsearch, and MySQL used in development and CI. These commands leverage Docker Compose to manage the service containers. You can specify which services to start or view logs for using the SERVICES environment variable.

```bash
make services-up
# To exclude mysql: SERVICES="redis elasticsearch" make services-up

make services-down

make services-logs
# For specific services: SERVICES='elasticsearch redis' make services-logs

make services-status
```

--------------------------------

### Install Python Dependencies with Pip

Source: https://github.com/openaustralia/morph/blob/main/provisioning/README.md

Installs Python project dependencies using pip, typically from a requirements file. This command ensures all necessary libraries are available for the project to run.

```bash
pip install -r provisioning/requirements.txt
```

--------------------------------

### Install Python Development Libraries on Ubuntu

Source: https://github.com/openaustralia/morph/blob/main/provisioning/README.md

Installs essential development libraries for Python on Ubuntu systems. These libraries, such as libreadline-dev and libsqlite3-dev, are often required for compiling Python modules or running certain Python applications.

```bash
sudo apt install libreadline-dev libsqlite3-dev
```

--------------------------------

### Configure Morph Development Environment

Source: https://github.com/openaustralia/morph/blob/main/README.md

Steps to configure the local development environment by copying example configuration files and editing them according to specific settings. This includes database connection details and environment variables.

```bash
cp config/database.yml.example config/database.yml
cp env-example .env
cp env-staging-example .env.staging # if needed
cp env-staging-example .env.vagrant # if needed

# Edit config/database.yml with your database settings
# Edit .env, .env.staging, .env.vagrant with your local environment and vagrant development settings
```

--------------------------------

### Run Guard for Live Reloading

Source: https://github.com/openaustralia/morph/blob/main/README.md

Starts Guard, which automatically reloads the web page when view files are edited during development. This is useful for rapid design and development cycles.

```shell
bundle exec guard
```

--------------------------------

### Install Mailcatcher for Development Mails

Source: https://github.com/openaustralia/morph/blob/main/README.md

Installs the Mailcatcher gem, which is used to catch and display emails sent during development, preventing them from being sent to actual recipients.

```shell
gem install mailcatcher
```

--------------------------------

### Install Ansible Roles with Ansible Galaxy

Source: https://github.com/openaustralia/morph/blob/main/provisioning/README.md

Installs the necessary Ansible roles required for provisioning. This command should be run within the 'provisioning' directory. The roles are typically managed separately and not committed to the main repository.

```bash
cd provisioning
ansible-galaxy install -r requirements.yml -p roles
```

--------------------------------

### Configure RSpec GitHub Integration Test Exclusion

Source: https://github.com/openaustralia/morph/blob/main/TESTING.md

This snippet explains how to exclude tests related to GitHub integration. These tests require a specific private key file ('config/morph-github-app.private-key.pem'). Setting the DONT_RUN_GITHUB_TESTS environment variable to '1' will force the exclusion of these tests, which is useful during initial setup or when GitHub integration is not being actively tested.

```shell
DONT_RUN_GITHUB_TESTS=1
```

--------------------------------

### Install Capistrano Gem

Source: https://github.com/openaustralia/morph/blob/main/provisioning/README.md

Installs Capistrano, a remote server automation and deployment tool, as a Ruby gem. This is a dependency for certain deployment or management tasks within the project.

```bash
gem install capistrano
```

--------------------------------

### Search and Discovery with Elasticsearch

Source: https://context7.com/openaustralia/morph/llms.txt

Implements full-text search functionality across scrapers using Elasticsearch. This section provides a controller example for handling search queries.

```ruby
# In SearchController (app/controllers/search_controller.rb)

```

--------------------------------

### Background Job Processing with Sidekiq

Source: https://context7.com/openaustralia/morph/llms.txt

Queues scraper runs and processing tasks using Sidekiq. Includes examples for enqueuing runs, auto-run scheduling, and webhook deliveries. Handles errors by updating run status.

```ruby
# In RunWorker (app/workers/run_worker.rb)
class RunWorker
  include Sidekiq::Worker

  def perform(run_id)
    run = Run.find(run_id)
    runner = Morph::Runner.new(run)
    runner.go do |timestamp, stream, text|
      # Log output to database
      run.log_lines.create!(
        timestamp: timestamp,
        stream: stream,
        text: text
      )
    end
  rescue => e
    run.update!(
      status_code: 999,
      finished_at: Time.zone.now
    )
    raise
  end
end

# Queue a run
RunWorker.perform_async(run.id)

# Auto-run scheduling
ScraperAutoRunWorker.perform_async(scraper.id)

# Webhook delivery
DeliverWebhookWorker.perform_async(webhook_delivery.id)
```

--------------------------------

### Obtain Production SSL Certificate (Certbot)

Source: https://github.com/openaustralia/morph/blob/main/provisioning/README.md

Obtains SSL certificates for production domains using Certbot with the Nginx plugin. This command should be run as root on a fresh install.

```bash
certbot --nginx certonly -m contact@oaf.org.au --agree-tos
```

--------------------------------

### Create Scraper from GitHub Repository

Source: https://context7.com/openaustralia/morph/llms.txt

Import an existing GitHub repository to create a new scraper on Morph.io. This process involves creating a Scraper object and queuing a background worker for asynchronous processing. An example API call is also provided.

```ruby
# In ScrapersController#create_github
scraper = Scraper.new_from_github("username/repo_name", current_user)
if scraper.save
  scraper.create_create_scraper_progress!(
    heading: "Adding from GitHub",
    message: "Queuing",
    progress: 5
  )
  CreateFromGithubWorker.perform_async(scraper.id)
  redirect_to scraper
end

# Example API call
# POST /scrapers/github
# Parameters: scraper[full_name] = "openaustralia/test-scraper"
```

--------------------------------

### Clean Project Development Artifacts

Source: https://github.com/openaustralia/morph/blob/main/README.md

Commands to clean up various development artifacts, including virtual environments, installed roles, cached data, and Docker resources. The 'clobber' target is particularly destructive, removing all generated files and logs.

```bash
make clean
make clobber
make docker-clean
```

--------------------------------

### Localhost Empty Response Error

Source: https://github.com/openaustralia/morph/wiki/Windows-install-instructions

Indicates a 'ERR_EMPTY_RESPONSE' error when trying to access http://127.0.0.1:3000. This means the server sent no data back, often due to the server not running correctly or an issue with the application's response handling.

```html
The 127.0.0.1 page isn’t working

127.0.0.1 didn’t send any data.
ERR_EMPTY_RESPONSE
```

--------------------------------

### Docker-Based Scraper Execution with Morph.io

Source: https://context7.com/openaustralia/morph/llms.txt

Execute scrapers within isolated Docker containers, enforcing resource limits and providing environment variables. This snippet demonstrates compiling, starting, attaching logs, and finishing a Dockerized scraper run. It supports custom labels and platform configurations.

```ruby
# In Morph::DockerRunner (app/lib/morph/docker_runner.rb:60)
container = Morph::DockerRunner.compile_and_start_run(
  repo_path: "/path/to/scraper/code",
  env_variables: {
    "MORPH_API_KEY" => "secret_key_123",
    "DATABASE_NAME" => "data.sqlite"
  },
  container_labels: {
    "io.morph.scraper_id" => "42",
    "io.morph.run_id" => "1337"
  },
  max_lines: 10000,
  platform: "heroku-18",
  memory: 512 * 1024 * 1024  # 512 MB
) do |timestamp, stream, text|
  puts "[#{stream}] #{text}"
end

# Attach to running container and stream logs
Morph::DockerRunner.attach_to_run(container) do |timestamp, stream, text|
  LogLine.create!(
    run_id: run.id,
    stream: stream,
    text: text,
    timestamp: timestamp
  )
end

# Cleanup and extract results
result = Morph::DockerRunner.finish(container, ["data.sqlite"])
puts "Exit code: #{result.status_code}"
puts "CPU time: #{result.time_params[:cpu_time]}"
```

--------------------------------

### Set RAILS_ENV for Test Environment

Source: https://github.com/openaustralia/morph/wiki/Windows-install-instructions

Sets the RAILS_ENV environment variable to 'test'. This is typically done to isolate testing from development or production environments and to avoid loading certain development-specific gems like 'rack-mini-profiler'.

```bash
SET RAILS_ENV=test
```

--------------------------------

### Configure RSpec Slow Test Execution

Source: https://github.com/openaustralia/morph/blob/main/TESTING.md

This snippet shows how to configure RSpec to run tests tagged as slow. By default, these tests are skipped. Setting the RUN_SLOW_TESTS environment variable to '1' will enable their execution. This is useful for comprehensive testing when needed.

```shell
RUN_SLOW_TESTS=1
```

--------------------------------

### Configure RSpec Docker Test Exclusion

Source: https://github.com/openaustralia/morph/blob/main/TESTING.md

This snippet demonstrates how to prevent RSpec from running tests that depend on a live Docker server. These tests are not explicitly tagged as slow but can significantly increase test execution time. Setting the DONT_RUN_DOCKER_TESTS environment variable to '1' will exclude these tests.

```shell
DONT_RUN_DOCKER_TESTS=1
```

--------------------------------

### Launch and Provision Vagrant VM

Source: https://github.com/openaustralia/morph/blob/main/README.md

Make targets to manage the local Vagrant virtual machine used for development. This includes launching the VM, provisioning it with Ansible, and deploying the application to it.

```bash
make vagrant-up
make vagrant-provision
make vagrant-deploy
```

--------------------------------

### Provision and Deploy to Production

Source: https://github.com/openaustralia/morph/blob/main/README.md

Make targets for provisioning the production environment using Ansible and deploying the application to production. These commands are intended for use in a production deployment workflow.

```bash
make production-provision
make production-deploy
```

--------------------------------

### Obtain Staging SSL Certificate (Certbot Manual)

Source: https://github.com/openaustralia/morph/blob/main/provisioning/README.md

Obtains SSL certificates for the local development VM (dev.morph.io) using Certbot in manual mode with DNS challenges. Requires adding TXT records to DNS.

```bash
sudo certbot certonly --manual -d dev.morph.io --preferred-challenges dns -d api.dev.morph.io -d faye.dev.morph.io -d help.dev.morph.io
```

--------------------------------

### Authentication and Authorization

Source: https://context7.com/openaustralia/morph/llms.txt

Explains the GitHub OAuth integration for authentication and the use of CanCanCan for authorization.

```APIDOC
## Authentication and Authorization

GitHub OAuth integration with CanCanCan authorization.

### OAuth Callback

Automatically handles the GitHub OAuth flow.

**Endpoint:** `/users/auth/github`

### Authorization Checks

Checks are performed using `authorize!` within controllers.

**Example (ScrapersController):**

```ruby
# app/controllers/scrapers_controller.rb
class ScrapersController < ApplicationController
  def show
    @scraper = Scraper.friendly.find(params[:id])
    authorize! :show, @scraper  # Raises CanCan::AccessDenied if unauthorized
  end

  def update
    authorize! :update, @scraper
    @scraper.update!(scraper_params)
  end
end
```

### Ability Definitions

Ability definitions specify user permissions.

**Example (ScraperAbility):**

```ruby
# app/abilities/scraper_ability.rb
ability = ScraperAbility.new(current_user)
ability.can?(:show, scraper)      # Check if user can view
ability.can?(:update, scraper)    # Check if user can modify
ability.can?(:destroy, scraper)   # Check if user can delete
ability.can?(:data, scraper)      # Check if user can access data
```
```

--------------------------------

### Deploy Application

Source: https://github.com/openaustralia/morph/blob/main/provisioning/README.md

Deploys the application to the created Vagrant box using Capistrano. This command should be run after the Vagrant environment is up and provisioned.

```bash
cap local deploy
```

--------------------------------

### Webhook Delivery System

Source: https://context7.com/openaustralia/morph/llms.txt

System for triggering HTTP callbacks when scraper runs complete.

```APIDOC
## Webhook Delivery System

Trigger HTTP callbacks when scraper runs complete.

### Create Webhook

Associate a webhook with a scraper.

**Example:**

```ruby
# app/models/webhook.rb
webhook = scraper.webhooks.create!(url: "https://example.com/api/scraper_done")
```

### Automatic Delivery

Webhooks are delivered automatically after a run completes.

**Trigger:** `scraper.deliver_webhooks(run)`

**Note:** Webhook delivery is asynchronous and includes retry logic.

### Incoming Webhook Request

**Method:** POST

**URL:** `https://example.com/api/scraper_done` (or the URL configured for the webhook)

**Headers:**
- `Content-Type: application/json`

**Body:**

```json
{
  "scraper_name": "council_scraper",
  "run_id": 1337,
  "status_code": 0,
  "finished_at": "2025-12-30T23:45:00Z",
  "records_added": 42,
  "wall_time": 12.5
}
```
```

--------------------------------

### Scraper Creation from GitHub

Source: https://context7.com/openaustralia/morph/llms.txt

Import existing GitHub repositories as scrapers on the Morph.io platform. This process typically involves an API call to initiate the import and background workers to handle the creation.

```APIDOC
## POST /scrapers/github

### Description
Imports a GitHub repository as a new scraper on Morph.io. This endpoint initiates the process, and background workers handle the actual creation and setup.

### Method
POST

### Endpoint
`/scrapers/github`

### Request Body
- **scraper[full_name]** (string) - Required - The full name of the GitHub repository in `username/repo_name` format.

### Request Example
```json
{
  "scraper[full_name]": "openaustralia/test-scraper"
}
```

### Response
#### Success Response (200 or 302 Redirect)
- Typically redirects to the newly created scraper's page or returns a success status.

#### Response Example
(No specific JSON response example provided, often involves a redirect)
```

--------------------------------

### Webhook Delivery System for Scraper Completion

Source: https://context7.com/openaustralia/morph/llms.txt

Enables triggering HTTP callbacks when scraper runs complete. Shows how to create webhooks, trigger automatic delivery after a run, and outlines the expected JSON payload format for the webhook request.

```ruby
# Create webhook (app/models/webhook.rb)
webhook = scraper.webhooks.create!(url: "https://example.com/api/scraper_done")

# Automatic delivery after run (app/models/scraper.rb:358)
scraper.deliver_webhooks(run)

# Webhook is delivered asynchronously with retry logic
# POST https://example.com/api/scraper_done
# Content-Type: application/json
#
# {
#   "scraper_name": "council_scraper",
#   "run_id": 1337,
#   "status_code": 0,
#   "finished_at": "2025-12-30T23:45:00Z",
#   "records_added": 42,
#   "wall_time": 12.5
# }
```

--------------------------------

### Deploy to Production using Capistrano

Source: https://github.com/openaustralia/morph/blob/main/README.md

Deploys the morph.io application to a production server using Capistrano. This command initiates the deployment process.

```shell
cap production deploy
```

--------------------------------

### Configure Mise for Idiomatic Version Files

Source: https://github.com/openaustralia/morph/blob/main/provisioning/README.md

Configures the 'mise' tool to automatically use version files (e.g., .ruby-version, .python-version) for managing tool versions. This ensures that the correct versions of Ruby and Python are used for the project.

```toml
[settings]
idiomatic_version_file_enable_tools = ["ruby","python"]
```

--------------------------------

### Search Scrapers by Name and Description (Ruby)

Source: https://context7.com/openaustralia/morph/llms.txt

This snippet demonstrates how to search for scrapers using keywords in their name, description, or scraped domain names. It filters results to include only scrapers with associated data and paginates the output. Dependencies include the Morph.io client library. Inputs are search terms and pagination parameters; outputs are formatted scraper details.

```ruby
results = Scraper.search(
  "council meetings",
  fields: [:full_name, :description, :scraped_domain_names],
  where: { data?: true },  # Only scrapers with data
  page: 1,
  per_page: 20
)

results.each do |scraper|
  puts "#{scraper.full_name}: #{scraper.description}"
  puts "  Domains: #{scraper.scraped_domain_names.join(', ')}"
  puts "  Records: #{scraper.sqlite_total_rows}"
end
```

--------------------------------

### Environment Variables and Secret Management

Source: https://context7.com/openaustralia/morph/llms.txt

Stores and injects configuration into scraper runs using environment variables. Demonstrates creating variables, how they are injected as environment variables during a run, and how to access them within scraper code (e.g., Python).

```ruby
# Create variables (app/models/variable.rb)
scraper.variables.create!(name: "API_KEY", value: "secret_123")
scraper.variables.create!(name: "BASE_URL", value: "https://api.example.com")

# Variables are injected as environment variables during run
run = scraper.runs.create!(queued_at: Time.zone.now)
env_vars = run.env_variables
# => {"API_KEY" => "secret_123", "BASE_URL" => "https://api.example.com"}

# Access in scraper code (Python example)
# import os
# api_key = os.environ['API_KEY']
# base_url = os.environ['BASE_URL']
```

--------------------------------

### Access and Run Commands within Docker Containers

Source: https://github.com/openaustralia/morph/blob/main/README.md

Instructions for obtaining a bash shell within the running web container for interactive work or for executing commands in a temporary container. This is useful for debugging and performing container-specific tasks.

```bash
# Get a bash shell in the running web container:
docker compose exec web bash -i

# Run commands in a temporary container:
docker compose run web --rm -it bash -i
```

--------------------------------

### Run Ansible Playbooks

Source: https://github.com/openaustralia/morph/blob/main/provisioning/README.md

Executes Ansible playbooks to provision or update the production infrastructure. This command is used after making changes to Ansible configurations.

```bash
make ansible
```

--------------------------------

### Build mitmdump Docker Image

Source: https://github.com/openaustralia/morph/blob/main/provisioning/README.md

Rebuilds the mitmdump Docker image with custom SSL certificates. This involves cloning the buildstep repository, copying the CA certificate, linking the Dockerfile, and building the image.

```bash
cd
git clone https://github.com/openaustralia/buildstep.git
cd buildstep
cp /var/www/current/docker_images/morph-mitmdump/mitmproxy/mitmproxy-ca-cert.pem .
ln -s Dockerfile.heroku-24  Dockerfile
docker image build -t openaustralia/buildstep:latest .
```

--------------------------------

### Environment Variables and Secret Management

Source: https://context7.com/openaustralia/morph/llms.txt

How to store and inject environment variables and secrets into scraper runs.

```APIDOC
## Environment Variables and Secret Management

Store and inject configuration into scraper runs.

### Create Variables

Define environment variables associated with a scraper.

**Example:**

```ruby
# app/models/variable.rb
scraper.variables.create!(name: "API_KEY", value: "secret_123")
scraper.variables.create!(name: "BASE_URL", value: "https://api.example.com")
```

### Variable Injection

Variables are automatically injected as environment variables during a scraper run.

**Example:**

```ruby
run = scraper.runs.create!(queued_at: Time.zone.now)
env_vars = run.env_variables
# => {"API_KEY" => "secret_123", "BASE_URL" => "https://api.example.com"}
```

### Accessing Variables in Scraper Code

Variables can be accessed using standard environment variable methods in the scraper's language.

**Example (Python):**

```python
import os
api_key = os.environ['API_KEY']
base_url = os.environ['BASE_URL']
```
```

--------------------------------

### Search and Discovery

Source: https://context7.com/openaustralia/morph/llms.txt

Information on the full-text search functionality for scrapers using Elasticsearch.

```APIDOC
## Search and Discovery

Full-text search across scrapers using Elasticsearch.

### Search Functionality

Provides the ability to search for scrapers using a full-text search engine.

**Example (Controller):**

```ruby
# In SearchController (app/controllers/search_controller.rb)
# ... search logic using Elasticsearch ...
```
```

--------------------------------

### Manually Trigger Scraper Run

Source: https://context7.com/openaustralia/morph/llms.txt

Initiate a scraper run programmatically using the Scraper model or via an API endpoint. Includes checking the status of a scraper run, such as if it's running, finished successfully, or failed with errors.

```ruby
# In Scraper model (app/models/scraper.rb:305)
scraper = Scraper.friendly.find("username/scraper_name")
scraper.queue!  # Creates run and queues worker

# Or via API endpoint
# POST /username/scraper_name/run
# Requires authentication and proper permissions

# Check run status
run = scraper.last_run
if run.running?
  puts "Currently running"
elsif run.finished_successfully?
  puts "Completed successfully with #{run.status_code} exit code"
  puts "Wall time: #{run.wall_time} seconds"
elsif run.finished_with_errors?
  puts "Failed with errors: #{run.error_text}"
end
```

--------------------------------

### Run Tests and Lint Code

Source: https://github.com/openaustralia/morph/blob/main/README.md

Commands to execute the project's tests using RSpec and to perform code linting. These are essential steps for maintaining code quality and ensuring functionality during development.

```bash
bin/rake db:test:prepare
bin/rake

make test
make lint
```

--------------------------------

### Database Statistics

Source: https://context7.com/openaustralia/morph/llms.txt

Retrieves and displays various statistics about the database, such as total rows, table names, database size, and validity.

```ruby
puts "Total rows: #{database.sqlite_total_rows}"
puts "Tables: #{database.table_names.join(', ')}"
puts "Database size: #{database.sqlite_db_size} bytes"
puts "Valid database: #{database.valid?}"
```

--------------------------------

### GitHub OAuth and CanCanCan Authorization

Source: https://context7.com/openaustralia/morph/llms.txt

Integrates GitHub OAuth for authentication and uses CanCanCan for authorization. Automatically handles the OAuth callback flow and performs authorization checks for controller actions and ability definitions.

```ruby
# OAuth callback (app/controllers/users/omniauth_callbacks_controller.rb)
# Automatically handles GitHub OAuth flow
# Visit: /users/auth/github

# Authorization checks
class ScrapersController < ApplicationController
  def show
    @scraper = Scraper.friendly.find(params[:id])
    authorize! :show, @scraper  # Raises CanCan::AccessDenied if unauthorized
  end

  def update
    authorize! :update, @scraper
    @scraper.update!(scraper_params)
  end
end

# Ability definitions (app/abilities/scraper_ability.rb)
ability = ScraperAbility.new(current_user)
ability.can?(:show, scraper)      # Check if user can view
ability.can?(:update, scraper)    # Check if user can modify
ability.can?(:destroy, scraper)   # Check if user can delete
ability.can?(:data, scraper)      # Check if user can access data
```

--------------------------------

### Manual Scraper Run Execution

Source: https://context7.com/openaustralia/morph/llms.txt

Manually trigger a scraper run programmatically. This endpoint allows you to execute a scraper and then check its status.

```APIDOC
## POST /username/scraper_name/run

### Description
Triggers an immediate execution of a specified scraper. This endpoint requires authentication and appropriate permissions.

### Method
POST

### Endpoint
`/username/scraper_name/run`

### Parameters
(No explicit parameters listed for this endpoint, but authentication is required)

### Request Example
```bash
# Example using curl (authentication details would be needed)
curl -X POST https://morph.io/username/scraper_name/run
```

### Response
#### Success Response (200 or 202 Accepted)
- Indicates that the run has been queued successfully.

#### Response Example
(No specific JSON response example provided)

### Checking Run Status
After triggering a run, you can check the status of the last run:

```ruby
scraper = Scraper.friendly.find("username/scraper_name")
run = scraper.last_run

if run.running?
  puts "Currently running"
elsif run.finished_successfully?
  puts "Completed successfully with #{run.status_code} exit code"
elsif run.finished_with_errors?
  puts "Failed with errors: #{run.error_text}"
end
```
```

--------------------------------

### Background Job Processing

Source: https://context7.com/openaustralia/morph/llms.txt

Details on how background jobs are processed using Sidekiq, including running scrapers, auto-run scheduling, and webhook delivery.

```APIDOC
## Background Job Processing

Queue scraper runs and processing tasks with Sidekiq.

### Example: Running a Scraper

```ruby
# In RunWorker (app/workers/run_worker.rb)
class RunWorker
  include Sidekiq::Worker

  def perform(run_id)
    run = Run.find(run_id)
    runner = Morph::Runner.new(run)
    runner.go do |timestamp, stream, text|
      # Log output to database
      run.log_lines.create!(
        timestamp: timestamp,
        stream: stream,
        text: text
      )
    end
  rescue => e
    run.update!(
      status_code: 999,
      finished_at: Time.zone.now
    )
    raise
  end
end

# Queue a run
RunWorker.perform_async(run.id)
```

### Example: Auto-run Scheduling

```ruby
ScraperAutoRunWorker.perform_async(scraper.id)
```

### Example: Webhook Delivery

```ruby
DeliverWebhookWorker.perform_async(webhook_delivery.id)
```
```

--------------------------------

### Remote Code Execution API with cURL

Source: https://context7.com/openaustralia/morph/llms.txt

Provides a command-line interface using cURL to execute uploaded code remotely. Demonstrates creating a tarball of scraper code, uploading it via POST request with an API key, and handling streaming JSON line responses for stdout and stderr.

```bash
# Create tarball of scraper code
tar -czf scraper.tar.gz scraper.py requirements.txt

# Upload and run (streaming response)
curl -X POST https://morph.io/run \
  -H "X-API-KEY: your_api_key_here" \
  -F "code=@scraper.tar.gz" \
  --no-buffer

# Response format (streaming JSON lines):
# {"stream":"internalout","text":"Injecting configuration and compiling...\n"}
# {"stream":"stdout","text":"Starting scraper...\n"}
# {"stream":"stdout","text":"Scraped 42 records\n"}
# {"stream":"stderr","text":"Warning: Rate limit approaching\n"}
```

--------------------------------

### Scraper Model Operations and Metadata

Source: https://context7.com/openaustralia/morph/llms.txt

Manages core scraper operations, including creation, retrieval of statistics (run time, CPU time, disk usage, download count), recent run information, status checks (requires attention, running, runnable), and language detection.

```ruby
# In Scraper model (app/models/scraper.rb)
scraper = Scraper.create!(
  name: "council_scraper",
  full_name: "myuser/council_scraper",
  owner: current_user,
  description: "Scrapes council meeting data",
  auto_run: true,
  private: false,
  original_language_key: "python"
)

# Run statistics
puts "Average run time: #{scraper.average_successful_wall_time} seconds"
puts "Total CPU time: #{scraper.cpu_time} seconds"
puts "Total disk usage: #{scraper.total_disk_usage} bytes"
puts "Download count: #{scraper.download_count}"

# Recent runs
scraper.successful_runs.limit(10).each do |run|
  puts "Run #{run.id}: #{run.finished_at} - #{run.wall_time}s"
end

# Check status
puts "Requires attention: #{scraper.requires_attention?}"
puts "Currently running: #{scraper.running?}"
puts "Can run: #{scraper.runnable?}"

# Language detection
language = scraper.language
puts "Language: #{language.to_s}" if language
puts "Main scraper file: #{scraper.main_scraper_filename}"
```

--------------------------------

### Scraper Model Operations

Source: https://context7.com/openaustralia/morph/llms.txt

Core operations for managing scrapers, including creation, retrieving statistics, and checking status.

```APIDOC
## Scraper Model Operations

Core scraper management and metadata.

### Create Scraper

Creates a new scraper record.

**Example:**

```ruby
# In Scraper model (app/models/scraper.rb)
scraper = Scraper.create!(
  name: "council_scraper",
  full_name: "myuser/council_scraper",
  owner: current_user,
  description: "Scrapes council meeting data",
  auto_run: true,
  private: false,
  original_language_key: "python"
)
```

### Scraper Statistics

Retrieve various statistics about a scraper.

**Example:**

```ruby
puts "Average run time: #{scraper.average_successful_wall_time} seconds"
puts "Total CPU time: #{scraper.cpu_time} seconds"
puts "Total disk usage: #{scraper.total_disk_usage} bytes"
puts "Download count: #{scraper.download_count}"
```

### Recent Runs

Access recent successful runs of a scraper.

**Example:**

```ruby
scraper.successful_runs.limit(10).each do |run|
  puts "Run #{run.id}: #{run.finished_at} - #{run.wall_time}s"
end
```

### Status Checks

Check the operational status of a scraper.

**Example:**

```ruby
puts "Requires attention: #{scraper.requires_attention?}"
puts "Currently running: #{scraper.running?}"
puts "Can run: #{scraper.runnable?}"
```

### Language Detection

Determine the primary language of the scraper.

**Example:**

```ruby
language = scraper.language
puts "Language: #{language.to_s}" if language
puts "Main scraper file: #{scraper.main_scraper_filename}"
```
```

--------------------------------

### Database Query Interface for SQLite

Source: https://context7.com/openaustralia/morph/llms.txt

Interact with SQLite databases provided by Morph.io scrapers. This includes executing SQL queries with streaming support for large result sets, executing small queries that load all results into memory, and performing safe queries with error handling for potentially corrupted databases or invalid tables.

```ruby
# In Morph::Database (app/lib/morph/database.rb)
database = Morph::Database.new("/data/username/scraper_name")

# Stream large result sets
database.sql_query_streaming("SELECT * FROM data WHERE country = 'AU'") do |row|
  puts "Name: #{row['name']}, Value: #{row['value']}"
end

# Small queries (keep in memory)
results = database.sql_query("SELECT COUNT(*) as total FROM data")
puts "Total records: #{results.first['total']}"

# Safe queries with error handling
if results = database.sql_query_safe("SELECT * FROM potentially_bad_table")
  results.each { |row| process_row(row) }
else
  puts "Query failed or database corrupted"
end
```

--------------------------------

### Access Scraper Data via API

Source: https://context7.com/openaustralia/morph/llms.txt

Retrieve scraper data in various formats (JSON, CSV, SQLite, Atom) using the Morph.io API. Supports SQL queries and JSONP callbacks. Requires an API key for authentication.

```bash
# Get data as JSON with SQL query
curl -H "X-API-KEY: your_api_key_here" \
  "https://morph.io/username/scraper_name/data.json?query=select%20*%20from%20data%20limit%2010"

# Get data as CSV
curl -H "X-API-KEY: your_api_key_here" \
  "https://morph.io/username/scraper_name/data.csv?query=select%20*%20from%20data"

# Download entire SQLite database
curl -H "X-API-KEY: your_api_key_here" \
  "https://morph.io/username/scraper_name/data.sqlite" \
  -o scraper_data.sqlite

# Get data as Atom feed (requires title, content, link, date columns)
curl -H "X-API-KEY: your_api_key_here" \
  "https://morph.io/username/scraper_name/data.atom?query=select%20title%2C%20content%2C%20link%2C%20date%20from%20data"

# JSONP callback support
curl "https://morph.io/username/scraper_name/data.json?query=select%20*%20from%20data&callback=myCallback&key=your_api_key"
```

--------------------------------

### Remote Code Execution API

Source: https://context7.com/openaustralia/morph/llms.txt

API endpoint for remotely executing uploaded code via the command-line client.

```APIDOC
## Remote Code Execution API

Run uploaded code remotely via the command-line client.

### Method
POST

### Endpoint
`/run`

### Parameters
#### Request Body
- **code** (file) - Required - A tarball (`.tar.gz`) containing the scraper code and its dependencies.

### Request Example

```bash
# Create tarball of scraper code
tar -czf scraper.tar.gz scraper.py requirements.txt

# Upload and run (streaming response)
curl -X POST https://morph.io/run \
  -H "X-API-KEY: your_api_key_here" \
  -F "code=@scraper.tar.gz" \
  --no-buffer
```

### Response Format
Streaming JSON lines.

#### Response Example

```json
{"stream":"internalout","text":"Injecting configuration and compiling...\n"}
{"stream":"stdout","text":"Starting scraper...\n"}
{"stream":"stdout","text":"Scraped 42 records\n"}
{"stream":"stderr","text":"Warning: Rate limit approaching\n"}
```
```

--------------------------------

### Restart Discourse Container

Source: https://github.com/openaustralia/morph/blob/main/provisioning/README.md

Restarts the Discourse application container. This command is used when the Discourse container stops unexpectedly and needs to be rebuilt and restarted.

```bash
root@morph:/var/discourse# ./launcher rebuild app
```

--------------------------------

### Disable Docker Tests with Guard

Source: https://github.com/openaustralia/morph/blob/main/README.md

Runs Guard while disabling slow integration tests that run against a Docker server. This is achieved by setting the DONT_RUN_DOCKER_TESTS environment variable.

```shell
DONT_RUN_DOCKER_TESTS=1 bundle exec guard
```

--------------------------------

### Search Scrapers API

Source: https://context7.com/openaustralia/morph/llms.txt

This endpoint allows you to search for scrapers based on keywords in their name, description, or scraped domains. It returns a list of matching scrapers with their details.

```APIDOC
## GET /search

### Description
Searches for scrapers by name, description, or scraped domains.

### Method
GET

### Endpoint
/search

### Parameters
#### Query Parameters
- **q** (string) - Required - The search query string.
- **fields** (array of strings) - Optional - Fields to include in the results (e.g., `full_name`, `description`, `scraped_domain_names`).
- **where** (object) - Optional - Filtering criteria for scrapers (e.g., `{ data?: true }`).
- **page** (integer) - Optional - The page number for pagination. Defaults to 1.
- **per_page** (integer) - Optional - The number of results per page. Defaults to 20.

### Request Example
```
GET /search?q=council+meetings&fields[]=full_name&fields[]=description&where={ "data?": true }&page=1&per_page=20
```

### Response
#### Success Response (200)
- **scrapers** (array) - An array of scraper objects.
  - **full_name** (string) - The full name of the scraper.
  - **description** (string) - The description of the scraper.
  - **scraped_domain_names** (array of strings) - A list of domains scraped by the scraper.
  - **sqlite_total_rows** (integer) - The total number of rows in the scraper's database.

#### Response Example
```json
{
  "scrapers": [
    {
      "full_name": "Example Scraper",
      "description": "This scraper collects data about council meetings.",
      "scraped_domain_names": ["example.com", "council.gov"],
      "sqlite_total_rows": 1500
    }
  ]
}
```
```

--------------------------------

### API Data Access Endpoint

Source: https://context7.com/openaustralia/morph/llms.txt

Query and retrieve scraper data in multiple formats (JSON, CSV, SQLite, Atom feed) with streaming support. Requires an API key for authentication.

```APIDOC
## GET /username/scraper_name/data.<format>

### Description
Retrieves scraper data in a specified format (e.g., .json, .csv, .sqlite, .atom). Supports filtering and ordering via SQL queries in the `query` parameter. JSONP callback is supported.

### Method
GET

### Endpoint
`/username/scraper_name/data.<format>`

### Query Parameters
- **query** (string) - Optional - An SQL query to filter and order the data.
- **callback** (string) - Optional - JSONP callback function name.
- **key** (string) - Required - Your API key for authentication.

### Request Example
```bash
curl -H "X-API-KEY: your_api_key_here" \
  "https://morph.io/username/scraper_name/data.json?query=select%20*%20from%20data%20limit%2010"
```

### Response
#### Success Response (200)
- Data in the requested format (JSON, CSV, SQLite, Atom feed).

#### Response Example (JSON)
```json
{
  "example": "[{"column1": "value1", "column2": "value2"}]"
}
```

#### Response Example (CSV)
```csv
column1,column2
value1,value2
```
```

--------------------------------

### Disable Spring in RubyMine for RSpec

Source: https://github.com/openaustralia/morph/blob/main/README.md

Configures the RSpec run configuration in RubyMine to disable Spring by setting the DISABLE_SPRING=1 environment variable. This can help prevent unexpected behavior during testing.

```shell
Environment variable: DISABLE_SPRING=1
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.