### Example Scraper Implementation in Ruby

Source: https://context7.com/openaustralia/morph-cli/llms.txt

A complete Ruby scraper that demonstrates how to extract development application data from a government website and store it in an SQLite database using the ScraperWiki library. This example showcases web scraping techniques, data cleaning, and conditional saving of records.

```ruby
require 'scraperwiki'
require 'mechanize'

starting_url = 'https://www2.bmcc.nsw.gov.au/datracking/Modules/applicationmaster/default.aspx?page=exhibit'
comment_url = 'http://www.bmcc.nsw.gov.au/sustainableliving/developmentapplicationsinnotification'

def clean_whitespace(text)
  text.gsub("\r", ' ').gsub("\n", ' ').squeeze(" ").strip
end

def scrape_table(doc, comment_url)
  doc.search('table tbody tr').each do |tr|
    tds = tr.search('td')
    h = tds.map { |td| td.inner_html }

    record = {
      'info_url' => (doc.uri + tds[0].at('a')['href']).to_s,
      'comment_url' => comment_url,
      'council_reference' => clean_whitespace(h[1]),
      'on_notice_from' => Date.strptime(clean_whitespace(h[2]), "%d/%m/%Y").to_s,
      'on_notice_to' => Date.strptime(clean_whitespace(h[3]), "%d/%m/%Y").to_s,
      'address' => clean_whitespace(h[4].split('<br>')[0]) + ", " + clean_whitespace(h[5]) + ", NSW",
      'description' => clean_whitespace(h[4].split('<br>')[1..-1].join),
      'date_scraped' => Date.today.to_s
    }

    # Check if record already exists
    if (ScraperWiki.select("* from data where `council_reference`='#{record['council_reference']}'").empty? rescue true)
      puts "Saving: #{record['council_reference']}"
      ScraperWiki.save_sqlite(['council_reference'], record)
    else
      puts "Skipping already saved record #{record['council_reference']}"
    end
  end
end

# Initialize Mechanize with SSL verification disabled
agent = Mechanize.new do |a|
  a.verify_mode = OpenSSL::SSL::VERIFY_NONE
end

doc = agent.get(starting_url)
scrape_table(doc, comment_url)

# Output: Data saved to data.sqlite
```

--------------------------------

### Morph CLI Command-Line Interface Examples

Source: https://context7.com/openaustralia/morph-cli/llms.txt

Provides examples of common commands used with the Morph CLI. These commands allow users to run scrapers, enable development mode, specify directories, and access help information. The Thor-based CLI is designed for flexible and efficient scraper development.

```bash
# Run scraper in current directory
morph

# Run against development server
morph --dev

# Specify directory explicitly
morph --directory=/path/to/scraper

# Show version
morph version
# Output: Morph CLI 0.2.5

# Get help
morph help
```

--------------------------------

### Morph CLI Logging Example

Source: https://context7.com/openaustralia/morph-cli/llms.txt

Demonstrates how the Morph CLI handles internal logging and outputs messages to standard output (stdout) and standard error (stderr). This is useful for understanding how to debug and monitor scraper execution within the Morph environment.

```ruby
internal_line = '{"stream":"internalout","text":"Container started"}'
MorphCLI.log(internal_line)
# Outputs to $stdout: "Container started"
```

--------------------------------

### Find All Scraper Files with Ruby

Source: https://context7.com/openaustralia/morph-cli/llms.txt

Recursively discovers all files within a specified directory, excluding hidden directories (those starting with a dot). It returns an array of relative paths suitable for packaging into an upload archive. This function is essential for identifying all necessary files for a scraper. It requires the 'morph-cli' gem.

```ruby
require 'morph-cli'

directory = "/path/to/scraper"
all_paths = MorphCLI.all_paths(directory)

# Returns array of relative paths:
# ["scraper.rb", "lib/helpers.rb", "config.yaml", "README.md"]

# Verify scraper file exists
scraper_file = all_paths.find { |file| /scraper\.[\S]+$/ =~ file }
if scraper_file
  puts "Found scraper: #{scraper_file}"
else
  puts "Error: No scraper file found (expected scraper.rb, scraper.py, etc.)"
  exit(1)
end
```

--------------------------------

### Configuration Management with Ruby

Source: https://context7.com/openaustralia/morph-cli/llms.txt

Manages user configuration, including API keys and server URLs, storing them securely in `~/.morph` with 0600 permissions. This function allows loading existing configurations, modifying them, and saving the updated settings. It requires the 'morph-cli' gem.

```ruby
require 'morph-cli'

# Load existing or default configuration
config = MorphCLI.load_config

# Default structure:
# {
#   development: { base_url: "http://127.0.0.1:3000" },
#   production: { base_url: "https://morph.io" }
# }

# Add API key and save
config[:production][:api_key] = "sk_live_abc123xyz789"
MorphCLI.save_config(config)

# Configuration file is created at ~/.morph with permissions 0600
config_path = MorphCLI.config_path
puts "Config saved to: #{config_path}"
```

--------------------------------

### Create Tar Archive with Ruby

Source: https://context7.com/openaustralia/morph-cli/llms.txt

Creates a tar archive from a specified directory and list of file paths for uploading to the Morph server. It handles temporary file creation and ensures proper cleanup. This function requires the 'morph-cli' and 'archive/tar/minitar' gems. The output is a `File` object ready for HTTP upload.

```ruby
require 'morph-cli'
require 'archive/tar/minitar'

directory = "/path/to/scraper"
paths = ["scraper.rb", "lib/helpers.rb", "config.yaml"]

# Create tar archive
tar_file = MorphCLI.create_tar(directory, paths)

# Result is a File object ready for upload
# The tar is created at /tmp/out
puts "Archive created: #{tar_file.path}"
puts "Archive size: #{File.size(tar_file.path)} bytes"

# Use in HTTP upload
RestClient::Request.execute(
  method: :post,
  url: "https://morph.io/run",
  payload: {
    api_key: "your_api_key",
    code: tar_file
  }
)
```

--------------------------------

### Calculate Directory Size with Ruby

Source: https://context7.com/openaustralia/morph-cli/llms.txt

Computes the total size of all files specified by a list of paths within a given directory and formats this size into a human-readable string (e.g., KB, MB). This provides user feedback during the upload process. It requires the 'morph-cli' gem.

```ruby
require 'morph-cli'

directory = "/path/to/scraper"
paths = ["scraper.rb", "lib/parser.rb", "data/seeds.json"]

size = MorphCLI.get_dir_size(directory, paths)
puts "Uploading #{size}..."

# Example output: "Uploading 45.3 KB..."
# or: "Uploading 2.1 MB..."
```

--------------------------------

### Execute Morph Scraper with Ruby

Source: https://context7.com/openaustralia/morph-cli/llms.txt

Executes a scraper by uploading its code to the Morph.io server and streaming back the results in real-time. It handles authentication, error handling for unauthorized access and connection issues, and requires the 'morph-cli' and 'rest-client' gems. Input is the directory of the scraper, a boolean for development mode, and environment configuration. Output streams to stdout/stderr.

```ruby
require 'morph-cli'

# Execute scraper in current directory
directory = Dir.pwd
development = false
env_config = {
  base_url: "https://morph.io",
  api_key: "your_api_key_here",
  timeout: 600  # 10 minutes
}

begin
  MorphCLI.execute(directory, development, env_config)
rescue RestClient::Unauthorized
  puts "Invalid API key - please check your credentials"
rescue Errno::ECONNREFUSED => e
  puts "Cannot connect to Morph server at #{env_config[:base_url]}"
end

# Output streams to stdout/stderr in real-time:
# {"stream":"stdout","text":"Scraping started..."}
# {"stream":"stdout","text":"Found 25 records"}
# {"stream":"internalout","text":"Scraper completed successfully"}
```

--------------------------------

### Stream and Parse Log Output with Ruby

Source: https://context7.com/openaustralia/morph-cli/llms.txt

Processes streaming log output received from the Morph server. It parses each line as JSON and routes the content to the appropriate output stream (stdout or stderr). This function is crucial for displaying real-time feedback and errors during scraper execution. It requires the 'morph-cli' and 'json' gems.

```ruby
require 'morph-cli'
require 'json'

# Example log line from server
log_line = '{"stream":"stdout","text":"Processing record 1 of 100"}'

MorphCLI.log(log_line)
# Outputs to $stdout: "Processing record 1 of 100"

error_line = '{"stream":"stderr","text":"Warning: Rate limit approaching"}'
MorphCLI.log(error_line)
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.