### Install Dependencies with Make

Source: https://github.com/elastic/crawler/blob/main/docs/DEVELOPER_GUIDE.md

Install project dependencies using the make install command. This command bundles Ruby and Java dependencies.

```bash
make install
```

--------------------------------

### Get Crawler Version

Source: https://github.com/elastic/crawler/blob/main/docs/CLI.md

Retrieves the product version of the Crawler. Run this command to verify the installed version.

```bash
$ bin/crawler version

> v0.2.0
```

--------------------------------

### Example Sitemap XML

Source: https://github.com/elastic/crawler/blob/main/docs/features/CRAWLER_DIRECTIVES.md

This is an example of a sitemap file in XML format. It lists URLs to be crawled.

```xml
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://shop.example.com/products/1/</loc>
  </url>
  <url>
    <loc>https://shop.example.com/products/2/</loc>
  </url>
  <url>
    <loc>https://shop.example.com/products/3/</loc>
  </url>
</urlset>
```

--------------------------------

### Print Crawler Version

Source: https://context7.com/elastic/crawler/llms.txt

Use this command to check the installed version of the crawler. No setup is required.

```bash
bin/crawler version
# v0.2.0
```

--------------------------------

### Boot Faux Site Locally

Source: https://github.com/elastic/crawler/blob/main/vendor/faux/README.md

Run the Faux-generated website locally using the 'rackup' command. Ensure you have Bundler installed and the necessary gems. The site will be accessible at http://localhost:9393.

```shell
  $ bundle exec rackup
```

--------------------------------

### Install Dependencies with Environment Management

Source: https://github.com/elastic/crawler/blob/main/docs/DEVELOPER_GUIDE.md

Install dependencies while automatically managing and verifying the correct Ruby and Java environment versions using rbenv and jenv. Set CRAWLER_MANAGE_ENV to true to enable this.

```bash
CRAWLER_MANAGE_ENV=true make install
```

--------------------------------

### Run Test Crawl with Docker

Source: https://github.com/elastic/crawler/blob/main/README.md

Use this command to test your Docker setup and run a basic crawl against example.com, printing results to the console. It requires a crawl-config.yml file in the current directory.

```bash
cat > crawl-config.yml << EOF
output_sink: console
domains:
  - url: https://example.com
EOF

docker run \
  -v "$(pwd)":/config \
  -it docker.elastic.co/integrations/crawler:latest jruby \
  bin/crawler crawl /config/crawl-config.yml
```

--------------------------------

### Standalone Elasticsearch Configuration (`elasticsearch.yml`)

Source: https://context7.com/elastic/crawler/llms.txt

This example demonstrates a standalone `elasticsearch.yml` file for connection settings. Settings here are overridden by `crawler.yml` if present.

```yaml
# elasticsearch.yml — standalone ES connection config

elasticsearch.host: https://my-deployment.es.us-east-1.aws.elastic.cloud
elasticsearch.port: 443
elasticsearch.api_key: <%= ENV['ES_API_KEY'] %>

# SSL
elasticsearch.ssl_verify: true
elasticsearch.ca_file: /path/to/ca.crt

# Client behavior
elasticsearch.request_timeout: 10
elasticsearch.retry_on_failure: 3
elasticsearch.delay_on_retry: 2000
elasticsearch.reload_on_failure: false
elasticsearch.compression: true

# Bulk API
elasticsearch.bulk_api.max_items: 10
elasticsearch.bulk_api.max_size_bytes: 1_048_576

# Ingest pipeline
elasticsearch.pipeline: ent-search-generic-ingestion
elasticsearch.pipeline_enabled: true
elasticsearch.pipeline_params._reduce_whitespace: true
elasticsearch.pipeline_params._run_ml_inference: true
elasticsearch.pipeline_params._extract_binary_content: true

```

--------------------------------

### Check Ruby and Java Versions

Source: https://github.com/elastic/crawler/blob/main/docs/DEVELOPER_GUIDE.md

Verify that the correct Ruby and Java versions are being used by checking against the .ruby-version and .java-version files. This is a prerequisite for installing dependencies.

```bash
ruby --version

java -version
```

--------------------------------

### Full Annotated Crawler Configuration (`crawler.yml`)

Source: https://context7.com/elastic/crawler/llms.txt

This example shows a comprehensive `crawler.yml` configuration file with explanations for various settings. It supports ERB template syntax for environment variables.

```yaml
# crawler.yml — full annotated example

# Output sink: console | file | elasticsearch
output_sink: elasticsearch
output_index: my-site-index

# Domains to crawl (crawled concurrently, with shared output)
domains:
  - url: https://example.com
    seed_urls:                          # Entry points (default: domain root)
      - https://example.com/blog
      - https://example.com/docs
    sitemap_urls:
      - https://example.com/sitemap.xml
    auth:
      type: basic                       # basic | raw
      username: myuser
      password: <%= ENV['SITE_PASS'] %>
    exclude_tags:                       # HTML5 tags to strip from body
      - header
      - footer
      - nav

# Crawl depth and URL limits
max_crawl_depth: 5
max_unique_url_count: 100_000
max_url_length: 2048
max_url_segments: 16
max_url_params: 10
max_redirects: 10

# Content field size limits
max_title_size: 1000
max_body_size: 5_242_880               # 5 MB
max_description_size: 512
max_headings_count: 10
max_indexed_links_count: 10

# Performance tuning
threads_per_crawl: 10
url_queue_size_limit: 10_000
compression_enabled: true
head_requests_enabled: false

# Connection settings
connect_timeout: 10
socket_timeout: 10
request_timeout: 60

# Proxy (optional)
http_proxy_host: proxy.internal
http_proxy_port: 8888
http_proxy_protocol: http

# SSL
ssl_verification_mode: full            # full | none
ssl_ca_certificates:
  - /path/to/ca.pem

# Purge outdated documents after primary crawl
purge_crawl_enabled: true

# Scheduling (used with `crawler schedule`)
schedule:
  pattern: "0 2 * * *"                 # Daily at 2am

# Logging
log_level: info
system_logs_to_file: true
event_logs_to_file: false
log_file_directory: "./logs"
log_file_rotation_policy: weekly       # daily | weekly | monthly

# Elasticsearch connection (can also be in a separate elasticsearch.yml)
elasticsearch:
  host: https://my-deployment.es.us-east-1.aws.elastic.cloud
  port: 443
  api_key: <%= ENV['ES_API_KEY'] %>
  pipeline: ent-search-generic-ingestion
  pipeline_enabled: true
  pipeline_params:
    _reduce_whitespace: true
    _run_ml_inference: true
    _extract_binary_content: true
  bulk_api:
    max_items: 10
    max_size_bytes: 1_048_576          # 1 MB

sink_lock_retry_interval: 1
sink_lock_max_retries: 120

```

--------------------------------

### Run Crawler Locally

Source: https://github.com/elastic/crawler/blob/main/docs/DEVELOPER_GUIDE.md

Execute the crawler locally by providing a path to the configuration file. This command starts the crawling process.

```bash
bin/crawler crawl path/to/config.yml
```

--------------------------------

### Start a Scheduled Crawl Job via CLI

Source: https://github.com/elastic/crawler/blob/main/docs/features/SCHEDULING.md

Execute this Docker command to start a scheduled crawl job using a YAML configuration file. Ensure the YAML file is mounted into the container.

```bash
docker run \
  -v ./scheduled-crawl.yml:/scheduled-crawl.yml \
  -it docker.elastic.co/integrations/crawler:latest jruby bin/crawler schedule /scheduled-crawl.yml
```

--------------------------------

### Robots Meta Tag Examples

Source: https://github.com/elastic/crawler/blob/main/docs/features/CRAWLER_DIRECTIVES.md

Examples demonstrating the use of `noindex`, `nofollow`, and a combination of both directives in robots meta tags.

```html
<meta name="robots" content="noindex">
```

```html
<meta name="robots" content="nofollow">
```

```html
<meta name="robots" content="noindex, nofollow">
```

--------------------------------

### Generate NOTICE.txt

Source: https://github.com/elastic/crawler/blob/main/script/licenses/README.md

Execute this script to generate the NOTICE.txt file, which aggregates all third-party dependency licenses.

```bash
./script/licenses/generate_notice_txt.rb
```

--------------------------------

### Canonical URL Link Tag Example

Source: https://github.com/elastic/crawler/blob/main/docs/features/CRAWLER_DIRECTIVES.md

An example of a canonical URL link tag pointing to a specific product page.

```html
<link rel="canonical" href="https://example.com/categories/dresses/starlet-red-medium">
```

--------------------------------

### Data Attributes for Content Exclusion Example

Source: https://github.com/elastic/crawler/blob/main/docs/features/CRAWLER_DIRECTIVES.md

This example shows how to use the `data-elastic-exclude` attribute to prevent specific HTML content within a div from being indexed.

```html
<body>
  <p>This is your page content, which will be indexed by the web crawler.
  <div data-elastic-exclude>Content in this div will be excluded from the search index</div>
</body>
```

--------------------------------

### Create Crawler Configuration File (Manual)

Source: https://github.com/elastic/crawler/blob/main/README.md

Manually creates a `crawl-config.yml` file, replacing placeholders with your specific Elasticsearch and target website details. Use this if environment variables are not set or not working.

```bash
cat > crawl-config.yml << 'EOF'
output_sink: elasticsearch
output_index: web-crawl-test

elasticsearch:
  host: https://your-deployment.es.region.aws.elastic.cloud  # Your ES_HOST
  port: 443                                                   # Your ES_PORT (443 for cloud, 9200 for localhost)  
  api_key: your_encoded_api_key_here                          # Your ES_API_KEY from Step 2
  pipeline_enabled: false

domains:
  - url: https://your-website.com                             # Your target website
EOF
```

--------------------------------

### robots.txt Sitemap Directive

Source: https://github.com/elastic/crawler/blob/main/docs/features/CRAWLER_DIRECTIVES.md

This is an example of a Sitemap directive in a robots.txt file. It points the crawler to the location of your sitemap.

```txt
Sitemap: https://shop.example.com/sitemap.xml
```

--------------------------------

### Define a Simple Website with Faux

Source: https://github.com/elastic/crawler/blob/main/vendor/faux/README.md

Use Faux::Base to define pages, sitemaps, and robots.txt rules for a website. Each 'page' block defines a URL and its content, while 'sitemap' and 'robots' blocks configure site-wide directives.

```ruby
class SimpleSite < Faux::Base
  page '/foo' do
    status 200
    link_to '/foobar'
  end

  page '/bar' do
    status 200
    link_to '/bang'
    link_to '/baz'
  end

  sitemap '/sitemap.xml' do
    link_to 'http://localhost:9393/foo'
    link_to '/bar'
  end

  # Adds a /robots.txt file with the specified rules.
  robots do
    user_agent '*'
    disallow '/foo'
    sitemap 'http://localhost:9393/sitemap.xml'
  end
end
```

--------------------------------

### Utility function to get URL parameters

Source: https://github.com/elastic/crawler/blob/main/spec/fixtures/uncrate.com.html

A JavaScript function to extract URL query parameters. It handles URL encoding and replaces '+' with spaces.

```javascript
function getParameterByName(name) { name = name.replace(/[\[\]]/g, "\$& "); var regex = new RegExp("[?&]" + name + "=([^&#]*)"), results = regex.exec(location.search); return results === null ? "" : decodeURIComponent(results[1].replace(/\+/g, " ")); }
```

--------------------------------

### Initialize Quantcast Tracking

Source: https://github.com/elastic/crawler/blob/main/spec/fixtures/uncrate.com.html

This script initializes the Quantcast tracking service. It should be included on pages where Quantcast analytics are desired.

```javascript
var _qevents = _qevents || [];
(function() {
    var elem = document.createElement('script');
    elem.src = (document.location.protocol == "https:" ? "https://secure" : "http://edge") + ".quantserve.com/quant.js";
    elem.async = true;
    elem.type = "text/javascript";
    var scpt = document.getElementsByTagName('script')[0];
    scpt.parentNode.insertBefore(elem, scpt);
})();
_qevents.push({ qacct:"p-92CBSYgectFl6" });
```

--------------------------------

### Deny specific paths using 'begins' rule

Source: https://github.com/elastic/crawler/blob/main/docs/features/CRAWL_RULES.md

Use this configuration to deny crawling of any URLs that begin with a specified path. Ensure the pattern starts with '/'.

```yaml
domains:
  - url: http://example.com
    crawl_rules:
      - policy: deny
        type: begins
        pattern: /blog
```

--------------------------------

### Website Configuration and Settings

Source: https://github.com/elastic/crawler/blob/main/spec/fixtures/gilacountyaz.gov.html

This JavaScript block initializes and populates a global RZ object with various website configuration settings, including page type, URLs, and permissions.

```javascript
if (typeof(RZ) == 'undefined') RZ = {}; if (!RZ.link) RZ.link = new Object(); RZ.pagetype = 'template' RZ.baseurlprefix = '../../' RZ.baseurlpath = 'government/assessor/' RZ.protocolRelativeRevizeBaseUrl='//cms3.revize.com' RZ.pagetemplatename = 'freeform' RZ.pagetemplateid = '3' RZ.pagemodule = 'links' RZ.pagemoduleid = '1' RZ.pagerecordid = '65'; RZ.pageid = 'links-65'; RZ.pageparentid = '3'; RZ.pagesectionid = '65'; RZ.pagesectionname = 'assessor'; RZ.pagesectionlevel = '2'; RZ.pagesectionfolder = 'government/assessor/'; RZ.pagesectionfilter = 'sectionid=65'; RZ.pagelinkfilter = 'linklevel=2 and linksectionid=65'; RZ.pagelinklevel = '2'; RZ.pagelinkid = '65'; RZ.pageidfield = ''; RZ.isnewrecord = false; RZ.editmodule = ''; RZ.editrecordid = '65'; RZ.editrecordversion = ''; RZ.editversion = ''; //same as editrecordversion RZ.editaction = ''; //----- Permission data RZ.page_roles = '' RZ.page_users = '03a89693f0c6f105c078cc0e58c52847ddaa8d5cebf062d43652999ca8849d3f|1139a951bca375abe2142ea6ed631a1de10006c7745684cd584c91e9afc0c87e|b2ef173eb697462e4dc1bc45b188d68d015a03419fe4249511a36f690d87c3ef|4d68e1c1becde718eda2906f5ea57d3a7c1b54b39c4f71fa7d819e971c663644|93d8882556a2bd32bf98e3b95f81429edb6ffbb19271875c533e155680d095b7' RZ.page_key = 'freeform[65]' RZ.parent_key = 'freeform[3]' RZ.inherit_key = 'freeform[65]' RZ.workflowname = '' RZ.permissions_options = 'warningsOFF' RZ.permissions_module = 'webspace_page_permissions' RZ.webspace = 'gilaaz' RZ.webspacedesc = "tinymce"; RZ.featurespattern = '(EZ)'; null RZ.webspacelinksurl = '' RZ.webspacelinksurl = './webspace/links.html' RZ.workflowlist = '' RZ.revizeserverurl = 'https://cms3.revize.com/revize/gilaaz'; if (!RZ.nextseq) RZ.nextseq = {linknames:{},modules:{}}; //----- Field values from webspace_config module RZ.webspace_config = new Object(); RZ.webspace_config.admin_email = "noreply@revize.com"; RZ.webspace_config.calendar_options = "max_events=6"; RZ.webspace_config.form_captcha = "yes"; RZ.webspace_config.form_server = "revizeserver"; RZ.webspace_config.form_technology = "jsp"; RZ.webspace_config.formwizard_ftp_password = ""; RZ.webspace_config.formwizard_ftp_port = ""; RZ.webspace_config.formwizard_ftp_server = ""; RZ.webspace_config.formwizard_ftp_user = ""; RZ.webspace_config.formwizard_setup = "yes"; RZ.webspace_config.formwizard_smtp_password = "xxx"; RZ.webspace_config.formwizard_smtp_server = "xxx"; RZ.webspace_config.formwizard_smtp_user = "xxx"; RZ.webspace_config.google_analytics = "UA-33716902-1"; RZ.webspace_config.home_pageid = "0"; RZ.webspace_config.home_sectionid = "0"; RZ.webspace_config.home_template = "home,index,sitemap"; RZ.webspace_config.icon_email = "<img src=\"//cms3.revize.com/revize/images/email_icon_yellow.gif\" width=\"16\" border=\"0\" height=\"17\" title=\"Request email notification when page changes\" alt=\"Request email notification when page changes\" style=\"\" />"; RZ.webspace_config.image_maxheight = "3000"; RZ.webspace_config.image_maxwidth = "3000"; RZ.webspace_config.image_setup = "yes"; RZ.webspace_config.livesite_url = ""; RZ.webspace_config.menu_links_module = "links"; RZ.webspace_config.menu_newsection = ""; RZ.webspace_config.menu_nexturl = "webspace_menu-editform.jsp"; RZ.webspace_config.menu_sections_module = "sections"; RZ.webspace_config.menu_system = "custom"; RZ.webspace_config.m
```

--------------------------------

### Extract Publish Year from URL

Source: https://github.com/elastic/crawler/blob/main/docs/features/EXTRACTION_RULES.md

This rule extracts the publication year from blog post URLs. It applies to URLs starting with '/blog' and uses a regex to capture the year.

```yaml
domains:
  - url: https://totally-real-rpg.com
    extraction_rulesets:
      - url_filters:
          - type: "begins"
            pattern: "/blog"
        rules:
          - action: "extract"
            field_name: "publish_year"
            selector: "blog\/([0-9]{4})"
            join_as: "string"
            source: "url"
```

--------------------------------

### Run the Crawler

Source: https://github.com/elastic/crawler/blob/main/README.md

Executes the Elastic Crawler Docker image to crawl the target website and ingest data into Elasticsearch based on the provided configuration file.

```bash
docker run \
  -v "$(pwd)":/config \
  -it docker.elastic.co/integrations/crawler:latest jruby \
  bin/crawler crawl /config/crawl-config.yml
```

--------------------------------

### Create Crawler Configuration File (Auto-populated)

Source: https://github.com/elastic/crawler/blob/main/README.md

Generates a `crawl-config.yml` file using environment variables for Elasticsearch connection and target website. This is the recommended method if environment variables are set.

```bash
cat > crawl-config.yml << EOF
output_sink: elasticsearch
output_index: web-crawl-test

elasticsearch:
  host: $ES_HOST
  port: $ES_PORT
  api_key: $ES_API_KEY
  pipeline_enabled: false

domains:
  - url: $TARGET_WEBSITE
EOF
```