FSCrawler

https://fscrawler.readthedocs.io/en/latest/
Admin
FSCrawler is a file system crawler for Elasticsearch that indexes binary documents like PDF, Open...
Tokens:70,316
Snippets:868
Trust Score:9.6
Update:5 months ago
Show doc for...
Tokens:
Raw
### Install Documentation Dependencies

Source: https://fscrawler.readthedocs.io/en/latest/dev/doc.html

Installs the necessary Python packages for building documentation using pip.

```bash
$ pip install -r docs/requirements.txt

```

--------------------------------

### Install Spotless Git Pre-Push Hook

Source: https://fscrawler.readthedocs.io/en/latest/_sources/dev/build.rst.txt

Install a Git pre-push hook to automatically run the Spotless formatting check before each push.

```bash
mvn spotless:install-git-pre-push-hook
```

--------------------------------

### Running Scan Checkpoint Example

Source: https://fscrawler.readthedocs.io/en/latest/_sources/admin/status.rst.txt

This JSON structure illustrates a checkpoint file while a scan is in progress. It shows the current scan ID, state, start time, directories being processed, and file counts.

```json
{
  "scan_id": "abc123-def456",
  "state": "RUNNING",
  "scan_start_time": "2025-07-01T12:00:00",
  "current_path": "/data/documents/subfolder",
  "pending_paths": ["/data/documents/other"],
  "completed_paths": ["/data/documents", "/data/documents/processed"],
  "files_processed": 50,
  "files_deleted": 0,
  "retry_count": 0
}
```

--------------------------------

### Setup New FSCrawler Job

Source: https://fscrawler.readthedocs.io/en/latest/_sources/admin/cli-options.rst.txt

Use the --setup option to create a default configuration file for a new FSCrawler job.

```sh
bin/fscrawler --setup
```

--------------------------------

### Setup FSCrawler Configuration

Source: https://fscrawler.readthedocs.io/en/latest/_sources/user/getting_started.rst.txt

Use the --setup command to create the default FSCrawler configuration file if it does not exist. The path to the settings file is displayed after execution.

```sh
$ bin/fscrawler --setup
17:40:33,905 INFO  [f.console] You can edit the settings in [~/.fscrawler/fscrawler/_settings.yaml]. Then, you can run again fscrawler without the --setup option.
```

--------------------------------

### Example Tika Configuration File

Source: https://fscrawler.readthedocs.io/en/latest/admin/fs/local-fs.html

An example XML file demonstrating how to configure Tika parsers. This file can be used with the 'tika_config_path' setting.

```xml
<?xml version="1.0" encoding="UTF-8"?>
<properties>
  <service-loader dynamic="true"/>
  <service-loader loadErrorHandler="IGNORE"/>
  <parsers>
    <!-- Use Default Parser for files, but Default Parser will never use HTML parser -->
    <parser class="org.apache.tika.parser.DefaultParser">
      <parser-exclude class="org.apache.tika.parser.html.JsoupParser"/>
    </parser>
    <!-- Use a different parser for XHTML -->
    <parser class="org.apache.tika.parser.xml.XMLParser">
      <mime>application/xhtml+xml</mime>
    </parser>
  </parsers>
</properties>

```

--------------------------------

### Install Sphinx Dependencies

Source: https://fscrawler.readthedocs.io/en/latest/_sources/dev/doc.rst.txt

Installs the necessary Python packages for documentation generation using pip. Ensure Python3 is installed beforehand.

```bash
$ pip install -r docs/requirements.txt
```

--------------------------------

### Start FSCrawler

Source: https://fscrawler.readthedocs.io/en/latest/user/getting_started.html

Execute this command to start FSCrawler. It will attempt to read its configuration from a default local file.

```bash
bin/fscrawler

```

--------------------------------

### Example Static Metadata Content

Source: https://fscrawler.readthedocs.io/en/latest/_sources/admin/fs/tags.rst.txt

Define static metadata in a YAML file. This example adds `hostname` and `environment` fields to all documents.

```yaml
external:
  hostname: "server001"
  environment: "production"
```

--------------------------------

### Start Local Elasticsearch Node

Source: https://fscrawler.readthedocs.io/en/latest/admin/fs/elasticsearch.html

Use the provided curl command to download and start a local Elasticsearch node. The connection URL and API key will be available in the generated .env file.

```bash
curl -fsSL https://elastic.co/start-local | sh
```

--------------------------------

### MP3 File Metadata Example

Source: https://fscrawler.readthedocs.io/en/latest/_sources/admin/fs/local-fs.rst.txt

Example of metadata extracted from an MP3 file. Note that all fields are indexed as text by default.

```json
{
  "xmpDM:genre" : "Vocal",
  "X-Parsed-By" : "org.apache.tika.parser.DefaultParser",
  "creator" : "David Pilato",
  "xmpDM:album" : "FS Crawler",
  "xmpDM:trackNumber" : "1",
  "xmpDM:releaseDate" : "2016",
  "meta:author" : "David Pilato",
  "xmpDM:artist" : "David Pilato",
  "dc:creator" : "David Pilato",
  "xmpDM:audioCompressor" : "MP3",
  "title" : "Test Tika",
  "xmpDM:audioChannelType" : "Stereo",
  "version" : "MPEG 3 Layer III Version 1",
  "xmpDM:logComment" : "Hello but reverted",
  "xmpDM:audioSampleRate" : "44100",
  "channels" : "2",
  "dc:title" : "Test Tika",
  "Author" : "David Pilato",
  "xmpDM:duration" : "1018.775146484375",
  "Content-Type" : "audio/mpeg",
  "samplerate" : "44100"
}
```

--------------------------------

### Docker Compose Example with OTel Collector

Source: https://fscrawler.readthedocs.io/en/latest/_sources/admin/otel.rst.txt

Example configuration for a Docker Compose stack including FSCrawler and an OTel collector. Ensure the OTEL_EXPORTER_OTLP_ENDPOINT is set to the collector's address within the Docker network.

```yaml
services:
  fscrawler:
    image: dadoonet/fscrawler:latest
    environment:
      - OTEL_EXPORTER_OTLP_ENDPOINT=http://edot-collector:4318
      - OTEL_SERVICE_NAME=fscrawler
      - OTEL_RESOURCE_ATTRIBUTES=deployment.environment=docker
```

--------------------------------

### Start FSCrawler REST Service

Source: https://fscrawler.readthedocs.io/en/latest/user/rest.html

Use the `--rest` option to start FSCrawler with its REST service. Combining it with `--loop 0` allows it to continuously listen for REST requests without indexing local files. The service typically starts on http://127.0.0.1:8080/fscrawler.

```bash
$ bin/fscrawler --loop 0 --rest
18:55:37,851 INFO  [f.p.e.c.f.FsCrawlerImpl] Starting FS Crawler
18:55:39,237 INFO  [f.p.e.c.f.FsCrawlerImpl] FS crawler Rest service started on [http://127.0.0.1:8080/fscrawler]

```

--------------------------------

### Add filesize as tag

Source: https://fscrawler.readthedocs.io/en/latest/admin/fs/rest.html

This example shows how to dynamically add the filesize of an uploaded document as a tag within the 'external' object. It uses shell commands to get the file size and format it into the JSON payload.

```sh
echo "This is my text" > test.txt
curl -F "file=@test.txt" 
-F "tags={\"file\":{\"filesize\":$(ls -l test.txt | awk ‘{print $5}’)}}" "http://127.0.0.1:8080/_document"
```

--------------------------------

### Example FSCrawler Job File Specification

Source: https://fscrawler.readthedocs.io/en/latest/_sources/admin/fs/index.rst.txt

A comprehensive YAML job file example detailing various settings for file system crawling, including path, includes/excludes, file size limits, content indexing, and metadata options.

```yaml
# optional: the name of the crawler. Defaults to the job directory name.
name: "test"

# required
fs:

  # define a "local" file path crawler, if running inside a docker container this must be the path INSIDE the container (/tmp/es)
  url: "/path/to/docs"
  follow_symlinks: false
  remove_deleted: true
  continue_on_error: false

  # scan every 5 minutes for changes in url defined above
  update_rate: "5m"

  # opional: define includes and excludes, "~" files are excluded by default if not defined below
  includes:
  - "*.doc"
  - "*.xls"
  excludes:
  - "resume.doc"

  # optional: do not send big files to TIKA
  ignore_above: "512mb"

  # special handling of JSON files, should only be used if ALL files are JSON
  json_support: false
  add_as_inner_object: false

  # special handling of XML files, should only be used if ALL files are XML
  xml_support: false

  # use MD5 from filename (instead of filename) if set to false
  filename_as_id: true

  # include size ot file in index
  add_filesize: true

  # inlcude user/group of file only if needed
  attributes_support: false

  # collect ACL metadata when available
  acl_support: false

  # do you REALLY want to store every file as a copy in the index ? Then set this to true
  store_source: false

  # you may want to store (partial) content of the file (see indexed_chars)    
  index_content: true

  # how much data from the content of the file should be indexed (and stored inside the index), set to 0 if you need checksum, but no content at all to be indexed
  #indexed_chars: "0"
  indexed_chars: "10000.0"

  # usually file metadata will be stored in separate fields, if you want to keep the original set, set this to true
  raw_metadata: false

  # optional: add checksum meta (requires index_content to be set to true)
  checksum: "MD5"

  # recommmended, but will create another index

```

--------------------------------

### FTP Plugin Usage

Source: https://fscrawler.readthedocs.io/en/latest/admin/fs/rest.html

Demonstrates how to use the FTP plugin to read files from an FTP server. It shows examples with and without pre-configured server settings.

```APIDOC
## FTP Plugin

The `ftp` plugin reads a file from an FTP server. It accepts the following parameters:

### Parameters

* `path` (required): path to the file on the remote server
* `hostname` (optional): FTP server hostname. If not provided, uses the `server.hostname` from job settings.
* `port` (optional): FTP server port. If not provided, uses the `server.port` from job settings.
* `username` (optional): FTP username. If not provided, uses the `server.username` from job settings.
* `password` (optional): FTP password. If not provided, uses the `server.password` from job settings.

### Request Example (with full details)
```bash
curl -XPOST http://127.0.0.1:8080/_document -H 'Content-Type: application/json' -d 
'{ 
  "type": "ftp",
  "ftp": {
    "hostname": "ftp.example.com",
    "port": 21,
    "username": "myuser",
    "password": "mypassword",
    "path": "/documents/document.pdf"
  }
}'
```

### Request Example (using job settings)
```bash
curl -XPOST http://127.0.0.1:8080/_document -H 'Content-Type: application/json' -d 
'{ 
  "type": "ftp",
  "ftp": {
    "path": "/documents/document.pdf"
  }
}'
```
```

--------------------------------

### Start FSCrawler

Source: https://fscrawler.readthedocs.io/en/latest/_sources/user/getting_started.rst.txt

Run FSCrawler using the provided script. It will read settings from a default configuration file.

```sh
bin/fscrawler
```

--------------------------------

### Setup FSCrawler Job

Source: https://fscrawler.readthedocs.io/en/latest/_sources/user/tutorial.rst.txt

Use this command to set up a new FSCrawler job. It creates a sample configuration file for the specified job name.

```sh
# On Linux/Mac
bin/fscrawler --setup resumes
# On Windows
.\bin\fscrawler --setup resumes
```

--------------------------------

### Example .meta.yml for Adding External Fields

Source: https://fscrawler.readthedocs.io/en/latest/admin/fs/tags.html

Shows how to declare fields like 'foo' under the 'external' key in `.meta.yml` to make them available for indexing. These can then be processed by ingest pipelines.

```yaml
external:
  foo: "bar"
  myTitle: "My document title"

```

--------------------------------

### Example .meta.yml for Overwriting Fields

Source: https://fscrawler.readthedocs.io/en/latest/_sources/admin/fs/tags.rst.txt

The `.meta.yml` file can overwrite existing document fields. This example shows how the `content` field will be replaced with 'HIDDEN'.

```yaml
content: "HIDDEN"
```

--------------------------------

### Metadata Merging Example

Source: https://fscrawler.readthedocs.io/en/latest/_sources/admin/fs/tags.rst.txt

Demonstrates how static metadata and `.meta.yml` content are merged. Values in `.meta.yml` take precedence over static metadata if keys conflict.

```yaml
external:
   category: "general"
   owner: "team-a"

```

```yaml
external:
   owner: "team-b"
   priority: "high"

```

```yaml
external:
   category: "general"
   owner: "team-b"
   priority: "high"

```

--------------------------------

### Example MP3 Raw Metadata

Source: https://fscrawler.readthedocs.io/en/latest/admin/fs/local-fs.html

Illustrates the structure of raw metadata extracted from an MP3 audio file.

```json
{
   "xmpDM:genre" : "Vocal",
   "X-Parsed-By" : "org.apache.tika.parser.DefaultParser",
   "creator" : "David Pilato",
   "xmpDM:album" : "FS Crawler",
   "xmpDM:trackNumber" : "1",
   "xmpDM:releaseDate" : "2016",
   "meta:author" : "David Pilato",
   "xmpDM:artist" : "David Pilato",
   "dc:creator" : "David Pilato",
   "xmpDM:audioCompressor" : "MP3",
   "title" : "Test Tika",
   "xmpDM:audioChannelType" : "Stereo",
   "version" : "MPEG 3 Layer III Version 1",
   "xmpDM:logComment" : "Hello but reverted",
   "xmpDM:audioSampleRate" : "44100",
   "channels" : "2",
   "dc:title" : "Test Tika",
   "Author" : "David Pilato",
   "xmpDM:duration" : "1018.775146484375",
   "Content-Type" : "audio/mpeg",
   "samplerate" : "44100"
}

```

--------------------------------

### Example .meta.yml Content

Source: https://fscrawler.readthedocs.io/en/latest/_sources/admin/fs/tags.rst.txt

Use this format in a `.meta.yml` file to add custom metadata to documents. Only supported fields, especially those under the `external` key, will be indexed.

```yaml
external:
  myTitle: "My document title"
```

--------------------------------

### Start FSCrawler Indexing

Source: https://fscrawler.readthedocs.io/en/latest/_sources/user/tutorial.rst.txt

Run FSCrawler with the job name to start indexing documents from the configured directory. This command initiates the crawling and indexing process.

```sh
# On Linux/Mac
bin/fscrawler resumes
# On Windows
.\bin\fscrawler resumes
```

--------------------------------

### Example PDF Raw Metadata

Source: https://fscrawler.readthedocs.io/en/latest/admin/fs/local-fs.html

Illustrates the structure of raw metadata extracted from a PDF document.

```json
{
   "date" : "2016-07-07T08:37:42Z",
   "pdf:PDFVersion" : "1.5",
   "xmp:CreatorTool" : "Microsoft Word",
   "Keywords" : "keyword1, keyword2",
   "access_permission:modify_annotations" : "true",
   "access_permission:can_print_degraded" : "true",
   "subject" : "Test Tika Object",
   "dc:creator" : "David Pilato",
   "dcterms:created" : "2016-07-07T08:37:42Z",
   "Last-Modified" : "2016-07-07T08:37:42Z",
   "dcterms:modified" : "2016-07-07T08:37:42Z",
   "dc:format" : "application/pdf; version=1.5",
   "title" : "Test Tika title",
   "Last-Save-Date" : "2016-07-07T08:37:42Z",
   "access_permission:fill_in_form" : "true",
   "meta:save-date" : "2016-07-07T08:37:42Z",
   "pdf:encrypted" : "false",
   "dc:title" : "Test Tika title",
   "modified" : "2016-07-07T08:37:42Z",
   "cp:subject" : "Test Tika Object",
   "Content-Type" : "application/pdf",
   "X-Parsed-By" : "org.apache.tika.parser.DefaultParser",
   "creator" : "David Pilato",
   "meta:author" : "David Pilato",
   "dc:subject" : "keyword1, keyword2",
   "meta:creation-date" : "2016-07-07T08:37:42Z",
   "created" : "Thu Jul 07 10:37:42 CEST 2016",
   "access_permission:extract_for_accessibility" : "true",
   "access_permission:assemble_document" : "true",
   "xmpTPg:NPages" : "2",
   "Creation-Date" : "2016-07-07T08:37:42Z",
   "access_permission:extract_content" : "true",
   "access_permission:can_print" : "true",
   "meta:keyword" : "keyword1, keyword2",
   "Author" : "David Pilato",
   "access_permission:can_modify" : "true"
}

```

--------------------------------

### Start FSCrawler REST Service

Source: https://fscrawler.readthedocs.io/en/latest/_sources/user/rest.rst.txt

Use this command to start FSCrawler with the REST service enabled and set to loop indefinitely. This is useful for continuous listening to incoming REST requests.

```sh
$ bin/fscrawler --loop 0 --rest
18:55:37,851 INFO  [f.p.e.c.f.FsCrawlerImpl] Starting FS Crawler
18:55:39,237 INFO  [f.p.e.c.f.FsCrawlerImpl] FS crawler Rest service started on [http://127.0.0.1:8080/fscrawler]
```

--------------------------------

### Run FSCrawler with Data

Source: https://fscrawler.readthedocs.io/en/latest/_sources/user/getting_started.rst.txt

Start FSCrawler after creating a directory with files to index. The crawler connects to Elasticsearch and enters watch mode.

```sh
$ bin/fscrawler
17:41:45,395 INFO  [f.p.e.c.f.FsCrawlerImpl] FSCrawler is now connected to Elasticsearch version [9.0.0]
17:41:45,395 INFO  [f.p.e.c.f.FsCrawlerImpl] FSCrawler started in watch mode. It will run unless you stop it with CTRL+C.
17:41:45,395 INFO  [f.p.e.c.f.FsParser] FS crawler started for [fscrawler] for [/tmp/es] every [15m]
```

--------------------------------

### Mounting External Directory with Docker

Source: https://fscrawler.readthedocs.io/en/latest/_sources/admin/layout.rst.txt

Example command to run FSCrawler using Docker, specifically mounting the 'external' directory.

```sh
docker run -it --rm \
        -v ~/.fscrawler:/root/.fscrawler \
        -v ~/tmp:/tmp/es:ro \
        -v "$PWD/external:/usr/share/fscrawler/external" \
        dadoonet/fscrawler
```

--------------------------------

### Example Static Metadata File

Source: https://fscrawler.readthedocs.io/en/latest/admin/fs/tags.html

Define static metadata in a YAML file, such as `/path/to/static_metadata.yml`, to apply common fields like 'hostname' and 'environment' to all indexed documents.

```yaml
external:
  hostname: "server001"
  environment: "production"

```

--------------------------------

### Example .meta.yml for Adding New External Fields

Source: https://fscrawler.readthedocs.io/en/latest/_sources/admin/fs/tags.rst.txt

To add a field like 'foo' that is not inherently supported, first define it under the `external` key in your `.meta.yml` file.

```yaml
external:
  foo: "bar"
  myTitle: "My document title"
```

--------------------------------

### Docker Compose Environment Variables

Source: https://fscrawler.readthedocs.io/en/latest/_sources/installation.rst.txt

Example .env file for Docker Compose, defining settings for Elasticsearch and FSCrawler.

```sh
# Password for the 'elastic' user (at least 6 characters)
ES_LOCAL_PASSWORD=changeme

# Version of Elastic products
ES_LOCAL_VERSION=|ES_stack_version|

# Set the ES container name
ES_LOCAL_CONTAINER_NAME=es-fscrawler

# Set to 'basic' or 'trial' to automatically start the 30-day trial
ES_LOCAL_LICENSE=basic
#ES_LOCAL_LICENSE=trial

# Port to expose Elasticsearch HTTP API to the host
ES_LOCAL_PORT=9200
ES_LOCAL_DISK_SPACE_REQUIRED=1gb
ES_LOCAL_JAVA_OPTS="-XX:UseSVE=0 -Xms128m -Xmx2g"

# Project namespace (defaults to the current folder name if not set)
COMPOSE_PROJECT_NAME=fscrawler

# FSCrawler Settings
FSCRAWLER_VERSION=|FSCrawler_version|
FSCRAWLER_PORT=8080

# Optionally, you can change the log level settings
FS_JAVA_OPTS="-DLOG_LEVEL=debug -DDOC_LEVEL=debug"
```

--------------------------------

### Docker Compose Service Definition

Source: https://fscrawler.readthedocs.io/en/latest/_sources/installation.rst.txt

Example docker-compose.yml file defining the Elasticsearch service for FSCrawler.

```yaml
---
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:${ES_LOCAL_VERSION}
    container_name: ${ES_LOCAL_CONTAINER_NAME}
    volumes: 
      - dev-elasticsearch:/usr/share/elasticsearch/data
    ports:
      - 127.0.0.1:${ES_LOCAL_PORT}:9200
    environment:
      - discovery.type=single-node
      - ELASTIC_PASSWORD=${ES_LOCAL_PASSWORD}
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=false
      - xpack.license.self_generated.type=${ES_LOCAL_LICENSE}
      - xpack.ml.use_auto_machine_memory_percent=true
      - ES_JAVA_OPTS=${ES_LOCAL_JAVA_OPTS}
      - cluster.routing.allocation.disk.watermark.low=${ES_LOCAL_DISK_SPACE_REQUIRED}
      - cluster.routing.allocation.disk.watermark.high=${ES_LOCAL_DISK_SPACE_REQUIRED}
      - cluster.routing.allocation.disk.watermark.flood_stage=${ES_LOCAL_DISK_SPACE_REQUIRED}
    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test:
        [
          "CMD-SHELL",

```

--------------------------------

### Configure JAVA_HOME

Source: https://fscrawler.readthedocs.io/en/latest/_sources/user/getting_started.rst.txt

Set the JAVA_HOME environment variable to point to your Java installation directory. This is required for FSCrawler to run.

```sh
export JAVA_HOME="~/.sdkman/candidates/java/current"
```

--------------------------------

### Define Include and Exclude Patterns

Source: https://fscrawler.readthedocs.io/en/latest/_sources/admin/fs/local-fs.rst.txt

Use 'fs.includes' and 'fs.excludes' to specify which files to index and which to ignore. Wildcards can be used. Files starting with '~' are excluded by default.

```yaml
name: "test"
fs:
  includes:
  - "*/*.doc"
  - "*/*.pdf"
  excludes:
  - "*/resume*"
```

--------------------------------

### Run Tests with Custom Options

Source: https://fscrawler.readthedocs.io/en/latest/_sources/dev/build.rst.txt

Use these options with `mvn install -rf :fscrawler-it` to customize test execution, such as output verbosity, locale, timezone, and leaving temporary files.

```bash
mvn install -rf :fscrawler-it \
  -Dtests.output=always \
  -Dtests.locale=fr-FR \
  -Dtests.timezone=CEST \
  -Dtests.verbose \
  -Dtests.leaveTemporary \
  -Dtests.seed=E776CE45185A6E7A
```

--------------------------------

### Configure Includes and Excludes

Source: https://fscrawler.readthedocs.io/en/latest/admin/fs/local-fs.html

Defines 'fs.includes' and 'fs.excludes' patterns to selectively index files and directories. Excludes files starting with '~' by default.

```yaml
name: "test"
fs:
  includes:
  - "*/*.doc"
  - "*/*.pdf"
  excludes:
  - "*/resume*"

```

--------------------------------

### Run FSCrawler CLI Test Script

Source: https://fscrawler.readthedocs.io/en/latest/_sources/dev/test-fscrawler-cli.rst.txt

Execute the full FSCrawler CLI test script, which includes building the distribution, starting the Docker stack, and crawling with OTel tracing.

```bash
./distribution/test-scripts/test-fscrawler-cli.sh
```

--------------------------------

### Configure Static Metadata File

Source: https://fscrawler.readthedocs.io/en/latest/_sources/admin/fs/tags.rst.txt

Use the `tags.staticMetaFilename` setting to specify a file containing metadata that applies to all indexed documents. This example uses `/path/to/static_metadata.yml`.

```yaml
fs:
  url: "/path/to/docs"
tags:
  staticMetaFilename: "/path/to/static_metadata.yml"
```

--------------------------------

### Simulate Document Ingestion

Source: https://fscrawler.readthedocs.io/en/latest/_sources/admin/fs/rest.rst.txt

This example demonstrates how to simulate document ingestion without actually indexing the content into Elasticsearch. This is useful for debugging and verifying the extraction process.

```APIDOC
## POST /_document?simulate=true

### Description
Simulates document ingestion, returning extracted content and metadata without indexing.

### Method
POST

### Endpoint
http://127.0.0.1:8080/_document?simulate=true

### Parameters
#### Query Parameters
- **simulate** (boolean) - Optional - If true, simulates ingestion.
- **debug** (boolean) - Optional - Enables debug mode.

#### Request Body
- **file** (file) - Required - The file to upload.

### Request Example
```sh
echo "This is my text" > test.txt
curl -F "file=@test.txt" "http://127.0.0.1:8080/_document?debug=true&simulate=true"
```
```

--------------------------------

### Run FSCrawler with Docker Compose

Source: https://fscrawler.readthedocs.io/en/latest/_sources/installation.rst.txt

Use this command to start the FSCrawler service along with its dependencies when using Docker Compose. Ensure your documents are in the 'docs' directory.

```sh
docker-compose up
```

--------------------------------

### FSCrawler Status Response Example

Source: https://fscrawler.readthedocs.io/en/latest/_sources/admin/fs/rest.rst.txt

A typical JSON response from the GET / endpoint, detailing FSCrawler and Elasticsearch versions, along with detailed configuration settings.

```json
{
  "ok" : true,
  "version" : "|FSCrawler_version|",
  "elasticsearch" : "|ES_stack_version|",
  "settings" : {
    "name" : "fscrawler",
    "fs" : {
      "url" : "/tmp/es",
      "update_rate" : "15m",
      "excludes" : [ "*/~*" ],
      "json_support" : false,
      "add_as_inner_object" : false,
      "xml_support" : false,
      "follow_symlinks" : false,
      "remove_deleted" : true,
      "continue_on_error" : false,
      "filename_as_id" : false,
      "add_filesize" : true,
      "attributes_support" : false,
      "store_source" : false,
      "index_content" : true,
      "acl_support" : false,
      "raw_metadata" : true,
      "index_folders" : true,
      "lang_detect" : false,
      "ocr" : {
        "enabled" : true,
        "language" : "eng",
        "output_type" : "txt",
        "pdf_strategy" : "ocr_and_text",
        "page_seg_mode" : 1,
        "preserve_interword_spacing" : false
      }
    },
    "server" : {
      "port" : 0,
      "protocol" : "local"
    },
    "elasticsearch" : {
      "urls" : [ "http://es-fscrawler:9200" ],
      "index" : "fscrawler_docs",
      "index_folder" : "fscrawler_folder",
      "bulk_size" : 100,
      "flush_interval" : "5s",
      "byte_size" : "10mb",
      "username" : "elastic",
      "ssl_verification" : true,
      "push_templates" : true,
      "semantic_search" : true
    },
    "rest" : {
      "url" : "http://127.0.0.1:8080",
      "enable_cors" : false
    },
    "tags" : {
      "meta_filename" : ".meta.yml"
    }
  }
}
```

--------------------------------

### Build HTML Documentation

Source: https://fscrawler.readthedocs.io/en/latest/dev/doc.html

Navigates to the docs directory and builds the HTML version of the documentation using make.

```bash
$ cd docs
$ make html

```

--------------------------------

### Example Generated JSON Document

Source: https://fscrawler.readthedocs.io/en/latest/admin/fs/elasticsearch.html

This is a sample JSON document structure as generated by FSCrawler, including content, metadata, file information, and path details.

```json
{
   "content":"This is a sample text available in page 1\n\nThis second part of the text is in Page 2\n\n",
   "content_semantic":"This is a sample text available in page 1\n\nThis second part of the text is in Page 2\n\n",
   "meta":{
      "author":"David Pilato",
      "title":"Test Tika title",
      "date":"2016-07-07T16:37:00.000+0000",
      "keywords":[
         "keyword1",
         "  keyword2"
      ],
      "language":"en",
      "description":"Comments",
      "created":"2016-07-07T16:37:00.000+0000"
   },
   "file":{
      "extension":"odt",
      "content_type":"application/vnd.oasis.opendocument.text",
      "created":"2018-07-30T11:35:08.000+0000",
      "last_modified":"2018-07-30T11:35:08.000+0000",
      "last_accessed":"2018-07-30T11:35:08.000+0000",
      "indexing_date":"2018-07-30T11:35:19.781+0000",
      "filesize":6236,
      "filename":"test.odt",
      "url":"file:///tmp/test.odt"
   },
   "path":{
      "root":"7537e4fb47e553f110a1ec312c2537c0",
      "virtual":"/test.odt",
      "real":"/tmp/test.odt"
   }
}


```

--------------------------------

### Example .meta.yml for External Tags

Source: https://fscrawler.readthedocs.io/en/latest/admin/fs/tags.html

Use a `.meta.yml` file in a directory to add custom metadata to documents within that directory. Only supported fields can be added; others are ignored unless explicitly declared as external tags.

```yaml
external:
  myTitle: "My document title"

```

--------------------------------

### Upload and Index a Text File via REST

Source: https://fscrawler.readthedocs.io/en/latest/_sources/user/rest.rst.txt

This example demonstrates how to create a text file and then upload it to FSCrawler using curl for indexing into Elasticsearch. The response includes the filename and the Elasticsearch URL of the created document.

```sh
echo "This is my text" > test.txt
curl -F "file=@test.txt" "http://127.0.0.1:8080/fscrawler/_document"
```

--------------------------------

### Build HTML Documentation

Source: https://fscrawler.readthedocs.io/en/latest/_sources/dev/doc.rst.txt

Navigates to the docs directory and builds the HTML version of the documentation using the make command. The output can be found in target/html/index.html.

```bash
$ cd docs
$ make html
```

--------------------------------

### Docker Compose Example with EDOT Collector

Source: https://fscrawler.readthedocs.io/en/latest/admin/otel.html

A docker-compose configuration for running FSCrawler with an EDOT Collector, Elasticsearch, and Kibana. Tracing is configured via environment variables within the FSCrawler service definition.

```yaml
services:
  fscrawler:
    image: dadoonet/fscrawler:latest
    environment:
      - OTEL_EXPORTER_OTLP_ENDPOINT=http://edot-collector:4318
      - OTEL_SERVICE_NAME=fscrawler
      - OTEL_RESOURCE_ATTRIBUTES=deployment.environment=docker

```

--------------------------------

### Configure FTP Provider with SSH Hostname

Source: https://fscrawler.readthedocs.io/en/latest/_sources/admin/fs/local-fs.rst.txt

Example of migrating from a deprecated server protocol setting to the recommended fs.provider setting for FTP crawling. Note the inclusion of the server hostname.

```yaml
name: "test"
fs:
  provider: "ftp"
  url: "/path/to/data/dir"
server:
  hostname: "mynode.mydomain.com"
```

--------------------------------

### Check Code Formatting with Spotless

Source: https://fscrawler.readthedocs.io/en/latest/dev/build.html

Use Spotless to ensure consistent code formatting across the project. Run `spotless:check` to verify and `spotless:apply` to fix.

```bash
mvn spotless:check

```

```bash
mvn spotless:check -DratchetFrom=NONE

```

```bash
mvn spotless:apply

```

```bash
mvn spotless:install-git-pre-push-hook

```

--------------------------------

### Apply Code Formatting with Spotless

Source: https://fscrawler.readthedocs.io/en/latest/_sources/dev/build.rst.txt

Automatically fix code formatting issues by running `mvn spotless:apply`. It is recommended to run this before committing.

```bash
mvn spotless:apply
```

--------------------------------

### Check Code Formatting with Spotless

Source: https://fscrawler.readthedocs.io/en/latest/_sources/dev/build.rst.txt

Run `mvn spotless:check` to verify code formatting. Use `-DratchetFrom=NONE` to check the entire codebase, not just changes since `origin/master`.

```bash
mvn spotless:check
```

```bash
mvn spotless:check -DratchetFrom=NONE
```

--------------------------------

### Get FSCrawler Status

Source: https://fscrawler.readthedocs.io/en/latest/_sources/admin/fs/rest.rst.txt

Call the root endpoint GET / to retrieve an overview of the running FSCrawler service. The response includes version information and current settings.

```sh
curl http://127.0.0.1:8080/
```

--------------------------------

### Get Crawler Status

Source: https://fscrawler.readthedocs.io/en/latest/_sources/admin/fs/rest.rst.txt

Retrieves the current status of the FSCrawler.

```APIDOC
## GET /_crawler/status

### Description
Retrieves the current status of the FSCrawler, including its state, progress, and statistics.

### Method
GET

### Endpoint
/_crawler/status

### Response
#### Success Response (200)
- **state** (string) - The current state of the crawler (e.g., RUNNING, PAUSED, STOPPED).
- **scan_id** (string) - The unique identifier for the current scanning process.
- **current_path** (string) - The directory currently being scanned.
- **pending_directories** (integer) - The number of directories yet to be scanned.
- **completed_directories** (integer) - The number of directories that have been scanned.
- **files_processed** (integer) - The total number of files processed.
- **files_deleted** (integer) - The number of files deleted during the process.
- **scan_start_time** (string) - The timestamp when the scan started.
- **scan_end_time** (string) - The timestamp when the scan ended (null if ongoing).
- **next_check** (string) - The time for the next scheduled check (null if not applicable).
- **elapsed_time** (string) - The duration the scan has been running.
- **retry_count** (integer) - The number of retries performed.
- **last_error** (string) - Information about the last encountered error (null if none).

#### Response Example
```json
{
  "state" : "RUNNING",
  "scan_id" : "abc123-def456",
  "current_path" : "/data/documents/subfolder",
  "pending_directories" : 42,
  "completed_directories" : 158,
  "files_processed" : 1523,
  "files_deleted" : 12,
  "scan_start_time" : "2024-01-15T10:30:00",
  "scan_end_time" : null,
  "next_check" : null,
  "elapsed_time" : "15m 32s",
  "retry_count" : 0,
  "last_error" : null
}
```
```

--------------------------------

### FSCrawler Status

Source: https://fscrawler.readthedocs.io/en/latest/_sources/admin/fs/rest.rst.txt

Retrieve an overview of the running FSCrawler service by calling the GET / endpoint.

```APIDOC
## GET /

### Description
Retrieves an overview of the running FSCrawler service.

### Method
GET

### Endpoint
/

### Response
#### Success Response (200)
- **ok** (boolean) - Indicates if the request was successful.
- **version** (string) - The version of FSCrawler.
- **elasticsearch** (string) - The version of Elasticsearch.
- **settings** (object) - Contains the FSCrawler settings.

#### Response Example
```json
{
  "ok" : true,
  "version" : "|FSCrawler_version|",
  "elasticsearch" : "|ES_stack_version|",
  "settings" : { ... }
}
```
```

--------------------------------

### Live Reload Documentation Builds

Source: https://fscrawler.readthedocs.io/en/latest/dev/doc.html

Uses sphinx-autobuild to automatically rebuild and serve documentation when source files change, enabling hot reloading.

```bash
$ sphinx-autobuild source target/html

```

--------------------------------

### View Elasticsearch Component Templates

Source: https://fscrawler.readthedocs.io/en/latest/admin/fs/elasticsearch.html

Retrieve all component templates starting with 'fscrawler*' to inspect their content.

```json
GET _component_template/fscrawler*

```

--------------------------------

### Enable Live Reloading for Documentation

Source: https://fscrawler.readthedocs.io/en/latest/_sources/dev/doc.rst.txt

Uses sphinx-autobuild to serve documentation with live reloading. Changes made to documentation files will automatically update in the browser.

```bash
$ sphinx-autobuild source target/html
```

--------------------------------

### Run Tests with Custom Options

Source: https://fscrawler.readthedocs.io/en/latest/dev/build.html

Configure test execution with options like output verbosity, locale, timezone, and temporary file handling.

```bash
mvn install -rf :fscrawler-it \
  -Dtests.output=always \
  -Dtests.locale=fr-FR \
  -Dtests.timezone=CEST \
  -Dtests.verbose \
  -Dtests.leaveTemporary \
  -Dtests.seed=E776CE45185A6E7A

```

--------------------------------

### Resume the Crawler

Source: https://fscrawler.readthedocs.io/en/latest/_sources/admin/fs/rest.rst.txt

This endpoint resumes a paused FSCrawler, allowing it to start its next scheduled or on-demand run.

```APIDOC
## POST /_crawler/resume

### Description
Resumes a paused FSCrawler. If the crawler is not paused, this action has no effect.

### Method
POST

### Endpoint
http://127.0.0.1:8080/_crawler/resume

### Response
#### Success Response (200)
- **ok** (boolean) - Indicates success.
- **message** (string) - Confirmation message.

Example (resuming from pause):
```json
{
  "ok" : true,
  "message" : "Crawler resumed."
}
```

Example (crawler not paused):
```json
{
  "ok" : true,
  "message" : "Crawler is not paused. No action needed."
}
```

#### Error Response (400)
- **ok** (boolean) - Indicates failure.
- **message** (string) - Error message when the crawler is closed.

Example:
```json
{
  "ok" : false,
  "message" : "Crawler is closed. Cannot resume."
}
```
```

--------------------------------

### FSCrawler CLI Test with Options

Source: https://fscrawler.readthedocs.io/en/latest/dev/test-fscrawler-cli.html

Demonstrates various ways to run the FSCrawler CLI test script using different flags to control the build, Docker stack, and OTel tracing.

```bash
./distribution/test-scripts/test-fscrawler-cli.sh --skip-docker
```

```bash
./distribution/test-scripts/test-fscrawler-cli.sh --no-otel
```

```bash
./distribution/test-scripts/test-fscrawler-cli.sh --skip-build --skip-docker
```

```bash
./distribution/test-scripts/test-fscrawler-cli.sh --skip-build --skip-docker --log-level=debug
```

--------------------------------

### Activate REST Service

Source: https://fscrawler.readthedocs.io/en/latest/admin/fs/rest.html

Launch FSCrawler with the `--rest` option to enable its REST service.

```bash
fscrawler --rest
```

--------------------------------

### Using Placeholders in Job Files

Source: https://fscrawler.readthedocs.io/en/latest/_sources/admin/fs/index.rst.txt

Define job file settings using environment variables or system properties. Placeholders like ${VAR_NAME} are supported, with optional default values using :=.

```yaml
fs:
  url: "${HOME}/docs"
elasticsearch:
  nodes:
  - url: "${ES_NODE1:=https://127.0.0.1:9200}"
  api_key: "${ES_API_KEY}"
```

--------------------------------

### Example Elasticsearch Document Structure

Source: https://fscrawler.readthedocs.io/en/latest/_sources/admin/fs/elasticsearch.rst.txt

This is a typical JSON document generated by the crawler, showcasing content, metadata, and file information.

```json
{
   "content":"This is a sample text available in page 1\n\nThis second part of the text is in Page 2\n\n",
   "content_semantic":"This is a sample text available in page 1\n\nThis second part of the text is in Page 2\n\n",
   "meta":{
      "author":"David Pilato",
      "title":"Test Tika title",
      "date":"2016-07-07T16:37:00.000+0000",
      "keywords":[
         "keyword1",
         "  keyword2"
      ],
      "language":"en",
      "description":"Comments",
      "created":"2016-07-07T16:37:00.000+0000"
   },
   "file":{
      "extension":"odt",
      "content_type":"application/vnd.oasis.opendocument.text",
      "created":"2018-07-30T11:35:08.000+0000",
      "last_modified":"2018-07-30T11:35:08.000+0000",
      "last_accessed":"2018-07-30T11:35:08.000+0000",
      "indexing_date":"2018-07-30T11:35:19.781+0000",
      "filesize":6236,
      "filename":"test.odt",
      "url":"file:///tmp/test.odt"
   },
   "path":{
      "root":"7537e4fb47e553f110a1ec312c2537c0",
      "virtual":"/test.odt",
      "real":"/tmp/test.odt"
   }
}
```

--------------------------------

### Execute Project Release Script

Source: https://fscrawler.readthedocs.io/en/latest/_sources/dev/release.rst.txt

Run the release script to automate the project release process. This script handles versioning, testing, building, deployment, and announcements. You can pass Maven options like `-DskipTests` to customize the build.

```bash
$ release.sh
```

--------------------------------

### Get Crawler Status using REST

Source: https://fscrawler.readthedocs.io/en/latest/_sources/admin/fs/rest.rst.txt

Use this curl command to retrieve the current operational status of the FSCrawler instance.

```sh
curl http://127.0.0.1:8080/_crawler/status
```

--------------------------------

### Run Tests with External Cluster and API Key

Source: https://fscrawler.readthedocs.io/en/latest/_sources/dev/build.rst.txt

When using an external cluster, provide an API key for authentication if your cluster doesn't use default credentials. This is the recommended approach for security.

```bash
mvn verify -pl fr.pilato.elasticsearch.crawler:fscrawler-it \
    -Dtests.cluster.apiKey=APIKEYHERE \
    -Dtests.cluster.url=https://localhost:9200 \

```

--------------------------------

### Configure FTP Settings

Source: https://fscrawler.readthedocs.io/en/latest/admin/fs/ftp.html

Use this configuration to index files from a remote FTP server. Ensure all FTP server details are correctly provided.

```yaml
name: "test"
fs:
  provider: "ftp"
  url: "/path/to/data/dir/on/server"
server:
  hostname: "mynode.mydomain.com"
  port: 21
  username: "username"
  password: "password"

```

--------------------------------

### Extract Latest SNAPSHOT Version

Source: https://fscrawler.readthedocs.io/en/latest/installation.html

Example XML snippet showing how to find the latest SNAPSHOT version for downloading FSCrawler distribution.

```xml
<snapshotVersion>
  <extension>zip</extension>
  <value>2.10-20250801.161301-75</value>
  <updated>20250801161301</updated>
</snapshotVersion>

```

--------------------------------

### Run Project Release Script

Source: https://fscrawler.readthedocs.io/en/latest/dev/release.html

Execute the release script to automate the project release process. This script handles versioning, testing, building, deployment, and more. You can pass Maven options like -DskipTests to customize the build.

```bash
$ release.sh

```

--------------------------------

### Example Raw Metadata from PDF

Source: https://fscrawler.readthedocs.io/en/latest/_sources/admin/fs/local-fs.rst.txt

This JSON object illustrates the kind of raw metadata that can be extracted from a PDF document when `raw_metadata` is enabled.

```json
{
   "date" : "2016-07-07T08:37:42Z",
   "pdf:PDFVersion" : "1.5",
   "xmp:CreatorTool" : "Microsoft Word",
   "Keywords" : "keyword1, keyword2",
   "access_permission:modify_annotations" : "true",
   "access_permission:can_print_degraded" : "true",
   "subject" : "Test Tika Object",
   "dc:creator" : "David Pilato",
   "dcterms:created" : "2016-07-07T08:37:42Z",
   "Last-Modified" : "2016-07-07T08:37:42Z",
   "dcterms:modified" : "2016-07-07T08:37:42Z",
   "dc:format" : "application/pdf; version=1.5",
   "title" : "Test Tika title",
   "Last-Save-Date" : "2016-07-07T08:37:42Z",
   "access_permission:fill_in_form" : "true",
   "meta:save-date" : "2016-07-07T08:37:42Z",
   "pdf:encrypted" : "false",
   "dc:title" : "Test Tika title",
   "modified" : "2016-07-07T08:37:42Z",
   "cp:subject" : "Test Tika Object",
   "Content-Type" : "application/pdf",
   "X-Parsed-By" : "org.apache.tika.parser.DefaultParser",
   "creator" : "David Pilato",
   "meta:author" : "David Pilato",
   "dc:subject" : "keyword1, keyword2",
   "meta:creation-date" : "2016-07-07T08:37:42Z",
   "created" : "Thu Jul 07 10:37:42 CEST 2016",
   "access_permission:extract_for_accessibility" : "true",
   "access_permission:assemble_document" : "true"
}
```

--------------------------------

### Clear Checkpoint

Source: https://fscrawler.readthedocs.io/en/latest/admin/fs/rest.html

Call this endpoint to clear the current checkpoint, forcing a fresh start. The crawler must be paused or stopped before calling this endpoint.

```APIDOC
## DELETE /_crawler/checkpoint

### Description
Clears the FSCrawler's checkpoint, forcing a fresh start. The crawler must be paused or stopped prior to this operation.

### Method
DELETE

### Endpoint
/_crawler/checkpoint

### Request Example
```
curl -X DELETE http://127.0.0.1:8080/_crawler/checkpoint
```

### Response
#### Success Response (200)
- **ok** (boolean) - Indicates success.
- **message** (string) - Confirmation message.

#### Response Example
```json
{
  "ok" : true,
  "message" : "Checkpoint cleared"
}
```

#### Error Response (400)
- **ok** (boolean) - Indicates failure.
- **message** (string) - Error message if the crawler is running and not paused.

#### Response Example
```json
{
  "ok" : false,
  "message" : "Cannot clear checkpoint while crawler is running. Pause or stop it first."
}
```

#### Error Response (404)
- **ok** (boolean) - Indicates failure.
- **message** (string) - Error message if there is no active crawler.

#### Response Example
```json
{
  "ok" : false,
  "message" : "Failed to clear checkpoint as we don't have a checkpoint handler. This probably means there's no active crawler. Did you start with --loop 0?"
}
```

#### Server Error Response (500)
- **message** (string) - Error message indicating an I/O error during checkpoint clearing.

#### Response Example
```
Failed to clear checkpoint: [specific error message]
```
```

--------------------------------

### Enable XML Document Indexing

Source: https://fscrawler.readthedocs.io/en/latest/admin/fs/local-fs.html

Set `xml_support` to `true` to index XML files and convert them to JSON. Content is stored under `_source`. For synchronization, use `add_as_inner_object`.

```yaml
name: "test"
fs:
  xml_support: true

```
Tokens:
Raw