### Install Pre-commit Hooks

Source: https://github.com/arxiv/arxiv-search/blob/develop/README.md

Installs the necessary development dependencies and then installs the pre-commit hooks to run checks before each commit.

```bash
pipenv install --dev
pipenv run pre-commit install
```

--------------------------------

### Install Development Dependencies

Source: https://github.com/arxiv/arxiv-search/blob/develop/README.md

Installs all necessary development tools and dependencies for testing and quality assurance.

```bash
pipenv install --dev
```

--------------------------------

### Search for Multiple Categories - Wildcard Example

Source: https://github.com/arxiv/arxiv-search/blob/develop/search/domain/classic_api/README_classic_search_query.md

Illustrates how to search for entries within a category using a wildcard. Includes both working and failing examples.

```shell
# Works:
curl -si https://export.arxiv.org/api/query?search_query=cat:q-fin.%2A | egrep 'cat|<title'

# Fails:
curl -si http://localhost:8080/api/query?search_query=cat:q-fin.%2A | egrep 'cat|<title'
```

--------------------------------

### Install classic-api with Helm

Source: https://github.com/arxiv/arxiv-search/blob/develop/deploy/classic-api/README.md

Installs the classic-api using Helm, specifying image tag, namespaces, and Elasticsearch configuration.

```bash
helm install ./ --name=classic-api --set=image.tag=some_tag \
  --tiller-namespace=development --namespace=development  \
  --set=ingress.host=development.arxiv.org \
  --set=elasticsearch.host=foo.es.amazonaws.com \
  --set=elasticsearch.index=arxiv0.3
```

--------------------------------

### Install Dependencies and Create/Populate Index

Source: https://github.com/arxiv/arxiv-search/blob/develop/README.md

Installs development dependencies using pipenv and then runs scripts to create and populate the search index. Ensure you have access to the /docmeta endpoint, which is restricted to the CUL network.

```bash
pipenv install --dev
FLASK_APP=app.py FLASK_DEBUG=1 ELASTICSEARCH_SERVICE_HOST=127.0.0.1 pipenv run python create_index.py
FLASK_APP=app.py FLASK_DEBUG=1 ELASTICSEARCH_SERVICE_HOST=127.0.0.1 pipenv run python bulk_index.py
```

--------------------------------

### Install search-api with Helm

Source: https://github.com/arxiv/arxiv-search/blob/develop/deploy/api/README.md

Use this command to install the search-api to the development namespace. Customize image tag, Elasticsearch host and index, and replica count as needed.

```bash
helm install ./ --set=image.tag=some_tag \
  --tiller-namespace=development --namespace=development  \
  --set=ingress.host=development.arxiv.org \
  --set=elasticsearch.host=foo.es.amazonaws.com \
  --set=elasticsearch.index=arxiv0.3 \
  --set=scaling.replicas=2
```

--------------------------------

### Query Cannot Start with a Wildcard

Source: https://github.com/arxiv/arxiv-search/blob/develop/search/domain/classic_api/README_classic_search_query.md

Demonstrates an invalid query that starts with a wildcard character.

```shell
curl -is http://localhost:8080/api/query?search_query=%2A'
```

--------------------------------

### Ruby Simple Example

Source: https://github.com/arxiv/arxiv-search/blob/develop/docs/source/classic_api.md

This example demonstrates how to retrieve and print the Atom feed from the arXiv API using Ruby's built-in uri and net/http libraries.

```ruby
require 'uri'
require 'net/http'

query = URI.parse("http://arxiv.org/api/query?search_query=all:electron&id_list=&start=0&max_results=1")

response = Net::HTTP.start(query.host, query.port) do |http|
  http.get(query.path)
end

puts response.body

```

--------------------------------

### Fetch arXiv API data with Perl

Source: https://github.com/arxiv/arxiv-search/blob/develop/docs/source/classic_api.md

Uses the LWP module to make a GET request to the arXiv API and print the content. Ensure LWP is installed.

```default
use LWP;
use strict;

my $url = 'http://export.arxiv.org/api/query?search_query=all:electron&start=0&max_results=1';
my $browser = LWP::UserAgent->new();
my $response = $browser->get($url);
print $response->content();
```

--------------------------------

### Run Classic API in Dev Mode

Source: https://github.com/arxiv/arxiv-search/blob/develop/README.md

Starts the Flask development server for the classic API. Ensure FLASK_APP is set to 'classic-api.py'.

```bash
FLASK_APP=classic-api.py FLASK_DEBUG=1 ELASTICSEARCH_SERVICE_HOST=127.0.0.1 pipenv run flask run
```

--------------------------------

### Bypass Search Query Adaptor - Error Example

Source: https://github.com/arxiv/arxiv-search/blob/develop/search/domain/classic_api/README_classic_search_query.md

Shows an example of an invalid query string when using the raw=1 parameter.

```shell
curl -is http://localhost:8080/api/query?search_query=pineapple&raw=1
  <entry>
    <title>Error</title>
    <summary>Invalid query string: 'pineapple'</summary>
```

--------------------------------

### Run New Metadata API in Dev Mode

Source: https://github.com/arxiv/arxiv-search/blob/develop/README.md

Starts the Flask development server for the new metadata API. Ensure FLASK_APP is set to 'api.py'.

```bash
FLASK_APP=api.py FLASK_DEBUG=1 ELASTICSEARCH_SERVICE_HOST=127.0.0.1 pipenv run flask run
```

--------------------------------

### Typical Atom Response Example

Source: https://github.com/arxiv/arxiv-search/blob/develop/docs/source/classic_api.md

This is an example of a typical Atom response received from the arXiv API when querying for 'electron'. It includes metadata about the search and the first entry found.

```xml
<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom" xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/" xmlns:arxiv="http://arxiv.org/schemas/atom">
  <link xmlns="http://www.w3.org/2005/Atom" href="http://arxiv.org/api/query?search_query=all:electron&amp;id_list=&amp;start=0&amp;max_results=1" rel="self" type="application/atom+xml"/>
  <title xmlns="http://www.w3.org/2005/Atom">ArXiv Query: search_query=all:electron&amp;id_list=&amp;start=0&amp;max_results=1</title>
  <id xmlns="http://www.w3.org/2005/Atom">http://arxiv.org/api/cHxbiOdZaP56ODnBPIenZhzg5f8</id>
  <updated xmlns="http://www.w3.org/2005/Atom">2007-10-08T00:00:00-04:00</updated>
  <opensearch:totalResults xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/">1000</opensearch:totalResults>
  <opensearch:startIndex xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/">0</opensearch:startIndex>
  <opensearch:itemsPerPage xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/">1</opensearch:itemsPerPage>
  <entry xmlns="http://www.w3.org/2005/Atom" xmlns:arxiv="http://arxiv.org/schemas/atom">
    <id xmlns="http://www.w3.org/2005/Atom">http://arxiv.org/abs/hep-ex/0307015</id>
    <published xmlns="http://www.w3.org/2005/Atom">2003-07-07T13:46:39-04:00</published>
    <updated xmlns="http://www.w3.org/2005/Atom">2003-07-07T13:46:39-04:00</updated>
    <title xmlns="http://www.w3.org/2005/Atom">Multi-Electron Production at High Transverse Momenta in ep Collisions at
  HERA</title>
    <summary xmlns="http://www.w3.org/2005/Atom">  Multi-electron production is studied at high electron transverse momentum in
positron- and electron-proton collisions using the H1 detector at HERA. The
data correspond to an integrated luminosity of 115 pb-1. Di-electron and
tri-electron event yields are measured. Cross sections are derived in a
restricted phase space region dominated by photon-photon collisions. In general
good agreement is found with the Standard Model predictions. However, for
electron pair invariant masses above 100 GeV, three di-electron events and
three tri-electron events are observed, compared to Standard Model expectations
of 0.30 0.04 and 0.23 0.04, respectively.
</summary>
    <author xmlns="http://www.w3.org/2005/Atom">
      <name xmlns="http://www.w3.org/2005/Atom">H1 Collaboration</name>
    </author>
    <arxiv:comment xmlns:arxiv="http://arxiv.org/schemas/atom">23 pages, 8 figures and 4 tables</arxiv:comment>
    <arxiv:journal_ref xmlns:arxiv="http://arxiv.org/schemas/atom">Eur.Phys.J. C31 (2003) 17-29</arxiv:journal_ref>
    <link xmlns="http://www.w3.org/2005/Atom" href="http://arxiv.org/abs/hep-ex/0307015v1" rel="alternate" type="text/html"/>
    <link xmlns="http://www.w3.org/2005/Atom" title="pdf" href="http://arxiv.org/pdf/hep-ex/0307015v1" rel="related" type="application/pdf"/>
    <arxiv:primary_category xmlns:arxiv="http://arxiv.org/schemas/atom" term="hep-ex" scheme="http://arxiv.org/schemas/atom"/>
    <category term="hep-ex" scheme="http://arxiv.org/schemas/atom"/>
  </entry>
</feed>

```

--------------------------------

### Build Documentation Locally

Source: https://github.com/arxiv/arxiv-search/blob/develop/README.md

Builds the service documentation locally. Requires navigating to the docs directory, installing requirements, and running make with a specified format (e.g., html, latexpdf).

```bash
cd docs
pip install -r requirements.txt
make [format]
```

--------------------------------

### Bypass Search Query Adaptor - Working Example

Source: https://github.com/arxiv/arxiv-search/blob/develop/search/domain/classic_api/README_classic_search_query.md

Demonstrates a successful query using the raw=1 parameter, showing the modified title element.

```shell
curl -is http://localhost:8080/api/query?search_query=pineapple&raw=1
    ...
    <title>arXiv Query: search_query=all:pineapple
```

--------------------------------

### OpenSearch Start Index Element

Source: https://github.com/arxiv/arxiv-search/blob/develop/docs/source/classic_api.md

The <opensearch:startIndex> element specifies the starting index of the results, analogous to the 'start' parameter in API queries.

```xml
<opensearch:startIndex xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/">
   0
</opensearch:startIndex>
```

--------------------------------

### PHP Simple Example

Source: https://github.com/arxiv/arxiv-search/blob/develop/docs/source/classic_api.md

This snippet illustrates how to fetch and display the raw Atom response from the arXiv API using PHP's file_get_contents function.

```php
<?php

$query = "http://arxiv.org/api/query?search_query=all:electron&id_list=&start=0&max_results=1";
$response = file_get_contents($query);

if ($response === FALSE) {
    // Handle error
    echo "Error fetching data.";
} else {
    echo $response;
}

?>

```

--------------------------------

### Paging through search results

Source: https://github.com/arxiv/arxiv-search/blob/develop/docs/source/classic_api.md

Demonstrates how to construct URLs to retrieve results in chunks using the start and max_results parameters. This is useful for handling large result sets.

```default
http://export.arxiv.org/api/query?search_query=all:electron&start=0&max_results=10 (1)
http://export.arxiv.org/api/query?search_query=all:electron&start=10&max_results=10 (2)
http://export.arxiv.org/api/query?search_query=all:electron&start=20&max_results=10 (3)
```

--------------------------------

### Fuzzy search example

Source: https://github.com/arxiv/arxiv-search/blob/develop/search/domain/classic_api/README_classic_search_query.md

Demonstrates a fuzzy search using the '~' modifier, which appears to work on author and 'all' prefixes but not on the title field. The examples show it having no effect.

```shell
curl -si https://export.arxiv.org/api/query?search_query="fruit+identification"~3 | egrep '<name|<title'
```

```shell
curl -si https://export.arxiv.org/api/query?search_query="fruit+identification"~3 | egrep '<name|<title'
```

--------------------------------

### Python Simple Example

Source: https://github.com/arxiv/arxiv-search/blob/develop/docs/source/classic_api.md

This snippet shows how to fetch data from the arXiv API using Python's urllib library and display the Atom feed.

```python
import urllib

query = "http://arxiv.org/api/query?search_query=all:electron&id_list=&start=0&max_results=1"
response = urllib.urlopen(query)

print response.read()

```

--------------------------------

### Perl Simple Example

Source: https://github.com/arxiv/arxiv-search/blob/develop/docs/source/classic_api.md

This snippet demonstrates how to make a simple HTTP request to the arXiv API using Perl's LWP library and print the raw Atom response.

```perl
#!/usr/bin/perl

use strict;
use warnings;

use LWP::UserAgent;

my $ua = LWP::UserAgent->new;
my $query = "http://arxiv.org/api/query?search_query=all:electron&id_list=&start=0&max_results=1";
my $response = $ua->get($query);

if ($response->is_success) {
    print $response->decoded_content;
} else {
    die $response->status_line;
}

```

--------------------------------

### API Search Query with Sorting Parameters

Source: https://github.com/arxiv/arxiv-search/blob/develop/docs/source/classic_api.md

Example of an API search query demonstrating the use of `sortBy` and `sortOrder` parameters to control the result set ordering.

```default
http://export.arxiv.org/api/query?search_query=ti:"electron thermal conductivity"&sortBy=lastUpdatedDate&sortOrder=ascending
```

--------------------------------

### Run Flask Development Server

Source: https://github.com/arxiv/arxiv-search/blob/develop/README.md

Starts the Flask development server, which monitors Python files for changes and automatically restarts. Static files and templates are not monitored, requiring a manual restart for changes to take effect.

```bash
FLASK_APP=app.py FLASK_DEBUG=1 ELASTICSEARCH_SERVICE_HOST=127.0.0.1 pipenv run flask run
```

--------------------------------

### Search by DOI (no results)

Source: https://github.com/arxiv/arxiv-search/blob/develop/search/domain/classic_api/README_classic_search_query.md

Example of a curl command to search for a specific DOI that is expected to yield no results.

```shell
curl -is https://export.arxiv.org/api/query?search_query=doi:"10.1088/0004-637X/753/1/35"
```

--------------------------------

### Indexing Agent Log Output

Source: https://github.com/arxiv/arxiv-search/blob/develop/README.md

Example log output from the indexing agent showing it processing records. This output is informational and indicates the agent's progress.

```log
agent            | application 12/Apr/2018:15:49:24 +0000 - search.agent.consumer - None - [arxiv:null] - INFO: "Processing record 49583482484923667520018808447538184389617576917415755778"
agent            | application 12/Apr/2018:15:49:25 +0000 - search.agent.consumer - None - [arxiv:null] - INFO: "Processing record 49583482484923667520018808447539393315437191546590461954"
agent            | application 12/Apr/2018:15:49:25 +0000 - search.agent.consumer - None - [arxiv:null] - INFO: "Processing record 49583482484923667520018808447540602241256806175765168130"
agent            | application 12/Apr/2018:15:49:25 +0000 - search.agent.consumer - None - [arxiv:null] - INFO: "Processing record 49583482484923667520018808447541811167076420804939874306"
```

--------------------------------

### Date Filtering with submittedDate

Source: https://github.com/arxiv/arxiv-search/blob/develop/search/domain/classic_api/README_classic_search_query.md

Demonstrates filtering search results by date range using the 'submittedDate' parameter, including variations with brackets and legacy examples.

```shell
curl -is http://localhost:8080/api/query?search_query=submittedDate:"202301010600+TO+202301030600" | egrep -i 'date|publish|<id>'
Same with [ ]:
http://localhost:8080/api/query?search_query=submittedDate:%5b202301010600+TO+202301030600%5D | egrep -i '<id>' | sort
   ...
   <id>http://arxiv.org/abs/2210.05489v3</id>
   <updated>2023-01-02T20:15:04Z</updated>
   <published>2022-10-11T10:45:07Z</published>

# legacy:
curl -si https://export.arxiv.org/api/query?search_query=ti:%22Generating+Approximate+Ground+States+of+Molecules%22 | gi '<updated|<publish|<id>'
  <updated>2025-11-03T00:00:00-05:00</updated>
    <id>http://arxiv.org/abs/2210.05489v3</id>
    <updated>2023-01-02T05:37:45Z</updated>
    <published>2022-10-11T14:45:07Z</published>

curl -si https://export.arxiv.org/api/query?search_query=ti:%22Generating+Approximate+Ground+States+of+Molecules%22+AND+submittedDate:%5b202301010600+TO+202301030600%5D | gi '<updated|<publish|<id>'

curl -si https://export.arxiv.org/api/query?search_query=ti:%22Generating+Approximate+Ground+States+of+Molecules%22+AND+submittedDate:%5b202210100600+TO+202210120600%5D | gi '<updated|<publish|<id>'
    ...
    <id>http://arxiv.org/abs/2210.05489v3</id>

# Correct the python search to use published:
curl -si http://localhost:8080/api/query?search_query=ti:%22Generating+Approximate+Ground+States+of+Molecules%22+AND+submittedDate:%5b202210100600+TO+202210120600%5D | gi '<updated|<publish|<id>|<title'
    <id>http://arxiv.org/abs/2210.05489v3</id>
    <updated>2025-11-03T18:00:21Z</updated>
    <published>2022-10-11T10:45:07Z</published>
    <title>Generating Approximate Ground States of Mol...
```

--------------------------------

### Run Elasticsearch and Kibana with Docker Compose

Source: https://github.com/arxiv/arxiv-search/blob/develop/README.md

Starts Elasticsearch and Kibana using the provided docker-compose.yml file. This maps ES ports 9200/9300 and Kibana port 5601 to localhost. Ensure you have a recent version of docker-compose.

```bash
docker-compose up
```

--------------------------------

### v2 API search by all or DOI

Source: https://github.com/arxiv/arxiv-search/blob/develop/search/domain/classic_api/README_classic_search_query.md

Examples using the v2 API endpoint (localhost:8080) to search using the 'all' field or 'doi' field. Both 'all' and 'doi' fields accept quoted values.

```shell
curl -si http://localhost:8080/api/query?search_query=all:"10.1038/418838a"
```

```shell
curl -si http://localhost:8080/api/query?search_query=doi:"10.1038/418838a"
```

--------------------------------

### RSS Response from Classic API

Source: https://github.com/arxiv/arxiv-search/blob/develop/docs/source/migration.md

Example of an RSS response from the classic arXiv API, showing search results in XML format. This format was used by the older API.

```xml
<?xml version='1.0' encoding='UTF-8'?>
<feed xmlns:arxiv="http://arxiv.org/schemas/atom" xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/" xmlns="http://www.w3.org/2005/Atom">
<id>http://api.arxiv.org/</id>
<title>arXiv Query: size: 50; terms: AND title=universes; include_fields: ['canonical', 'title', 'paper_id', 'href', 'version', 'paper_id_v', 'abstract', 'submitted_date', 'updated_date', 'comments', 'journal_ref', 'doi', 'primary_classification', 'secondary_classification', 'authors']</title>
<updated>2019-06-04T06:01:31.365740+00:00</updated>
<link href="https://api.arxiv.org/" type="application/atom+xml"/>
<generator uri="http://lkiesow.github.io/python-feedgen" version="0.7.0">python-feedgen</generator>
<opensearch:itemsPerPage>50</opensearch:itemsPerPage>
<opensearch:totalResults>6</opensearch:totalResults>
<opensearch:startIndex>0</opensearch:startIndex>
<entry>
    <id>http://127.0.0.1:5000/astro-ph/0311033v4</id>
    <title>A new paradigm for the universe</title>
    <updated>2019-06-04T06:01:31.366602+00:00</updated>
    <link href="http://127.0.0.1:5000/astro-ph/0311033v4" rel="alternate" type="text/html"/>
    <summary>This book provides a completely new approach to understanding the universe. The main idea is that the principal objects in the universe form a spectrum unified by the presence of a massive or hypermassive black hole. These objects are variously called quasars, active galaxies and spiral galaxies. The key to understanding their dynamics is angular momentum and the key tool, and main innovative idea of this work, is a proper formulation of "Mach's principle" using Sciama's ideas. In essence, what is provided here is a totally new paradigm for the universe. In this paradigm, there is no big bang, and the universe is many orders of magnitude older than current estimates for its age. Indeed there is no natural limit for its age.</summary>
    <category term="astro-ph" scheme="http://arxiv.org://arxiv.org/schemas/atom"/>
    <published>2017-12-21T13:11:38-05:00</published>
    <arxiv:comment>198 pages, 81 figures (including jpg's). Version 4 has a good sketch for the source of the CMB (see appendix F)</arxiv:comment>
    <arxiv:primary_category term="astro-ph"/>
    <author>
    <name>Colin Rourke</name>
    </author>
</entry>
</feed>
```

--------------------------------

### Basic API Search Query

Source: https://github.com/arxiv/arxiv-search/blob/develop/docs/source/classic_api.md

Use this URL to search for articles containing a specific term in the title or abstract. This is a simple GET request.

```url
http://export.arxiv.org/api/query?search_query=all:electron
```

--------------------------------

### Fetch arXiv API data with Ruby

Source: https://github.com/arxiv/arxiv-search/blob/develop/docs/source/classic_api.md

Uses the net/http and uri modules from the Ruby standard library to retrieve data from the arXiv API. These are included in default Ruby installations.

```default
require 'net/http'
require 'uri'
url = URI.parse('http://export.arxiv.org/api/query?search_query=all:electron&start=0&max_results=1')
res = Net::HTTP.get_response(url)
print res.body
```

--------------------------------

### JSON Response from New Search API

Source: https://github.com/arxiv/arxiv-search/blob/develop/docs/source/migration.md

Example of a JSON response from the new arXiv Search API, illustrating metadata and search results. This format is preferred for modern web development.

```json
{
"metadata": {
    "end": 6,
    "query": [
    {
        "parameter": "title",
        "value": "universes"
    }
    ],
    "size": 50,
    "start": 0,
    "total": 6
},
"results": [
    {
    "canonical": "https://arxiv.org/abs/1801.01865v2",
    "href": "http://127.0.0.1:5000/1801.01865v2",
    "paper_id": "1801.01865",
    "paper_id_v": "1801.01865v2",
    "title": "Massless Particle Creation in Bianchi I Universes",
    "version": 2
    }
]
}
```

--------------------------------

### Simple phrase search in title

Source: https://github.com/arxiv/arxiv-search/blob/develop/search/domain/classic_api/README_classic_search_query.md

Searches for the exact word 'pineapple' within the title field. This is a basic example of using quotes for exact word matching.

```bash
curl -is https://export.arxiv.org/api/query?search_query=ti:%22pineapple%22 | gi '<title' | sort
```

--------------------------------

### Fetch arXiv API data with Python

Source: https://github.com/arxiv/arxiv-search/blob/develop/docs/source/classic_api.md

Uses the urllib module from the Python standard library to fetch data from the arXiv API. This is included in default Python installations.

```default
import urllib
url = 'http://export.arxiv.org/api/query?search_query=all:electron&start=0&max_results=1'
data = urllib.urlopen(url).read()
print data
```

--------------------------------

### API Search Query with AND operator

Source: https://github.com/arxiv/arxiv-search/blob/develop/docs/source/classic_api.md

Construct a search query using the AND operator to find articles that contain multiple specified terms. This is also a GET request.

```url
http://export.arxiv.org/api/query?search_query=all:electron+AND+all:proton
```

--------------------------------

### Retrieve Multiple Entries by ID List

Source: https://github.com/arxiv/arxiv-search/blob/develop/search/domain/classic_api/README_classic_search_query.md

Fetches multiple arXiv entries by providing a comma-separated list of identifiers in the `id_list` parameter. This is an efficient way to get specific papers.

```bash
https://export.arxiv.org/api/query?id_list=2511.00104,2511.00147
```

--------------------------------

### Author search with wildcard

Source: https://github.com/arxiv/arxiv-search/blob/develop/search/domain/classic_api/README_classic_search_query.md

Searches for authors whose last name starts with 'del mae', using a wildcard. The underscore is shown as a shortcut for quotes and spaces in author names.

```bash
curl -si https://export.arxiv.org/api/query?search_query=au:del_mae%2A | egrep '<name|<title'
```

--------------------------------

### Retrieving a specific slice of large result sets

Source: https://github.com/arxiv/arxiv-search/blob/develop/docs/source/classic_api.md

Shows how to request a specific range of results (e.g., matches 6001-8000) from a large result set by adjusting the start and max_results parameters. Note the server-side limit of 30000 results per call, with slices of at most 2000.

```default
http://export.arxiv.org/api/query?search_query=all:electron&start=6000&max_results=8000
```

--------------------------------

### Indexing Agent Startup Logs

Source: https://github.com/arxiv/arxiv-search/blob/develop/README.md

Observe the initial log output during the indexing agent's startup phase. This shows the agent attempting to connect to Kinesis and waiting for metadata.

```bash
agent            | application 12/Apr/2018:15:43:13 +0000 - search.agent.base - None - [arxiv:null] - INFO: "New consumer for MetadataIsAvailable (0)"
agent            | application 12/Apr/2018:15:43:13 +0000 - search.agent.base - None - [arxiv:null] - INFO: "Getting a new connection to Kinesis at https://localstack:4568 in region us-east-1, with SSL verification=False"
agent            | application 12/Apr/2018:15:43:13 +0000 - search.agent.base - None - [arxiv:null] - INFO: "Waiting for MetadataIsAvailable to be available"
agent            | application 12/Apr/2018:15:43:13 +0000 - search.agent.base - None - [arxiv:null] - ERROR: "Waiting for stream MetadataIsAvailable"
```

--------------------------------

### Stop Docker Compose Services

Source: https://github.com/arxiv/arxiv-search/blob/develop/README.md

Stops the Elasticsearch and Kibana containers that were started using docker-compose up.

```bash
docker-compose down
```

--------------------------------

### Run Main Test Suite with Coverage

Source: https://github.com/arxiv/arxiv-search/blob/develop/README.md

Executes the primary test suite and generates a coverage report.

```bash
pipenv run nose2 --with-coverage
```

--------------------------------

### Run Pydocstyle for Documentation Style

Source: https://github.com/arxiv/arxiv-search/blob/develop/README.md

Checks documentation style against NumPy conventions for the 'search' package, ignoring D401 errors.

```bash
pipenv run pydocstyle --convention=numpy --add-ignore=D401 search
```

--------------------------------

### Typical API Call Structure

Source: https://github.com/arxiv/arxiv-search/blob/develop/docs/source/classic_api.md

Illustrates the flow of a typical API call, from the client's request URL with parameters to the server's response.

```text
Request from url: http://export.arxiv.org/api/query  (1)
 with parameters: search_query=all:electron
                .
                .
                .
API server processes the request and sends the response
                .
                .
                .
Response received by client.  (2)
```

--------------------------------

### Basic Search Query with Parentheses

Source: https://github.com/arxiv/arxiv-search/blob/develop/search/domain/classic_api/README_classic_search_query.md

Demonstrates a basic search query using parentheses to group terms with the AND operator. This is useful for complex searches where specific term combinations are required.

```bash
curl -is https://export.arxiv.org/api/query?search_query=\(ti:a+AND+ti:b\) | gi '<title' | sort
```

--------------------------------

### Search with Title and Category

Source: https://github.com/arxiv/arxiv-search/blob/develop/search/domain/classic_api/README_classic_search_query.md

Demonstrates searching for papers with a specific title and category, including variations and limitations.

```shell
curl -si https://export.arxiv.org/api/query?search_query=ti:%22Periodicity+in+Cryptocurrency+Volatility+and+Liquidity%22+AND+cat:q-fin.PR | gi '<title|category|<id>'
    <id>... 2109.12142v2

# Also works: q-fin.TR
# Does not works if cs.SD
# Also works: cat:q-fin. ਵੀ

curl -si http://localhost:8080/api/query?search_query=ti:%22Periodicity+in+Cryptocurrency+Volatility+and+Liquidity%22+AND+cat:q-fin.PR | gi '<title|category|<id>'
# Also works: q-fin.TR
curl -si http://localhost:8080/api/query?search_query=ti:%22Periodicity+in+Cryptocurrency+Volatility+and+Liquidity%22+AND+cat:q-fin.PR | gi '<title|category|<id>'
# Does not works: q-fin. ਵੀ
```

--------------------------------

### Run Test Suite with Integration Tests

Source: https://github.com/arxiv/arxiv-search/blob/develop/README.md

Executes the test suite, including integration tests, and generates a coverage report.

```bash
WITH_INTEGRATION=1 pipenv run nose2 --with-coverage
```

--------------------------------

### Atom Feed Link Element

Source: https://github.com/arxiv/arxiv-search/blob/develop/docs/source/classic_api.md

The <link> element provides a URL to retrieve the feed again. The URL represents a canonicalized version of the query and can be used for GET requests.

```xml
<link xmlns="http://www.w3.org/2005/Atom" href="http://arxiv.org/api/query?search_query=all:electron&amp;id_list=&amp;start=0&amp;max_results=1" rel="self" type="application/atom+xml"/>
```

--------------------------------

### Indexing Agent Running Logs

Source: https://github.com/arxiv/arxiv-search/blob/develop/README.md

View the log output once the indexing agent and localstack are running. This indicates successful connection and readiness to process records.

```bash
agent            | application 12/Apr/2018:15:44:14 +0000 - search.agent.base - None - [arxiv:null] - ERROR: "Failed to get stream while waiting"
agent            | application 12/Apr/2018:15:44:14 +0000 - search.agent.base - None - [arxiv:null] - INFO: "Could not connect to stream; attempting to create"
agent            | application 12/Apr/2018:15:44:14 +0000 - search.agent.base - None - [arxiv:null] - INFO: "Created; waiting for MetadataIsAvailable again"
agent            | application 12/Apr/2018:15:44:14 +0000 - search.agent.base - None - [arxiv:null] - ERROR: "Waiting for stream MetadataIsAvailable"
localstack       | Ready.
agent            | application 12/Apr/2018:15:44:24 +0000 - search.agent.base - None - [arxiv:null] - INFO: "Ready to start"
agent            | application 12/Apr/2018:15:44:24 +0000 - search.agent.base - None - [arxiv:null] - INFO: "Starting processing from position 49583482132750299344823207796409748205413425533752967170 on stream MetadataIsAvailable and shard 0"
```

--------------------------------

### Search API Endpoint

Source: https://github.com/arxiv/arxiv-search/blob/develop/docs/source/classic_api.md

This endpoint allows users to search for articles on arXiv. Results are returned in Atom 1.0 format. It can be accessed via HTTP GET or POST.

```APIDOC
## GET /api/query

### Description
Allows searching for articles on arXiv using various query parameters. Results are returned in Atom 1.0 format.

### Method
GET, POST

### Endpoint
http://export.arxiv.org/api/query

### Parameters
#### Query Parameters
- **search_query** (string) - Required - The query string to search for articles. Can include terms like 'all:electron', 'all:electron+AND+all:proton'.

### Request Example
```json
{
  "example": "http://export.arxiv.org/api/query?search_query=all:electron"
}
```

### Response
#### Success Response (200)
- **response** (Atom 1.0 XML) - Contains the search results in Atom 1.0 format.

#### Response Example
```json
{
  "example": "[Atom 1.0 XML content]"
}
```
```

--------------------------------

### Submit Search Query via POST Request

Source: https://github.com/arxiv/arxiv-search/blob/develop/search/domain/classic_api/README_classic_search_query.md

Demonstrates how to submit a search query using an HTTP POST request. This method is suitable for longer or more complex queries that might exceed URL length limits.

```bash
curl -X POST http://localhost:8080/api/query -d "search_query=ti:pineappl"
```

--------------------------------

### Run Elasticsearch in Docker

Source: https://github.com/arxiv/arxiv-search/blob/develop/README.md

Builds a Docker image for Elasticsearch and runs it, mapping ports 9200 and 9300 to the host. This is necessary for the arXiv search service to connect to Elasticsearch.

```bash
docker build -t "arxiv/elasticsearch" -f ./Dockerfile-elasticsearch .
docker run -it \
    -e "http.host=0.0.0.0" \
    -e "transport.host=127.0.0.1" \
    -p 9200:9200 -p 9300:9300 arxiv/elasticsearch

echo $ES_BASE_URL; # http://127.0.0.1:9200
curl "${ES_BASE_URL}/_cat/health?v";
```

--------------------------------

### Extract Lucene implementation version

Source: https://github.com/arxiv/arxiv-search/blob/develop/search/domain/classic_api/README_classic_search_query.md

Extracts the MANIFEST.MF file from lucene.jar and prints the Implementation-Version.

```shell
cd arxiv-httpd/java/lucene_search/lib
jar xf lucene.jar META-INF/MANIFEST.MF
cat META-INF/MANIFEST.MF | grep Implementation-Version
    Implementation-Version: 2.3.2 652650 - buschmi - 2008-05-01 13:30:57
```

--------------------------------

### Run Pylint for Code Linting

Source: https://github.com/arxiv/arxiv-search/blob/develop/README.md

Executes Pylint on the 'search' package to assess code quality, aiming for a score of 9/10 or better.

```bash
pipenv run pylint search
```

--------------------------------

### Run Pre-commit Hooks Manually

Source: https://github.com/arxiv/arxiv-search/blob/develop/README.md

Manually executes all configured pre-commit hooks on the project's changed files.

```bash
pipenv run pre-commit run
```

--------------------------------

### Example Error Response for Malformed ID

Source: https://github.com/arxiv/arxiv-search/blob/develop/docs/source/classic_api.md

This snippet shows the XML Atom feed returned when an API call contains a malformed ID. The error details are found within the `<summary>` tag.

```default
<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom" xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/">
  <link xmlns="http://www.w3.org/2005/Atom" href="http://arxiv.org/api/query?search_query=&amp;id_list=1234.12345" rel="self" type="application/atom+xml"/>
  <title xmlns="http://www.w3.org/2005/Atom">ArXiv Query: search_query=&amp;id_list=1234.12345</title>
  <id xmlns="http://www.w3.org/2005/Atom">http://arxiv.org/api/kvuntZ8c9a4Eq5CF7KY03nMug+Q</id>
  <updated xmlns="http://www.w3.org/2005/Atom">2007-10-12T00:00:00-04:00</updated>
  <opensearch:totalResults xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/">1</opensearch:totalResults>
  <opensearch:startIndex xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/">0</opensearch:startIndex>

  <opensearch:itemsPerPage xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/">1</opensearch:itemsPerPage>
  <entry xmlns="http://www.w3.org/2005/Atom">
    <id xmlns="http://www.w3.org/2005/Atom">http://arxiv.org/api/errors#incorrect_id_format_for_1234.12345</id>
    <title xmlns="http://www.w3.org/2005/Atom">Error</title>
    <summary xmlns="http://www.w3.org/2005/Atom">incorrect id format for 1234.12345</summary>
    <updated xmlns="http://www.w3.org/2005/Atom">2007-10-12T00:00:00-04:00</updated>

    <link xmlns="http://www.w3.org/2005/Atom" href="http://arxiv.org/api/errors#incorrect_id_format_for_1234.12345" rel="alternate" type="text/html"/>
    <author xmlns="http://www.w3.org/2005/Atom">
      <name xmlns="http://www.w3.org/2005/Atom">arXiv api core</name>
    </author>
  </entry>
</feed>
```

--------------------------------

### Populate Index with Specific Paper ID

Source: https://github.com/arxiv/arxiv-search/blob/develop/README.md

Populates the search index with papers defined in tests/data/sample.json. This script can take several minutes to run. Optionally, specify a single paper ID using the --paper_id parameter.

```bash
FLASK_APP=app.py FLASK_DEBUG=1 ELASTICSEARCH_SERVICE_HOST=127.0.0.1 pipenv run python bulk_index.py --paper_id <paper_id>
```

--------------------------------

### Search by Category and Submission Date

Source: https://github.com/arxiv/arxiv-search/blob/develop/search/domain/classic_api/README_classic_search_query.md

Shows how to filter search results by category and a specific submission date range.

```shell
curl -si http://localhost:8080/api/query?search_query=cat:q-fin.%2A+AND+submittedDate:%22201602070000+TO+201602072359%22 | gi '<title|category|<id>'
```

--------------------------------

### Search for multi-last names with spaces and underscores

Source: https://github.com/arxiv/arxiv-search/blob/develop/search/domain/classic_api/README_classic_search_query.md

Demonstrates various ways to search for authors with multi-part last names like 'del Maestro', using spaces, underscores, and quotes.

```bash
curl -si https://export.arxiv.org/api/query?search_query=au:del+maestro | egrep '<name|<title'
```

```bash
curl -si https://export.arxiv.org/api/query?search_query=au:%22del+maestro%22 | egrep '<name|<title'
```

```bash
curl -si https://export.arxiv.org/api/query?search_query=au:del_maestro | egrep '<name|<title'
```

```bash
curl -si https://export.arxiv.org/api/query?search_query=au:%22del_maestro%22 | egrep '<name|<title'
```

```bash
curl -si http://localhost:8080/api/query?search_query=au:%22del_maestro%22 | egrep '<name|<title'
```

--------------------------------

### Check Lucene version in build file

Source: https://github.com/arxiv/arxiv-search/blob/develop/search/domain/classic_api/README_classic_search_query.md

Navigates to the lucene_search directory and checks the build.xml file for excluded Lucene JAR versions.

```shell
cd arxiv-httpd/java/lucene_search
grep lucene build.xml
    <exclude name="lucene-1.2.jar"/>
```

--------------------------------

### Title Search with Explicit OR

Source: https://github.com/arxiv/arxiv-search/blob/develop/search/domain/classic_api/README_classic_search_query.md

Demonstrates searching within titles for one of two terms using the explicit OR operator. Note the URL encoding for the OR operator.

```bash
curl -is https://export.arxiv.org/api/query?search_query=ti:("hawaii+OR+pineapple") | gi '<title' | sort
```

--------------------------------

### Display Search Results

Source: https://github.com/arxiv/arxiv-search/blob/develop/search/templates/search/search-macros.html

Renders a list of search results, including pagination, paper IDs, classifications, authors, titles, abstracts, submission dates, and comments. Handles highlighting for search terms within fields. Use this macro to display the output of a search query.

```Jinja2
{% macro search_results(form, results, metadata, url_for_page, url_for_author_search, is_current, hide_abstracts) %}

{% if metadata.total_pages > 1 %} {{ pagination(metadata, url_for_page) }} {% endif %}

{# Start index is 0-based. #} {% for result in results %}NaN.  {%- set display_paper_id = result.paper_id if is_current(result) else result.paper_id_v -%} [arXiv:{{ display_paper_id }}]({{ url_for('abs_by_id', paper_id=display_paper_id) }})  {{ paper_format_links(result, display_paper_id) }} 
      
      {{ result.primary_classification.category.id }} {% if result.secondary_classification %} {% for secondary in result.secondary_classification %} {{ secondary.category.id }} {% endfor %} {% endif -%}
      
      {% if result.doi %} {% for doi in result.doi %}
      
      doi [{{ doi }}](https://doi.org/{{ doi }})
      
      {% endfor %} {% endif %}
      
      {# Note: mathjax class should be applied to any element that requires MathJax processing #}
      
      {% if result.highlight.title %} {{ result.highlight.title | safe }} {% else %} {{ result.title }} {% endif %}
      
      Authors: {% for author in result.authors[0:25] %} {% if author %}[{{ author.first_name }} {{ author.last_name }}{% if author.suffix %} {{ author.suffix }}{%- endif -%}]({{ url_for_author_search(author.first_name, author.last_name) }}){{ ", " if not loop.last }}{% endif %} {% endfor -%} {% if result.authors | length > 25 %}, et al. ({{ result.authors | length - 25 }} additional authors not shown){% endif %}
      
      {% if not hide_abstracts %}
      
      Abstract: {{ result.preview.abstract | safe }} {%if result.truncated.abstract %}▽ More{% endif %} {% if result.highlight.abstract %}{{ result.highlight.abstract | safe }}{% else %}{{ result.abstract }}{% endif %} △ Less
      
      {% endif %}
      
      Submitted {{ result.submitted_date.strftime('%-d %B, %Y') }}; {% if result.version > 1 %}v1 submitted {{ result.submitted_date_first.strftime('%-d %B, %Y') }};{% endif %} originally announced {{ result.announced_date_first.strftime('%B %Y') }}. {% if not is_current(result) %}Latest version: [{{ result.latest }}]({{ url_for('abs_by_id', paper_id=result.latest) }}).{% endif %}
      
      {% if result.comments %}
      
      Comments: {% if result.highlight.comments %}{{ result.highlight.comments | safe }}{% else %}{{ result.comments }}{% endif %}
      
      {% endif %} {% if result.report_num or result.acm_class or result.msc_class %}
      
      {% if result.report_num %} Report number: {% if result.highlight.report_num %}{{ result.highlight.report_num | safe }}{% else %}{{ result.report_num }}{% endif %} {% endif %} {% if result.msc_class %} MSC Class: {% if result.highlight.msc_class %}{{ result.highlight.msc_class | safe }} ({{ result.msc_class }}){% else %}{{ result.msc_class }}{% endif %} {% endif %} {% if result.acm_class %} ACM Class: {% if result.highlight.acm_class %}{{ result.highlight.acm_class | safe }} ({{ result.acm_class }}){% else %}{{ result.acm_class }}{% endif %} {% endif %}
      
      {% endif %} {% if result.journal_ref %}
      

```

--------------------------------

### Search by DOI with wildcard

Source: https://github.com/arxiv/arxiv-search/blob/develop/search/domain/classic_api/README_classic_search_query.md

Shows how to use an asterisk (*) as a wildcard in a DOI search query. Quotes are not required for the DOI when using a wildcard.

```shell
curl https://export.arxiv.org/api/query?search_query=doi:10.1038/418838a*
```