### UrlStore Default Configuration
Source: https://github.com/adbar/courlan/blob/master/_autodocs/configuration.md
Minimal setup for UrlStore using default configuration. Use this for basic URL storage and addition.
```python
from courlan import UrlStore
# Minimal setup
store = UrlStore()
store.add_urls(['https://example.com/page1', 'https://example.com/page2'])
```
--------------------------------
### Install Development Dependencies
Source: https://github.com/adbar/courlan/blob/master/CONTRIBUTING.md
Install Courlan with its development dependencies from a local checkout. This command is necessary to set up the project for local development and testing.
```bash
pip install -e ".[dev]"
```
--------------------------------
### Network Configuration Example
Source: https://github.com/adbar/courlan/blob/master/_autodocs/DOCUMENTATION-MAP.md
Illustrates setting network configuration options such as the HTTP connection pool size, retry strategy, and acceptable HTTP status codes.
```python
from courlan.meta import HTTP_POOL, RETRY_STRATEGY, ACCEPTABLE_CODES
# Configure network settings
HTTP_POOL = 10
RETRY_STRATEGY = {"tries": 3, "delay": 1, "backoff": 2}
ACCEPTABLE_CODES = [200, 201, 301, 302]
```
--------------------------------
### Install coURLan using pip
Source: https://github.com/adbar/courlan/blob/master/README.md
Install the latest stable version of coURLan from PyPI. You can also upgrade to the latest version or install directly from the GitHub repository.
```bash
pip install courlan
```
```bash
pip install --upgrade courlan
```
```bash
pip install git+https://github.com/adbar/courlan.git
```
--------------------------------
### URL Storage and Crawling Example
Source: https://github.com/adbar/courlan/blob/master/_autodocs/DOCUMENTATION-MAP.md
Shows a basic workflow for using UrlStore to manage URLs for crawling. This is a core part of the URL Storage & Crawling use case.
```python
from courlan.urlstore import UrlStore
# Initialize UrlStore
store = UrlStore()
# Add URLs to the store
store.add_urls(["http://example.com", "http://example.org"])
# Get URLs to download
urls_to_download = store.get_download_urls(10)
# Process downloaded URLs (example)
for url in urls_to_download:
# ... download content ...
store.add_from_html(url, "Link")
```
--------------------------------
### URL Manipulation Example
Source: https://github.com/adbar/courlan/blob/master/_autodocs/DOCUMENTATION-MAP.md
Demonstrates using URL utility functions like extract_domain. This is part of the URL Manipulation use case.
```python
from courlan.urlutils import extract_domain
# Extract the domain from a URL
domain = extract_domain("http://www.example.com/path")
```
--------------------------------
### URL Validation Example
Source: https://github.com/adbar/courlan/blob/master/_autodocs/DOCUMENTATION-MAP.md
Demonstrates using the check_url function for validating URLs. This is part of the URL Validation use case.
```python
from courlan.core import check_url
# Check if a URL is valid
is_valid = check_url("http://example.com")
```
--------------------------------
### UrlStore Combined Configuration
Source: https://github.com/adbar/courlan/blob/master/_autodocs/configuration.md
Example of combining multiple configuration options for UrlStore, such as compressed storage, language filtering, strict mode, trailing slash normalization, and verbose output.
```python
# For German language crawling with strict filtering
store = UrlStore(
compressed=True, # Reduce memory usage
language='de', # Only German content
strict=True, # Filter major platforms
trailing_slash=False, # Normalize URL format
verbose=True # Graceful interruption
)
```
--------------------------------
### Example Usage of State Enum
Source: https://github.com/adbar/courlan/blob/master/_autodocs/types.md
Demonstrates how to use the State enum to check the visit status of a domain. This is useful for conditional logic within the UrlStore.
```python
from courlan.urlstore import State
domain_state = State.OPEN
if domain_state == State.OPEN:
print("Domain has unvisited URLs")
```
--------------------------------
### Get all known domains
Source: https://github.com/adbar/courlan/blob/master/_autodocs/api-reference/urlstore.md
Retrieves a list of all domain URLs that the UrlStore is currently aware of.
```python
def get_known_domains(self) -> list[str]:
```
--------------------------------
### Language Filtering Example
Source: https://github.com/adbar/courlan/blob/master/_autodocs/DOCUMENTATION-MAP.md
Shows how to use language filtering functions like lang_filter. This is part of the Language & Filtering use case.
```python
from courlan.filters import lang_filter
# Filter content based on language (example)
content = "This is English text."
if lang_filter(content, "en"):
print("Content is in English.")
```
--------------------------------
### Example of UrlStore Interaction with UrlPathTuple
Source: https://github.com/adbar/courlan/blob/master/_autodocs/types.md
Illustrates how to retrieve known URLs for a domain using UrlStore methods, which internally utilize UrlPathTuple objects. Direct manipulation of UrlPathTuple is typically for internal use.
```python
# Typically used internally by UrlStore
# To get full URLs, use UrlStore methods:
store.find_known_urls('https://example.com')
# Returns: ['https://example.com/page1', 'https://example.com/page2', ...]
```
--------------------------------
### Common Log Messages from Network Module
Source: https://github.com/adbar/courlan/blob/master/_autodocs/api-reference/meta.md
Examples of log messages from the network module, indicating network reachability issues and successful results.
```text
WARNING: cannot reach URL:
DEBUG: result found:
```
--------------------------------
### Common Log Messages from UrlStore
Source: https://github.com/adbar/courlan/blob/master/_autodocs/api-reference/meta.md
Examples of log messages from UrlStore, including invalid URLs, wrong languages, and discarded URLs.
```text
DEBUG: Invalid URL:
DEBUG: Wrong language:
WARNING: Discarding URL:
WARNING: discarded (size): urls:
DEBUG: %s objects in GC after UrlStore.discard
DEBUG: UrlStore reset, %s objects in GC
```
--------------------------------
### Link Extraction Example
Source: https://github.com/adbar/courlan/blob/master/_autodocs/DOCUMENTATION-MAP.md
Illustrates using the extract_links function to extract hyperlinks from a given URL. This is part of the Link Extraction use case.
```python
from courlan.core import extract_links
# Extract links from a URL
links = extract_links("http://example.com")
```
--------------------------------
### Configure Logging Verbosity
Source: https://github.com/adbar/courlan/blob/master/_autodocs/configuration.md
Set up Python's logging module to control the verbosity of Courlan's output. This example enables all debug messages.
```python
import logging
# Enable all debug messages
logging.basicConfig(
level=logging.DEBUG,
format='%(name)s - %(levelname)s - %(message)s'
)
from courlan import check_url
check_url('https://example.com') # Now shows debug logs
```
--------------------------------
### Get downloadable URLs with time limits
Source: https://github.com/adbar/courlan/blob/master/_autodocs/api-reference/urlstore.md
Fetches a list of URLs that are ready for immediate download. This method respects per-domain time limits to ensure polite crawling. It can return up to a specified maximum number of URLs.
```python
def get_download_urls(
self,
time_limit: float = 10.0,
max_urls: int = 10000,
) -> list[str]:
```
```python
# Get up to 100 URLs, respecting 10-second delay per domain
urls = store.get_download_urls(time_limit=10, max_urls=100)
```
--------------------------------
### Get Host and Path from URL
Source: https://github.com/adbar/courlan/blob/master/_autodocs/api-reference/url_utilities.md
Decomposes a URL into its host (scheme + netloc) and path (including query and fragment). Useful for organizing URLs by domain while preserving the full path.
```python
from courlan import get_host_and_path
get_host_and_path('https://www.un.org/en/about-us')
# ('https://www.un.org', '/en/about-us')
get_host_and_path('https://example.com:8080/path?id=1&lang=en#section')
# ('https://example.com:8080', '/path?id=1&lang=en#section')
get_host_and_path('https://example.com/')
# ('https://example.com', '/')
# Used internally by UrlStore for domain-based organization
host, path = get_host_and_path('https://blog.example.com/article/page1')
# host = 'https://blog.example.com'
# path = '/article/page1'
```
--------------------------------
### Example of SplitResult Usage
Source: https://github.com/adbar/courlan/blob/master/_autodocs/types.md
Shows the structure of a SplitResult object, which is returned by urllib.parse.urlsplit() and used for URL normalization. It contains components like scheme, netloc, path, query, and fragment.
```python
from urllib.parse import SplitResult
# A named tuple-like object with fields:
SplitResult(
scheme='https',
netloc='www.example.com',
path='/page',
query='id=1',
fragment='section'
)
```
--------------------------------
### Get Host Info (Domain and Base URL) from URL
Source: https://github.com/adbar/courlan/blob/master/_autodocs/api-reference/url_utilities.md
Combines domain extraction and base URL retrieval into a single function. Returns the extracted domain name and the full base URL (protocol + domain).
```python
from courlan import get_hostinfo
domain, base_url = get_hostinfo('https://www.un.org/en/about-us')
# domain = 'un.org'
# base_url = 'https://www.un.org'
domain, base_url = get_hostinfo('https://example.com')
# domain = 'example.com'
# base_url = 'https://example.com'
```
--------------------------------
### Initialize UrlStore
Source: https://github.com/adbar/courlan/blob/master/_autodocs/api-reference/urlstore.md
Instantiate the UrlStore class. Options include enabling compression, setting a language filter, enforcing strict URL filtering, managing trailing slashes, and enabling verbose mode for signal handlers.
```python
# Basic usage
store = UrlStore()
```
```python
# With compression for large URL collections
store = UrlStore(compressed=True)
```
```python
# With language filtering
store = UrlStore(language='en', strict=True)
```
```python
# With verbose mode (Unix/Linux only)
store = UrlStore(verbose=True)
```
--------------------------------
### Initialize and add URLs to UrlStore
Source: https://github.com/adbar/courlan/blob/master/_autodocs/api-reference/urlstore.md
Demonstrates the basic usage pattern for initializing an UrlStore with specific options and adding initial URLs to begin a crawling process.
```python
from courlan import UrlStore
import time
# Initialize
store = UrlStore(compressed=True, language='en', verbose=True)
# Add initial URLs
store.add_urls(['https://example.com', 'https://other.org'])
```
--------------------------------
### Sample URLs with Options
Source: https://github.com/adbar/courlan/blob/master/_autodocs/configuration.md
Use sample_urls to select a subset of URLs. Configure sample size and options to exclude domains based on URL count.
```python
from courlan import sample_urls
sample = sample_urls(
input_urls,
samplesize,
exclude_min = None,
exclude_max = None,
strict = False,
verbose = False,
)
```
--------------------------------
### Get Logger for a Specific Module
Source: https://github.com/adbar/courlan/blob/master/_autodocs/api-reference/meta.md
Obtain a logger instance for a particular Courlan module or the root logger.
```python
logger = logging.getLogger('courlan.core')
logger = logging.getLogger('courlan.urlstore')
logger = logging.getLogger('courlan') # Root logger
```
--------------------------------
### Configure Basic Logging
Source: https://github.com/adbar/courlan/blob/master/_autodocs/api-reference/meta.md
Set up basic logging to display all debug messages with a simple format.
```python
import logging
# Show all debug messages
logging.basicConfig(
level=logging.DEBUG,
format='%(name)s - %(levelname)s - %(message)s'
)
```
--------------------------------
### Get download counts per domain
Source: https://github.com/adbar/courlan/blob/master/_autodocs/api-reference/urlstore.md
Retrieves a list containing the number of URLs downloaded from each known domain.
```python
def get_all_counts(self) -> list[int]:
```
--------------------------------
### Get domains with unvisited URLs
Source: https://github.com/adbar/courlan/blob/master/_autodocs/api-reference/urlstore.md
Returns a list of domain URLs that still contain one or more unvisited URLs.
```python
def get_unvisited_domains(self) -> list[str]:
```
--------------------------------
### UrlStore Constructor
Source: https://github.com/adbar/courlan/blob/master/_autodocs/api-reference/urlstore.md
Initializes a new UrlStore instance. Options include enabling compression, setting a language for filtering, enforcing strict URL filtering, preserving trailing slashes, and enabling verbose mode for signal handlers.
```APIDOC
## UrlStore()
### Description
Initializes a new UrlStore instance with configurable options for compression, language filtering, strictness, trailing slash handling, and verbose mode.
### Parameters
- **compressed** (bool) - Optional - Default: `False` - Enable compression (bz2 or zlib) for stored URLs to reduce memory usage.
- **language** (str | None) - Optional - Default: `None` - ISO 639-1 language code for language-aware URL filtering when adding URLs.
- **strict** (bool) - Optional - Default: `False` - Apply strict URL filtering when adding URLs.
- **trailing_slash** (bool) - Optional - Default: `True` - Preserve trailing slashes when storing URLs.
- **verbose** (bool) - Optional - Default: `False` - Enable signal handlers (SIGINT, SIGTERM) to dump unvisited URLs on interruption (Unix only).
### Examples
```python
# Basic usage
store = UrlStore()
# With compression for large URL collections
store = UrlStore(compressed=True)
# With language filtering
store = UrlStore(language='en', strict=True)
# With verbose mode (Unix/Linux only)
store = UrlStore(verbose=True)
```
```
--------------------------------
### UrlStore Constructor Configuration
Source: https://github.com/adbar/courlan/blob/master/_autodocs/API-INDEX.md
Configure UrlStore with options for compression, language filtering, strictness, trailing slashes, and verbosity.
```python
UrlStore(
compressed=False, # Compress stored URLs
language=None, # ISO 639-1 language code
strict=False, # Strict filtering
trailing_slash=True, # Preserve trailing slashes
verbose=False # Signal handlers for graceful exit
)
```
--------------------------------
### Common Log Messages from check_url()
Source: https://github.com/adbar/courlan/blob/master/_autodocs/api-reference/meta.md
Examples of DEBUG messages generated by the check_url() function, indicating URL rejection reasons.
```text
DEBUG: rejected, basic filter:
DEBUG: rejected, type filter:
DEBUG: rejected, lang filter:
DEBUG: rejected, validation test:
DEBUG: rejected, extension filter:
DEBUG: rejected, domain name:
DEBUG: rejected, path filter:
DEBUG: discarded URL:
```
--------------------------------
### Get total number of stored URLs
Source: https://github.com/adbar/courlan/blob/master/_autodocs/api-reference/urlstore.md
Returns the aggregate count of all URLs currently stored within the UrlStore, across all domains.
```python
def total_url_number(self) -> int:
```
--------------------------------
### Initialize UrlStore for Web Crawling
Source: https://github.com/adbar/courlan/blob/master/_autodocs/configuration.md
Configure UrlStore for efficient web crawling of news sites. Use compression for large collections, filter by language, enforce strictness to avoid spam, and normalize URLs.
```python
from courlan import UrlStore
store = UrlStore(
compressed=True, # Handle large collections
language='en', # Focus on English content
strict=True, # Filter spam and ads
trailing_slash=False # Normalize URLs
)
```
--------------------------------
### News Crawler Workflow
Source: https://github.com/adbar/courlan/blob/master/_autodocs/API-INDEX.md
Set up a news crawler using UrlStore with strict filtering and compression. It adds seed URLs, establishes a download schedule, fetches content, and adds extracted links back to the store.
```python
from courlan import UrlStore
import time
import requests
# Strict filtering for news quality
store = UrlStore(language='en', strict=True, compressed=True)
# Add seed URLs
# Assuming news_domains is defined elsewhere
store.add_urls(news_domains)
# Crawl loop with scheduling
while store.unvisited_websites_number() > 0:
schedule = store.establish_download_schedule(max_urls=20)
for delay, url in schedule:
time.sleep(delay)
response = requests.get(url)
store.add_from_html(response.text, url)
```
--------------------------------
### Loading a Persisted UrlStore
Source: https://github.com/adbar/courlan/blob/master/_autodocs/DOCUMENTATION-MAP.md
Demonstrates how to load a previously persisted UrlStore object from a file.
```python
from courlan.urlstore import UrlStore
# Load a persisted UrlStore
store = UrlStore.load_store("path/to/your/store.json")
```
--------------------------------
### write
Source: https://github.com/adbar/courlan/blob/master/_autodocs/api-reference/urlstore.md
Saves the current state of the UrlStore to disk as a pickled object.
```APIDOC
## write()
### Description
Saves the UrlStore to disk as a pickled object.
### Method
```python
def write(self, filename: str) -> None
```
### Parameters
- **filename** (str) - Required - The name of the file to save the UrlStore to
### Example
```python
store.write('my_urls.pkl')
```
```
--------------------------------
### Logging Module Configuration
Source: https://github.com/adbar/courlan/blob/master/_autodocs/DOCUMENTATION-MAP.md
Shows how to configure the logging module for the library, including setting the logging level.
```python
import logging
# Configure logging level
logging.basicConfig(level=logging.INFO)
```
--------------------------------
### Basic URL Format Filter
Source: https://github.com/adbar/courlan/blob/master/_autodocs/api-reference/filters.md
Checks if a URL starts with 'http' and falls within a length range of 10 to 500 characters.
```python
def basic_filter(url: str) -> bool:
# Checks basic URL format: starts with `http` and is 10-500 characters long.
```
--------------------------------
### Load UrlStore State from File
Source: https://github.com/adbar/courlan/blob/master/_autodocs/configuration.md
Loads a previously saved UrlStore state from a pickle file. The loaded store will resume with the same configuration as when it was saved.
```python
from courlan import load_store
store = load_store('urls.pkl')
# Resumes with same configuration as when saved
```
--------------------------------
### Basic Courlan Command-Line Usage
Source: https://github.com/adbar/courlan/blob/master/README.md
Demonstrates basic command-line usage for processing a list of URLs from an input file to an output file.
```bash
$ courlan --inputfile url-list.txt --outputfile cleaned-urls.txt
```
--------------------------------
### is_navigation_page()
Source: https://github.com/adbar/courlan/blob/master/_autodocs/api-reference/filters.md
Detects if a URL points to a navigation or overview page rather than a content page. Examples include pagination, category listings, and archives.
```APIDOC
## is_navigation_page()
### Description
Detects if a URL points to a navigation or overview page rather than a content page. Examples include pagination (`/page/1/`), category listings (`/category/tech/`), archives (`/2024/`), etc.
### Parameters
- **url** (str) - Required - URL to check.
### Return Value
`True` if the URL matches navigation page patterns, `False` otherwise.
### Navigation Patterns Detected
- Archives: `/archives/`
- Authors: `/author/`, `/auth/`
- Categories/Tags: `/category/`, `/cat/`, `/tag/`, `/tags/`, `/topic/`, `/topics/`, `/schlagwort/`, `/kategorie/`
- Pagination: `/page/`, `/paged/`, `/seite/`, `/paged`
- Query-based pagination: `?p=123` (numeric)
### Examples
```python
from courlan import is_navigation_page
is_navigation_page('https://www.randomblog.net/category/myposts')
# True
is_navigation_page('https://example.com/page/2/')
# True
is_navigation_page('https://example.com/article/my-post')
# False
is_navigation_page('https://blog.com/2024/01/article')
# False (date-based URL structure is content)
is_navigation_page('https://example.com/archives')
# True
```
```
--------------------------------
### Crawl with UrlStore
Source: https://github.com/adbar/courlan/blob/master/_autodocs/API-INDEX.md
Initialize UrlStore, add seed URLs, and then iteratively fetch URLs, extract links from their HTML content, and add them back to the store until no unvisited URLs remain.
```python
from courlan import UrlStore
store = UrlStore(language='en', strict=True)
store.add_urls(['https://example.com'])
while store.unvisited_websites_number() > 0:
url = store.get_url('https://example.com')
if url:
html = fetch(url) # Assuming fetch is defined elsewhere
store.add_from_html(html, url)
```
--------------------------------
### Extract Links with Options
Source: https://github.com/adbar/courlan/blob/master/_autodocs/configuration.md
Use extract_links to get links from page content. Configure options like external link filtering, language, and strictness.
```python
from courlan import extract_links
links = extract_links(
pagecontent,
url,
external_bool = False,
no_filter = False,
language = None,
strict = True,
trailing_slash = True,
with_nav = False,
redirects = False,
reference = None,
)
```
--------------------------------
### Include Navigation Pages
Source: https://github.com/adbar/courlan/blob/master/_autodocs/api-reference/check_url.md
Demonstrates how to include navigation and overview pages, such as pagination, by setting with_nav=True. By default, these pages are filtered out.
```python
# Include pagination pages
check_url('http://www.example.org/page/10/', with_nav=True)
# Returns: ('http://www.example.org/page/10', 'example.org')
# Default: exclude pagination
check_url('http://www.example.org/page/10/')
# Returns: None
```
--------------------------------
### sample_urls()
Source: https://github.com/adbar/courlan/blob/master/_autodocs/API-INDEX.md
Samples URLs based on their associated domains.
```APIDOC
## sample_urls()
### Description
Samples URLs by domain. This function selects a subset of URLs, organized or filtered by their respective domains.
### Method
Not applicable (function call)
### Parameters
None explicitly defined in this reference.
### Returns
- `List of URLs`: A list containing sampled URLs.
```
--------------------------------
### Import UrlStore and load_store
Source: https://github.com/adbar/courlan/blob/master/_autodocs/api-reference/urlstore.md
Import the necessary classes from the courlan library. This is typically the first step before using UrlStore.
```python
from courlan import UrlStore, load_store
```
--------------------------------
### Get crawl delay from robots.txt
Source: https://github.com/adbar/courlan/blob/master/_autodocs/api-reference/urlstore.md
Retrieves the crawl delay specified in a website's robots.txt file. If no delay is specified or rules are not found, a default value is returned.
```python
def get_crawl_delay(self, website: str, default: float = 5) -> float:
```
```python
delay = store.get_crawl_delay('https://example.com', default=5)
# Returns delay from robots.txt or 5 seconds
```
--------------------------------
### Implement Checkpointing for Backup and Recovery
Source: https://github.com/adbar/courlan/blob/master/_autodocs/configuration.md
Implement a function to create regular checkpoints of the UrlStore state. This is crucial for backup and recovery in long-running crawling processes.
```python
# Regular checkpoints
import shutil
import time
def checkpoint(store, interval=3600):
last_save = time.time()
while True:
if time.time() - last_save > interval:
store.write('urls_backup.pkl')
last_save = time.time()
# ... crawling ...
```
--------------------------------
### Using RobotFileParser with UrlStore
Source: https://github.com/adbar/courlan/blob/master/_autodocs/types.md
Demonstrates how to parse robots.txt rules using the standard library's RobotFileParser and store them within the UrlStore. This allows the UrlStore to respect crawling restrictions.
```python
from urllib.robotparser import RobotFileParser
rules = RobotFileParser()
rules.set_url('https://example.com/robots.txt')
rules.read()
store.store_rules('https://example.com', rules)
# Later, check if URL is allowed
if rules.can_fetch('*', 'https://example.com/page'):
# Can fetch this URL
pass
```
--------------------------------
### URL Utility Functions
Source: https://github.com/adbar/courlan/blob/master/README.md
Provides various utility functions for URL manipulation, including getting base URLs, host and path, host information, and fixing relative URLs.
```python
from courlan import *
url = 'https://www.un.org/en/about-us'
get_base_url(url)
```
```python
get_host_and_path(url)
```
```python
get_hostinfo(url)
```
```python
fix_relative_urls('https://www.un.org', 'en/about-us')
```
--------------------------------
### Public API Exports from __init__.py
Source: https://github.com/adbar/courlan/blob/master/_autodocs/MODULE-STRUCTURE.md
Imports core functions like check_url and extract_links from the courlan.core module, and sample_urls from courlan.sampling.
```python
from courlan.core import (
check_url,
extract_links,
filter_links,
)
from courlan.sampling import sample_urls
```
--------------------------------
### Focused Crawling with Robots.txt
Source: https://github.com/adbar/courlan/blob/master/_autodocs/API-INDEX.md
Demonstrates how to perform focused crawling by integrating robots.txt rules for filtering links. This approach allows for controlled crawling based on website policies.
```python
from courlan import check_url, extract_links, filter_links
import robots
# Manual control for special cases
robots_parser = robots.RobotFileParser()
robots_parser.set_url('https://example.com/robots.txt')
robots_parser.read()
html = requests.get(url).text
links, nav = filter_links(
html, url,
lang='en',
rules=robots_parser,
strict=True
)
# Process nav links first
for link in nav:
# ... process navigation ...
```
--------------------------------
### Get Base URL from URL String or Parsed Object
Source: https://github.com/adbar/courlan/blob/master/_autodocs/api-reference/url_utilities.md
Strips a URL to its base (scheme + netloc), removing path, query, and fragments. Accepts both string URLs and urllib.parse.SplitResult objects.
```python
from courlan import get_base_url
get_base_url('https://www.un.org/en/about-us')
# 'https://www.un.org'
get_base_url('https://example.com:8080/path?query=1#frag')
# 'https://example.com:8080'
# With parsed URL
from urllib.parse import urlsplit
parsed = urlsplit('https://example.com/path')
get_base_url(parsed)
# 'https://example.com'
```
--------------------------------
### Get an unvisited URL from a domain
Source: https://github.com/adbar/courlan/blob/master/_autodocs/api-reference/urlstore.md
Retrieves a single URL from the specified domain. Optionally marks the URL as visited immediately upon retrieval. Returns None if no unvisited URLs are available for the domain.
```python
def get_url(self, domain: str, as_visited: bool = True) -> str | None:
```
```python
url = store.get_url('https://example.com')
# Retrieves first unvisited URL and marks it visited
# Returns None if all URLs visited
```
--------------------------------
### Establish a download schedule with backoff
Source: https://github.com/adbar/courlan/blob/master/_autodocs/api-reference/urlstore.md
Generates a list of URLs with associated delays, suitable for implementing a polite crawling schedule. The output is sorted by URL, and each tuple contains the delay in seconds and the URL itself.
```python
def establish_download_schedule(
self,
max_urls: int = 100,
time_limit: int = 10,
) -> list[str]:
```
```python
schedule = store.establish_download_schedule(max_urls=10, time_limit=5)
# [(0.0, 'https://example.com/1'),
# (5.0, 'https://example.com/2'),
# (0.0, 'https://other.org/page'),
# ...]
for delay, url in schedule:
sleep(delay)
download(url)
```
--------------------------------
### Robots.txt Parsing and Storage
Source: https://github.com/adbar/courlan/blob/master/_autodocs/MODULE-STRUCTURE.md
Demonstrates how to use `urllib.robotparser.RobotFileParser` to fetch and parse a website's `robots.txt` file, and then store these rules within a Courlan `UrlStore`.
```python
from urllib.robotparser import RobotFileParser
from courlan import UrlStore
rules = RobotFileParser()
rules.set_url(url + '/robots.txt')
rules.read()
store = UrlStore()
store.store_rules(url, rules)
```
--------------------------------
### check_url Returns None for Invalid URLs
Source: https://github.com/adbar/courlan/blob/master/_autodocs/errors.md
Provides examples of check_url returning None for various invalid URL formats, including invalid IP addresses, too short URLs, and incorrect schemes.
```python
from courlan import check_url
# Invalid URLs return None
print(check_url('http://666.0.0.1/')) # None (invalid IP)
print(check_url('http://invalid')) # None (too short)
print(check_url('ftp://example.com')) # None (wrong scheme)
```
--------------------------------
### Sample URLs by Domain
Source: https://github.com/adbar/courlan/blob/master/_autodocs/README.md
Uses `sample_urls` to select a specified number of URLs per domain from a larger list, useful for targeted crawling.
```python
from courlan import sample_urls
urls = ['https://example.com/' + str(i) for i in range(100)]
sample = sample_urls(urls, 10) # 10 URLs per domain
```
--------------------------------
### UrlStore Full Constructor Signature
Source: https://github.com/adbar/courlan/blob/master/_autodocs/configuration.md
This shows the complete signature for the UrlStore constructor, including all available parameters and their default values.
```python
from courlan import UrlStore
store = UrlStore(
compressed: bool = False,
language: str | None = None,
strict: bool = False,
trailing_slash: bool = True,
verbose: bool = False,
)
```
--------------------------------
### redirection_test(url: str)
Source: https://github.com/adbar/courlan/blob/master/_autodocs/api-reference/meta.md
Performs an HTTP HEAD request to a URL and follows redirects to get the final canonical URL. This function is used internally by `check_url(with_redirects=True)` and is crucial for resolving shortened URLs or complex redirect chains.
```APIDOC
## redirection_test(url: str)
### Description
Performs an HTTP HEAD request to a URL and follows redirects to get the final canonical URL. Used internally by `check_url(with_redirects=True)`.
### Parameters
#### Path Parameters
- **url** (str) - Required - URL to test for redirects
### Return Value
The final URL after following redirects (may differ from input URL).
### Raises
- **ValueError**: If the URL cannot be reached or returns a non-acceptable HTTP status
### Acceptable Status Codes
The function accepts these HTTP status codes:
- 200 (OK)
- 300, 301, 302, 303, 304, 305, 306, 307, 308 (Redirect codes)
Any other status (4xx, 5xx, etc.) raises `ValueError`.
### Network Configuration
- **HTTP Pool**: urllib3 PoolManager with:
- 100 concurrent connection pools
- 10-second timeout per request
- Automatic retry strategy (2 retries for specific status codes)
- SSL warnings disabled
- **Retry Strategy**: Retries on status codes: 429, 499, 500, 502, 503, 504, 509, 520-527, 530, 598
- Max 2 retries with exponential backoff
- Backoff factor: 1 second
### Examples
```python
from courlan.network import redirection_test
# URL with redirect
final_url = redirection_test('https://example.com/redirect')
# May return different URL if there were redirects
# Shortener expansion
final_url = redirection_test('https://bit.ly/example')
# Returns the full expanded URL
# Failed request raises ValueError
try:
final_url = redirection_test('https://unreachable.example.org')
except ValueError as e:
print(f"Cannot reach URL: {e}")
```
### Internal Usage
Called automatically by `check_url()` when `with_redirects=True`:
```python
from courlan import check_url
url, domain = check_url(
'https://httpbin.org/redirect-to?url=https://example.com',
with_redirects=True
)
# Uses redirection_test() internally
```
### Source
`courlan/network.py`: lines 46-71
```
--------------------------------
### Check for Blacklisted Domains in Strict Mode
Source: https://github.com/adbar/courlan/blob/master/_autodocs/configuration.md
Demonstrates how to check if a domain is in the BLACKLIST and if strict mode is enabled.
```python
from courlan.settings import BLACKLIST
# Set of problematic domains to exclude in strict mode
# Examples: 'facebook', 'google', 'instagram', 'twitter', 'amazon', etc.
if domain in BLACKLIST and strict_mode:
# Domain is blocked
pass
```
--------------------------------
### Polite Crawling with Rate Limiting
Source: https://github.com/adbar/courlan/blob/master/_autodocs/README.md
Utilize `UrlStore` to manage URLs for polite crawling. Establish a download schedule with maximum URLs and time limits to respect server rates, then process responses and add new URLs.
```python
from courlan import UrlStore
import time
store = UrlStore()
store.add_urls(initial_urls)
while store.unvisited_websites_number() > 0:
schedule = store.establish_download_schedule(
max_urls=10,
time_limit=5
)
for delay, url in schedule:
time.sleep(delay)
response = requests.get(url)
store.add_from_html(response.text, url)
```
--------------------------------
### Crawling Loop with URL Store
Source: https://github.com/adbar/courlan/blob/master/_autodocs/api-reference/urlstore.md
Demonstrates a typical crawling loop using the URL store. It establishes a download schedule, fetches pages, extracts new links, and optionally stores robots.txt rules. This pattern is useful for managing large-scale web crawling efficiently.
```python
while store.unvisited_websites_number() > 0:
# Get batch of URLs with schedule
schedule = store.establish_download_schedule(max_urls=10, time_limit=5)
for delay, url in schedule:
time.sleep(delay)
# Fetch page
response = requests.get(url)
# Extract and add new links
store.add_from_html(response.text, url, lang='en')
# Optional: store robots.txt rules
# store.store_rules(get_base_url(url), parse_robots(url))
```
```python
# Save state for recovery
store.write('urls_checkpoint.pkl')
```
--------------------------------
### establish_download_schedule
Source: https://github.com/adbar/courlan/blob/master/_autodocs/api-reference/urlstore.md
Returns URLs with backoff schedule tuples for polite crawling, ensuring delays between requests to the same domain.
```APIDOC
## establish_download_schedule()
### Description
Returns URLs with backoff schedule tuples for polite crawling.
### Method
```python
def establish_download_schedule(
self,
max_urls: int = 100,
time_limit: int = 10,
) -> list[str]
```
### Parameters
- **max_urls** (int) - Optional - Maximum number of URLs to return (default: `100`)
- **time_limit** (int) - Optional - Minimum seconds to wait before re-crawling a domain (default: `10`)
### Return
List of tuples `(delay_seconds, url)`, sorted by URL
### Example
```python
schedule = store.establish_download_schedule(max_urls=10, time_limit=5)
# [(0.0, 'https://example.com/1'),
# (5.0, 'https://example.com/2'),
# (0.0, 'https://other.org/page'),
# ...]
for delay, url in schedule:
sleep(delay)
download(url)
```
```
--------------------------------
### Configure UrlStore for Research Data Collection
Source: https://github.com/adbar/courlan/blob/master/_autodocs/configuration.md
Set up UrlStore for research purposes, allowing broader content with strict=False and enabling verbose output for graceful interruption handling. Includes periodic state saving.
```python
store = UrlStore(
strict=False, # Allow broader content
verbose=True # Handle interruption gracefully
)
# Periodically save state
import time
while crawling:
# ... crawl URLs ...
if time.time() % 3600 < 1: # Every hour
store.write('checkpoint.pkl')
```
--------------------------------
### Graceful Interruption Handling with UrlStore
Source: https://github.com/adbar/courlan/blob/master/_autodocs/README.md
Initialize `UrlStore` with `verbose=True` to automatically print unvisited URLs to stdout upon interruption (e.g., Ctrl+C). This allows for saving progress or resuming crawls.
```python
from courlan import UrlStore
store = UrlStore(verbose=True)
# Now on Ctrl+C, unvisited URLs are printed to stdout
# Redirect to file: python script.py > unvisited.txt
```
--------------------------------
### Research Data Collection Workflow
Source: https://github.com/adbar/courlan/blob/master/_autodocs/API-INDEX.md
Use UrlStore for broad data collection with compression and verbose logging. Save the store's state periodically and later sample the dumped URLs for analysis.
```python
from courlan import UrlStore, sample_urls
# Collect broadly, then sample
store = UrlStore(compressed=True, verbose=True)
# Assuming all_urls is defined elsewhere
store.add_urls(all_urls)
# Save checkpoint
store.write('research_urls.pkl')
# Later: sample for analysis
sample = sample_urls(store.dump_urls(), 100)
```
--------------------------------
### Language-Specific Crawling with UrlStore
Source: https://github.com/adbar/courlan/blob/master/_autodocs/README.md
Initialize `UrlStore` with a specific language and strict mode. Add initial URLs and then filter extracted links from HTML content based on the specified language.
```python
from courlan import UrlStore, filter_links
store = UrlStore(language='de', strict=True)
store.add_urls(german_urls)
html = requests.get(url).text
links, nav = filter_links(html, url, lang='de')
store.add_urls(links, appendleft=nav)
```
--------------------------------
### Manage URLs for Crawling with UrlStore
Source: https://github.com/adbar/courlan/blob/master/_autodocs/README.md
Demonstrates using `UrlStore` to add, retrieve, and process URLs for a web crawl, including adding links extracted from fetched HTML.
```python
from courlan import UrlStore
store = UrlStore(language='en', strict=True)
store.add_urls(['https://example.com'])
while store.unvisited_websites_number() > 0:
url = store.get_url('https://example.com')
if url:
html = requests.get(url).text
store.add_from_html(html, url)
```
--------------------------------
### sample_urls()
Source: https://github.com/adbar/courlan/blob/master/_autodocs/api-reference/sample_urls.md
Samples URLs from a list using domain-based stratified sampling. Creates a balanced sample across multiple domains, optionally filtering by domain size constraints. Useful for creating representative URL datasets without bias toward large domains.
```APIDOC
## sample_urls(input_urls: list[str], samplesize: int, exclude_min: int | None = None, exclude_max: int | None = None, strict: bool = False, verbose: bool = False)
### Description
Samples URLs from a list using domain-based stratified sampling. Creates a balanced sample across multiple domains, optionally filtering by domain size constraints. Useful for creating representative URL datasets without bias toward large domains.
### Parameters
#### Path Parameters
- None
#### Query Parameters
- None
#### Request Body
- None
### Parameters
- **input_urls** (list[str]) - Required - List of URLs to sample from.
- **samplesize** (int) - Required - Number of URLs to sample per domain.
- **exclude_min** (int | None) - Optional - Exclude domains with fewer than this many URLs. Defaults to None.
- **exclude_max** (int | None) - Optional - Exclude domains with more than this many URLs. Defaults to None.
- **strict** (bool) - Optional - Enable strict URL filtering when organizing by domain. Defaults to False.
- **verbose** (bool) - Optional - Enable debug logging output. Defaults to False.
### Return Value
A list of sampled URLs (sorted alphabetically).
### Examples
#### Basic sampling
```python
from courlan import sample_urls
urls = [
'https://example1.org/' + str(i) for i in range(100)
] + [
'https://example2.org/' + str(i) for i in range(50)
]
# Sample 10 URLs per domain
sample = sample_urls(urls, 10)
# Returns ~20 URLs total (10 from each domain)
print(len(sample)) # Approximately 20
```
#### Filter by domain size
```python
# Only sample from domains with 20-80 URLs
sample = sample_urls(urls, 10, exclude_min=20, exclude_max=80)
# Excludes domains outside this size range
```
#### Exclude small domains
```python
# Only sample from domains with at least 50 URLs
sample = sample_urls(urls, 10, exclude_min=50)
# Domains with fewer than 50 URLs are skipped
```
#### Exclude large domains
```python
# Only sample from domains with at most 100 URLs
sample = sample_urls(urls, 10, exclude_max=100)
# Domains with more than 100 URLs are skipped
```
#### Strict filtering
```python
# Apply strict URL validation during sampling
sample = sample_urls(urls, 10, strict=True)
# URLs that fail strict validation are excluded before sampling
```
#### Verbose output
```python
# Enable logging to see which domains are processed
sample = sample_urls(urls, 10, verbose=True)
# Prints debug information about sampling process
```
```
--------------------------------
### load_store()
Source: https://github.com/adbar/courlan/blob/master/_autodocs/API-INDEX.md
Loads a saved UrlStore object from disk.
```APIDOC
## load_store()
### Description
Load saved UrlStore from disk. This function deserializes a UrlStore object that was previously saved to a file.
### Method
Not applicable (function call)
### Parameters
None explicitly defined in this reference.
### Returns
- `UrlStore object`: The loaded UrlStore instance.
```
--------------------------------
### URL Utilities Exports from __init__.py
Source: https://github.com/adbar/courlan/blob/master/_autodocs/MODULE-STRUCTURE.md
Imports URL utility functions such as extract_domain, filter_urls, fix_relative_urls, get_base_url, get_host_and_path, get_hostinfo, and is_external from the courlan.urlutils module.
```python
from courlan.urlutils import (
extract_domain,
filter_urls,
fix_relative_urls,
get_base_url,
get_host_and_path,
get_hostinfo,
is_external,
)
```
--------------------------------
### UrlStore Configuration for Memory Constraints
Source: https://github.com/adbar/courlan/blob/master/_autodocs/configuration.md
Optimize UrlStore for memory-constrained environments by using maximum compression and disabling verbose mode to reduce overhead. Includes periodic dumping and reloading to clear caches.
```python
store = UrlStore(
compressed=True, # Maximum compression
verbose=False # Disable signal handlers overhead
)
# Periodically dump and reload to clear caches
for batch in batches:
store.add_urls(batch)
# ... process ...
store.reset() # Clear memory
```
--------------------------------
### Courlan Command-Line Interface Help
Source: https://github.com/adbar/courlan/blob/master/README.md
Displays the help message for the Courlan command-line utility, outlining available options for input/output, filtering, and sampling.
```bash
$ courlan --help
usage: courlan [-h] -i INPUTFILE -o OUTPUTFILE [-d DISCARDEDFILE] [-v]
[-p PARALLEL] [--strict] [-l LANGUAGE] [-r] [--sample SAMPLE]
[--exclude-max EXCLUDE_MAX] [--exclude-min EXCLUDE_MIN]
Command-line interface for Courlan
options:
-h, --help show this help message and exit
I/O:
Manage input and output
-i INPUTFILE, --inputfile INPUTFILE
name of input file (required)
-o OUTPUTFILE, --outputfile OUTPUTFILE
name of output file (required)
-d DISCARDEDFILE, --discardedfile DISCARDEDFILE
name of file to store discarded URLs (optional)
-v, --verbose increase output verbosity
-p PARALLEL, --parallel PARALLEL
number of parallel processes (not used for sampling)
Filtering:
Configure URL filters
--strict perform more restrictive tests
-l LANGUAGE, --language LANGUAGE
use language filter (ISO 639-1 code)
-r, --redirects check redirects
Sampling:
Use sampling by host, configure sample size
--sample SAMPLE size of sample per domain
--exclude-max EXCLUDE_MAX
exclude domains with more than n URLs
--exclude-min EXCLUDE_MIN
exclude domains with less than n URLs
```
--------------------------------
### Follow Redirects with check_url
Source: https://github.com/adbar/courlan/blob/master/_autodocs/api-reference/check_url.md
Shows how to use the with_redirects=True option to have check_url() follow HTTP redirects and return the final URL. This requires the URL to be reachable.
```python
# Follow HTTP redirects
url, domain = check_url(
'https://httpbin.org/redirect-to?url=http%3A%2F%2Fexample.org',
with_redirects=True
)
# May return the final URL after redirects
```
--------------------------------
### Custom URL Storage Implementation
Source: https://github.com/adbar/courlan/blob/master/_autodocs/MODULE-STRUCTURE.md
Shows how to extend the `UrlStore` class from Courlan to implement custom storage logic. This is useful for modifying how URLs are retrieved or managed.
```python
from courlan import UrlStore
class MyUrlStore(UrlStore):
def get_url(self, domain, **kwargs):
# Custom selection logic
return super().get_url(domain, **kwargs)
```
--------------------------------
### Storage Classes Exports from __init__.py
Source: https://github.com/adbar/courlan/blob/master/_autodocs/MODULE-STRUCTURE.md
Imports storage-related classes and functions like UrlStore and load_store from the courlan.urlstore module.
```python
from courlan.urlstore import (
UrlStore,
load_store,
)
```
--------------------------------
### load_store()
Source: https://github.com/adbar/courlan/blob/master/_autodocs/DOCUMENTATION-MAP.md
Loads a persisted UrlStore object from a file.
```APIDOC
## load_store()
### Description
Loads a previously persisted `UrlStore` object from a file. This allows you to resume a crawling session or access previously stored URL data.
### Method
`load_store(filepath)`
### Parameters
#### Path Parameters
- **filepath** (string) - Required - The path to the file from which to load the `UrlStore`.
### Request Example
```python
from courlan.urlstore import load_store
# Load the UrlStore from a file
loaded_store = load_store('path/to/your/urlstore_backup.json')
# Now you can use the loaded_store object
print(f"Loaded store with {loaded_store.total_url_number()} URLs.")
```
### Response
#### Success Response (200)
- **UrlStore object** - The loaded `UrlStore` instance.
#### Response Example
```json
{
"example": "UrlStore object instance"
}
```
```
--------------------------------
### Enable Memory Recovery for Unvisited URLs
Source: https://github.com/adbar/courlan/blob/master/_autodocs/errors.md
Configure `UrlStore` with `verbose=True` to enable memory recovery. On receiving SIGINT or SIGTERM, it will print unvisited URLs, which can be redirected to a file for later processing.
```python
from courlan import UrlStore
# Enable with verbose=True
store = UrlStore(verbose=True)
# On Ctrl+C, will call print_unvisited_urls() before exiting
# Pipe output to file for later processing:
# python script.py 2>/dev/null > unvisited.txt
```
--------------------------------
### Print all URLs and their status
Source: https://github.com/adbar/courlan/blob/master/_autodocs/api-reference/urlstore.md
Outputs all stored URLs to the console, along with their corresponding visited status, formatted as 'URL\tvisited_status'.
```python
def print_urls(self) -> None:
```
--------------------------------
### Handle ValueError for Incomplete URL
Source: https://github.com/adbar/courlan/blob/master/_autodocs/errors.md
Illustrates catching a ValueError from get_host_and_path when an empty URL is provided, indicating an incomplete URL.
```python
from courlan import get_host_and_path
try:
get_host_and_path('') # Empty URL
except ValueError as e:
print(e) # "incomplete URL: "
```
--------------------------------
### Retrieve robots.txt rules
Source: https://github.com/adbar/courlan/blob/master/_autodocs/api-reference/urlstore.md
Fetches the stored robots.txt parsing rules for a specific website. Returns None if no rules have been stored for the given website.
```python
def get_rules(self, website: str) -> RobotFileParser | None:
```
--------------------------------
### Basic filter_links() Usage
Source: https://github.com/adbar/courlan/blob/master/_autodocs/api-reference/filter_links.md
Demonstrates basic usage of filter_links() without robots.txt. It shows how regular content links and priority navigation links are separated. Note that non-crawlable links like 'login' and 'contact' are automatically filtered.
```python
from courlan import filter_links
from urllib.robotparser import RobotFileParser
html = '''
Article
Category
Page 2
Login
Contact
'''
# Without robots.txt
links, priority_links = filter_links(html, 'https://example.org/')
print(links)
# ['/article/123']
print(priority_links)
# ['/category/tech', '/page/2']
# Note: login and contact are filtered as not-crawlable
```
--------------------------------
### Save UrlStore State to File
Source: https://github.com/adbar/courlan/blob/master/_autodocs/configuration.md
Saves the current state of the UrlStore to a file using Python's pickle format. This binary format is not human-readable but is efficient for storage and retrieval.
```python
store.write('urls.pkl') # Binary format, not human-readable
```
--------------------------------
### Sampling URLs by Domain
Source: https://github.com/adbar/courlan/blob/master/README.md
Use the sample_urls function to select a specified number of URLs from a larger list, useful for creating representative samples of web content.
```python
from courlan import sample_urls
my_urls = ['https://example.org/' + str(x) for x in range(100)]
my_sample = sample_urls(my_urls, 10)
```
--------------------------------
### Format Code with Ruff
Source: https://github.com/adbar/courlan/blob/master/CONTRIBUTING.md
Apply code formatting to the Courlan project and tests directory using Ruff. This ensures consistent code style across the project.
```bash
ruff format courlan tests
```
--------------------------------
### Logging Integration with Courlan
Source: https://github.com/adbar/courlan/blob/master/_autodocs/MODULE-STRUCTURE.md
Explains how to integrate Python's standard logging module with Courlan. All Courlan modules utilize the `logging` module for output, allowing for configurable verbosity.
```python
import logging
logging.basicConfig(level=logging.DEBUG)
# All courlan modules use logging for output
```
--------------------------------
### Cleaning Functions Exports from __init__.py
Source: https://github.com/adbar/courlan/blob/master/_autodocs/MODULE-STRUCTURE.md
Imports URL cleaning and normalization functions such as clean_url, normalize_url, and scrub_url from the courlan.clean module.
```python
from courlan.clean import (
clean_url,
normalize_url,
scrub_url,
)
```