### Install waybackpack using pip Source: https://github.com/jsvine/waybackpack/blob/master/README.md Install the waybackpack package using pip. Ensure you have Python 3.3+ and pip installed. ```sh pip install waybackpack ``` -------------------------------- ### CLI: Download URL snapshots with waybackpack Source: https://context7.com/jsvine/waybackpack/llms.txt Use the `waybackpack` CLI to download snapshots of a URL to a specified directory. Options include filtering by date, downloading only unique content, listing URLs without downloading, and controlling rate limiting and error handling. ```sh # Download all snapshots of a URL to a local directory waybackpack http://www.dol.gov/ -d ~/Downloads/dol-wayback ``` ```sh # Download only snapshots from 1996 waybackpack http://www.dol.gov/ -d ~/Downloads/dol-1996 --to-date 1996 ``` ```sh # Download snapshots between two dates, unique content only waybackpack https://example.com/ \ -d ~/Downloads/example-archive \ --from-date 20100101 \ --to-date 20201231 \ --uniques-only ``` ```sh # List all archived snapshot URLs (no download) waybackpack http://www.dol.gov/ --list ``` ```sh # List raw (unprocessed) archive URLs waybackpack http://www.dol.gov/ --list --raw ``` ```sh # Download with rate limiting and error tolerance waybackpack https://example.com/ \ -d ~/Downloads/example \ --delay 2 \ --delay-retry 10 \ --max-retries 5 \ --ignore-errors \ --no-clobber \ --progress ``` ```sh # Collapse to one snapshot per month, follow redirects waybackpack https://example.com/ \ -d ~/Downloads/example-monthly \ --collapse timestamp:6 \ --follow-redirects \ --user-agent "waybackpack (research-project; user@example.com)" ``` ```sh # Download raw bytes without toolbar stripping waybackpack https://example.com/ -d ~/Downloads/example-raw --raw ``` -------------------------------- ### Waybackpack Command-Line Usage Source: https://github.com/jsvine/waybackpack/blob/master/README.md This shows the general usage and available options for the waybackpack command-line tool. Options include specifying directories, listing URLs, raw fetching, date ranges, user agents, and error handling. ```sh usage: waybackpack [-h] [--version] (-d DIR | --list) [--raw] [--root ROOT] [--from-date FROM_DATE] [--to-date TO_DATE] [--user-agent USER_AGENT] [--follow-redirects] [--uniques-only] [--collapse COLLAPSE] [--ignore-errors] [--max-retries MAX_RETRIES] [--no-clobber] [--quiet] [--progress] [--delay DELAY] [--delay-retry DELAY_RETRY] url ``` -------------------------------- ### Python: Archive URL snapshots with Waybackpack Source: https://context7.com/jsvine/waybackpack/llms.txt Use the `Pack` class to discover and download snapshots of a URL. You can provide explicit timestamps or let Pack auto-discover them via the CDX API. Configure session settings for retries and delays. ```python import tempfile, shutil from waybackpack import Pack, Session, search session = Session( user_agent="waybackpack (myproject; user@example.com)", max_retries=3, delay_retry=5, ) # Discover snapshots, then build a Pack with explicit timestamps url = "https://www.dol.gov/" snapshots = search(url, from_date="20010101", to_date="20011231", session=session) timestamps = [snap["timestamp"] for snap in snapshots] pack = Pack(url, timestamps=timestamps, session=session) print(f"Found {len(pack.assets)} snapshots") # Download all snapshots to a local directory output_dir = "/tmp/dol-archive" pack.download_to( output_dir, raw=False, # Strip Wayback toolbar injections ignore_errors=True, # Log non-HTTP errors and continue no_clobber=True, # Skip already-downloaded files delay=1, # Sleep 1 second between fetches progress=False, # Set True if tqdm is installed ) # Creates: /tmp/dol-archive/20010115120000/www.dol.gov/index.html, etc. # Alternatively, let Pack auto-discover timestamps via CDX pack_auto = Pack(url, session=session) print(pack_auto.timestamps[:3]) # ['19961102145216', '19961103063843', '19961222171647'] ``` -------------------------------- ### waybackpack.search() Source: https://context7.com/jsvine/waybackpack/llms.txt Queries the Internet Archive's CDX API to retrieve a list of snapshot metadata dictionaries for a given URL. Supports filtering by date range, collapsing results, and returning only unique snapshots. ```APIDOC ## waybackpack.search() ### Description Queries the Internet Archive's CDX API and returns a list of snapshot metadata dicts for a given URL, each containing fields like `timestamp`, `statuscode`, `mimetype`, and `dupecount`. Supports optional date range filtering, collapsing of results (e.g., one per month), and deduplication via `uniques_only`. ### Method Signature ```python waybackpack.search(url: str, from_date: str = None, to_date: str = None, uniques_only: bool = False, collapse: list = None) ``` ### Parameters #### Query Parameters - **url** (str) - Required - The URL to search for snapshots. - **from_date** (str) - Optional - Filter snapshots from this date (format: YYYYMMDDhhss). - **to_date** (str) - Optional - Filter snapshots up to this date (format: YYYYMMDDhhss). - **uniques_only** (bool) - Optional - If True, return only unique snapshots based on content hash. - **collapse** (list) - Optional - List of fields to collapse results by (e.g., `["timestamp:6"]` for one per month). ### Request Example ```python import waybackpack # Fetch all snapshots of a URL snapshots = waybackpack.search("http://www.dol.gov/") print(len(snapshots)) # Filter by date range snapshots_1996 = waybackpack.search( "http://www.dol.gov/", from_date="19960101", to_date="1996" ) # Return only unique snapshots uniques = waybackpack.search( "http://www.dol.gov/", to_date="1996", uniques_only=True ) # Collapse to at most one result per month monthly = waybackpack.search( "https://example.com/", collapse=["timestamp:6"] ) ``` ### Response #### Success Response - Returns a list of dictionaries, where each dictionary represents a snapshot and contains fields such as `timestamp`, `statuscode`, `mimetype`, and `dupecount`. ### Response Example ```json { "urlkey": "gov,dol)/", "timestamp": "19961102145216", "statuscode": "200", "mimetype": "text/html", "dupecount": "1" } ``` ``` -------------------------------- ### Query CDX API for Snapshots with waybackpack.search() Source: https://context7.com/jsvine/waybackpack/llms.txt Use this function to retrieve metadata for all historical snapshots of a URL from the Wayback Machine. Supports filtering by date range, returning only unique snapshots, and collapsing results to one per month. ```python import waybackpack # Fetch all snapshots of a URL snapshots = waybackpack.search("http://www.dol.gov/") print(len(snapshots)) # e.g., 2000+ print(snapshots[0]) # {'urlkey': 'gov,dol)/', 'timestamp': '19961102145216', 'statuscode': '200', ...} ``` ```python # Filter by date range (format: YYYYMMDDhhss, trailing digits optional) snapshots_1996 = waybackpack.search( "http://www.dol.gov/", from_date="19960101", to_date="1996" ) print(len(snapshots_1996)) # 5 ``` ```python # Return only unique snapshots (skip duplicates by content hash) uniques = waybackpack.search( "http://www.dol.gov/", to_date="1996", uniques_only=True ) print(len(uniques)) # 2 ``` ```python # Collapse to at most one result per month monthly = waybackpack.search( "https://example.com/", collapse=["timestamp:6"] ) print([s["timestamp"][:6] for s in monthly]) # ['199601', '199602', ...] ``` -------------------------------- ### List URLs of Archived Snapshots Source: https://github.com/jsvine/waybackpack/blob/master/README.md Use this command to list all available archived snapshot URLs for a given URL without downloading the content. ```sh waybackpack http://www.dol.gov/ --list ``` -------------------------------- ### Download Wayback Machine Archive for a URL Source: https://github.com/jsvine/waybackpack/blob/master/README.md Use this command to download all archived snapshots for a given URL within a specified date range. The output is saved to a specified directory. ```sh waybackpack http://www.dol.gov/ -d ~/Downloads/dol-wayback --to-date 1996 ``` -------------------------------- ### Create HTTP Session with Retry Logic using waybackpack.Session Source: https://context7.com/jsvine/waybackpack/llms.txt Instantiate a Session object to manage HTTP requests with automatic retries for 4XX/5XX errors and connection issues. Customize user agent, redirect following, max retries, and retry delay. This session can be passed to other API components to share connection settings. ```python from waybackpack import Session # Create a session with custom settings session = Session( user_agent="waybackpack/0.6.4 (myproject; user@example.com)", follow_redirects=True, # Follow Wayback Machine JS redirects max_retries=5, # Retry up to 5 times on failure delay_retry=10, # Wait 10 seconds between retries ) # Use the session for a direct GET (returns requests.Response or None on exhausted retries) res = session.get("https://web.archive.org/cdx/search/cdx", params={ "url": "https://example.com/", "output": "json", "limit": "1" }) if res is not None: print(res.status_code) # 200 print(res.json()) else: print("Max retries reached, request skipped.") ``` -------------------------------- ### waybackpack.Session Source: https://context7.com/jsvine/waybackpack/llms.txt Wraps `requests.get` with automatic retry logic for handling transient network errors and server-side issues (4XX/5XX responses). Allows for shared connection settings across multiple API calls. ```APIDOC ## waybackpack.Session ### Description Wraps `requests.get` with automatic retry-on-failure behavior for 4XX/5XX responses and connection errors. All other API components accept an optional `Session` instance so connection settings are shared. Key parameters: `user_agent`, `follow_redirects`, `max_retries`, `delay_retry`. ### Method Signature ```python Session(user_agent: str = None, follow_redirects: bool = False, max_retries: int = 3, delay_retry: int = 5) ``` ### Parameters #### Initialization Parameters - **user_agent** (str) - Optional - Custom User-Agent string for requests. - **follow_redirects** (bool) - Optional - Whether to follow Wayback Machine JavaScript redirects. Defaults to `False`. - **max_retries** (int) - Optional - Maximum number of retries for failed requests. Defaults to 3. - **delay_retry** (int) - Optional - Delay in seconds between retries. Defaults to 5. ### Method #### `get(url: str, params: dict = None)` Performs an HTTP GET request using the configured session. Returns a `requests.Response` object on success or `None` if max retries are exhausted. ### Request Example ```python from waybackpack import Session # Create a session with custom settings session = Session( user_agent="waybackpack/0.6.4 (myproject; user@example.com)", follow_redirects=True, max_retries=5, delay_retry=10, ) # Use the session for a direct GET res = session.get("https://web.archive.org/cdx/search/cdx", params={ "url": "https://example.com/", "output": "json", "limit": "1" }) if res is not None: print(res.status_code) print(res.json()) else: print("Max retries reached, request skipped.") ``` ### Response #### Success Response - Returns a `requests.Response` object on successful request. #### Failure Response - Returns `None` if the maximum number of retries is reached without success. ``` -------------------------------- ### Represent and Fetch Archived Snapshot with waybackpack.Asset Source: https://context7.com/jsvine/waybackpack/llms.txt Use the Asset class to represent a specific archived URL at a given timestamp. It constructs the archive URL and can fetch the content, optionally stripping Wayback Machine injections from HTML. Set `raw=True` to retrieve unprocessed bytes. ```python from waybackpack import Asset, Session session = Session() # Create an asset for a specific URL + timestamp asset = Asset("http://www.dol.gov/", timestamp="19961102145216") # Get the Wayback Machine URL for this snapshot print(asset.get_archive_url()) # https://web.archive.org/web/19961102145216/http://www.dol.gov/ # Get the raw (unmodified) archive URL print(asset.get_archive_url(raw=True)) # https://web.archive.org/web/19961102145216id_/http://www.dol.gov/ # Fetch the content (returns bytes; strips toolbar HTML by default) content = asset.fetch(session=session, raw=False) if content: print(len(content), "bytes") print(content[:200]) # Fetch raw, unprocessed bytes raw_content = asset.fetch(session=session, raw=True) ``` -------------------------------- ### waybackpack.Asset Source: https://context7.com/jsvine/waybackpack/llms.txt Represents a single archived snapshot of a URL at a specific point in time. Provides methods to construct the archive URL and fetch the content, with options for post-processing. ```APIDOC ## waybackpack.Asset ### Description Represents one snapshot of a URL at a specific Wayback Machine timestamp. Constructs the archive URL and fetches the content, optionally stripping Wayback Machine toolbar/analytics injections from HTML. Pass `raw=True` to bypass all post-processing and retrieve the exact bytes stored by the archive. ### Method Signature ```python Asset(url: str, timestamp: str) ``` ### Parameters #### Initialization Parameters - **url** (str) - Required - The original URL of the archived resource. - **timestamp** (str) - Required - The timestamp of the snapshot (format: YYYYMMDDhhss). ### Methods #### `get_archive_url(raw: bool = False)` Constructs the full URL to access the snapshot on the Wayback Machine. - **raw** (bool) - Optional - If True, returns the raw archive URL without modifications. #### `fetch(session: Session = None, raw: bool = False)` Fetches the content of the archived snapshot. - **session** (Session) - Optional - An instance of `waybackpack.Session` to use for the request. - **raw** (bool) - Optional - If True, fetches the raw, unprocessed bytes from the archive, bypassing any HTML stripping. ### Request Example ```python from waybackpack import Asset, Session session = Session() # Create an asset for a specific URL + timestamp asset = Asset("http://www.dol.gov/", timestamp="19961102145216") # Get the Wayback Machine URL for this snapshot print(asset.get_archive_url()) # Get the raw archive URL print(asset.get_archive_url(raw=True)) # Fetch the content (strips toolbar HTML by default) content = asset.fetch(session=session, raw=False) if content: print(len(content), "bytes") # Fetch raw, unprocessed bytes raw_content = asset.fetch(session=session, raw=True) ``` ### Response #### Success Response - `get_archive_url()`: Returns a string representing the archive URL. - `fetch()`: Returns bytes representing the content of the snapshot. Returns `None` if the fetch fails after retries. ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.