### Install Wayback via Pip Source: https://github.com/edgi-govdata-archiving/wayback/blob/main/docs/source/installation.md Install the Wayback Python library using pip. This is the recommended method for most users. ```bash $ pip install wayback ``` -------------------------------- ### Set Up Local Development Environment Source: https://github.com/edgi-govdata-archiving/wayback/blob/main/CONTRIBUTING.rst Install the project and its development dependencies in a virtual environment. ```bash cd wayback/ python -m venv .venv source .venv/bin/activate pip install -e '.[dev,docs]' ``` -------------------------------- ### Instantiate WaybackClient Source: https://github.com/edgi-govdata-archiving/wayback/blob/main/docs/source/usage.md Initialize the WaybackClient to interact with the Wayback Machine. No specific setup is required beyond importing the library. ```python from wayback import WaybackClient wayback = WaybackClient() ``` -------------------------------- ### Install Wayback Package Source: https://github.com/edgi-govdata-archiving/wayback/blob/main/README.rst Install the Wayback package using pip. This command should be run in your terminal or command prompt. ```bash pip install wayback ``` -------------------------------- ### RateLimit Usage Examples Source: https://context7.com/edgi-govdata-archiving/wayback/llms.txt Illustrates various ways to use the `RateLimit` class, including standalone limiting within loops using a context manager, direct `wait()` calls, disabling limits, and sharing limits across threads. ```python import wayback import time import threading # Standalone usage: limit a loop to 2 iterations per second limit = wayback.RateLimit(per_second=2) for i in range(6): with limit: print(f"Processing item {i} at {time.time():.2f}") # Items are spaced ~0.5 seconds apart # Use wait() directly limit = wayback.RateLimit(per_second=1) for url in ['https://a.com', 'https://b.com', 'https://c.com']: limit.wait() # blocks until 1 second has passed since last call print(f"Fetching {url}") # Disable rate limiting (per_second=0) no_limit = wayback.RateLimit(per_second=0) no_limit.wait() # returns immediately # Shared limit across threads shared = wayback.RateLimit(per_second=5) def worker(thread_id): for _ in range(3): with shared: print(f"Thread {thread_id} working at {time.time():.3f}") threads = [threading.Thread(target=worker, args=(i,)) for i in range(3)] for t in threads: t.start() for t in threads: t.join() # All 9 operations together are capped at 5 per second across all threads ``` -------------------------------- ### Process Mementos with Error Handling Source: https://context7.com/edgi-govdata-archiving/wayback/llms.txt This example shows how to iterate through a list of URLs and timestamps, fetching mementos using WaybackClient. It includes comprehensive error handling for various exceptions like NoMementoError, MementoPlaybackError, RateLimitError, and more. Ensure wayback, time, and specific exceptions are imported. ```python import wayback import time from wayback.exceptions import ( WaybackException, UnexpectedResponseFormat, BlockedByRobotsError, BlockedSiteError, MementoPlaybackError, NoMementoError, WaybackRetryError, RateLimitError, ) with wayback.WaybackClient() as client: urls_to_process = [ ('https://www.epa.gov/', '20200601'), ('https://www.noaa.gov/', '20190101'), ] for url, ts in urls_to_process: try: memento = client.get_memento(url, timestamp=ts) print(f"OK: {memento.url} at {memento.timestamp}") except NoMementoError: # URL has never been archived at all — do not retry other timestamps print(f"SKIP: {url} has never been archived") except MementoPlaybackError as e: # Archived but the specific capture can't be served print(f"PLAYBACK_ERROR: {url} — {e}") except BlockedByRobotsError: # robots.txt disallows access via Wayback CDX print(f"ROBOTS_BLOCKED: {url}") except BlockedSiteError: # Administrative/DMCA block print(f"ADMIN_BLOCKED: {url}") except RateLimitError as e: # 429 response — pause and retry wait = e.retry_after or 60 print(f"RATE_LIMITED: sleeping {wait}s") time.sleep(wait) except WaybackRetryError as e: # All retries exhausted — log and continue print(f"RETRY_EXHAUSTED after {e.retries} tries f"({e.time:.1f}s): {e.cause}") except UnexpectedResponseFormat as e: # CDX API returned unparseable data print(f"PARSE_ERROR: {e}") except WaybackException as e: # Catch-all for any other Wayback-specific error print(f"WAYBACK_ERROR: {e}") ``` -------------------------------- ### Search with Multiple Filters in Wayback Source: https://github.com/edgi-govdata-archiving/wayback/blob/main/docs/source/release-history.md Apply multiple filters to a search query using a list or tuple for the `filter_field` parameter. This example searches for captures at 'nasa.gov/' with a 404 status and 'feature' in the URL. ```python client.search('nasa.gov/', match_type='prefix', filter_field=['statuscode:404', 'urlkey:.*feature.*']) ``` -------------------------------- ### Get Earliest Memento for a URL Source: https://github.com/edgi-govdata-archiving/wayback/blob/main/docs/source/usage.md Retrieve the earliest archived copy (memento) of a given URL. The results are returned as a generator, allowing for lazy loading of data. ```python results = wayback.search("nasa.gov") first_record = next(results) print(f"Earliest memento timestamp: {first_record.timestamp}") ``` -------------------------------- ### Example Memento Links Structure Source: https://github.com/edgi-govdata-archiving/wayback/blob/main/docs/source/release-history.md This dictionary structure represents the `links` property of a `wayback.Memento` object, showing relationships like 'original', 'first memento', 'prev memento', 'next memento', and 'last memento'. ```json { 'original': { 'url': 'https://www.fws.gov/birds/', 'rel': 'original' }, 'first memento': { 'url': 'https://web.archive.org/web/20050323155300id_/http://www.fws.gov:80/birds', 'rel': 'first memento', 'datetime': 'Wed, 23 Mar 2005 15:53:00 GMT' }, 'prev memento': { 'url': 'https://web.archive.org/web/20210125125216id_/https://www.fws.gov/birds/', 'rel': 'prev memento', 'datetime': 'Mon, 25 Jan 2021 12:52:16 GMT' }, 'next memento': { 'url': 'https://web.archive.org/web/20210321180831id_/https://www.fws.gov/birds', 'rel': 'next memento', 'datetime': 'Sun, 21 Mar 2021 18:08:31 GMT' }, 'last memento': { 'url': 'https://web.archive.org/web/20221006031005id_/https://fws.gov/birds', 'rel': 'last memento', 'datetime': 'Thu, 06 Oct 2022 03:10:05 GMT' } } ``` -------------------------------- ### Initialize WaybackClient Source: https://context7.com/edgi-govdata-archiving/wayback/llms.txt Demonstrates basic and preferred usage of the WaybackClient. The client manages sessions for interacting with the Wayback Machine. Using it as a context manager ensures the session is closed automatically. ```python import wayback from datetime import date # Basic usage — client manages its own session client = wayback.WaybackClient() # Preferred: use as a context manager so the session is closed on exit with wayback.WaybackClient() as client: for record in client.search('http://nasa.gov', to_date=date(1999, 1, 1)): print(record.timestamp, record.original) # Advanced: pass a custom session session = wayback.WaybackSession(retries=10, backoff=3, timeout=30) client = wayback.WaybackClient(session=session) client.close() # Must close manually when not using context manager ``` -------------------------------- ### Initialize Wayback Client Source: https://github.com/edgi-govdata-archiving/wayback/blob/main/README.rst Import the wayback library and create an instance of the WaybackClient. This client is used to interact with the Wayback Machine API. ```python import wayback client = wayback.WaybackClient() ``` -------------------------------- ### Retrieve Most Recent Captures Source: https://context7.com/edgi-govdata-archiving/wayback/llms.txt Retrieves a specified number of the most recent captures for a URL. The `limit` parameter with a negative value and `fast_latest=True` are used to efficiently get the latest records. ```python # 4. Retrieve only the most recent 5 captures (negative limit) recent = list(islice( client.search('https://nasa.gov/', limit=-5, fast_latest=True), 5 )) print(recent[-1].timestamp) # most recent ``` -------------------------------- ### wayback.WaybackClient.get_memento() Updates Source: https://github.com/edgi-govdata-archiving/wayback/blob/main/docs/source/release-history.md This section details the breaking changes and new features introduced to the `get_memento` method in version 0.3.0 Alpha 1. It covers changes in parameters, return types, and usage examples. ```APIDOC ## get_memento() ### Description Retrieves a specific archived version (memento) of a given URL. This method has undergone significant changes in parameter handling and return types to improve usability and provide richer information. ### Method Signature (v0.3.0 Alpha 1) ```python def get_memento(self, url, datetime=None, mode=Mode.original, *, exact=True, exact_redirects=None, target_window=24 * 60 * 60, follow_redirects=True) ``` ### Key Changes and New Features: 1. **Parameter Handling**: * All parameters except `url` are now keyword-only. Positional arguments for these parameters are deprecated and will cause errors. * The `url` parameter now accepts either a standard URL or a `wayback.CdxRecord` object. * New parameters `datetime` and `mode` have been added for more precise retrieval. 2. **New Parameters**: * `datetime` (datetime object): Specifies the desired capture time for the memento. If timezone-unaware, it defaults to UTC. * `mode` (wayback.Mode enum): Specifies the playback mode for the memento (e.g., `Mode.original`, `Mode.view`). Defaults to `Mode.original`. * `follow_redirects` (bool): Determines whether to follow historical redirects. Defaults to `True`. 3. **Return Type**: * The method now returns a `wayback.Memento` object instead of a Requests response object. This object provides more structured information about the memento. ### Usage Examples: **Using new parameters for datetime and URL:** ```python import datetime client.get_memento('http://www.noaa.gov/', datetime=datetime.datetime(2018, 8, 1)) ``` **Using a `wayback.CdxRecord`:** ```python for record in client.search('http://www.noaa.gov/'): client.get_memento(record) ``` **Specifying playback mode:** ```python client.get_memento('http://www.noaa.gov/', datetime=datetime.datetime(2018, 8, 1), mode=wayback.Mode.view) ``` **Using original Wayback URLs (still supported):** ```python client.get_memento('http://web.archive.org/web/20180801000000id_/http://www.noaa.gov/') ``` ### `wayback.Memento` Object: The returned `Memento` object provides attributes like: * `memento.url`: The original URL captured. * `memento.timestamp`: The capture timestamp. * `memento.mode`: The playback mode. * `memento.memento_url`: The full Wayback Machine URL for the memento. ``` -------------------------------- ### Clone the Wayback Repository Source: https://github.com/edgi-govdata-archiving/wayback/blob/main/CONTRIBUTING.rst Clone your forked repository locally to begin development. ```bash git clone git@github.com:your_name_here/wayback.git ``` -------------------------------- ### Configuring Rate Limits for WaybackClient Source: https://github.com/edgi-govdata-archiving/wayback/blob/main/docs/source/release-history.md Shows how to configure rate limits for `get_memento()` calls when instantiating `WaybackSession`. The default values help prevent temporary blocks by the Wayback Machine servers. ```python client = WaybackClient(WaybackSession(memento_calls_per_second=2)) # These now take a minimum of 0.5 seconds, even if the Wayback Machine # responds instantly (there's no delay on the first call): client.get_memento('http://www.noaa.gov/', timestamp='20180816111911') client.get_memento('http://www.noaa.gov/', timestamp='20180829092926') ``` -------------------------------- ### Using CdxRecord with get_memento() Source: https://github.com/edgi-govdata-archiving/wayback/blob/main/docs/source/release-history.md Illustrates how to pass a 'wayback.CdxRecord' object directly to get_memento(), simplifying the process of retrieving a memento when you already have a record from a search. ```python for record in client.search('http://www.noaa.gov/'): client.get_memento(record) ``` -------------------------------- ### Run Linters and Tests Source: https://github.com/edgi-govdata-archiving/wayback/blob/main/CONTRIBUTING.rst Ensure your changes pass code quality checks and tests before committing. ```bash flake8 wayback tests pytest -v . ``` -------------------------------- ### Specifying Playback Mode for Mementos Source: https://github.com/edgi-govdata-archiving/wayback/blob/main/docs/source/release-history.md Demonstrates how to use the 'mode' parameter to control how a memento is presented, with 'wayback.Mode.view' offering a browsable format and 'wayback.Mode.original' returning the exact archived response. ```python client.get_memento('http://www.noaa.gov/', datetime=datetime.datetime(2018, 8, 1), mode=wayback.Mode.view) ``` -------------------------------- ### Wayback User Agent String Source: https://github.com/edgi-govdata-archiving/wayback/blob/main/docs/source/release-history.md Illustrates the default user agent string used by the Wayback client, which includes the package version and repository URL. ```python wayback/0.2.2 (+https://github.com/edgi-govdata-archiving/wayback) ``` -------------------------------- ### Keyword-Only Parameters for get_memento() Source: https://github.com/edgi-govdata-archiving/wayback/blob/main/docs/source/release-history.md Demonstrates the correct way to call get_memento() with keyword arguments, which is now required for all parameters except 'url'. This ensures compatibility with newer versions of the library. ```python client.get_memento('http://web.archive.org/web/20180816111911id_/http://www.noaa.gov/', exact=False, exact_redirects=False, target_window=3600) ``` ```python # This will now cause you some trouble :( client.get_memento('http://web.archive.org/web/20180816111911id_/http://www.noaa.gov/', False, False, 3600) ``` -------------------------------- ### Share Rate Limits Across Multiple Clients Source: https://context7.com/edgi-govdata-archiving/wayback/llms.txt Demonstrates how to create shared rate limiters for `WaybackSession` instances to enforce limits across multiple clients in a threaded application. Ensure sessions are closed when done. ```python shared_search_limit = wayback.RateLimit(per_second=0.5) shared_memento_limit = wayback.RateLimit(per_second=2) session_a = wayback.WaybackSession( search_calls_per_second=shared_search_limit, memento_calls_per_second=shared_memento_limit, ) session_b = wayback.WaybackSession( search_calls_per_second=shared_search_limit, memento_calls_per_second=shared_memento_limit, ) # session_a and session_b share the same rate limit counters client_a = wayback.WaybackClient(session=session_a) client_b = wayback.WaybackClient(session=session_b) # Reset network connections without closing the session session_a.reset() # Always close when done client_a.close() client_b.close() ``` -------------------------------- ### Using Different Memento Fetching Modes Source: https://context7.com/edgi-govdata-archiving/wayback/llms.txt Illustrates how to use the `Mode` enum with `WaybackClient.get_memento` to control the format of the returned archived content. Each mode offers different URL rewriting and toolbar options. ```python import wayback from datetime import datetime, timezone ts = datetime(2020, 6, 1, tzinfo=timezone.utc) with wayback.WaybackClient() as client: # Mode.original (default) — exact bytes as stored by the crawler original = client.get_memento('https://www.epa.gov/', timestamp=ts, mode=wayback.Mode.original) print(original.mode) # 'id_' # Mode.view — HTML rewritten for browser display (links, images, scripts # all point to Wayback equivalents); includes Wayback navigation toolbar view = client.get_memento('https://www.epa.gov/', timestamp=ts, mode=wayback.Mode.view) print(view.mode) # '' # Mode.javascript — URL-rewriting applied to JS content js = client.get_memento( 'https://www.example.com/app.js', timestamp=ts, mode=wayback.Mode.javascript, ) print(js.mode) # 'js_' # Mode.css — URL-rewriting applied to CSS content css = client.get_memento( 'https://www.example.com/style.css', timestamp=ts, mode=wayback.Mode.css, ) print(css.mode) # 'cs_' # Mode.image — image-specific rewriting img = client.get_memento( 'https://www.example.com/logo.png', timestamp=ts, mode=wayback.Mode.image, ) print(img.mode) # 'im_' # All Mode values for m in wayback.Mode: print(m.name, m.value) # original id_ # view # javascript js_ # css cs_ # image im_ ``` -------------------------------- ### Create a New Branch Source: https://github.com/edgi-govdata-archiving/wayback/blob/main/CONTRIBUTING.rst Create a new branch for your bug fixes or feature development. ```bash git checkout -b name-of-your-bugfix-or-feature ``` -------------------------------- ### WaybackClient Source: https://context7.com/edgi-govdata-archiving/wayback/llms.txt The main entry point for interacting with the Wayback Machine. It manages sessions and provides methods for searching and retrieving archived content. It can be used as a context manager for automatic session closing. ```APIDOC ## WaybackClient The main entry point for all Wayback Machine interactions. Manages an underlying `WaybackSession` and exposes `search()` and `get_memento()`. Can be used as a context manager to ensure the underlying session is closed automatically. ```python import wayback from datetime import date # Basic usage — client manages its own session client = wayback.WaybackClient() # Preferred: use as a context manager so the session is closed on exit with wayback.WaybackClient() as client: for record in client.search('http://nasa.gov', to_date=date(1999, 1, 1)): print(record.timestamp, record.original) # Advanced: pass a custom session session = wayback.WaybackSession(retries=10, backoff=3, timeout=30) client = wayback.WaybackClient(session=session) client.close() # Must close manually when not using context manager ``` ``` -------------------------------- ### Iterate and Retrieve CDX Records Source: https://context7.com/edgi-govdata-archiving/wayback/llms.txt Shows how to search the CDX index for records within a date range and retrieve the corresponding memento content. It also demonstrates deduplicating records by digest to avoid re-downloading identical content. ```python import wayback from datetime import date with wayback.WaybackClient() as client: for record in client.search( 'https://www.nasa.gov/', from_date=date(1996, 1, 1), to_date=date(1997, 1, 1), limit=5, ): # Core fields print(record.urlkey) # SURT-formatted URL key, e.g. 'gov,nasa)/' print(record.timestamp) # datetime(1996, 12, 31, 23, 58, 47, tzinfo=utc) print(record.original) # 'http://www.nasa.gov/' print(record.mimetype) # 'text/html' print(record.statuscode) # 200 (None for revisit records) print(record.digest) # 'ABCDEFGHIJKLMNOPQRSTUVWXYZ234567' (base32 SHA-1) print(record.length) # 12345 (bytes stored on disk, None if unknown) # Convenience URL properties print(record.raw_url) # 'https://web.archive.org/web/19961231235847id_/http://www.nasa.gov/' print(record.view_url) # 'https://web.archive.org/web/19961231235847/http://www.nasa.gov/' # Pass directly to get_memento — no need to re-specify timestamp memento = client.get_memento(record) print(len(memento.content)) # Deduplicate by digest to avoid re-downloading identical content seen_digests = set() for record in client.search('https://www.epa.gov/', from_date=date(2015, 1, 1)): if record.digest in seen_digests: continue seen_digests.add(record.digest) memento = client.get_memento(record, exact=False) if memento and memento.ok: print(f"{record.timestamp}: {len(memento.content)} bytes") ``` -------------------------------- ### Commit and Push Changes Source: https://github.com/edgi-govdata-archiving/wayback/blob/main/CONTRIBUTING.rst Stage, commit, and push your local changes to your feature branch on GitHub. ```bash git add . git commit -m "Your detailed description of your changes." git push origin name-of-your-bugfix-or-feature ``` -------------------------------- ### Configure WaybackSession for Network Behavior Source: https://context7.com/edgi-govdata-archiving/wayback/llms.txt Constructs a WaybackSession with custom network settings like retries, backoff delays, timeouts, user agent, and per-endpoint call rates. This session can then be passed to WaybackClient for fine-grained control. ```python import wayback # Custom session with tighter rate limits and more retries session = wayback.WaybackSession( retries=10, backoff=4, # delays: 0s, 4s, 8s, 16s, 32s, ... timeout=30, # 30-second socket timeout user_agent='my-research-bot/1.0 (contact@example.com)', search_calls_per_second=0.5, # 1 CDX search every 2 seconds memento_calls_per_second=2, # 2 memento fetches per second ) client = wayback.WaybackClient(session=session) ``` -------------------------------- ### Accessing and Inspecting Memento Data Source: https://context7.com/edgi-govdata-archiving/wayback/llms.txt Demonstrates how to fetch a Memento object, access its properties like URL, timestamp, headers, and body, and handle redirect history. It also shows manual connection management. ```python import wayback from datetime import datetime, timezone with wayback.WaybackClient() as client: memento = client.get_memento( 'https://www.noaa.gov/home', timestamp=datetime(2018, 8, 16, 11, 19, 11, tzinfo=timezone.utc), exact=False, ) # Identity / location print(memento.url) # 'https://www.noaa.gov/' (original captured URL) print(memento.memento_url) # full Wayback URL used to fetch the memento print(memento.timestamp) # datetime(2018, 8, 29, 8, 8, 49, tzinfo=utc) print(memento.mode) # 'id_' (playback mode string) # HTTP response data print(memento.status_code) # 200 print(memento.ok) # True (status < 400) print(memento.is_redirect) # False (status in 300–399) print(memento.encoding) # 'utf-8' # Headers are case-insensitive print(memento.headers['content-type']) # 'text/html; charset=utf-8' print(memento.headers['Content-Type']) # same value for name in memento.headers: print(f"{name}: {memento.headers[name]}") # Body html_text = memento.text # decoded string raw_bytes = memento.content # raw bytes # Redirect history print(memento.history) # tuple of Memento objects (only actual memento redirects) print(memento.debug_history) # tuple of all Wayback URLs traversed # Related mementos via links if 'first memento' in memento.links: print(memento.links['first memento']['url']) print(memento.links['first memento']['datetime']) if 'last memento' in memento.links: print(memento.links['last memento']['url']) # repr is informative in notebooks / REPLs print(repr(memento)) # # Manual connection management m = client.get_memento('https://nasa.gov/', timestamp='20200601') try: process(m.text) finally: m.close() # safe to call multiple times ``` -------------------------------- ### Configure Independent Rate Limits for Wayback Sessions Source: https://github.com/edgi-govdata-archiving/wayback/blob/main/docs/source/release-history.md Use `wayback.RateLimit` to create shared rate limits across multiple `WaybackSession` instances. This allows for independent rate limiting behavior when sessions are not actively used together. ```python from wayback import WaybackSession, RateLimit # Each of these sessions will be limited to 1 search request every 2 # seconds. During a period where only one of these sessions is in use, it # will continue to operate at 1 request every 2 seconds: session1 = WaybackSession(search_calls_per_second=0.5) session2 = WaybackSession(search_calls_per_second=0.5) # These sessions will be limited to 1 search request each second as a # combined total. During a period where only one of these sessions is in # use, it will make 1 request per second, but if both are actively in use, each will make 1 request every 2 seconds: rate = RateLimit(per_second=1) session1 = WaybackSession(search_calls_per_second=rate) session2 = WaybackSession(search_calls_per_second=rate) ``` -------------------------------- ### WaybackClient.search() Source: https://context7.com/edgi-govdata-archiving/wayback/llms.txt Queries the Wayback CDX API for all captures of a given URL and yields `CdxRecord` objects. Results are automatically paginated and support date-range filtering, match types, field filters, and collapse options. ```APIDOC ## WaybackClient.search() Queries the Wayback CDX API for all captures of a given URL and yields `CdxRecord` objects. Results are automatically paginated using resume keys — the generator transparently fetches subsequent pages as needed. Supports date-range filtering, match types, field filters, and collapse options. The `StopIteration` value is the total count of yielded records. ```python import wayback from datetime import date, datetime, timezone from itertools import islice with wayback.WaybackClient() as client: # 1. Basic exact-URL search with a date range for record in client.search( 'https://www.epa.gov/', from_date=date(2020, 1, 1), to_date=date(2021, 1, 1), ): print(f"{record.timestamp.isoformat()} {record.original} {record.statuscode} {record.mimetype}") # e.g.: 2020-03-14T12:34:56+00:00 https://www.epa.gov/ 200 text/html # 2. Prefix match — all URLs under a path for record in client.search('https://epa.gov/climate-change/*'): print(record.urlkey, record.digest) # 3. Domain match — all subdomains for record in client.search('*.epa.gov'): print(record.original) # 4. Retrieve only the most recent 5 captures (negative limit) recent = list(islice( client.search('https://nasa.gov/', limit=-5, fast_latest=True), 5 )) print(recent[-1].timestamp) # most recent # 5. Filter by status code and deduplicate by digest for record in client.search( 'https://www.noaa.gov/', filter_field=['statuscode:200', 'mimetype:text/html'], collapse='digest', from_date=date(2018, 1, 1), to_date=date(2019, 1, 1), ): print(record.raw_url) # direct link to original archived content print(record.view_url) # public Wayback viewer URL # 6. Error handling try: results = client.search('https://example-blocked-site.com/') for record in results: pass except wayback.exceptions.BlockedByRobotsError: print("Blocked by robots.txt") except wayback.exceptions.BlockedSiteError: print("Site is administratively blocked (takedown notice, etc.)") except wayback.exceptions.UnexpectedResponseFormat as e: print(f"CDX response could not be parsed: {e}") ``` ``` -------------------------------- ### Search Wayback Machine with Prefix Match Source: https://context7.com/edgi-govdata-archiving/wayback/llms.txt Searches for all archived URLs under a specific path using a prefix match. This is useful for retrieving all captures of pages within a particular section of a website. ```python # 2. Prefix match — all URLs under a path for record in client.search('https://epa.gov/climate-change/*'): print(record.urlkey, record.digest) ``` -------------------------------- ### Search and Download Mementos Source: https://github.com/edgi-govdata-archiving/wayback/blob/main/README.rst Search for historical web page records (mementos) for a given URL before a specified date and download them. Requires the 'date' object from the 'datetime' module. ```python for record in client.search('http://nasa.gov', to_date=date(1999, 1, 1)): memento = client.get_memento(record) ``` -------------------------------- ### Handling Historical Redirects with follow_redirects Source: https://github.com/edgi-govdata-archiving/wayback/blob/main/docs/source/release-history.md Explains the 'follow_redirects' parameter, which controls whether historical redirects are followed during memento retrieval. The default is True, preserving previous behavior. ```python client.get_memento('http://web.archive.org/web/20180801000000id_/http://www.noaa.gov/') ``` -------------------------------- ### Handle Wayback Machine API Errors Source: https://context7.com/edgi-govdata-archiving/wayback/llms.txt Demonstrates error handling for various potential issues when querying the Wayback Machine API, such as being blocked by robots.txt, administrative blocks, or unexpected response formats. ```python # 6. Error handling try: results = client.search('https://example-blocked-site.com/') for record in results: pass except wayback.exceptions.BlockedByRobotsError: print("Blocked by robots.txt") except wayback.exceptions.BlockedSiteError: print("Site is administratively blocked (takedown notice, etc.)") except wayback.exceptions.UnexpectedResponseFormat as e: print(f"CDX response could not be parsed: {e}") ``` -------------------------------- ### Accessing Memento Details with Wayback.Memento Object Source: https://github.com/edgi-govdata-archiving/wayback/blob/main/docs/source/release-history.md Shows how to use the new 'wayback.Memento' object returned by get_memento(). It provides attributes like 'url', 'timestamp', 'mode', and 'memento_url' for easier access to memento information. ```python memento = client.get_memento('http://www.noaa.gov/home', datetime(2018, 8, 16, 11, 19, 11), exact=False) # These values were previously not available except by parsing # `memento.url`. The old `memento.url` is now `memento.memento_url`. memento.url == 'http://www.noaa.gov/' memento.timestamp == datetime(2018, 8, 29, 8, 8, 49, tzinfo=timezone.utc) memento.mode == 'id_' # Used to be `memento.url`: memento.memento_url == 'http://web.archive.org/web/20180816111911id_/http://www.noaa.gov/' # Used to be a list of `Response` objects, now a *tuple* of Mementos. It # lists only the redirects that are actual Mementos and not part of ``` -------------------------------- ### Using Datetime for Specific Memento Capture Source: https://github.com/edgi-govdata-archiving/wayback/blob/main/docs/source/release-history.md Shows how to retrieve a memento from a specific date and time using the new 'datetime' parameter. If no timezone is specified, it defaults to UTC. ```python client.get_memento('http://www.noaa.gov/', datetime.datetime(2018, 8, 1)) ``` -------------------------------- ### Download Memento Content Source: https://github.com/edgi-govdata-archiving/wayback/blob/main/docs/source/usage.md Fetch the original content of a specific memento. Use the `wayback.Mode.original` for analysis purposes. This client provides additional tools for handling Wayback Machine servers. ```python content = wayback.download("nasa.gov", timestamp="19960314031500", mode=wayback.Mode.original) print(f"Downloaded content length: {len(content)}") ``` -------------------------------- ### Accessing Memento Attributes Source: https://github.com/edgi-govdata-archiving/wayback/blob/main/docs/source/release-history.md Demonstrates how to access various attributes of a Memento object, including its URL, history, debug history, headers, status code, and content. ```python memento.memento_url == 'http://web.archive.org/web/20180816111911id_/http://www.noaa.gov/' ``` ```python memento.history == (Memento,) ``` ```python memento.debug_history == ('http://web.archive.org/web/20180816111911id_/http://noaa.gov/home', 'http://web.archive.org/web/20180829092926id_/http://noaa.gov/home', 'http://web.archive.org/web/20180829092926id_/http://noaa.gov/') ``` ```python memento.headers = {'header_name': 'header_value', 'another_header': 'another_value', 'and': 'so on'} ``` ```python memento.status_code ``` ```python memento.ok ``` ```python memento.is_redirect ``` ```python memento.encoding ``` ```python memento.content ``` ```python memento.text ``` -------------------------------- ### Use Separate Parameters for URL and Datetime in get_memento() Source: https://github.com/edgi-govdata-archiving/wayback/blob/main/docs/source/release-history.md The `get_memento` method now accepts a URL and a `datetime` object separately, simplifying the process of retrieving a specific memento. If no timezone is specified for `datetime`, it defaults to UTC. ```python client.get_memento('http://web.archive.org/web/20180801000000id_/http://www.noaa.gov/') ``` ```python client.get_memento('http://www.noaa.gov/', datetime.datetime(2018, 8, 1)) ``` -------------------------------- ### Expandable FAQ Functionality Source: https://github.com/edgi-govdata-archiving/wayback/blob/main/src/wayback/tests/test_files/fws-gov-birds.txt This jQuery script handles the click events for an expandable FAQ section. It animates the height of the answer container to show or hide the content. ```javascript $(document).ready(function() { $('.faq_question').click(function() { if ($(this).parent().is('.open')){ $(this).closest('.faq').find('.faq_answer_container').animate({'height':'0'},500); $(this).closest('.faq').removeClass('open'); }else{ var newHeight =$(this).closest('.faq').find('.faq_answer').height() +'px'; $(this).closest('.faq').find('.faq_answer_container').animate({'height':newHeight},500); $(this).closest('.faq').addClass('open'); } }); }); ``` -------------------------------- ### Accessing Memento Headers Case-Insensitively Source: https://github.com/edgi-govdata-archiving/wayback/blob/main/docs/source/release-history.md Demonstrates how to access memento headers. Header lookups are case-insensitive, but the original casing is preserved when iterating through the headers. ```default list(memento.headers) == ['Content-Type', 'Date'] memento.headers['Content-Type'] == memento.headers['content-type'] ```