### Install waybackpy using pip

Source: https://github.com/akamhy/waybackpy/blob/master/README.md

Recommended installation method using pip from PyPI. Use the -U flag to ensure the latest version is installed.

```bash
pip install waybackpy -U
```

--------------------------------

### Install waybackpy from git repository

Source: https://github.com/akamhy/waybackpy/blob/master/README.md

Installation directly from the git repository. This method is not recommended for general use.

```bash
pip install git+https://github.com/akamhy/waybackpy.git
```

--------------------------------

### Install waybackpy using conda

Source: https://github.com/akamhy/waybackpy/blob/master/README.md

Recommended installation method using conda from the conda-forge channel.

```bash
conda install -c conda-forge waybackpy
```

--------------------------------

### CDX Server API with Prefix Match Type

Source: https://github.com/akamhy/waybackpy/wiki/Python-package-docs

Retrieves archive snapshots for a given URL prefix. This example demonstrates using the 'prefix' match type to get all results under a specific path.

```python
from waybackpy import WaybackMachineCDXServerAPI
url = "archive.org/about/"
user_agent = "Your-user-agent"

cdx = WaybackMachineCDXServerAPI(url=url, user_agent=user_agent, match_type="prefix")

snapshots = cdx.snapshots()

for snapshot in snapshots:
    print(snapshot.archive_url)
```

--------------------------------

### Save API Response Headers Example

Source: https://github.com/akamhy/waybackpy/wiki/Wayback-Machine-APIs

This example shows the typical response headers returned by the Save API when archiving a web page. The saved URL needs to be parsed from these headers.

```text
{'Server': 'nginx/1.19.10', 'Date': 'Sun, 02 Jan 2022 10:54:09 GMT', 'Content-Type': 'text/html; charset=UTF-8', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'x-archive-orig-date': 'Sun, 02 Jan 2022 10:46:06 GMT', 'x-archive-orig-server': 'mw1385.eqiad.wmnet', 'x-archive-orig-x-content-type-options': 'nosniff', 'x-archive-orig-p3p': 'CP="See https://en.wikipedia.org/wiki/Special:CentralAutoLogin/P3P for more info."', 'x-archive-orig-content-language': 'en', 'x-archive-orig-vary': 'Accept-Encoding,Cookie,Authorization', 'x-archive-orig-last-modified': 'Sun, 02 Jan 2022 09:30:45 GMT', 'x-archive-orig-content-encoding': 'gzip', 'x-archive-orig-age': '2', 'x-archive-orig-x-cache': 'cp4030 miss, cp4027 hit/1', 'x-archive-orig-x-cache-status': 'hit-front', 'x-archive-orig-server-timing': 'cache;desc="hit-front", host;desc="cp4027"', 'x-archive-orig-strict-transport-security': 'max-age=106384710; includeSubDomains; preload', 'x-archive-orig-report-to': '{ "group": "wm_nel", "max_age": 86400, "endpoints": [{ "url": "https://intake-logging.wikimedia.org/v1/events?stream=w3c.reportingapi.network_error&schema_uri=/w3c/reportingapi/network_error/1.0.0" }] }', 'x-archive-orig-nel': '{ "report_to": "wm_nel", "max_age": 86400, "failure_fraction": 0.05, "success_fraction": 0.0}', 'x-archive-orig-permissions-policy': 'interest-cohort=()', 'x-archive-orig-x-client-ip': '207.241.232.35', 'x-archive-orig-cache-control': 'private, s-maxage=0, max-age=0, must-revalidate', 'x-archive-orig-accept-ranges': 'bytes', 'x-archive-orig-content-length': '164995', 'x-archive-orig-connection': 'keep-alive', 'x-archive-guessed-content-type': 'text/html', 'x-archive-guessed-charset': 'utf-8', 'memento-datetime': 'Sun, 02 Jan 2022 10:46:08 GMT', 'link': '<https://en.wikipedia.org/wiki/Social_media>; rel="original", <https://web.archive.org/web/timemap/link/https://en.wikipedia.org/wiki/Social_media>; rel="timemap"; type="application/link-format", <https://web.archive.org/web/https://en.wikipedia.org/wiki/Social_media>; rel="timegate", <https://web.archive.org/web/20051215000000/http://en.wikipedia.org/wiki/Social_media>; rel="first memento"; datetime="Thu, 15 Dec 2005 00:00:00 GMT", <https://web.archive.org/web/20220101114012/https://en.wikipedia.org/wiki/Social_media>; rel="prev memento"; datetime="Sat, 01 Jan 2022 11:40:12 GMT", <https://web.archive.org/web/20220102104608/https://en.wikipedia.org/wiki/Social_media>; rel="memento"; datetime="Sun, 02 Jan 2022 10:46:08 GMT", <https://web.archive.org/web/20220102104608/https://en.wikipedia.org/wiki/Social_media>; rel="last memento"; datetime="Sun, 02 Jan 2022 10:46:08 GMT"', 'content-security-policy': "default-src 'self' 'unsafe-eval' 'unsafe-inline' data: blob: archive.org web.archive.org analytics.archive.org pragma.archivelab.org", 'x-archive-src': 'spn2-20220102093111-wwwb-spn10.us.archive.org-8000.warc.gz', 'server-timing': 'captures_list;dur=275.334598, exclusion.robots;dur=0.096415, exclusion.robots.policy;dur=0.088356, RedisCDXSource;dur=1.634125, esindex;dur=0.008082, LoadShardBlock;dur=81.607259, PetaboxLoader3.datanode;dur=51.631773, CDXLines.iter;dur=18.885269, load_resource;dur=19.971806', 'x-app-server': 'wwwb-app204', 'x-ts': '200', 'x-tr': '910', 'X-location': 'All', 'X-Cache-Key': 'httpsweb.archive.org/web/20220102104608/https://en.wikipedia.org/wiki/Social_mediaIN', 'X-RL': '0', 'X-NA': '0', 'X-Page-Cache': 'MISS', 'X-NID': '-', 'Referrer-Policy': 'no-referrer-when-downgrade', 'Permissions-Policy': 'interest-cohort=()', 'Content-Encoding': 'gzip'}
```

--------------------------------

### Iterate through snapshots within a date range using CDXServerAPI

Source: https://github.com/akamhy/waybackpy/blob/master/README.md

Fetch and iterate over all snapshots for a given URL within a specified date range (start_timestamp and end_timestamp). This example prints the archive URL for each snapshot.

```python
>>> from waybackpy import WaybackMachineCDXServerAPI
>>> url = "https://pypi.org"
>>> user_agent = "Mozilla/5.0 (Windows NT 5.1; rv:40.0) Gecko/20100101 Firefox/40.0"
>>> cdx = WaybackMachineCDXServerAPI(url, user_agent, start_timestamp=2016, end_timestamp=2017)
>>> for item in cdx.snapshots():
...     print(item.archive_url)
...
https://web.archive.org/web/20160110011047/http://pypi.org/
https://web.archive.org/web/20160305104847/http://pypi.org/
. 
. # URLS REDACTED FOR READABILITY
. 
https://web.archive.org/web/20171127171549/https://pypi.org/
https://web.archive.org/web/20171206002737/http://pypi.org:80/
```

--------------------------------

### Get oldest availability using WaybackMachineAvailabilityAPI

Source: https://github.com/akamhy/waybackpy/blob/master/README.md

Retrieve the URL of the oldest available snapshot for a given URL using the WaybackMachineAvailabilityAPI. This method is noted to have performance issues.

```python
>>> availability_api.oldest()
https://web.archive.org/web/19981111184551/http://google.com:80/
```

--------------------------------

### Get oldest snapshot using CDXServerAPI

Source: https://github.com/akamhy/waybackpy/blob/master/README.md

Retrieve the oldest available snapshot for a URL using the CDXServerAPI. The result object provides various attributes of the snapshot.

```python
>>> cdx_api.oldest()
com,google)/ 19981111184551 http://google.com:80/ text/html 200 HOQ2TGPYAEQJPNUA6M4SMZ3NGQRBXDZ3 381
>>> oldest = cdx_api.oldest()
>>> oldest
com,google)/ 19981111184551 http://google.com:80/ text/html 200 HOQ2TGPYAEQJPNUA6M4SMZ3NGQRBXDZ3 381
>>> oldest.archive_url
'https://web.archive.org/web/19981111184551/http://google.com:80/'
>>> oldest.original
'http://google.com:80/'
>>> oldest.urlkey
'com,google)/'
>>> oldest.timestamp
'19981111184551'
>>> oldest.datetime_timestamp
datetime.datetime(1998, 11, 11, 18, 45, 51)
>>> oldest.statuscode
'200'
>>> oldest.mimetype
'text/html'
```

--------------------------------

### Filter by Mimetype and Digest (Regex)

Source: https://github.com/akamhy/waybackpy/wiki/CLI-docs

Filter capture results by mimetype and digest. This example excludes 'text/html' mimetypes and matches a specific digest, printing the archive URL, mimetype, and digest.

```bash
waybackpy --url archive.org --user-agent user-agent-example --cdx  --filter \!mimetype:text/html --filter digest:2WAXX5NUWNNCS2BDKCO5OVDQBJVNKIVV --cdx-print archiveurl  --cdx-print mimetype --cdx-print digest
```

--------------------------------

### Get newest availability using WaybackMachineAvailabilityAPI

Source: https://github.com/akamhy/waybackpy/blob/master/README.md

Retrieve the URL of the newest available snapshot for a given URL using the WaybackMachineAvailabilityAPI. This method might return a more recent snapshot than the CDXServerAPI's newest() method.

```python
>>> availability_api.newest()
https://web.archive.org/web/20220118150444/https://www.google.com/
```

--------------------------------

### Get newest snapshot using CDXServerAPI

Source: https://github.com/akamhy/waybackpy/blob/master/README.md

Retrieve the newest available snapshot for a URL using the CDXServerAPI. This snippet demonstrates accessing the archive URL and timestamp of the newest snapshot.

```python
>>> newest = cdx_api.newest()
>>> newest
com,google)/ 20220217234427 http://@google.com/ text/html 301 Y6PVK4XWOI3BXQEXM5WLLWU5JKUVNSFZ 563
>>> newest.archive_url
'https://web.archive.org/web/20220217234427/http://@google.com/'
>>> newest.timestamp
'20220217234427'
```

--------------------------------

### Filter by Status Code (Regex)

Source: https://github.com/akamhy/waybackpy/wiki/CLI-docs

Filter capture results to exclude those with a specific status code. The '!' prefix inverts the match. This example prints the archive URL and status code.

```bash
waybackpy --url archive.org --user-agent user-agent-example --cdx --filter \!statuscode:200 --cdx-print archiveurl --cdx-print statuscode
```

--------------------------------

### Get Newest Archive for a URL

Source: https://github.com/akamhy/waybackpy/wiki/CLI-docs

Retrieves the most recent archive for a given URL using the --newest flag. Requires a URL and a user agent.

```bash
waybackpy --url "https://en.wikipedia.org/wiki/YouTube" --user_agent "my-unique-user-agent" --newest
```

```bash
Archive URL:
https://web.archive.org/web/20220101184323/https://en.wikipedia.org/wiki/YouTube
```

--------------------------------

### Get Archive Near a Specific Time

Source: https://github.com/akamhy/waybackpy/wiki/CLI-docs

Fetches an archive closest to a specified year, month, day, and hour using the --near flag. Requires a URL, user agent, and time components.

```bash
waybackpy --url google.com --user_agent "my-unique-user-agent" --near --year 2008 --month 8 --day 8 --hour 8
```

```bash
Archive URL:
https://web.archive.org/web/20080808014003/http://www.google.com:80/
```

--------------------------------

### Retrieve Exact Match Archives for a Twitter Profile

Source: https://github.com/akamhy/waybackpy/wiki/Get-all-twitter-posts-by-any-user-|-CDX-API-use-case

Use this snippet to get all archived versions of a specific URL, such as a Twitter profile page. It fetches only exact matches for the provided URL.

```python
from waybackpy import WaybackMachineCDXServerAPI

user_agent = "Mozilla/5.0 (MacBook Air; M1 Mac OS X 11_4) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.1 Safari/604.1"
url = "https://twitter.com/jack"

wayback = WaybackMachineCDXServerAPI(url=url, user_agent=user_agent) # only the exact match

snapshots = wayback.snapshots() # <class 'generator'>

for snapshot in snapshots:
    print(snapshot.archive_url)
```

```bash
waybackpy --url "https://twitter.com/jack" --user-agent "Mozilla/5.0 (MacBook Air; M1 Mac OS X 11_4) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.1 Safari/604.1" --cdx --cdx-print archiveurl
```

--------------------------------

### Retrieve Tweets by User with Date Range

Source: https://github.com/akamhy/waybackpy/wiki/Get-all-twitter-posts-by-any-user-|-CDX-API-use-case

Use this Python snippet to fetch all archived tweets for a given user within a specified start and end timestamp. Requires the waybackpy library and a user agent.

```python
from waybackpy import WaybackMachineCDXServerAPI

user_agent = "Mozilla/5.0 (MacBook Air; M1 Mac OS X 11_4) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.1 Safari/604.1"
url = "https://twitter.com/jack"

wayback = WaybackMachineCDXServerAPI(url=url, user_agent=user_agent, match_type="prefix", collapses=["urlkey"], start_timestamp="2010", end_timestamp="2014") 
#  timeframe bound prefix matching enabled along with active urlkey based collapsing

snapshots = wayback.snapshots() # <class 'generator'>

for snapshot in snapshots:
    print(snapshot.archive_url)
```

--------------------------------

### Initialize CDXServerAPI

Source: https://github.com/akamhy/waybackpy/blob/master/README.md

Initialize the WaybackMachineCDXServerAPI for querying CDX data. Requires the URL and a user agent string.

```python
>>> from waybackpy import WaybackMachineCDXServerAPI
>>> url = "https://google.com"
>>> user_agent = "my new app's user agent"
>>> cdx_api = WaybackMachineCDXServerAPI(url, user_agent)
```

--------------------------------

### Basic CDX Server API Usage

Source: https://github.com/akamhy/waybackpy/wiki/Python-package-docs

Fetches and prints all archives for a given GitHub repository prefix. Requires a user agent and the URL of the repository.

```python
from waybackpy import WaybackMachineCDXServerAPI
url = "https://github.com/akamhy/*"
user_agent = "Your-user-agent"

cdx = WaybackMachineCDXServerAPI(url=url, user_agent=user_agent)
snapshots = cdx.snapshots()

for snapshot in snapshots:
    print(snapshot)
```

```text
com,github)/akamhy/antispam 20210113054521 https://github.com/akamhy/antispam text/html 404 DOVRV3NM56PCPIQ2IH2RUINLRDDFXXZO 17318
com,github)/akamhy/dhashpy 20211001180207 https://github.com/akamhy/dhashpy text/html 200 56W6EQISXHZ4PXBCRN7G7ZGWPV2YEMQG 37087
com,github)/akamhy/dhashpy/code_menu_contents/main 20211001180209 https://github.com/akamhy/dhashpy/code_menu_contents/main text/html 200 TC4HVXHKIJIZVWOUIT2NP6M6CE7KGB4S 4929
.
. # Many URLs redacted for readability
.
com,github)/akamhy/waybackpy/workflows/ci/badge.svg 20211029162652 https://github.com/akamhy/waybackpy/workflows/CI/badge.svg warc/revisit - W3FRZ5W6JL4BXITTRZNIKH4YD5D7XCIA 1372
com,github)/akamhy/waybackpy/workflows/ci/badge.svg 20211208051152 https://github.com/akamhy/waybackpy/workflows/CI/badge.svg warc/revisit - W3FRZ5W6JL4BXITTRZNIKH4YD5D7XCIA 1371
com,github)/akamhy/youtubereviewbot 20200909031810 https://github.com/akamhy/YouTubeReviewBot text/html 404 K6CZ667YBWFMRSN3OKTPDD4HQS66TM3U 89082
```

--------------------------------

### Initialize WaybackMachineAvailabilityAPI

Source: https://github.com/akamhy/waybackpy/blob/master/README.md

Initialize the WaybackMachineAvailabilityAPI for checking URL availability. Note that the CDXServerAPI is generally recommended over this API due to performance.

```python
>>> from waybackpy import WaybackMachineAvailabilityAPI
>>> 
>>> url = "https://google.com"
>>> user_agent = "Mozilla/5.0 (Windows NT 5.1; rv:40.0) Gecko/20100101 Firefox/40.0"
>>> 
>>> availability_api = WaybackMachineAvailabilityAPI(url, user_agent)
```

--------------------------------

### Save Webpage with Headers

Source: https://github.com/akamhy/waybackpy/wiki/CLI-docs

Demonstrates how to retrieve response headers when saving a webpage. Use the `--headers` flag to include headers in the output.

```bash
waybackpy --url "https://en.wikipedia.org/wiki/Social_media" --user_agent "my-unique-user-agent" --save --headers
```

--------------------------------

### CDX Server API Basic Usage

Source: https://github.com/akamhy/waybackpy/wiki/CLI-docs

Queries the CDX server API to retrieve archives for a given URL prefix. Uses the --cdx flag and requires a URL and user agent.

```bash
waybackpy --url "https://github.com/akamhy/*" --user-agent "Your-user-agent" --cdx
```

```bash
com,github)/akamhy/akamhy/waybackpy 20220210225324 https://github.com/akamhy/akamhy/waybackpy text/html 404 7NTMXPAOO2NTAH3EDOYQOGQBBS7YTZVM 113680
com,github)/akamhy/antispam 20210113054521 https://github.com/akamhy/antispam text/html 404 DOVRV3NM56PCPIQ2IH2RUINLRDDFXXZO 17318
com,github)/akamhy/dhashpy 20211001180207 https://github.com/akamhy/dhashpy text/html 200 56W6EQISXHZ4PXBCRN7G7ZGWPV2YEMQG 37087
.
. # Many URLs redacted for readability
.
com,github)/akamhy/waybackpy/workflows/tests/badge.svg 20220310220909 https://github.com/akamhy/waybackpy/workflows/Tests/badge.svg image/svg+xml 200 YQ7L3MX5WXNUY4BZIL4INNDVZF4JXZXJ 2459
com,github)/akamhy/waybackpy/workflows/tests/badge.svg 20220315150044 https://github.com/akamhy/waybackpy/workflows/Tests/badge.svg warc/revisit - YQ7L3MX5WXNUY4BZIL4INNDVZF4JXZXJ 1375
com,github)/akamhy/waybackpy/workflows/tests/badge.svg 20220315194257 https://github.com/akamhy/waybackpy/workflows/Tests/badge.svg warc/revisit - YQ7L3MX5WXNUY4BZIL4INNDVZF4JXZXJ 1374
```

--------------------------------

### Fetch All Known URLs for a Domain

Source: https://github.com/akamhy/waybackpy/wiki/CLI-docs

Retrieves all URLs known by the Wayback Machine for a given domain. Use the --subdomain flag to include subdomains. Results are saved to a file in the current directory.

```bash
waybackpy --url akamhy.github.io --user_agent "my-user-agent" --known_urls
# Prints all known URLs under akamhy.github.io
```

```bash
waybackpy --url akamhy.github.io --user_agent "my-user-agent" --known_urls --subdomain
# Prints all known URLs under akamhy.github.io including subdomain
```

--------------------------------

### Format Code with black

Source: https://github.com/akamhy/waybackpy/blob/master/CONTRIBUTING.md

Apply the 'black' code style formatter to the entire codebase.

```bash
black .
```

--------------------------------

### Find nearest availability using WaybackMachineAvailabilityAPI

Source: https://github.com/akamhy/waybackpy/blob/master/README.md

Find the URL of a snapshot nearest to a specified date and time using the WaybackMachineAvailabilityAPI. This method is also subject to performance considerations.

```python
>>> availability_api.near(year=2010, month=10, day=10, hour=10)
https://web.archive.org/web/20101010101708/http://www.google.com/
```

--------------------------------

### Collapse by URL Key (Prefix Match)

Source: https://github.com/akamhy/waybackpy/wiki/CLI-docs

Collapse search results to show only unique URLs using a prefix match. This filters out captures except for the first one encountered for a given URL key.

```bash
waybackpy --url archive.org --user-agent "i'm-user-agent" --cdx --match-type prefix --collapse urlkey --cdx-print archiveurl
```

--------------------------------

### Type Checking with mypy

Source: https://github.com/akamhy/waybackpy/blob/master/CONTRIBUTING.md

Perform static type checking on the waybackpy and tests directories using mypy.

```bash
mypy -p waybackpy -p tests
```

--------------------------------

### Save a Webpage using waybackpy CLI

Source: https://github.com/akamhy/waybackpy/wiki/CLI-docs

Save a given URL to the Wayback Machine using the CLI. The `--save` flag initiates the save operation. You can also specify a custom user agent with `--user_agent`.

```bash
waybackpy --url "https://en.wikipedia.org/wiki/Social_media" --user_agent "my-unique-user-agent" --save
```

--------------------------------

### Collapse CDX Snapshots by URL Key (Prefix Match)

Source: https://github.com/akamhy/waybackpy/wiki/Python-package-docs

Collapse CDX results to show only the first capture for each unique URL when using a prefix match type. This filters out subsequent captures of the same URL.

```python
from waybackpy import Cdx
url = "archive.org"
user_agent = "Your-apps-user-agent"

cdx = Cdx(url=url, user_agent=user_agent, collapses=["urlkey"], match_type="prefix")
snapshots = cdx.snapshots()

for snapshot in snapshots:
    print(snapshot.archive_url)
```

--------------------------------

### Save a Web Page Archive with Headers

Source: https://github.com/akamhy/waybackpy/wiki/CLI-docs

Saves a web page to the Wayback Machine and displays the archive URL along with the HTTP headers from the save operation. This is useful for verifying save details and debugging.

```bash
waybackpy --url "https://en.wikipedia.org/wiki/Social_media" --user_agent "my-unique-user-agent" --save --headers
```

```text
Archive URL:
https://web.archive.org/web/20220102104608/https://en.wikipedia.org/wiki/Social_media
Cached save:
True
Save API headers:
{'Server': 'nginx/1.19.10', 'Date': 'Sun, 02 Jan 2022 10:54:09 GMT', 'Content-Type': 'text/html; charset=UTF-8', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'x-archive-orig-date': 'Sun, 02 Jan 2022 10:46:06 GMT', 'x-archive-orig-server': 'mw1385.eqiad.wmnet', 'x-archive-orig-x-content-type-options': 'nosniff', 'x-archive-orig-p3p': 'CP="See https://en.wikipedia.org/wiki/Special:CentralAutoLogin/P3P for more info."', 'x-archive-orig-content-language': 'en', 'x-archive-orig-vary': 'Accept-Encoding,Cookie,Authorization', 'x-archive-orig-last-modified': 'Sun, 02 Jan 2022 09:30:45 GMT', 'x-archive-orig-content-encoding': 'gzip', 'x-archive-orig-age': '2', 'x-archive-orig-x-cache': 'cp4030 miss, cp4027 hit/1', 'x-archive-orig-x-cache-status': 'hit-front', 'x-archive-orig-server-timing': 'cache;desc="hit-front", host;desc="cp4027"', 'x-archive-orig-strict-transport-security': 'max-age=106384710; includeSubDomains; preload', 'x-archive-orig-report-to': '{ "group": "wm_nel", "max_age": 86400, "endpoints": [{ "url": "https://intake-logging.wikimedia.org/v1/events?stream=w3c.reportingapi.network_error&schema_uri=/w3c/reportingapi/network_error/1.0.0" }] }', 'x-archive-orig-nel': '{ "report_to": "wm_nel", "max_age": 86400, "failure_fraction": 0.05, "success_fraction": 0.0}', 'x-archive-orig-permissions-policy': 'interest-cohort=()', 'x-archive-orig-x-client-ip': '207.241.232.35', 'x-archive-orig-cache-control': 'private, s-maxage=0, max-age=0, must-revalidate', 'x-archive-orig-accept-ranges': 'bytes', 'x-archive-orig-content-length': '164995', 'x-archive-orig-connection': 'keep-alive', 'x-archive-guessed-content-type': 'text/html', 'x-archive-guessed-charset': 'utf-8', 'memento-datetime': 'Sun, 02 Jan 2022 10:46:08 GMT', 'link': '<https://en.wikipedia.org/wiki/Social_media>; rel="original", <https://web.archive.org/web/timemap/link/https://en.wikipedia.org/wiki/Social_media>; rel="timemap"; type="application/link-format", <https://web.archive.org/web/https://en.wikipedia.org/wiki/Social_media>; rel="timegate", <https://web.archive.org/web/20051215000000/http://en.wikipedia.org/wiki/Social_media>; rel="first memento"; datetime="Thu, 15 Dec 2005 00:00:00 GMT", <https://web.archive.org/web/20220101114012/https://en.wikipedia.org/wiki/Social_media>; rel="prev memento"; datetime="Sat, 01 Jan 2022 11:40:12 GMT", <https://web.archive.org/web/20220102104608/https://en.wikipedia.org/wiki/Social_media>; rel="memento"; datetime="Sun, 02 Jan 2022 10:46:08 GMT", <https://web.archive.org/web/20220102104608/https://en.wikipedia.org/wiki/Social_media>; rel="last memento"; datetime="Sun, 02 Jan 2022 10:46:08 GMT"', 'content-security-policy': "default-src 'self' 'unsafe-eval' 'unsafe-inline' data: blob: archive.org web.archive.org analytics.archive.org pragma.archivelab.org", 'x-archive-src': 'spn2-20220102093111-wwwb-spn10.us.archive.org-8000.warc.gz', 'server-timing': 'captures_list;dur=275.334598, exclusion.robots;dur=0.096415, exclusion.robots.policy;dur=0.088356, RedisCDXSource;dur=1.634125, esindex;dur=0.008082, LoadShardBlock;dur=81.607259, PetaboxLoader3.datanode;dur=51.631773, CDXLines.iter;dur=18.885269, load_resource;dur=19.971806', 'x-app-server': 'wwwb-app204', 'x-ts': '200', 'x-tr': '910', 'X-location': 'All', 'X-Cache-Key': 'httpsweb.archive.org/web/20220102104608/https://en.wikipedia.org/wiki/Social_mediaIN', 'X-RL': '0', 'X-NA': '0', 'X-Page-Cache': 'MISS', 'X-NID': '-', 'Referrer-Policy': 'no-referrer-when-downgrade', 'Permissions-Policy': 'interest-cohort=()', 'Content-Encoding': 'gzip'}
```

--------------------------------

### Run Tests with pytest

Source: https://github.com/akamhy/waybackpy/blob/master/CONTRIBUTING.md

Execute all tests in the tests directory using pytest.

```bash
pytest
```

--------------------------------

### Collapse by Timestamp

Source: https://github.com/akamhy/waybackpy/wiki/CLI-docs

Collapse search results to show at most one capture per hour by comparing the first 10 digits of the timestamp. Adjacent captures with matching prefixes are filtered out.

```bash
waybackpy --url "google.com" --user-agent "Your-apps-user-agent" --cdx --collapse "timestamp:10"
```

--------------------------------

### Collapse CDX Snapshots by Timestamp

Source: https://github.com/akamhy/waybackpy/wiki/Python-package-docs

Collapse CDX results to show at most one capture per hour by comparing the first 10 digits of the timestamp. Useful for reducing dense capture data.

```python
from waybackpy import Cdx
url = "google.com"
user_agent = "Your-apps-user-agent"

cdx = Cdx(url=url, user_agent=user_agent, collapses=["timestamp:10"])
snapshots = cdx.snapshots()

for snapshot in snapshots:
    print(snapshot.archive_url)
```

--------------------------------

### CDX Server API URL Match Scope

Source: https://github.com/akamhy/waybackpy/wiki/CLI-docs

Specifies the matching scope for CDX server API queries using the --match-type parameter. Options include 'exact', 'prefix', 'host', and 'domain'.

```bash
waybackpy --url "archive.org/about/" --user-agent "your-user-agent" --cdx --match-type "prefix" --cdx-print "archiveurl"
```

--------------------------------

### Save a URL using WaybackMachineSaveAPI

Source: https://github.com/akamhy/waybackpy/blob/master/README.md

Use WaybackMachineSaveAPI to save a given URL to the Wayback Machine. This snippet shows how to initialize the API, call the save method, and access the result and timestamp.

```python
>>> from waybackpy import WaybackMachineSaveAPI
>>> url = "https://github.com"
>>> user_agent = "Mozilla/5.0 (Windows NT 5.1; rv:40.0) Gecko/20100101 Firefox/40.0"
>>> 
>>> save_api = WaybackMachineSaveAPI(url, user_agent)
>>> save_api.save()
https://web.archive.org/web/20220118125249/https://github.com/
>>> save_api.cached_save
False
>>> save_api.timestamp()
datetime.datetime(2022, 1, 18, 12, 52, 49)
```

--------------------------------

### Retrieve the Oldest Archive of a Web Page

Source: https://github.com/akamhy/waybackpy/wiki/CLI-docs

Fetches the URL of the oldest available archive for a given web page using the `--oldest` flag. This is useful for historical research or when needing to access the earliest recorded version of a page.

```bash
waybackpy --url "https://en.wikipedia.org/wiki/SpaceX" --user_agent "my-unique-user-agent" --oldest
```

```text
Archive URL:
https://web.archive.org/web/20040803000845/http://en.wikipedia.org:80/wiki/SpaceX
```

--------------------------------

### Find snapshot near a specific date using CDXServerAPI

Source: https://github.com/akamhy/waybackpy/blob/master/README.md

Find a snapshot closest to a specified date and time using the CDXServerAPI. You can provide date components or a Unix timestamp.

```python
>>> near = cdx_api.near(year=2010, month=10, day=10, hour=10, minute=10)
>>> near.archive_url
'https://web.archive.org/web/20101010101435/http://google.com/'
>>> near
com,google)/ 20101010101435 http://google.com/ text/html 301 Y6PVK4XWOI3BXQEXM5WLLWU5JKUVNSFZ 391
>>> near.timestamp
'20101010101435'
>>> near.timestamp
'20101010101435'
>>> near = cdx_api.near(wayback_machine_timestamp=2008080808)
>>> near.archive_url
'https://web.archive.org/web/20080808051143/http://google.com/'
>>> near = cdx_api.near(unix_timestamp=1286705410)
>>> near
com,google)/ 20101010101435 http://google.com/ text/html 301 Y6PVK4XWOI3BXQEXM5WLLWU5JKUVNSFZ 391
>>> near.archive_url
'https://web.archive.org/web/20101010101435/http://google.com/'
```

--------------------------------

### Archive a Webpage with Waybackpy

Source: https://github.com/akamhy/waybackpy/wiki/Python-package-docs

Use the WaybackMachineSaveAPI to save a given URL to the Internet Archive. This requires a user agent string. The result includes the archive URL and metadata.

```python
>>> import waybackpy
>>> 
>>> url = "https://en.wikipedia.org/wiki/Multivariable_calculus"
>>> user_agent = "Mozilla/5.0 (Windows NT 5.1; rv:40.0) Gecko/20100101 Firefox/40.0"
>>> 
>>> save_api = waybackpy.WaybackMachineSaveAPI(url, user_agent=user_agent)
>>> save_api.save()
'https://web.archive.org/web/20220122102014/https://en.wikipedia.org/wiki/Multivariable_calculus'
>>> save_api.cached_save
False
>>> save_api.headers
{'Server': 'nginx/1.19.5', 'Date': 'Sat, 22 Jan 2022 10:20:19 GMT', 'Content-Type': 'text/html; charset=UTF-8', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'x-archive-orig-date': 'Fri, 21 Jan 2022 23:32:39 GMT', 'x-archive-orig-server': 'mw1407.eqiad.wmnet', 'x-archive-orig-x-content-type-options': 'nosniff', 'x-archive-orig-p3p': 'CP="See https://en.wikipedia.org/wiki/Special:CentralAutoLogin/P3P for more info."', 'x-archive-orig-content-language': 'en', 'x-archive-orig-vary': 'Accept-Encoding,Cookie,Authorization', 'x-archive-orig-last-modified': 'Fri, 21 Jan 2022 23:16:22 GMT', 'x-archive-orig-content-encoding': 'gzip', 'x-archive-orig-age': '38855', 'x-archive-orig-x-cache': 'cp4027 miss, cp4030 hit/2', 'x-archive-orig-x-cache-status': 'hit-front', 'x-archive-orig-server-timing': 'cache;desc="hit-front", host;desc="cp4030"', 'x-archive-orig-strict-transport-security': 'max-age=106384710; includeSubDomains; preload', 'x-archive-orig-report-to': '{ "group": "wm_nel", "max_age": 86400, "endpoints": [{ "url": "https://intake-logging.wikimedia.org/v1/events?stream=w3c.reportingapi.network_error&schema_uri=/w3c/reportingapi/network_error/1.0.0" }] }', 'x-archive-orig-nel': '{ "report_to": "wm_nel", "max_age": 86400, "failure_fraction": 0.05, "success_fraction": 0.0}', 'x-archive-orig-permissions-policy': 'interest-cohort=()', 'x-archive-orig-set-cookie': 'WMF-Last-Access=22-Jan-2022;Path=/;HttpOnly;secure;Expires=Wed, 23 Feb 2022 00:00:00 GMT, WMF-Last-Access-Global=22-Jan-2022;Path=/;Domain=.wikipedia.org;HttpOnly;secure;Expires=Wed, 23 Feb 2022 00:00:00 GMT, GeoIP=US:CA:San_Francisco:37.78:-122.47:v4; Path=/; secure; Domain=.wikipedia.org', 'x-archive-orig-x-client-ip': '207.241.227.105', 'x-archive-orig-cache-control': 'private, s-maxage=0, max-age=0, must-revalidate', 'x-archive-orig-accept-ranges': 'bytes', 'x-archive-orig-content-length': '28504', 'x-archive-orig-connection': 'keep-alive', 'x-archive-guessed-content-type': 'text/html', 'x-archive-guessed-charset': 'utf-8', 'memento-datetime': 'Sat, 22 Jan 2022 10:20:14 GMT', 'link': '<https://en.wikipedia.org/wiki/Multivariable_calculus>; rel="original", <https://web.archive.org/web/timemap/link/https://en.wikipedia.org/wiki/Multivariable_calculus>; rel="timemap"; type="application/link-format", <https://web.archive.org/web/https://en.wikipedia.org/wiki/Multivariable_calculus>; rel="timegate", <https://web.archive.org/web/20050422130129/http://en.wikipedia.org:80/wiki/Multivariable_calculus>; rel="first memento"; datetime="Fri, 22 Apr 2005 13:01:29 GMT", <https://web.archive.org/web/20220118154923/https://en.wikipedia.org/wiki/Multivariable_calculus>; rel="prev memento"; datetime="Tue, 18 Jan 2022 15:49:23 GMT", <https://web.archive.org/web/20220122102014/https://en.wikipedia.org/wiki/Multivariable_calculus>; rel="memento"; datetime="Sat, 22 Jan 2022 10:20:14 GMT", <https://web.archive.org/web/20220122102014/https://en.wikipedia.org/wiki/Multivariable_calculus>; rel="last memento"; datetime="Sat, 22 Jan 2022 10:20:14 GMT"', 'content-security-policy': "default-src 'self' 'unsafe-eval' 'unsafe-inline' data: blob: archive.org web.archive.org analytics.archive.org pragma.archivelab.org", 'x-archive-src': 'spn2-20220122093153-wwwb-spn23.us.archive.org-8000.warc.gz', 'server-timing': 'captures_list;dur=138.943024, exclusion.robots;dur=0.124457, exclusion.robots.policy;dur=0.114278, cdx.remote;dur=0.091306, esindex;dur=0.011012, LoadShardBlock;dur=101.247564, PetaboxLoader3.datanode;dur=44.420167, CDXLines.iter;dur=25.235685, PetaboxLoader3.resolve;dur=25.677021, load_resource;dur=4.737038', 'x-app-server': 'wwwb-app201', 'x-ts': '200', 'x-tr': '315', 'X-location': 'All', 'X-Cache-Key': 'httpsweb.archive.org/web/20220122102014/https://en.wikipedia.org/wiki/Multivariable_calculusIN', 'X-RL': '0', 'X-NA': '0', 'X-Page-Cache': 'MISS', 'X-NID': '-', 'Referrer-Policy': 'no-referrer-when-downgrade', 'Permissions-Policy': 'interest-cohort=()', 'Content-Encoding': 'gzip'}
>>> save_api.timestamp()
datetime.datetime(2022, 1, 22, 10, 20, 14)
>>> save_api.archive_url
'https://web.archive.org/web/20220122102014/https://en.wikipedia.org/wiki/Multivariable_calculus'
```

--------------------------------

### Retrieve Oldest Archive for a URL

Source: https://github.com/akamhy/waybackpy/wiki/Python-package-docs

Use the `oldest()` method to find the earliest archived version of a webpage. Requires a user agent.

```python
>>> from waybackpy import WaybackMachineAvailabilityAPI
>>> url = "https://www.google.com"
>>> user_agent = "Any-user-agent-you-want"
>>> availability_api = WaybackMachineAvailabilityAPI(url, user_agent)
>>> availability_api.oldest()
https://web.archive.org/web/19981111184551/http://google.com:80/
>>> availability_api.archive_url
'https://web.archive.org/web/19981111184551/http://google.com:80/'
>>> availability_api.json
{'url': 'https://www.google.com', 'archived_snapshots': {'closest': {'status': '200', 'available': True, 'url': 'http://web.archive.org/web/19981111184551/http://google.com:80/', 'timestamp': '19981111184551'}}, 'timestamp': '199401221029'}
>>> availability_api.timestamp()
datetime.datetime(1998, 11, 11, 18, 45, 51)
```

--------------------------------

### Retrieve All Archived Posts for a Twitter User (Prefix Match)

Source: https://github.com/akamhy/waybackpy/wiki/Get-all-twitter-posts-by-any-user-|-CDX-API-use-case

This snippet retrieves all archived snapshots for a given Twitter user by enabling prefix matching. It's useful for gathering all posts, even if the URL structure changes slightly.

```python
from waybackpy import WaybackMachineCDXServerAPI

user_agent = "Mozilla/5.0 (MacBook Air; M1 Mac OS X 11_4) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.1 Safari/604.1"
url = "https://twitter.com/jack"

wayback = WaybackMachineCDXServerAPI(url=url, user_agent=user_agent, match_type="prefix") # prefix match enabled
snapshots = wayback.snapshots() # <class 'generator'>

for snapshot in snapshots:
    print(snapshot.archive_url)
```

```bash
waybackpy --url "https://twitter.com/jack" --user-agent "Mozilla/5.0 (MacBook Air; M1 Mac OS X 11_4) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.1 Safari/604.1" --cdx --cdx-print archiveurl --match-type prefix
```

--------------------------------

### Collapse by Digest

Source: https://github.com/akamhy/waybackpy/wiki/CLI-docs

Collapse search results to show only unique captures based on their digest. Only adjacent captures with the same digest are collapsed.

```bash
waybackpy --url "google.com" --user-agent "Your-apps-user-agent" --cdx --collapse "digest" --cdx-print "archiveurl"
```

--------------------------------

### Retrieve Archive Near Specific Date/Time

Source: https://github.com/akamhy/waybackpy/wiki/Python-package-docs

Use the `near()` method to find an archive close to a specified year, month, day, hour, and minute, or a UNIX timestamp. Requires a user agent for date/time specification.

```python
>>> from waybackpy import WaybackMachineAvailabilityAPI
>>> url = "https://www.facebook.com/zuck"
>>> user_agent = "YOUR USER AGENT"
>>> availability_api = WaybackMachineAvailabilityAPI(url, user_agent=user_agent)
>>> availability_api.near(year=2012, month=10, day=29, hour=12, minute=16)
https://web.archive.org/web/20121029122242/https://www.facebook.com/zuck
>>> availability_api.json
{'url': 'https://www.facebook.com/zuck', 'archived_snapshots': {'closest': {'status': '200', 'available': True, 'url': 'http://web.archive.org/web/20121029122242/https://www.facebook.com/zuck', 'timestamp': '20121029122242'}}, 'timestamp': '201210291216'}
>>> availability_api.timestamp()
datetime.datetime(2012, 10, 29, 12, 22, 42)
```

```python
>>> import waybackpy
>>> url = "https://www.google.com" 
>>> unix_time = 1200144258 # you can pass str, int or float.
>>> availability_api = waybackpy.WaybackMachineAvailabilityAPI(url)
>>> availability_api.near(unix_timestamp=unix_time)
https://web.archive.org/web/20080114115458/http://www.google.com/
>>> availability_api.archive_url
'https://web.archive.org/web/20080114115458/http://www.google.com/'
>>> availability_api.timestamp()
datetime.datetime(2008, 1, 14, 11, 54, 58)
>>> availability_api.json
{'url': 'https://www.google.com', 'archived_snapshots': {'closest': {'status': '200', 'available': True, 'url': 'http://web.archive.org/web/20080114115458/http://www.google.com/', 'timestamp': '20080114115458'}}, 'timestamp': '20080112132418'}
```

--------------------------------

### List Known Archived URLs for a Domain

Source: https://github.com/akamhy/waybackpy/wiki/Python-package-docs

Use the `known_urls()` method to retrieve a list of all archived URLs for a given domain. Set `subdomain=True` to include subdomains. This method is built on the CDX API.

```python
>>> import waybackpy
>>> user_agent = "This is an example user agent"
>>> url = "pypi.org"
>>> wayback = waybackpy.Url(url=url, user_agent=user_agent)
>>> known_urls = wayback.known_urls(subdomain=False)
>>> for url in known_urls:
...     print(url)
... 
https://pypi.org/project/coralillo/1.0.0/
https://pypi.org/project/coralillo/1.0.1/
https://pypi.org/project/coraline-eda/
https://pypi.org/project/coralinedb/
https://pypi.org/project/coralogix/
https://pypi.org/project/coralogix/0.2.5.10/
https://pypi.org/project/coralogix/0.2.6.0/
. 
. # Millions of other URLs redacted from the output for readability and size limit of this GitHub wiki page
. 
https://pypi.org/project/coralogix/0.2.6.5/
https://pypi.org/project/coralogix/0.2.6.6/
https://pypi.org/project/corappo/
https://pypi.org/project/coras/
https://pypi.org/project/corax/
https://pypi.org/project/cord-19-tools/
https://pypi.org/project/cord-robot/
https://pypi.org/project/cord-workflow-controller-client/
https://pypi.org/project/corda/
```