Wayback Tweets

https://waybacktweets.claromes.com
Admin
Retrieves archived tweets CDX data from the Wayback Machine, performs necessary parsing, and saves...
Tokens:7,748
Snippets:138
Trust Score:5.6
Update:3 months ago
Show doc for...
Tokens:
Raw
### Install Poetry

Source: https://waybacktweets.claromes.com/installation.html

Install Poetry if it is not already installed, required for managing dependencies from source.

```bash
pip3 install poetry
```

--------------------------------

### Example: Detailed Query with Options

Source: https://waybacktweets.claromes.com/cli.html

Shows a comprehensive example of using multiple options to filter and control the output, including date range, limit, and verbose logging.

```bash
waybacktweets --from 20200305 --to 20231231 --limit 300 --verbose jack

```

--------------------------------

### Install with pip

Source: https://waybacktweets.claromes.com/installation.html

Install Wayback Tweets using pip, the standard Python package installer.

```bash
pip3 install waybacktweets
```

--------------------------------

### Example: Short-Option Query

Source: https://waybacktweets.claromes.com/cli.html

Illustrates the use of short-form options for the same parameters as the detailed query example.

```bash
waybacktweets -f 20200305 -t 20231231 -l 300 -v jack

```

--------------------------------

### Install Dependencies with Poetry

Source: https://waybacktweets.claromes.com/installation.html

Install all project dependencies using Poetry after cloning the repository.

```bash
poetry install
```

--------------------------------

### Install Pre-commit Hooks

Source: https://waybacktweets.claromes.com/installation.html

Install pre-commit hooks for development workflows using Poetry.

```bash
poetry run pre-commit install
```

--------------------------------

### Install with pipx

Source: https://waybacktweets.claromes.com/installation.html

Use pipx for a clean, isolated installation of Wayback Tweets.

```bash
pipx install waybacktweets
```

--------------------------------

### Example: Simple Username Query

Source: https://waybacktweets.claromes.com/cli.html

Demonstrates a basic query using only the required username argument.

```bash
waybacktweets jack

```

--------------------------------

### CDX API Example: Collapse by URL Key with Prefix Match

Source: https://waybacktweets.claromes.com/cli.html

An example from the Wayback CDX Server API documentation demonstrating collapsing by URL key in conjunction with a prefix match type to show unique URLs.

```http
http://web.archive.org/cdx/search/cdx?url=archive.org&collapse=urlkey&matchType=prefix

```

--------------------------------

### CDX API Example: Collapse by Digest

Source: https://waybacktweets.claromes.com/cli.html

An example from the Wayback CDX Server API documentation showing how to collapse results by the SHA1 hash digest to find unique captures.

```http
http://web.archive.org/cdx/search/cdx?url=archive.org&collapse=digest

```

--------------------------------

### Run Streamlit App

Source: https://waybacktweets.claromes.com/installation.html

Start the Streamlit application after activating the Poetry shell.

```bash
streamlit run app/app.py
```

--------------------------------

### CDX API Example: Collapse by Timestamp

Source: https://waybacktweets.claromes.com/cli.html

An example from the Wayback CDX Server API documentation demonstrating how to collapse results based on a substring of the timestamp field, showing at most one capture per hour.

```http
http://web.archive.org/cdx/search/cdx?url=google.com&collapse=timestamp:10

```

--------------------------------

### Clone Repository

Source: https://waybacktweets.claromes.com/installation.html

Clone the Wayback Tweets repository from GitHub to install from source.

```bash
git clone git@github.com:claromes/waybacktweets.git
```

--------------------------------

### get_response

Source: https://waybacktweets.claromes.com/api.html

Sends a GET request to the specified URL and returns the response.

```APIDOC
## get_response

### Description
Sends a GET request to the specified URL and returns the response.

### Parameters
#### Query Parameters
- **url** (str) - Required - The URL to send the GET request to.
- **params** (dict) - Optional - The parameters to include in the GET request.

### Returns
- **Tuple[Optional[Response], Optional[str], Optional[str]]** - The response from the server.

### Raises
- **ReadTimeoutError** - If a read timeout occurs.
- **ConnectionError** - If a connection error occurs.
- **HTTPError** - If an HTTP error occurs.
- **EmptyResponseError** - If the response is empty.
```

--------------------------------

### Send GET Request to CDX API

Source: https://waybacktweets.claromes.com/api.html

Use the get() method on a WaybackTweets instance to send a GET request to the Internet Archive's CDX API. This retrieves archived tweets in JSON format.

```python
response = wayback_tweets.get()
```

--------------------------------

### Build Documentation

Source: https://waybacktweets.claromes.com/installation.html

Clean previous builds and generate HTML documentation using make.

```bash
make clean html
```

--------------------------------

### Navigate to Docs Directory

Source: https://waybacktweets.claromes.com/installation.html

Change directory to the docs folder to build documentation.

```bash
cd docs
```

--------------------------------

### Run CLI with Poetry

Source: https://waybacktweets.claromes.com/installation.html

Execute the Wayback Tweets command-line interface using Poetry.

```bash
poetry run waybacktweets [OPTIONS] USERNAME
```

--------------------------------

### Navigate to Project Directory

Source: https://waybacktweets.claromes.com/installation.html

Change your current directory to the cloned Wayback Tweets repository.

```bash
cd waybacktweets
```

--------------------------------

### Activate Poetry Shell

Source: https://waybacktweets.claromes.com/installation.html

Activate the virtual environment managed by Poetry to run applications.

```bash
poetry shell
```

--------------------------------

### Basic CLI Usage

Source: https://waybacktweets.claromes.com/cli.html

The fundamental command structure for the waybacktweets CLI. Requires a username argument.

```bash
waybacktweets [OPTIONS] USERNAME

```

--------------------------------

### Initialize WaybackTweets Request

Source: https://waybacktweets.claromes.com/api.html

Instantiate the WaybackTweets class to prepare for requesting data from the Wayback CDX Server API. Specify parameters like username, date range, and limits for your query.

```python
from waybacktweets.api import WaybackTweets

wayback_tweets = WaybackTweets(username="your_username",
                             timestamp_from="20230101",
                             timestamp_to="20231231",
                             limit=100)
```

--------------------------------

### Add with Poetry

Source: https://waybacktweets.claromes.com/installation.html

Add Wayback Tweets as a dependency to your Poetry-managed project.

```bash
poetry add waybacktweets
```

--------------------------------

### Initialize TweetsParser

Source: https://waybacktweets.claromes.com/api.html

Create a TweetsParser instance to process the raw response from the archived tweets API. Provide the response, username, and desired field options.

```python
from waybacktweets.api import TweetsParser

tweets_parser = TweetsParser(archived_tweets_response=response,
                             username="your_username",
                             field_options=["created_at", "text"])
```

--------------------------------

### Initialize JsonParser for JSON Mimetypes

Source: https://waybacktweets.claromes.com/api.html

Create a JsonParser instance to parse tweets when the mimetype is application/json. This class is experimental.

```python
from waybacktweets.api import JsonParser

json_parser = JsonParser(archived_tweet_url="http://web.archive.org/web/20230101000000*/https://example.com/tweet.json")
```

--------------------------------

### Initialize TwitterEmbed for Tweet Parsing

Source: https://waybacktweets.claromes.com/api.html

Instantiate TwitterEmbed with a tweet URL to parse tweets using the Twitter Publish service, checking for availability.

```python
from waybacktweets.api import TwitterEmbed

twitter_embed = TwitterEmbed(tweet_url="https://twitter.com/user/status/123456789")
```

--------------------------------

### Run Wayback Tweets from the Command Line

Source: https://waybacktweets.claromes.com/quickstart.html

Execute Wayback Tweets as a standalone command-line tool to fetch tweets within a specified date range and limit.

```bash
waybacktweets --from 20150101 --to 20191231 --limit 250 jack

```

--------------------------------

### create_dataframe()

Source: https://waybacktweets.claromes.com/api.html

Creates a pandas DataFrame from the transposed tweet data. This is a preparatory step before exporting.

```APIDOC
## Method: create_dataframe

### Description
Creates a DataFrame from the transposed data.

### Return Type
DataFrame

### Returns
The DataFrame representation of the data.
```

--------------------------------

### HTMLTweetsVisualizer.json_loader

Source: https://waybacktweets.claromes.com/api.html

A static method to load JSON data from a specified file path or directly from a JSON string.

```APIDOC
## Static Method HTMLTweetsVisualizer.json_loader

### Description
Reads and loads JSON data from a specified file path or JSON string.

### Parameters
* **json_path** (Union[str, List[str]]) - The path of the JSON file or the JSON data itself.

### Returns
* List[Dict[str, Any]] - The content of the JSON file or data.
```

--------------------------------

### check_url_scheme

Source: https://waybacktweets.claromes.com/api.html

Corrects the URL scheme if it contains more than two slashes following the scheme.

```APIDOC
## check_url_scheme

### Description
Corrects the URL scheme if it contains more than two slashes following the scheme. This function uses a regular expression to find ‘http:’ or ‘https:’ followed by two or more slashes. It then replaces this with the scheme followed by exactly two slashes.

### Parameters
#### Path Parameters
- **url** (str) - Required - The URL to be corrected.

### Returns
- **str** - The corrected URL.
```

--------------------------------

### HTMLTweetsVisualizer.save

Source: https://waybacktweets.claromes.com/api.html

Saves the generated HTML content to a specified file.

```APIDOC
## Method HTMLTweetsVisualizer.save

### Description
Saves the generated HTML string to a file.

### Parameters
* **html_content** (str) - The HTML string to be saved.

### Returns
* None
```

--------------------------------

### HTMLTweetsVisualizer Class

Source: https://waybacktweets.claromes.com/api.html

The HTMLTweetsVisualizer class is used to generate and save HTML files for visualizing tweet data. It takes user information, JSON data paths, and an optional output HTML file path as input.

```APIDOC
## Class HTMLTweetsVisualizer

### Description
Class responsible for generating an HTML file to visualize the parsed data.

### Parameters
* **username** (str) - The username associated with the tweets.
* **json_path** (Union[str, List[str]]) - The path of the JSON file or the JSON data itself.
* **html_file_path** (str, optional) - The path where the HTML file will be saved.
```

--------------------------------

### HTMLTweetsVisualizer.generate

Source: https://waybacktweets.claromes.com/api.html

Generates an HTML string representation of the parsed tweet data.

```APIDOC
## Method HTMLTweetsVisualizer.generate

### Description
Generates an HTML string that represents the parsed data.

### Returns
* str - The generated HTML string.
```

--------------------------------

### Use Wayback Tweets as a Python Module

Source: https://waybacktweets.claromes.com/quickstart.html

Integrate Wayback Tweets into your Python projects to retrieve, parse, and export archived tweets. Requires importing necessary classes.

```python
from waybacktweets import WaybackTweets, TweetsParser, TweetsExporter

USERNAME = "jack"

api = WaybackTweets(USERNAME)
archived_tweets = api.get()

if archived_tweets:
    field_options = [
        "archived_urlkey",
        "archived_timestamp",
        "parsed_archived_timestamp",
        "archived_tweet_url",
        "parsed_archived_tweet_url",
        "original_tweet_url",
        "parsed_tweet_url",
        "available_tweet_text",
        "available_tweet_is_RT",
        "available_tweet_info",
        "archived_mimetype",
        "archived_statuscode",
        "archived_digest",
        "archived_length",
        "resumption_key",
    ]

    parser = TweetsParser(archived_tweets, USERNAME, field_options)
    parsed_tweets = parser.parse()

    exporter = TweetsExporter(parsed_tweets, USERNAME, field_options)
    exporter.save_to_csv()
    exporter.save_to_json()
    exporter.save_to_html()


```

--------------------------------

### generate_json()

Source: https://waybacktweets.claromes.com/api.html

Generates a JSON-formatted string from the DataFrame. The data is not saved to a file by this method.

```APIDOC
## Method: generate_json

### Description
Generates JSON data from the DataFrame (without saving to a file).

### Return Type
str

### Returns
The JSON-formatted string of the DataFrame.
```

--------------------------------

### timestamp_parser

Source: https://waybacktweets.claromes.com/api.html

Parses a timestamp into a formatted string.

```APIDOC
## timestamp_parser

### Description
Parses a timestamp into a formatted string.

### Parameters
#### Path Parameters
- **timestamp** (str) - Required - The timestamp string to parse.

### Returns
- **Optional[str]** - Returns the parsed timestamp in strftime format, or None if parsing fails.
```

--------------------------------

### save_to_html()

Source: https://waybacktweets.claromes.com/api.html

Saves the current DataFrame content to an HTML file.

```APIDOC
## Method: save_to_html

### Description
Saves the DataFrame to an HTML file.

### Return Type
None
```

--------------------------------

### Add Resumption Key

Source: https://waybacktweets.claromes.com/api.html

Call add_resumption_key() to extract and append the resumption key from the last tweet response. This is used for continuing queries.

```python
tweets_parser.add_resumption_key()
```

--------------------------------

### save_to_csv()

Source: https://waybacktweets.claromes.com/api.html

Saves the current DataFrame content to a CSV file.

```APIDOC
## Method: save_to_csv

### Description
Saves the DataFrame to a CSV file.

### Return Type
None
```

--------------------------------

### TwitterEmbed Class

Source: https://waybacktweets.claromes.com/api.html

The TwitterEmbed class is designed to parse tweets using the Twitter Publish service, checking for their availability.

```APIDOC
## Class: TwitterEmbed

### Description
This class is responsible for parsing tweets using the Twitter Publish service.

### Parameters
* **tweet_url** (_str_) – The URL of the tweet to be parsed.

## Method: embed()

### Description
Parses the archived tweets when they are still available. This function goes through each archived tweet and checks if it is still available. If the tweet is available, it extracts the necessary information and adds it to the respective lists.

### Return Type
`Optional[Tuple[List[str], List[bool], List[str]]]`

### Returns
A tuple of three lists containing the tweet texts, availability statuses, and URLs, respectively. If no tweets are available, returns None.
```

--------------------------------

### clean_wayback_machine_url

Source: https://waybacktweets.claromes.com/api.html

Cleans a Wayback Machine URL by ensuring it is associated with the correct username and timestamp.

```APIDOC
## clean_wayback_machine_url

### Description
Cleans a Wayback Machine URL by ensuring it is associated with the correct username and timestamp.

### Parameters
#### Path Parameters
- **wayback_machine_url** (str) - Required - The Wayback Machine URL to clean.
- **archived_timestamp** (str) - Required - The timestamp to associate with the Wayback Machine URL.
- **username** (str) - Required - The username to associate with the Wayback Machine URL.

### Returns
- **str** - The cleaned Wayback Machine URL.
```

--------------------------------

### save_to_json()

Source: https://waybacktweets.claromes.com/api.html

Saves the current DataFrame content to a JSON file.

```APIDOC
## Method: save_to_json

### Description
Saves the DataFrame to a JSON file.

### Return Type
None
```

--------------------------------

### semicolon_parser

Source: https://waybacktweets.claromes.com/api.html

Replaces semicolons in a string with %3B.

```APIDOC
## semicolon_parser

### Description
Replaces semicolons in a string with %3B.

### Parameters
#### Path Parameters
- **string** (str) - Required - The string to replace semicolons in.

### Returns
- **str** - The string with semicolons replaced by %3B.
```

--------------------------------

### Process Response Data

Source: https://waybacktweets.claromes.com/api.html

The process_response() method handles the raw archived tweet response, adding relevant CDX data to the parser's internal structure.

```python
tweets_parser.process_response(response)
```

--------------------------------

### JsonParser Class

Source: https://waybacktweets.claromes.com/api.html

The JsonParser class is an experimental class for parsing tweets when the mimetype is application/json.

```APIDOC
## Class: JsonParser

### Description
This class is responsible for parsing tweets when the mimetype is application/json. Note: This class is in an experimental phase.

### Parameters
* **archived_tweet_url** (_str_) – The URL of the archived tweet to be parsed.

## Method: parse()

### Description
Parses the archived tweets in JSON format.

### Return Type
`str`

### Returns
The parsed tweet text.
```

--------------------------------

### Parse Archived Tweets CDX Data

Source: https://waybacktweets.claromes.com/api.html

Call the parse() method on a TweetsParser instance to structure the retrieved CDX data. Optionally, print progress during parsing.

```python
parsed_data = tweets_parser.parse(print_progress=True)
```

--------------------------------

### TweetsParser Class

Source: https://waybacktweets.claromes.com/api.html

The TweetsParser class handles the parsing of archived tweets data obtained from the Wayback CDX Server API. It structures the data and can append specific fields.

```APIDOC
## Class: TweetsParser

### Description
This class is responsible for the overall parsing of archived tweets.

### Parameters
* **archived_tweets_response** (_List[str]_) – The response from the archived tweets.
* **username** (_str_) – The username associated with the tweets.
* **field_options** (_List[str]_) – The fields to be included in the parsed data.

## Method: _add_field(key, value)

### Description
Appends a value to a list in the parsed data structure.

### Parameters
* **key** (_str_) – The key in the parsed data structure.
* **value** (_Any_) – The value to be appended.

### Return Type
`None`

## Method: _add_resumption_key()

### Description
Adds the resumption key from the last archived tweet response to the parsed tweets. This method extracts the resumption key from the last item in the archived tweets response list and appends it to the ‘resumption_key’ field in the parsed tweets dictionary. It also prints the resumption key with instructions on how to use it with the ‘limit’ option for continuing the query from the end of the previous query.

### Raises
**ValueError** – If the list of archived tweet responses is empty.

## Method: _process_response(response)

### Description
Processes the archived tweet’s response and adds the relevant CDX data.

### Parameters
* **response** (_List[str]_) – The response from the archived tweet.

### Return Type
`None`

## Method: parse(print_progress=False)

### Description
Parses the archived tweets CDX data and structures it.

### Parameters
* **print_progress** (_bool_) – A boolean indicating whether to print progress or not.

### Return Type
`Dict[str, List[Any]]`

### Returns
The parsed tweets data.
```

--------------------------------

### check_pattern_tweet

Source: https://waybacktweets.claromes.com/api.html

Extracts the URL from a tweet URL with patterns such as Reply, Link, or Twimg.

```APIDOC
## check_pattern_tweet

### Description
Extracts the URL from a tweet URL with patterns such as:
  * Reply: /status//
  * Link: /status///
  * Twimg: /status/https://pbs

### Parameters
#### Path Parameters
- **tweet_url** (str) - Required - The tweet URL to extract the URL from.

### Returns
- **str** - Only the extracted URL from a tweet.
```

--------------------------------

### WaybackTweets Request Class

Source: https://waybacktweets.claromes.com/api.html

The WaybackTweets class is used to request data from the Wayback CDX Server API. It allows specifying parameters such as username, time range, and limit for retrieving archived tweets.

```APIDOC
## Class: WaybackTweets

### Description
Class responsible for requesting data from the Wayback CDX Server API.

### Parameters
* **username** (_str_) – The username associated with the tweets.
* **collapse** (_str_, optional) – The field to collapse duplicate lines on.
* **timestamp_from** (_str_, optional) – The timestamp to start retrieving tweets from.
* **timestamp_to** (_str_, optional) – The timestamp to stop retrieving tweets at.
* **limit** (_int_, optional) – The maximum number of results to return.
* **resumption_key** (_int_, optional) – Key to continue the query from the end of the previous query.
* **matchtype** (_str_, optional) – Results matching a certain prefix, a certain host or all subdomains.

## Method: get()

### Description
Sends a GET request to the Internet Archive’s CDX API to retrieve archived tweets.

### Return Type
`Optional[Dict[str, Any]]`

### Returns
The response from the CDX API in JSON format, if successful. Otherwise, None.
```

--------------------------------

### check_double_status

Source: https://waybacktweets.claromes.com/api.html

Checks if a Wayback Machine URL contains two occurrences of '/status/' and if the original tweet does not contain 'twitter.com'.

```APIDOC
## check_double_status

### Description
Checks if a Wayback Machine URL contains two occurrences of "/status/" and if the original tweet does not contain "twitter.com".

### Parameters
#### Path Parameters
- **wayback_machine_url** (str) - Required - The Wayback Machine URL to check.
- **original_tweet_url** (str) - Required - The original tweet URL to check.

### Returns
- **bool** - True if the conditions are met, False otherwise.
```

--------------------------------

### is_tweet_url

Source: https://waybacktweets.claromes.com/api.html

Checks if the provided URL is a Twitter status URL.

```APIDOC
## is_tweet_url

### Description
Checks if the provided URL is a Twitter status URL. This function checks if the provided URL contains "/status/" exactly once, which is a common pattern in Twitter status URLs.

### Parameters
#### Path Parameters
- **twitter_url** (str) - Required - The URL to check.

### Returns
- **bool** - True if the URL is a Twitter status URL, False otherwise.
```

--------------------------------

### Embed Archived Tweets

Source: https://waybacktweets.claromes.com/api.html

Use the embed() method of TwitterEmbed to process archived tweets, extracting tweet texts, availability status, and URLs.

```python
result = twitter_embed.embed()
```

--------------------------------

### TweetsExporter Class

Source: https://waybacktweets.claromes.com/api.html

The TweetsExporter class is used to export parsed archived tweets. It allows users to specify the data, associated username, and desired fields for export.

```APIDOC
## Class: TweetsExporter

### Description
Class responsible for exporting parsed archived tweets.

### Parameters
* **data** (Dict[str, List[Any]]) - Required - The parsed archived tweets data.
* **username** (str) - Required - The username associated with the tweets.
* **field_options** (List[str]) - Required - The fields to be included in the exported data. For more details on each option, visit Field Options.
```

--------------------------------

### Add Field to Parsed Data

Source: https://waybacktweets.claromes.com/api.html

Use the add_field() method to append a specific value to a list within the parsed data structure, identified by a key.

```python
tweets_parser.add_field(key="custom_field", value="some_value")
```

--------------------------------

### Parse JSON Tweet Data

Source: https://waybacktweets.claromes.com/api.html

Call the parse() method on a JsonParser instance to retrieve the tweet text from a JSON-formatted archived tweet.

```python
tweet_text = json_parser.parse()
```

--------------------------------

### clean_tweet_url

Source: https://waybacktweets.claromes.com/api.html

Cleans a tweet URL by ensuring it is associated with the correct username.

```APIDOC
## clean_tweet_url

### Description
Cleans a tweet URL by ensuring it is associated with the correct username.

### Parameters
#### Path Parameters
- **tweet_url** (str) - Required - The tweet URL to clean.
- **username** (str) - Required - The username to associate with the tweet URL.

### Returns
- **str** - The cleaned tweet URL.
```

--------------------------------

### delete_tweet_pathnames

Source: https://waybacktweets.claromes.com/api.html

Removes any pathnames from a tweet URL.

```APIDOC
## delete_tweet_pathnames

### Description
Removes any pathnames from a tweet URL.

### Parameters
#### Path Parameters
- **tweet_url** (str) - Required - The tweet URL to remove pathnames from.

### Returns
- **str** - The tweet URL without any pathnames.
```
Tokens:
Raw