### Install ntscraper Library Source: https://github.com/bocchilorenzo/ntscraper/blob/main/README.md This snippet shows the command to install the ntscraper library using pip. It's the first step to using the library for scraping Nitter content. ```bash pip install ntscraper ``` -------------------------------- ### GET /profile/{username} Source: https://context7.com/bocchilorenzo/ntscraper/llms.txt Retrieves comprehensive profile information for a specific Twitter/X user. ```APIDOC ## GET /profile/{username} ### Description Retrieves user profile details including bio, stats, and optionally following/followers lists. ### Method GET ### Endpoint get_profile_info(username, mode='basic', instance=None) ### Parameters #### Path Parameters - **username** (string) - Required - The user handle. #### Query Parameters - **mode** (string) - Optional - Set to 'detail' to include following/followers lists. ### Response #### Success Response (200) - **name** (string) - Display name - **bio** (string) - Profile bio - **stats** (object) - Follower/Following counts #### Response Example { "name": "GitHub", "username": "github", "stats": {"followers": 1000000} } ``` -------------------------------- ### GET /instance/random Source: https://context7.com/bocchilorenzo/ntscraper/llms.txt Retrieves a random working Nitter instance from the tested pool. ```APIDOC ## GET /instance/random ### Description Returns a URL for a currently functional Nitter instance to be used in subsequent requests. ### Method GET ### Endpoint get_random_instance() ### Response #### Success Response (200) - **instance** (string) - The URL of the Nitter instance. ``` -------------------------------- ### Get User Profile Information in Python Source: https://github.com/bocchilorenzo/ntscraper/blob/main/README.md This Python code demonstrates how to fetch profile information for a given Twitter username. It supports fetching basic information or more detailed data including follower and following lists. ```python bezos_information = scraper.get_profile_info("JeffBezos") ``` -------------------------------- ### Get Tweet by ID Source: https://context7.com/bocchilorenzo/ntscraper/llms.txt Demonstrates how to fetch a specific tweet using its unique identifier. This functionality is essential for retrieving individual tweets when their ID is known. ```python from ntscraper import Nitter scraper = Nitter() # Example for fetching a tweet by ID would go here, assuming a method like get_tweet_by_id exists. ``` -------------------------------- ### GET /get_tweets (User Profile) Source: https://context7.com/bocchilorenzo/ntscraper/llms.txt Retrieves the timeline of a specific user, including options to include replies and date-specific filtering. ```APIDOC ## GET /get_tweets (mode='user') ### Description Fetches tweets from a specific user's profile timeline. ### Method GET ### Endpoint scraper.get_tweets(username, mode='user', ...) ### Parameters #### Query Parameters - **username** (string) - Required - The Twitter handle of the user. - **mode** (string) - Required - Must be 'user'. - **replies** (boolean) - Optional - Whether to include replies in the results. ### Response #### Success Response (200) - **tweets** (list) - List of user's tweets including metadata like is-retweet and is-pinned. ``` -------------------------------- ### Scrape User Profile Tweets Source: https://context7.com/bocchilorenzo/ntscraper/llms.txt Illustrates how to retrieve tweets from a user's profile timeline. Examples cover fetching all recent tweets, retrieving tweets within a specific date range including replies, and searching for tweets directed to a particular user. It also shows how to process user timeline results, identifying retweets, pinned tweets, and quoted posts. ```python from ntscraper import Nitter scraper = Nitter() # Get all recent tweets from a user user_tweets = scraper.get_tweets("elonmusk", mode='user') # Get user tweets with date range and replies user_tweets = scraper.get_tweets( "github", mode='user', number=50, since="2024-01-01", until="2024-03-31", replies=True # Include replies ) # Get tweets directed to a specific user mentions = scraper.get_tweets( "python", mode='term', to="github" # Tweets directed to @github ) # Process user timeline for tweet in user_tweets['tweets']: is_retweet = "RT" if tweet['is-retweet'] else "" is_pinned = "[PINNED]" if tweet['is-pinned'] else "" print(f"{is_pinned}{is_retweet} {tweet['date']}: {tweet['text'][:80]}...") # Check for quoted posts if tweet['quoted-post']: print(f" Quoted: @{tweet['quoted-post']['user']['username']}") ``` -------------------------------- ### GET /tweet/{tweet_id} Source: https://context7.com/bocchilorenzo/ntscraper/llms.txt Retrieves the full details of a specific tweet by its unique ID. ```APIDOC ## GET /tweet/{tweet_id} ### Description Fetches a specific tweet's content, metadata, and statistics using the tweet ID. ### Method GET ### Endpoint get_tweet_by_id(username, tweet_id, instance=None, max_retries=5) ### Parameters #### Path Parameters - **tweet_id** (string) - Required - The unique identifier of the tweet. - **username** (string) - Required - The handle of the tweet author. #### Query Parameters - **instance** (string) - Optional - Specific Nitter instance to use. - **max_retries** (int) - Optional - Number of retries on failure. ### Response #### Success Response (200) - **id** (string) - Tweet ID - **text** (string) - Content of the tweet - **stats** (object) - Engagement statistics (comments, retweets, likes) #### Response Example { "id": "1824507305389592885", "text": "Tweet content...", "stats": {"likes": 500} } ``` -------------------------------- ### Search Tweets by Hashtag Source: https://context7.com/bocchilorenzo/ntscraper/llms.txt Provides examples for searching tweets associated with a specific hashtag. It includes basic hashtag searches and searches with options like the number of tweets, language, filters for verified accounts, and date ranges. The processing of hashtag search results is also demonstrated. ```python from ntscraper import Nitter scraper = Nitter() # Search for hashtag results = scraper.get_tweets("github", mode='hashtag') # Search hashtag with options results = scraper.get_tweets( "python", mode='hashtag', number=100, language="en", filters=["verified"], # Only from verified accounts since="2024-01-01" ) # Process hashtag results print(f"Found {len(results['tweets'])} tweets") for tweet in results['tweets']: print(f"@{tweet['user']['username']}: {tweet['text'][:100]}...") print(f"Stats: {tweet['stats']['likes']} likes, {tweet['stats']['retweets']} RTs") ``` -------------------------------- ### GET /get_tweets (Search by Hashtag) Source: https://context7.com/bocchilorenzo/ntscraper/llms.txt Retrieves tweets associated with a specific hashtag, with options to filter by verification status or date. ```APIDOC ## GET /get_tweets (mode='hashtag') ### Description Retrieves recent tweets containing a specific hashtag. ### Method GET ### Endpoint scraper.get_tweets(hashtag, mode='hashtag', ...) ### Parameters #### Query Parameters - **hashtag** (string) - Required - The hashtag to search. - **mode** (string) - Required - Must be 'hashtag'. - **filters** (list) - Optional - List of filters (e.g., ['verified']). ### Response #### Success Response (200) - **tweets** (list) - List of tweet objects. ``` -------------------------------- ### Get a Random Working Nitter Instance using ntscraper Source: https://context7.com/bocchilorenzo/ntscraper/llms.txt Provides a function to retrieve a random, working Nitter instance from a pool of tested instances. This instance can then be used for subsequent scraping requests to distribute load or use specific available servers. ```python from ntscraper import Nitter scraper = Nitter() # Get a random working instance instance = scraper.get_random_instance() print(f"Random instance: {instance}") # Use the instance for subsequent requests profile = scraper.get_profile_info("github", instance=instance) tweets = scraper.get_tweets("python", mode='term', instance=instance) ``` -------------------------------- ### Tweet Data Structure Example for ntscraper Source: https://context7.com/bocchilorenzo/ntscraper/llms.txt Illustrates the complete structure of a tweet object as returned by the `get_tweets()` and `get_tweet_by_id()` methods. This includes fields for tweet ID, link, text, user information, date, statistics, media, and quoted post details. ```json { "id": "1824507305389592885", "link": "https://twitter.com/X/status/1824507305389592885", "text": "Tweet content with @mentions and https://links.com", "user": { "name": "Display Name", "username": "@username", "profile_id": "123456789", "avatar": "https://pbs.twimg.com/profile_images/..." }, "date": "Aug 16, 2024 ยท 6:03 PM UTC", "is-retweet": false, "is-pinned": false, "external-link": "https://example.com/shared-link", "replying-to": ["@user1", "@user2"], # Empty list if not a reply "quoted-post": { "link": "https://twitter.com/...", "text": "Quoted tweet content", "user": {"name": "...", "username": "...", "avatar": "..."}, "date": "...", "pictures": [], "videos": [], "gifs": [] }, # Empty dict if no quoted post "stats": { "comments": 100, "retweets": 50, "quotes": 25, "likes": 500 }, "pictures": ["https://pbs.twimg.com/media/..."], "videos": ["https://video.twimg.com/..."], "gifs": ["https://video.twimg.com/..."] } ``` -------------------------------- ### Get Twitter/X Profile Information using ntscraper Source: https://context7.com/bocchilorenzo/ntscraper/llms.txt Fetches profile information for a given Twitter/X username. It supports fetching basic profile details or detailed information including following and followers lists by setting the 'mode' parameter to 'detail'. Handles retries and optional instances. ```python from ntscraper import Nitter scraper = Nitter() # Get basic profile info profile = scraper.get_profile_info("github") if profile: print(f"Name: {profile['name']}") print(f"Username: {profile['username']}") print(f"ID: {profile['id']}") print(f"Bio: {profile['bio']}") print(f"Location: {profile['location']}") print(f"Website: {profile['website']}") print(f"Joined: {profile['joined']}") print(f"Profile Image: {profile['image']}") print(f"Stats:") print(f" Tweets: {profile['stats']['tweets']}") print(f" Following: {profile['stats']['following']}") print(f" Followers: {profile['stats']['followers']}") print(f" Likes: {profile['stats']['likes']}") print(f" Media: {profile['stats']['media']}") # Get detailed profile with following/followers lists detailed_profile = scraper.get_profile_info( "github", max_retries=5, instance=None, mode='detail' # Include following_list and followers_list ) if detailed_profile: print(f"Following: {detailed_profile['following_list'][:10]}...") print(f"Followers: {detailed_profile['followers_list'][:10]}...") ``` -------------------------------- ### GET /get_tweets (Search by Term) Source: https://context7.com/bocchilorenzo/ntscraper/llms.txt Retrieves tweets matching a specific search term with support for various filters, date ranges, and language settings. ```APIDOC ## GET /get_tweets (mode='term') ### Description Searches for tweets containing a specific term. Supports filtering by date, language, location, and media type. ### Method GET ### Endpoint scraper.get_tweets(term, mode='term', ...) ### Parameters #### Query Parameters - **term** (string) - Required - The search query term. - **mode** (string) - Required - Must be 'term'. - **number** (int) - Optional - Number of tweets to retrieve (-1 for unlimited). - **since** (string) - Optional - Start date (YYYY-MM-DD). - **until** (string) - Optional - End date (YYYY-MM-DD). - **language** (string) - Optional - ISO 639-1 language code. - **near** (string) - Optional - Location filter. - **filters** (list) - Optional - List of filters (e.g., ['media', 'images']). ### Response #### Success Response (200) - **tweets** (list) - List of tweet objects containing id, user, text, date, and stats. - **threads** (list) - List of connected tweet threads. ### Response Example { "tweets": [{"id": "123", "text": "Example tweet", "stats": {"likes": 10}}], "threads": [] } ``` -------------------------------- ### Scrape Tweets by Term, Hashtag, or User in Python Source: https://github.com/bocchilorenzo/ntscraper/blob/main/README.md This Python snippet illustrates how to scrape tweets using the Nitter scraper. It shows examples for searching tweets by a general term, a specific hashtag, and from a user's profile. The `mode` parameter controls the scraping behavior. ```python github_hash_tweets = scraper.get_tweets("github", mode='hashtag') bezos_tweets = scraper.get_tweets("JeffBezos", mode='user') ``` -------------------------------- ### Get Single Tweet by ID in Python Source: https://github.com/bocchilorenzo/ntscraper/blob/main/README.md This Python snippet shows how to retrieve a single tweet using its author's username and the tweet ID. It's a direct way to fetch specific tweet content. ```python tweet = scraper.get_tweet_by_id("x", "1826317783430303888") ``` -------------------------------- ### Initialize Nitter Scraper Instance Source: https://context7.com/bocchilorenzo/ntscraper/llms.txt Demonstrates how to create an instance of the Nitter scraper with various configuration options. This includes basic initialization, custom logging levels, disabling instance checks, and specifying single or multiple Nitter instances. ```python from ntscraper import Nitter # Basic initialization with default settings scraper = Nitter() # Initialize with custom settings scraper = Nitter( log_level=1, # 0=warnings only, 1=info (default), None=no logs skip_instance_check=False # Set True if using trusted/local instance ) # Initialize with a specific Nitter instance scraper = Nitter( instances="http://localhost:8080", skip_instance_check=True ) # Initialize with multiple instances to choose from scraper = Nitter( instances=["https://nitter.example.com", "https://nitter2.example.com"], skip_instance_check=False ) ``` -------------------------------- ### Initialize Nitter Scraper in Python Source: https://github.com/bocchilorenzo/ntscraper/blob/main/README.md This Python code demonstrates how to initialize the Nitter scraper. It includes options for setting the log level and whether to skip instance checks, which can be useful for performance or when using trusted instances. ```python from ntscraper import Nitter scraper = Nitter(log_level=1, skip_instance_check=False) ``` -------------------------------- ### Multiprocessing for Batch Operations with ntscraper Source: https://context7.com/bocchilorenzo/ntscraper/llms.txt Demonstrates how to use Python's multiprocessing to scrape multiple terms or profiles concurrently. This is essential for efficient batch processing. The code must be enclosed within an `if __name__ == "__main__":` block for multiprocessing to work correctly. ```python from ntscraper import Nitter # IMPORTANT: Multiprocessing code must run in __main__ block if __name__ == "__main__": scraper = Nitter() # Batch search multiple terms terms = ["python", "javascript", "rust"] results = scraper.get_tweets(terms, mode='term', number=20) # Results is a list of dictionaries, one per term for i, term_results in enumerate(results): print(f"\n{terms[i]}: {len(term_results['tweets'])} tweets") for tweet in term_results['tweets'][:3]: print(f" - {tweet['text'][:60]}...") # Batch fetch multiple profiles usernames = ["github", "python", "rustlang"] profiles = scraper.get_profile_info(usernames) for profile in profiles: if profile: print(f"{profile['name']}: {profile['stats']['followers']} followers") ``` -------------------------------- ### Apply Search Filters in Nitter Scraper Source: https://context7.com/bocchilorenzo/ntscraper/llms.txt Demonstrates how to initialize the Nitter scraper and apply specific filters and exclusions to tweet search queries. This allows for precise data extraction by including or removing categories like verified accounts, media types, or replies. ```python from ntscraper import Nitter scraper = Nitter() # Example: Only tweets with images from verified users results = scraper.get_tweets( "technology", mode='term', filters=["verified", "images"], exclude=["replies", "nativeretweets"] ) ``` -------------------------------- ### Scrape multiple profiles using multiprocessing Source: https://github.com/bocchilorenzo/ntscraper/blob/main/README.md Uses the get_profile_info method to scrape multiple user profiles concurrently. This must be executed within an if __name__ == "__main__" block and is limited by the number of CPU cores. ```python usernames = ["x", "github"] results = scraper.get_profile_info(usernames) ``` -------------------------------- ### Retrieve a random Nitter instance Source: https://github.com/bocchilorenzo/ntscraper/blob/main/README.md Fetches a random Nitter instance from the available list. This is useful for rotating instances if one is experiencing connectivity issues. ```python random_instance = scraper.get_random_instance() ``` -------------------------------- ### Scrape Multiple Terms Concurrently with Multiprocessing in Python Source: https://github.com/bocchilorenzo/ntscraper/blob/main/README.md This Python code demonstrates how to scrape tweets for multiple terms simultaneously using multiprocessing. It's important to note that this feature is best used with a local Nitter instance to avoid overloading public servers. ```python terms = ["github", "bezos", "musk"] results = scraper.get_tweets(terms, mode='term') # Note: This code needs to be within an if __name__ == "__main__": block. ``` -------------------------------- ### Search Tweets by Term Source: https://context7.com/bocchilorenzo/ntscraper/llms.txt Shows how to search for tweets containing specific terms using the Nitter scraper. It covers basic term searches and advanced searches with filters for number of tweets, date ranges, language, location, media, and exclusions. The output structure for tweets and threads is also illustrated. ```python from ntscraper import Nitter scraper = Nitter() # Basic term search results = scraper.get_tweets("python programming", mode='term') # Search with filters and date range results = scraper.get_tweets( "machine learning", mode='term', number=50, # Limit to 50 tweets (-1 for unlimited) since="2024-01-01", # Start date (YYYY-MM-DD) until="2024-06-01", # End date (YYYY-MM-DD) language="en", # ISO 639-1 language code near="San Francisco", # Location filter filters=["media", "images"], # Include only tweets with media/images exclude=["replies"], # Exclude replies max_retries=5, # Retry attempts per page instance=None # Use random instance ) # Access results for tweet in results['tweets']: print(f"ID: {tweet['id']}") print(f"User: {tweet['user']['username']}") print(f"Text: {tweet['text']}") print(f"Date: {tweet['date']}") print(f"Likes: {tweet['stats']['likes']}") print(f"Retweets: {tweet['stats']['retweets']}") print(f"Pictures: {tweet['pictures']}") print(f"Videos: {tweet['videos']}") print("---") # Access threads (connected tweets) for thread in results['threads']: print(f"Thread with {len(thread)} tweets") ``` -------------------------------- ### Fetch a Specific Tweet using ntscraper Source: https://context7.com/bocchilorenzo/ntscraper/llms.txt Retrieves a single tweet by its ID. It handles optional parameters like specific instances and retries. The output includes detailed tweet information such as author, text, date, links, and statistics. Returns None if the tweet is not found. ```python from ntscraper import Nitter scraper = Nitter() tweet = scraper.get_tweet_by_id( username="X", tweet_id="1824507305389592885", instance=None, # Optional specific instance max_retries=5 ) if tweet: print(f"Tweet ID: {tweet['id']}") print(f"Author: {tweet['user']['name']} ({tweet['user']['username']})") print(f"Avatar: {tweet['user']['avatar']}") print(f"Text: {tweet['text']}") print(f"Date: {tweet['date']}") print(f"Link: {tweet['link']}") print(f"External Link: {tweet['external-link']}") print(f"Replying to: {tweet['replying-to']}") print(f"Stats: {tweet['stats']['comments']} comments, " f"{tweet['stats']['retweets']} RTs, " f"{tweet['stats']['quotes']} quotes, " f"{tweet['stats']['likes']} likes") print(f"Media: {len(tweet['pictures'])} pics, " f"{len(tweet['videos'])} videos, " f"{len(tweet['gifs'])} gifs") else: print("Tweet not found") ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.