### Minimal YTFetcher Setup Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/MODULE_INDEX.md Provides the most basic setup for fetching YouTube data from a channel with minimal configuration. ```python from ytfetcher import YTFetcher fetcher = YTFetcher.from_channel("TheOffice", max_results=10) data = fetcher.fetch_youtube_data() ``` -------------------------------- ### Clone Repository and Install Dependencies Source: https://github.com/kaya70875/ytfetcher/blob/main/README.md Clones the YTFetcher repository and installs project dependencies using Poetry. This is the initial setup for development. ```bash git clone https://github.com/kaya70875/ytfetcher.git cd ytfetcher poetry install ``` -------------------------------- ### Python Code Example Conventions Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/README.md Code examples in this documentation are complete and runnable, demonstrating realistic usage patterns with error handling. They avoid test assertions and internal implementation details. ```python # ✓ Complete, runnable examples # ✓ Show realistic usage patterns # ✓ Include error handling where relevant # ✗ Do NOT use test assertions (no expect/should) # ✗ Do NOT show internal implementation details ``` -------------------------------- ### Install YTFetcher Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/QUICKSTART.md Install the ytfetcher library using pip. ```bash pip install ytfetcher ``` -------------------------------- ### Quick Start: Fetch Transcripts and Metadata Source: https://github.com/kaya70875/ytfetcher/blob/main/docs/cli.md Fetches a specified number of video transcripts and metadata from a YouTube channel and saves the output in JSON format. This is a common starting point for users wanting to quickly extract data. ```bash ytfetcher channel TheOffice -m 50 -f json ``` -------------------------------- ### Complete ytfetcher Example Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/QUICKSTART.md A comprehensive example demonstrating advanced usage of ytfetcher, including configuring fetch options with filters, fetching data with comments, exporting to JSON, preparing data for ML, and checking for failed transcripts. ```python from ytfetcher import YTFetcher from ytfetcher.config import FetchOptions from ytfetcher.filters import min_views, min_duration from ytfetcher.services import JSONExporter from ytfetcher.utils import channel_data_to_rows # Configure options = FetchOptions( languages=['en'], filters=[ min_views(1000), min_duration(300) ], cache_enabled=True ) # Fetch fetcher = YTFetcher.from_channel( channel_handle="TheOffice", max_results=100, options=options ) data = fetcher.fetch_with_comments(max_comments=20) # Export JSON exporter = JSONExporter( channel_data=data, allowed_metadata_list=['title', 'view_count'], filename='office_data' ) exporter.write() # Prepare for ML rows = channel_data_to_rows(data, include_comments=False) # Check failures failed = fetcher.get_failed_transcripts() print(f"Fetched: {len(data)}, Failed: {len(failed)}") ``` -------------------------------- ### JSONExporter Usage Example Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/api-reference/exporters.md Example demonstrating how to instantiate and use the JSONExporter to export YouTube data to a specified directory with selected metadata. ```python from ytfetcher.services import JSONExporter channel_data = fetcher.fetch_youtube_data() exporter = JSONExporter( channel_data=channel_data, allowed_metadata_list=['title', 'description'], timing=True, filename='my_export', output_dir='./exports' ) exporter.write() ``` -------------------------------- ### Example Usage of TXTExporter Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/api-reference/exporters.md Demonstrates how to instantiate and use the TXTExporter to export channel data with specified metadata and output settings. Ensure channel_data is defined before use. ```python from ytfetcher.services import TXTExporter exporter = TXTExporter( channel_data=channel_data, allowed_metadata_list=['title', 'description'], timing=True, filename='transcripts', output_dir='./output' ) exporter.write() ``` -------------------------------- ### CSVExporter Usage Example Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/api-reference/exporters.md Example demonstrating how to instantiate and use the CSVExporter to export YouTube data, specifying metadata fields and output path. ```python from ytfetcher.services import CSVExporter exporter = CSVExporter( channel_data=channel_data, allowed_metadata_list=['title', 'view_count'], timing=True, filename='transcripts', output_dir='./data' ) exporter.write() ``` -------------------------------- ### Build Docker Image Source: https://github.com/kaya70875/ytfetcher/blob/main/README.md Builds the Docker image for YTFetcher. Ensure you have Docker and Docker Compose installed. ```bash docker-compose build ``` -------------------------------- ### Install vaderSentiment Source: https://github.com/kaya70875/ytfetcher/blob/main/docs/examples.md This command installs the vaderSentiment library, which is required for performing sentiment analysis on YouTube comments. ```bash pip install vaderSentiment ``` -------------------------------- ### Channel Data Structure Example Source: https://github.com/kaya70875/ytfetcher/blob/main/docs/index.md Example of the structured data returned by fetch_youtube_data, including ChannelData, Transcript, and DLSnippet objects. ```python [ ChannelData( video_id='video1', transcripts=[ Transcript( text="Hey there", start=0.0, duration=1.54 ), Transcript( text="Happy coding!", start=1.56, duration=4.46 ) ] metadata=DLSnippet( video_id='video1', title='VideoTitle', description='VideoDescription', url='https://youtu.be/video1', duration=120, view_count=1000, thumbnails=[{'url': 'thumbnail_url'}] ) ), # Other ChannelData objects... ] ``` -------------------------------- ### ChannelData Structure Example Source: https://github.com/kaya70875/ytfetcher/blob/main/README.md Example structure of the ChannelData object returned by the fetch_youtube_data method, including video ID, transcripts, and metadata. ```python [ ChannelData( video_id='video1', transcripts=[ Transcript( text="Hey there", start=0.0, duration=1.54 ), Transcript( text="Happy coding!", start=1.56, duration=4.46 ) ] metadata=DLSnippet( video_id='video1', title='VideoTitle', description='VideoDescription', url='https://youtu.be/video1', duration=120, view_count=1000, thumbnails=[{'url': 'thumbnail_url'}] ) ), # Other ChannelData objects... ] ``` -------------------------------- ### Example Structure of Fetched Data with Transcripts Source: https://github.com/kaya70875/ytfetcher/blob/main/docs/index.md Illustrates the expected data structure when fetching comments with transcripts and metadata, showing ChannelData containing video ID, transcripts, metadata, and comments. ```python [ ChannelData( video_id='id1', transcripts=list[Transcript(...)], metadata=DLSnippet(...), comments=list[Comment( text='Comment one.', like_count=20, author='@author', time_text='8 days ago' )] ) ] ``` -------------------------------- ### Handle VideoUnavailable Exception Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/errors.md This example demonstrates how to handle cases where a video ID does not correspond to an available video, such as deleted, private, or non-existent videos. It outputs the unavailable video ID. ```python from ytfetcher import YTFetcher from ytfetcher.exceptions import VideoUnavailable try: fetcher = YTFetcher.from_video_ids(["dQw4w9WgXcQ", "deletedvideoXX"]) data = fetcher.fetch_youtube_data() except VideoUnavailable as e: print(f"Video not available: {e.video_id}") ``` -------------------------------- ### TXTExporter Output Format Example Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/api-reference/exporters.md Illustrates the structure of the plain text output generated by TXTExporter, including video metadata and transcript segments with timing. ```text Transcript for xyz123: title --> Video Title description --> Description text... 0.5 --> 3.0 Transcript text 3.0 --> 5.0 More transcript text Comments for xyz123 Comment --> Great video! Author --> @user Likes --> 15 Time Text --> 2 days ago ``` -------------------------------- ### Iterate Through Video Transcripts Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/QUICKSTART.md Access and print the start time and text of each transcript segment for a given video. Assumes channel_data is already fetched. ```python for video in channel_data: print(f"Video: {video.metadata.title}") for segment in video.transcripts: time = segment.start text = segment.text print(f"[{time:.1f}s] {text}") ``` -------------------------------- ### Run Tests with Poetry Source: https://github.com/kaya70875/ytfetcher/blob/main/README.md Executes all tests for the YTFetcher project using Poetry. Ensure all dependencies are installed before running. ```bash poetry run pytest ``` -------------------------------- ### Combine Multiple Video Filters Source: https://github.com/kaya70875/ytfetcher/blob/main/docs/cli.md Multiple filters can be combined using AND logic. This example filters videos by minimum views, minimum duration, and title inclusion. ```bash ytfetcher channel TheOffice -m 50 -f json \ --min-views 1000 \ --min-duration 300 \ --includes-title "tutorial" ``` -------------------------------- ### Configure Generic Proxy Settings Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/api-reference/config.md Use GenericProxyConfig to set up custom HTTP and HTTPS proxy servers for your requests. This is suitable for any standard proxy server setup. ```python from ytfetcher.config import GenericProxyConfig, FetchOptions from ytfetcher import YTFetcher proxy = GenericProxyConfig( http_proxy="http://user:pass@host:port", https_proxy="https://user:pass@host:port" ) options = FetchOptions(proxy_config=proxy) fetcher = YTFetcher.from_channel( channel_handle="TheOffice", options=options ) ``` -------------------------------- ### Convert Fetched Channel Data to Dictionary Example Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/types.md Demonstrates converting fetched ChannelData objects to dictionaries using the to_dict() method. Useful for debugging or data export. ```python channel_data = fetcher.fetch_youtube_data() for item in channel_data: data_dict = item.to_dict() print(data_dict) ``` -------------------------------- ### Preview with Comments Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/api-reference/preview.md This pattern demonstrates how to fetch channel data including comments and then render a preview. The 'limit' parameter affects both the number of videos and the detail shown for each. ```python fetcher = YTFetcher.from_channel("TheOffice", max_results=10) channel_data = fetcher.fetch_with_comments(max_comments=10) preview = PreviewRenderer() preview.render(data=channel_data, limit=3) ``` -------------------------------- ### Instantiate PreviewRenderer Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/api-reference/preview.md Create an instance of the PreviewRenderer class. No parameters are required. ```python from ytfetcher.services import PreviewRenderer preview = PreviewRenderer() ``` -------------------------------- ### Initialize YTFetcher with BaseYoutubeDLFetcher Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/api-reference/ytfetcher.md Demonstrates the basic instantiation of the YTFetcher class using a provided BaseYoutubeDLFetcher instance and optional FetchOptions. ```python from yt_dlp import YoutubeDL from ytfetcher import YTFetcher, FetchOptions # Assuming BaseYoutubeDLFetcher is a base class or protocol # For demonstration, we'll use a placeholder or a concrete implementation if available # Example using a hypothetical concrete fetcher: # from ytfetcher.fetchers import ChannelFetcher # youtube_dl_fetcher = ChannelFetcher(channel_handle='example') # Placeholder for a BaseYoutubeDLFetcher instance # In a real scenario, this would be an instance of ChannelFetcher, VideoListFetcher, etc. class MockYoutubeDLFetcher: pass youtube_dl_fetcher = MockYoutubeDLFetcher() options = FetchOptions( language='en', use_cache=True ) fetcher = YTFetcher(youtube_dl_fetcher, options) ``` -------------------------------- ### Preview First Few Results Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/api-reference/preview.md Use this pattern to preview a limited number of videos from a channel's data. It fetches YouTube data and then renders a preview, limiting the display to the first few videos. ```python from ytfetcher import YTFetcher from ytfetcher.services import PreviewRenderer fetcher = YTFetcher.from_channel("TheOffice", max_results=20) channel_data = fetcher.fetch_youtube_data() preview = PreviewRenderer() preview.render(data=channel_data, limit=2) # Show first 2 videos ``` -------------------------------- ### Initialize TranscriptFetcher Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/api-reference/transcript-fetcher.md Instantiate TranscriptFetcher with video IDs, language preferences, and manual creation flag. Ensure 'languages' is provided if 'manually_created' is True. ```python from ytfetcher._transcript_fetcher import TranscriptFetcher from ytfetcher.config import HTTPConfig fetcher = TranscriptFetcher( video_ids=['dQw4w9WgXcQ', 'jNQXAC9IVRw'], languages=['en'], manually_created=False ) result = fetcher.fetch() ``` -------------------------------- ### Display All CLI Options Source: https://github.com/kaya70875/ytfetcher/blob/main/docs/cli.md Displays all available commands and options for the YTFetcher CLI. Use this to understand the full range of functionalities and parameters. ```bash ytfetcher -h ``` -------------------------------- ### FailedTranscript Data Model and Usage Example Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/types.md Represents a record of a transcript fetch failure, including the video ID, reason, an optional message, and a flag for permanent exceptions. This example shows how to fetch failed transcripts and print their details. ```python class FailedTranscript(BaseModel): video_id: str reason: str message: str | None = None is_permanent_exception: bool = False ``` ```python from ytfetcher import YTFetcher fetcher = YTFetcher.from_channel("TheOffice", max_results=20) results = fetcher.fetch_youtube_data() failed = fetcher.get_failed_transcripts() for item in failed: print(f"Video {item.video_id}: {item.reason}") if item.message: print(f" Message: {item.message}") ``` -------------------------------- ### Enable and Configure Cache Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/QUICKSTART.md Enable caching with `FetchOptions` and set a time-to-live (TTL) for reusing cached transcripts. ```python options = FetchOptions(cache_enabled=True, cache_ttl=30) # Reuses cached transcripts for 30 days ``` -------------------------------- ### Handle YTFetcherError Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/errors.md Example of how to catch and handle the base YTFetcherError. This is useful for general error management in ytfetcher operations. ```python from ytfetcher.exceptions import YTFetcherError try: fetcher = YTFetcher.from_channel("TheOffice") data = fetcher.fetch_youtube_data() except YTFetcherError as e: print(f"YTFetcher error: {e}") ``` -------------------------------- ### Basic FetchOptions Configuration Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/api-reference/config.md Configure basic fetch options including preferred languages, cache enablement, and cache time-to-live. This is useful for standard operations where default settings are not sufficient. ```python from ytfetcher.config import FetchOptions from ytfetcher import YTFetcher options = FetchOptions( languages=['en', 'es'], cache_enabled=True, cache_ttl=7 ) fetcher = YTFetcher.from_channel( channel_handle="TheOffice", max_results=50, options=options ) ``` -------------------------------- ### Async Fetching with ThreadPoolExecutor Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/QUICKSTART.md Fetch data from multiple channels concurrently using ThreadPoolExecutor. Ensure you have the ytfetcher library installed. ```python from concurrent.futures import ThreadPoolExecutor from ytfetcher import YTFetcher def fetch_channel(channel_handle): fetcher = YTFetcher.from_channel(channel_handle, max_results=50) return fetcher.fetch_youtube_data() channels = ["TheOffice", "Parks and Recreation", "Brooklyn Nine-Nine"] with ThreadPoolExecutor(max_workers=3) as executor: results = list(executor.map(fetch_channel, channels)) all_data = [item for sublist in results for item in sublist] ``` -------------------------------- ### Get Default Cache Path Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/api-reference/cache.md Retrieves the default directory path where the cache is stored. This is useful for understanding or specifying the cache location. ```python from ytfetcher.config import default_cache_path path = default_cache_path() # Returns: ~/.cache/ytfetcher ``` -------------------------------- ### Basic Export with Custom Metadata Source: https://github.com/kaya70875/ytfetcher/blob/main/docs/cli.md Fetches a specified number of videos from a channel, exports them as JSON, excludes timings, and includes only title and description metadata. ```bash ytfetcher channel TheOffice -m 20 -f json --no-timing --metadata title description ``` -------------------------------- ### Get Default Cache Path Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/api-reference/config.md Retrieves the default directory path where ytfetcher stores its cache. This is useful for understanding or accessing cached data. ```python from ytfetcher.config import default_cache_path cache_dir = default_cache_path() print(f"Cache location: {cache_dir}") ``` -------------------------------- ### Initialize Fetcher with Search Query Source: https://github.com/kaya70875/ytfetcher/blob/main/README.md Initializes the YTFetcher to discover videos based on a search query. The number of results can be controlled using the max_results parameter. ```python from ytfetcher import YTFetcher # Searches for the top 10 videos matching 'Artificial Intelligence' fetcher = YTFetcher.from_search( query="Artificial Intelligence", max_results=10 ) ``` -------------------------------- ### video_ids Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/api-reference/ytfetcher.md Accesses a list of video IDs that have been fetched from the YouTube source. This property provides a quick way to get all the video identifiers processed by the fetcher. ```APIDOC ## video_ids ### Description List of video IDs fetched from the YouTube source (channel, playlist, search, or provided directly). ### Method ```python @property video_ids: list[str] ``` ### Returns `list[str]`: Video ID strings. ### Example ```python ids = fetcher.video_ids print(f"Fetched {len(ids)} video IDs") ``` ``` -------------------------------- ### Get Failed Transcripts Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/api-reference/ytfetcher.md Retrieves a list of transcripts that failed during the last fetch operation. Useful for identifying and debugging issues with specific video transcriptions. ```python fetcher = YTFetcher.from_channel(channel_handle="TheOffice", max_results=20) results = fetcher.fetch_youtube_data() failed = fetcher.get_failed_transcripts() for item in failed: print(f"Video {item.video_id}: {item.reason} - {item.message}") ``` -------------------------------- ### CLI: Fetch from Channel Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/QUICKSTART.md Use the CLI to fetch data from a YouTube channel, specifying the channel name, maximum results, and output format. ```bash # Fetch from channel ytfetcher channel TheOffice -m 50 -f json ``` -------------------------------- ### CLI: Fetch from Playlist Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/QUICKSTART.md Use the CLI to fetch data from a YouTube playlist, specifying the playlist ID, maximum results, and output format. ```bash # Fetch from playlist ytfetcher playlist PLxxxxxxx -m 20 -f csv ``` -------------------------------- ### Using Custom HTTP/HTTPS Proxies Source: https://github.com/kaya70875/ytfetcher/blob/main/docs/cli.md Specify custom HTTP and HTTPS proxy servers for requests. Ensure the proxy URL includes authentication if required. ```bash ytfetcher channel TheOffice -f json --http-proxy "http://user:pass@host:port" --https-proxy "https://user:pass@host:port" ``` -------------------------------- ### Fetch with Search Query via CLI Source: https://github.com/kaya70875/ytfetcher/blob/main/README.md Fetches data from YouTube search results using the CLI. Provide a search query, output format, and optionally max results. ```bash ytfetcher search "AI Getting Jobs" -f json -m 25 ``` -------------------------------- ### Configure Filtered Fetching with Proxies Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/api-reference/config.md Sets up FetchOptions to use a specific proxy (Webshare) and apply filters for minimum views and duration, with caching enabled. ```python from ytfetcher.filters import min_views, min_duration from ytfetcher.config import WebshareProxyConfig, FetchOptions proxy = WebshareProxyConfig( username="user", password="pass" ) options = FetchOptions( proxy_config=proxy, filters=[min_views(5000), min_duration(300)], cache_enabled=True ) ``` -------------------------------- ### Access Fetched Video IDs Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/api-reference/ytfetcher.md Accesses the list of video IDs that were successfully fetched from the YouTube source. Use this to get a count or iterate through the IDs of the fetched content. ```python ids = fetcher.video_ids print(f"Fetched {len(ids)} video IDs") ``` -------------------------------- ### Fetch Channel Videos via CLI Source: https://github.com/kaya70875/ytfetcher/blob/main/README.md Fetches data from the 'videos' tab of a YouTube channel using the CLI. Defaults to videos if tab is not specified. ```bash # Default: videos ytfetcher channel TheOffice -m 20 --tab videos -f json ``` -------------------------------- ### Configure Language Fallback Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/api-reference/config.md Sets up FetchOptions to prioritize a primary language and fall back to a secondary language if the primary is unavailable. ```python options = FetchOptions( languages=['de', 'en'] # Try German first, fall back to English ) ``` -------------------------------- ### Convert Channel Data to Pandas DataFrame Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/api-reference/utils.md Converts ChannelData into a pandas DataFrame, excluding comments. This is ideal for data manipulation and analysis using pandas. Ensure pandas is installed. ```python import pandas as pd from ytfetcher import YTFetcher from ytfetcher.utils import channel_data_to_rows fetcher = YTFetcher.from_channel("TheOffice", max_results=20) channel_data = fetcher.fetch_youtube_data() rows = channel_data_to_rows(channel_data, include_comments=False) df = pd.DataFrame(rows) print(df[['video_id', 'title', 'view_count']]) ``` -------------------------------- ### Example Structure of Fetched Comments Only Source: https://github.com/kaya70875/ytfetcher/blob/main/docs/index.md Shows the data structure returned when fetching only comments, consisting of a list of Comment objects, each with text, like count, author, and time text. ```python [ Comment( text='Comment one.', like_count=20, author='@author', time_text='8 days ago' ) ## OTHER COMMENT OBJECTS... ] ``` -------------------------------- ### Initialize HTTPConfig with Default Headers Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/api-reference/config.md Use HTTPConfig without arguments to automatically apply realistic browser-like headers to your requests. This is useful for mimicking browser behavior. ```python from ytfetcher.config import HTTPConfig # Uses default browser-like headers config = HTTPConfig() options = FetchOptions(http_config=config) ``` -------------------------------- ### Fetch Channel Shorts via CLI Source: https://github.com/kaya70875/ytfetcher/blob/main/README.md Fetches data from the 'shorts' tab of a YouTube channel using the CLI. ```bash # Fetch from the Shorts tab ytfetcher channel TheOffice -m 20 --tab shorts -f json ``` -------------------------------- ### Complete YTFetcher Workflow Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/MODULE_INDEX.md Demonstrates a full workflow including configuration with filters and caching, fetching data from a channel, exporting to JSON, and converting data for machine learning. ```python from ytfetcher import YTFetcher from ytfetcher.config import FetchOptions from ytfetcher.services import JSONExporter from ytfetcher.utils import channel_data_to_rows from ytfetcher.filters import min_duration, min_views # Configure options = FetchOptions( filters=[min_views(1000), min_duration(300)], cache_enabled=True ) # Fetch fetcher = YTFetcher.from_channel("TheOffice", max_results=50, options=options) channel_data = fetcher.fetch_youtube_data() # Export exporter = JSONExporter(channel_data=channel_data) exporter.write() # Convert for ML rows = channel_data_to_rows(channel_data) ``` -------------------------------- ### Transcript Model Definition Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/types.md Defines the structure for a single transcript segment, including text, start time, and duration. Used to represent timed text data from YouTube videos. ```python class Transcript(BaseModel): text: str start: float duration: float ``` -------------------------------- ### Configure Fetcher with Proxy Source: https://github.com/kaya70875/ytfetcher/blob/main/README.md Initializes YTFetcher with combined proxy configurations for fetching data. Supports generic and Webshare proxies. ```python from ytfetcher import YTFetcher from ytfetcher.config import GenericProxyConfig, WebshareProxyConfig, FetchOptions options = FetchOptions( proxy_config=GenericProxyConfig() | WebshareProxyConfig() ) fetcher = YTFetcher.from_channel( channel_handle="TheOffice", max_results=3, options=options ) ``` -------------------------------- ### Export TXT with Comments Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/api-reference/exporters.md Use TXTExporter to export channel data, including comments, in TXT format. This example shows how to fetch data with comments and specify metadata fields for the export. ```python fetcher = YTFetcher.from_channel("TheOffice", max_results=5) channel_data = fetcher.fetch_with_comments(max_comments=10) exporter = TXTExporter( channel_data=channel_data, allowed_metadata_list=['title', 'description', 'view_count'], filename='with_comments' ) exporter.write() ``` -------------------------------- ### Export CSV Without Timing Information Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/api-reference/exporters.md Use CSVExporter to export channel data in CSV format. This example demonstrates exporting only the 'title' metadata and disabling timing information for a simpler output. ```python from ytfetcher.services import CSVExporter exporter = CSVExporter( channel_data=channel_data, allowed_metadata_list=['title'], timing=False, filename='simple_export' ) exporter.write() ``` -------------------------------- ### FetchOptions with Proxy Configuration Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/api-reference/config.md Configure proxy settings for routing requests through a gateway, using either a specific proxy provider like Webshare or a generic configuration. This is essential for managing network traffic or accessing geo-restricted content. ```python from ytfetcher.config import FetchOptions, WebshareProxyConfig, GenericProxyConfig from ytfetcher import YTFetcher # Using Webshare proxy proxy = WebshareProxyConfig( username="your_username", password="your_password" ) options = FetchOptions(proxy_config=proxy) fetcher = YTFetcher.from_channel( channel_handle="TheOffice", options=options ) ``` -------------------------------- ### Basic CLI Channel Fetch Source: https://github.com/kaya70875/ytfetcher/blob/main/README.md Fetches data from a YouTube channel using the CLI. Specify channel handle, max results, and output format. ```bash ytfetcher channel -m -f ``` -------------------------------- ### Fetch and Print Video Transcripts Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/types.md Fetches YouTube channel data and iterates through the first three transcript segments of each video, printing their start time, end time, and text. Requires YTFetcher and VideoTranscript. ```python from ytfetcher import YTFetcher fetcher = YTFetcher.from_channel("TheOffice", max_results=1) channel_data = fetcher.fetch_youtube_data() if channel_data: video = channel_data[0] for segment in video.transcripts[:3]: end_time = segment.start + segment.duration print(f"[{segment.start:.1f}s - {end_time:.1f}s]: {segment.text}") ``` -------------------------------- ### Get Cached Transcripts and Failures Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/api-reference/cache.md Retrieve both successful transcripts and failure records from the cache for a given list of video IDs and a specific cache key. Expired entries are ignored. Returns empty lists if no cache hits. ```python from ytfetcher.cache import SQLiteCache cache = SQLiteCache(cache_dir="~/.cache/ytfetcher") video_ids = ['video1', 'video2', 'video3'] cache_key = "en|manually_created=False" successes, failures = cache.get_cached_states( video_ids=video_ids, cache_key=cache_key ) print(f"Cache hits: {len(successes)} transcripts, {len(failures)} failures") ``` -------------------------------- ### Use Webshare Proxy via CLI Source: https://github.com/kaya70875/ytfetcher/blob/main/README.md Fetches data using a Webshare proxy through the CLI. Requires channel handle, output format, and Webshare username/password. ```bash ytfetcher -f json --webshare-proxy-username "" --webshare-proxy-password "" ``` -------------------------------- ### CLI: Fetch with Filters Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/QUICKSTART.md Apply filters such as minimum views and minimum duration when fetching data from a channel using the CLI. ```bash # With filters ytfetcher channel TheOffice -m 50 --min-views 1000 --min-duration 300 ``` -------------------------------- ### Apply Multiple Filters to Fetch YouTube Data Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/api-reference/filters.md Use this snippet to fetch YouTube data from a channel, applying multiple filters simultaneously. Filters include minimum views, minimum duration, and title keywords. Ensure 'ytfetcher' library is installed. ```python from ytfetcher import YTFetcher from ytfetcher.config import FetchOptions from ytfetcher.filters import min_duration, min_views, filter_by_title options = FetchOptions( filters=[ min_views(5000), # At least 5000 views min_duration(600), # At least 10 minutes filter_by_title("tutorial") # Title contains "tutorial" ] ) fetcher = YTFetcher.from_channel( channel_handle="TheOffice", max_results=50, options=options ) channel_data = fetcher.fetch_youtube_data() ``` -------------------------------- ### FetchOptions with Filters Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/api-reference/config.md Apply filters to fetch only videos meeting specific criteria, such as minimum duration and minimum views. This is useful for targeting specific types of content. ```python from ytfetcher.config import FetchOptions from ytfetcher.filters import min_duration, min_views from ytfetcher import YTFetcher options = FetchOptions( filters=[ min_duration(600), # At least 10 minutes min_views(1000) # At least 1000 views ] ) fetcher = YTFetcher.from_channel( channel_handle="TheOffice", options=options ) ``` -------------------------------- ### Preview Search Results Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/api-reference/preview.md This usage pattern shows how to fetch YouTube data based on a search query and then render a preview of the results. The 'limit' parameter controls the amount of detail displayed for each search result. ```python fetcher = YTFetcher.from_search("How to code in Python", max_results=15) channel_data = fetcher.fetch_youtube_data() preview = PreviewRenderer() preview.render(data=channel_data, limit=5) ``` -------------------------------- ### Initialize Fetcher with Custom Video IDs Source: https://github.com/kaya70875/ytfetcher/blob/main/README.md Initializes the YTFetcher to target specific videos directly using a list of their IDs. This is the most efficient method when video IDs are already known. ```python from ytfetcher import YTFetcher fetcher = YTFetcher.from_video_ids( video_ids=['video1', 'video2', 'video3'] ) # Rest is same ... ``` -------------------------------- ### CLI: Fetch from Video IDs Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/QUICKSTART.md Use the CLI to fetch data from specific video IDs, listing them after the 'video' command and specifying the output format. ```bash # Fetch from video IDs ytfetcher video video1 video2 video3 -f txt ``` -------------------------------- ### CLI: Search Videos Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/QUICKSTART.md Use the CLI to search for videos based on a query, specifying the search term, maximum results, and output format. ```bash # Search ytfetcher search "Python tutorial" -m 10 -f json ``` -------------------------------- ### CLI: Custom Metadata Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/QUICKSTART.md Specify custom metadata fields like 'title' and 'view_count' to be fetched for each video using the `--metadata` flag. ```bash # Custom metadata ytfetcher channel TheOffice -m 50 --metadata title view_count ``` -------------------------------- ### PreviewRenderer Constructor Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/api-reference/preview.md Initializes a new instance of the PreviewRenderer class. This object is used to render previews of YouTube data. ```APIDOC ## Constructor PreviewRenderer() ### Description Initializes a new instance of the PreviewRenderer class. ### Parameters None ### Returns `PreviewRenderer` instance ### Example ```python from ytfetcher.services import PreviewRenderer preview = PreviewRenderer() ``` ``` -------------------------------- ### Import All Configuration Components Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/api-reference/config.md Imports all available configuration classes and functions from the ytfetcher.config module for easy access. ```python from ytfetcher.config import ( FetchOptions, HTTPConfig, GenericProxyConfig, WebshareProxyConfig, ProxyConfig, default_cache_path, setup_logging ) ``` -------------------------------- ### Fetch Data Using Search Query Source: https://github.com/kaya70875/ytfetcher/blob/main/docs/cli.md Fetches transcripts and metadata based on a search query, mimicking YouTube's search functionality. Allows specifying the maximum number of results and output format. ```bash ytfetcher search -m ``` ```bash ytfetcher search "AI Getting Jobs" -m 20 -f json ``` -------------------------------- ### TranscriptFetcher Constructor Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/api-reference/transcript-fetcher.md Initializes the TranscriptFetcher with a list of video IDs and optional configuration for HTTP, proxies, languages, and manually created transcripts. ```APIDOC ## TranscriptFetcher Constructor ### Description Initializes the TranscriptFetcher with a list of video IDs and optional configuration for HTTP, proxies, languages, and manually created transcripts. ### Parameters #### Parameters - **video_ids** (list[str]) - Required - List of YouTube video IDs to fetch transcripts for - **http_config** (HTTPConfig | None) - Optional - HTTP configuration for headers and timeout settings. Defaults to HTTPConfig(). - **proxy_config** (ProxyConfig | None) - Optional - Optional proxy configuration for requests. Defaults to None. - **languages** (Iterable[str] | None) - Optional - Language codes in priority order (e.g., ['de', 'en']). Defaults to None. - **manually_created** (bool) - Optional - If True, only fetch manually created transcripts (requires languages). Defaults to False. ### Raises - `TranscriptFetchError`: If `manually_created=True` but no `languages` are specified ### Example ```python from ytfetcher._transcript_fetcher import TranscriptFetcher from ytfetcher.config import HTTPConfig fetcher = TranscriptFetcher( video_ids=['dQw4w9WgXcQ', 'jNQXAC9IVRw'], languages=['en'], manually_created=False ) result = fetcher.fetch() ``` ``` -------------------------------- ### YouTube Data Fetching with Configuration Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/README.md Fetches YouTube data with custom configurations including filters, language preferences, and caching. Requires importing FetchOptions and filter functions. ```python from ytfetcher import YTFetcher from ytfetcher.config import FetchOptions from ytfetcher.filters import min_views options = FetchOptions( filters=[min_views(5000)], languages=['en'], cache_enabled=True ) fetcher = YTFetcher.from_channel("TheOffice", options=options) data = fetcher.fetch_youtube_data() ``` -------------------------------- ### Fetch by Video IDs via CLI Source: https://github.com/kaya70875/ytfetcher/blob/main/README.md Fetches data for specific video IDs using the CLI. Provide one or more video IDs and the desired output format. ```bash ytfetcher video video_id1 video_id2 ... -f json ``` -------------------------------- ### Initialize SQLiteCache Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/api-reference/cache.md Instantiate the SQLiteCache class. Specify the directory for the database file and optionally set the time-to-live (TTL) for cache entries in days. The cache directory will be created if it doesn't exist. ```python from ytfetcher.cache import SQLiteCache # Using default cache location cache = SQLiteCache(cache_dir="~/.cache/ytfetcher") # Using custom location cache = SQLiteCache(cache_dir="/custom/cache/path", ttl=30) ``` -------------------------------- ### Import Shortcuts for ytfetcher Components Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/QUICKSTART.md Provides a comprehensive list of import shortcuts for various classes and functions within the ytfetcher library, including core components, configuration options, filters, services, utilities, and exceptions. ```python # All exports from ytfetcher import YTFetcher, ChannelData, VideoTranscript, DLSnippet from ytfetcher.config import FetchOptions, HTTPConfig, GenericProxyConfig, WebshareProxyConfig from ytfetcher.filters import min_duration, max_duration, min_views, max_views, filter_by_title from ytfetcher.services import JSONExporter, CSVExporter, TXTExporter, PreviewRenderer from ytfetcher.utils import channel_data_to_rows from ytfetcher.exceptions import YTFetcherError, ChannelNotFound, TranscriptFetchError ``` -------------------------------- ### Initialize Fetcher from Playlist ID Source: https://github.com/kaya70875/ytfetcher/blob/main/README.md Initializes the YTFetcher to retrieve metadata and transcripts for all videos within a specified YouTube playlist ID. ```python from ytfetcher import YTFetcher fetcher = YTFetcher.from_playlist_id( playlist_id="playlistid1254" ) # Rest is same ... ``` -------------------------------- ### Enable and Configure Cache in FetchOptions Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/api-reference/cache.md Sets up FetchOptions to enable caching, specifying the cache path and TTL. This ensures that fetched data is stored and retrieved from the cache. ```python from ytfetcher import YTFetcher from ytfetcher.config import FetchOptions options = FetchOptions( cache_enabled=True, cache_path="~/.cache/ytfetcher", cache_ttl=7 ) fetcher = YTFetcher.from_channel( channel_handle="TheOffice", options=options ) # First call: fetches from YouTube, caches results data1 = fetcher.fetch_youtube_data() # Second call: returns cached results for same config data2 = fetcher.fetch_youtube_data() ``` -------------------------------- ### CLI: Combine Multiple Filters Source: https://github.com/kaya70875/ytfetcher/blob/main/README.md Command to apply multiple filters simultaneously, including minimum views, minimum duration, and title substring. This allows for precise video selection. ```bash ytfetcher channel TheOffice -m 50 -f json --min-views 1000 --min-duration 300 --includes-title "tutorial" ``` -------------------------------- ### Prepare YouTube Channel Data for ML with Pandas Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/MODULE_INDEX.md Fetches YouTube data, converts it into a row-based format suitable for machine learning, and saves it as a CSV file using Pandas. Requires importing YTFetcher, channel_data_to_rows, and pandas. ```python from ytfetcher.utils import channel_data_to_rows import pandas as pd fetcher = YTFetcher.from_channel("TheOffice", max_results=100) data = fetcher.fetch_youtube_data() rows = channel_data_to_rows(data, include_comments=False) df = pd.DataFrame(rows) df.to_csv("transcripts.csv") ``` -------------------------------- ### Use Custom HTTP Proxy via CLI Source: https://github.com/kaya70875/ytfetcher/blob/main/README.md Fetches data using custom HTTP and HTTPS proxies via the CLI. Specify the proxy URLs with authentication details if needed. ```bash ytfetcher -f json --http-proxy "http://user:pass@host:port" --https-proxy "https://user:pass@host:port" ``` -------------------------------- ### Initialize YTFetcher from a Search Query Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/api-reference/ytfetcher.md Use this method to create a YTFetcher instance for retrieving data based on a global YouTube search query. You can specify the maximum number of search results to process. ```python fetcher = YTFetcher.from_search( query="Artificial Intelligence", max_results=10 ) ``` -------------------------------- ### Type Hinting for Video Data Processing Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/QUICKSTART.md Illustrates type hints for processing video data, including metadata, transcripts, and comments. Also shows how to handle failed transcript data. ```python from ytfetcher import YTFetcher, ChannelData, DLSnippet, Transcript, Comment from ytfetcher.config import FetchOptions from ytfetcher.models import FailedTranscript def process_videos(data: list[ChannelData]) -> None: for video in data: metadata: DLSnippet = video.metadata transcripts: list[Transcript] = video.transcripts comments: list[Comment] = video.comments def handle_failures(failed: list[FailedTranscript]) -> None: for item in failed: print(f"{item.video_id}: {item.reason}") ``` -------------------------------- ### Fetch Manually Created Transcripts Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/QUICKSTART.md Configure `FetchOptions` to specifically fetch manually created transcripts, which is slower but may be necessary in some cases. The default `manually_created=False` fetches any available transcript. ```python # Default: fetches any available transcript (faster) options = FetchOptions(manually_created=False) # Required: only manually created (slower) options = FetchOptions(manually_created=True, languages=['en']) ``` -------------------------------- ### Initialize HTTPConfig with Custom Headers Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/api-reference/config.md Provide a dictionary of custom HTTP headers to the HTTPConfig constructor to override default headers. This allows for specific request customization. ```python from ytfetcher.config import HTTPConfig, FetchOptions from ytfetcher import YTFetcher config = HTTPConfig( headers={ "User-Agent": "My Custom User Agent", "Accept-Language": "en-US,en;q=0.9" } ) options = FetchOptions(http_config=config) fetcher = YTFetcher.from_channel( channel_handle="TheOffice", options=options ) ``` -------------------------------- ### Prepare Video Content for AI Summarization Source: https://github.com/kaya70875/ytfetcher/blob/main/docs/examples.md This function fetches a video's transcript and metadata, strips out timing information, and saves it as a clean .txt file suitable for AI tools. It uses TXTExporter with timing disabled and specifies allowed metadata. ```python from ytfetcher import YTFetcher from ytfetcher.services import TXTExporter def prepare_for_ai(video_ids: list[str]): # 1. Initialize using the video URL factory fetcher = YTFetcher.from_video_ids(video_ids=video_ids) print(f"Downloading transcript for AI processing...") channel_data = fetcher.fetch_youtube_data() # 2. Extract the text only for a clean 'context' string full_transcript = "" for video in channel_data: # We join the transcript segments into one clean block of text full_transcript = " ".join([t.text for t in video.transcripts]) title = video.metadata.title if video.metadata else "Unknown Title" # 3. Use your TXTExporter with specific settings # We turn off 'timing' to make it readable for an AI exporter = TXTExporter( channel_data=channel_data, filename="ai_context_ready", timing=False, allowed_metadata_list=['title', 'url', 'description'] ) exporter.write() print(f"--- PREVIEW FOR AI ---") print(f"Title: {title}") print(f"Transcript Length: {len(full_transcript)} characters") print(f"File saved to: ./exports/ai_context_ready.txt") if __name__ == "__main__": url = ['NKnZYvZA7w4'] prepare_for_ai(url) ``` -------------------------------- ### CLI: Fetch with Comments Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/QUICKSTART.md Fetch data from a channel and include a specified number of comments using the `--comments` flag. ```bash # With comments ytfetcher channel TheOffice -m 20 --comments 10 -f json ``` -------------------------------- ### Clean Cache via CLI Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/api-reference/cache.md Demonstrates how to clean the cache using the ytfetcher command-line interface. The --cache-path option can be used to specify a non-default cache location. ```bash ytfetcher cache --clean ytfetcher cache --clean --cache-path ~/.cache/ytfetcher ``` -------------------------------- ### Using Proxy for Rate Limit Avoidance Source: https://github.com/kaya70875/ytfetcher/blob/main/docs/cli.md Utilizes Webshare proxy service with provided credentials to avoid rate limits when fetching a large number of videos. ```bash ytfetcher channel TheOffice -m 100 -f json \ --webshare-proxy-username "your_username" \ --webshare-proxy-password "your_password" ``` -------------------------------- ### Import PreviewRenderer Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/api-reference/preview.md This snippet shows the necessary import statement for using the PreviewRenderer class from the ytfetcher.services module. ```python from ytfetcher.services import PreviewRenderer ``` -------------------------------- ### FetchOptions with Custom Cache Path and TTL Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/api-reference/config.md Specify a custom directory for storing cached transcripts and set a longer cache time-to-live. This is useful for managing cache location and data freshness policies. ```python options = FetchOptions( cache_path="/custom/cache/directory", cache_ttl=30 # 30-day TTL ) ``` -------------------------------- ### Configure Fetcher with Custom HTTP Headers Source: https://github.com/kaya70875/ytfetcher/blob/main/README.md Initializes YTFetcher with custom HTTP headers to mimic browser behavior. Useful for overriding default User-Agent. ```python from ytfetcher import YTFetcher from ytfetcher.config import HTTPConfig, FetchOptions custom_config = HTTPConfig( headers={"User-Agent": "ytfetcher/1.0"} ) options = FetchOptions( http_config=custom_config ) fetcher = YTFetcher.from_channel( channel_handle="TheOffice", max_results=10, options=options ) ``` -------------------------------- ### Fetch Channel Data from Specific Tabs Source: https://github.com/kaya70875/ytfetcher/blob/main/docs/cli.md Demonstrates how to fetch data from different tabs of a YouTube channel, including regular uploads, shorts, and live streams. Use the `--tab` option to specify the desired feed. ```bash # Regular uploads (default) ytfetcher channel TheOffice -m 20 --tab videos -f json ``` ```bash # Shorts feed ytfetcher channel TheOffice -m 20 --tab shorts -f json ``` ```bash # Live/Streams feed ytfetcher channel TheOffice -m 20 --tab streams -f json ``` -------------------------------- ### Fetch YouTube Data with Custom Proxy Configuration Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/MODULE_INDEX.md Fetches YouTube data using a custom proxy server, configured with username and password. Requires importing YTFetcher, FetchOptions, and WebshareProxyConfig. ```python from ytfetcher.config import FetchOptions, WebshareProxyConfig proxy = WebshareProxyConfig(username="user", password="pass") options = FetchOptions(proxy_config=proxy) fetcher = YTFetcher.from_channel("TheOffice", options=options) data = fetcher.fetch_youtube_data() ``` -------------------------------- ### Fetch Channel Streams via CLI Source: https://github.com/kaya70875/ytfetcher/blob/main/README.md Fetches data from the 'streams' (live/past streams) tab of a YouTube channel using the CLI. ```bash # Fetch from the Live/Streams tab ytfetcher channel TheOffice -m 20 --tab streams -f json ``` -------------------------------- ### TXTExporter Constructor Source: https://github.com/kaya70875/ytfetcher/blob/main/_autodocs/api-reference/exporters.md Initializes the TXTExporter to export data as plain text. It takes channel data and optional parameters for metadata, timing, filename, and output directory. ```APIDOC ## TXTExporter Constructor ### Description Initializes the TXTExporter to export data as plain text. It takes channel data and optional parameters for metadata, timing, filename, and output directory. ### Parameters - **channel_data** (list[ChannelData]) - Required - The channel data to export. - **allowed_metadata_list** (Sequence[METADATA_LIST]) - Optional - Defaults to `DEFAULT_METADATA`. A sequence of metadata fields to include. - **timing** (bool) - Optional - Defaults to `True`. Whether to include timing information. - **filename** (str) - Optional - Defaults to 'data'. The base name for the output file. - **output_dir** (str | None) - Optional - Defaults to `None`. The directory to save the output file. If None, saves to the current directory. ### Example ```python from ytfetcher.services import TXTExporter exporter = TXTExporter( channel_data=channel_data, allowed_metadata_list=['title', 'description'], timing=True, filename='transcripts', output_dir='./output' ) ``` ``` -------------------------------- ### Fetch Comments with Transcripts Source: https://github.com/kaya70875/ytfetcher/blob/main/docs/cli.md Fetches videos, includes a specified number of top comments per video, exports the data as CSV, and saves it to a specified directory. ```bash ytfetcher channel TheOffice -m 10 --comments 5 -f csv -o ./data ```