### Install ultimate-sitemap-parser with pip Source: https://github.com/gatenlp/ultimate-sitemap-parser/blob/main/README.rst Use this command to install the library using pip. ```sh pip install ultimate-sitemap-parser ``` -------------------------------- ### Install Ultimate Sitemap Parser Source: https://context7.com/gatenlp/ultimate-sitemap-parser/llms.txt Install the library using pip or conda. ```bash pip install ultimate-sitemap-parser ``` ```bash conda install -c conda-forge ultimate-sitemap-parser ``` -------------------------------- ### Build Documentation Locally Source: https://github.com/gatenlp/ultimate-sitemap-parser/blob/main/docs/contributing.rst Build the project documentation locally using Sphinx and uv. This command starts a live build server for previewing changes. ```bash cd docs uv run make livehtml ``` -------------------------------- ### Install ultimate-sitemap-parser with conda Source: https://github.com/gatenlp/ultimate-sitemap-parser/blob/main/README.rst Use this command to install the library using conda from the conda-forge channel. ```sh conda install -c conda-forge ultimate-sitemap-parser ``` -------------------------------- ### Get Sitemap Tree for Homepage with Options Source: https://github.com/gatenlp/ultimate-sitemap-parser/blob/main/docs/changelog.rst Call usp.tree.sitemap_tree_for_homepage() with optional arguments to control sitemap discovery. Set disable_robots_txt=True or disable_known_paths=True to exclude specific discovery methods. ```python usp.tree.sitemap_tree_for_homepage(disable_robots_txt=True) ``` ```python usp.tree.sitemap_tree_for_homepage(disable_known_paths=True) ``` -------------------------------- ### Parse Robots.txt Source: https://github.com/gatenlp/ultimate-sitemap-parser/blob/main/docs/reference/formats.rst Example of a robots.txt file. This parser implements logic to detect 'Sitemap' keys, case-insensitively supporting 'Sitemap' or 'Site-map'. ```text User-agent: * Disallow: /deny/ Allow: /allow/ Sitemap: https://example.com/sitemap.xml Site-map: https://example.com/another-sitemap.xml ``` -------------------------------- ### List Sitemap Pages - Pages Format Source: https://github.com/gatenlp/ultimate-sitemap-parser/blob/main/docs/reference/cli.rst Example of listing sitemap pages using the --format pages option for a flat list of URLs. ```shell-session $ usp ls https://example.org/ --format pages https://example.org/page1.html ``` -------------------------------- ### Navigate Sitemap Tree Source: https://context7.com/gatenlp/ultimate-sitemap-parser/llms.txt Builds a sitemap tree from a homepage URL and iterates through all sitemaps and pages within the tree. This example requires importing specific sitemap objects for type checking and handling. ```python from usp.tree import sitemap_tree_for_homepage from usp.objects.sitemap import ( AbstractSitemap, IndexWebsiteSitemap, IndexXMLSitemap, IndexRobotsTxtSitemap, PagesXMLSitemap, PagesTextSitemap, PagesRSSSitemap, PagesAtomSitemap, InvalidSitemap ) tree = sitemap_tree_for_homepage('https://www.example.org/') # The root is always IndexWebsiteSitemap print(f"Root URL: {tree.url}") print(f"Root type: {type(tree).__name__}") # Iterate over all sitemaps in the tree for sitemap in tree.all_sitemaps(): print(f"\nSitemap: {sitemap.url}") print(f"Type: {type(sitemap).__name__}") # Check sitemap type and handle accordingly if isinstance(sitemap, InvalidSitemap): print(f"Invalid sitemap reason: {sitemap.reason}") elif isinstance(sitemap, IndexRobotsTxtSitemap): print(f"robots.txt with {len(sitemap.sub_sitemaps)} sitemaps declared") elif isinstance(sitemap, IndexXMLSitemap): print(f"XML index with {len(sitemap.sub_sitemaps)} sub-sitemaps") elif isinstance(sitemap, PagesXMLSitemap): print(f"XML sitemap with {len(sitemap.pages)} pages") elif isinstance(sitemap, PagesRSSSitemap): print(f"RSS feed with {len(sitemap.pages)} items") # Access direct children only (not recursive) for sub_sitemap in tree.sub_sitemaps: print(f"Direct child: {sub_sitemap.url}") # Access pages in a specific sitemap (not recursive) for sub_sitemap in tree.sub_sitemaps: for page in sub_sitemap.pages: print(f"Page in {sub_sitemap.url}: {page.url}") ``` -------------------------------- ### List Sitemap Pages - Strip URL Option Source: https://github.com/gatenlp/ultimate-sitemap-parser/blob/main/docs/reference/cli.rst Example of listing sitemap pages with the --strip-url option. This removes the base URL from the output. ```shell-session $ usp ls https://example.org/ --strip-url https://example.org/ /robots.txt /sitemap.xml /page1.html ``` -------------------------------- ### List Sitemap Pages - Default Output Source: https://github.com/gatenlp/ultimate-sitemap-parser/blob/main/docs/reference/cli.rst Example of listing sitemap pages with default tab-indented format. Shows nested structure including robots.txt and sitemap.xml. ```shell-session $ usp ls https://example.org/ https://example.org/ https://example.org/robots.txt https://example.org/sitemap.xml https://example.org/page1.html ``` -------------------------------- ### Check pyexpat Version Source: https://github.com/gatenlp/ultimate-sitemap-parser/blob/main/docs/guides/security.rst Use this command to check the installed version of the pyexpat module. Ensure it is greater than 2.4.0 for security. ```python-console >>> import pyexpat >>> pyexpat.EXPAT_VERSION 'expat_2.4.7' ``` -------------------------------- ### Parse XML Sitemap Index Source: https://github.com/gatenlp/ultimate-sitemap-parser/blob/main/docs/reference/formats.rst Example of an XML sitemap index file. This parser supports the standard Sitemap index format, allowing for flexible parsing of timestamps and ignoring unexpected tags. ```xml https://example.com/sitemap1.xml.gz 2023-01-01T12:00:00+00:00 https://example.com/sitemap2.xml 2023-01-02T12:00:00+00:00 ``` -------------------------------- ### Parse Plain Text Sitemap Source: https://github.com/gatenlp/ultimate-sitemap-parser/blob/main/docs/reference/formats.rst Example of a plain text sitemap. The parser reads line by line, identifying valid URLs with HTTP/HTTPS protocols and non-empty hostnames, ignoring non-URL content. ```text https://example.com/page1.html http://example.com/page2.html This is not a URL and will be ignored. https://example.com/page3.html ``` -------------------------------- ### Parse RSS 2.0 XML Source: https://github.com/gatenlp/ultimate-sitemap-parser/blob/main/docs/reference/formats.rst This snippet shows an example of an RSS 2.0 feed. Items without a title or description are ignored, and items without a link are also ignored. ```xml Example Feed http://www.example.com This is an example feed. Example Item 1 http://www.example.com/item1 This is the first example item. Mon, 01 Jan 2024 12:00:00 GMT Example Item 2 http://www.example.com/item2 This is the second example item. Tue, 02 Jan 2024 13:00:00 GMT ``` -------------------------------- ### Parse XML URL Set Source: https://github.com/gatenlp/ultimate-sitemap-parser/blob/main/docs/reference/formats.rst Example of an XML sitemap URL set. The parser handles standard URL set formats, including non-standard features like truncated files and flexible date parsing. ```xml https://example.com/page1.html 2023-01-01T12:00:00+00:00 weekly 0.8 https://example.com/page2.html 2023-01-02T12:00:00+00:00 ``` -------------------------------- ### Parse Google News Sitemap Extension Source: https://github.com/gatenlp/ultimate-sitemap-parser/blob/main/docs/reference/formats.rst Example of an XML sitemap with the Google News extension. This extension provides additional information about news stories. If present, data is stored as a SitemapNewsStory object. ```xml https://example.com/article1.html The Example Times en 2023-10-26T10:00:00+00:00 Example Article Title ``` -------------------------------- ### Parse Google Image Sitemap Extension Source: https://github.com/gatenlp/ultimate-sitemap-parser/blob/main/docs/reference/formats.rst Example of an XML sitemap with the Google Image extension. This extension provides additional information about images on a page. If present, data is stored as a list of SitemapImage objects. ```xml https://example.com/page1.html https://example.com/image1.jpg A descriptive caption for image 1. Image Title 1 https://example.com/image2.png ``` -------------------------------- ### Fetch Sitemap Tree for a Homepage Source: https://github.com/gatenlp/ultimate-sitemap-parser/blob/main/docs/get-started.rst Use this function to get a tree representation of a website's sitemap structure. It lazily loads sitemap data to reduce memory usage for large sitemaps. ```python from usp.tree import sitemap_tree_for_homepage tree = sitemap_tree_for_homepage('https://example.org/') ``` -------------------------------- ### Filter Sub-Sitemaps by Index Content Source: https://github.com/gatenlp/ultimate-sitemap-parser/blob/main/docs/guides/fetch-parse.rst Use a recurse_list_callback to filter the list of sub-sitemap URLs within an index sitemap. The callback receives the list of URLs and should return a filtered list to fetch. This example fetches only if both 'blog' and 'products' sub-sitemaps are present. ```python from usp.tree import sitemap_tree_for_homepage def filter_list_callback(urls: List[str], recursion_level: int, parent_urls: Set[str]) -> List[str]: if any('blog' in url for url in urls) and any('products' in url for url in urls): return urls return [] tree = sitemap_tree_for_homepage( 'https://www.example.org/', recurse_list_callback=filter_list_callback, ) ``` -------------------------------- ### Configure Requests Client Options Source: https://github.com/gatenlp/ultimate-sitemap-parser/blob/main/docs/guides/http-client.rst Instantiate RequestsWebClient with custom options like wait time and random wait, then set a timeout. Pass the configured client to sitemap_tree_for_homepage. ```python from usp.web_client.requests_client import RequestsWebClient from usp.tree import sitemap_tree_for_homepage client = RequestsWebClient(wait=5.0, random_wait=True) client.set_timeout(30) tree = sitemap_tree_for_homepage('https://www.example.org/', web_client=client) ``` -------------------------------- ### Ultimate Sitemap Parser CLI Usage Source: https://github.com/gatenlp/ultimate-sitemap-parser/blob/main/docs/reference/cli.rst Basic usage of the usp command-line tool. Use -h for help. ```none usage: usp [-h] [-v] ... Ultimate Sitemap Parser options: -h, --help show this help message and exit -v, --version show program's version number and exit commands: ls List sitemap pages ``` -------------------------------- ### Implement Custom HTTPX Client Source: https://github.com/gatenlp/ultimate-sitemap-parser/blob/main/docs/guides/http-client.rst Create custom client classes by subclassing AbstractWebClient and related response classes to integrate libraries like HTTPX. Instantiate the custom client and pass it to sitemap_tree_for_homepage. ```python from usp.web_client.abstract_client import AbstractWebClient, AbstractWebClientSuccessResponse, WebClientErrorResponse class HttpxWebClientSuccessResponse(AbstractWebClientSuccessResponse): ... class HttpxWebClientErrorResponse(WebClientErrorResponse): pass class HttpxWebClient(AbstractWebClient): ... client = HttpxWebClient() tree = sitemap_tree_for_homepage('https://www.example.org/', web_client=client) ``` -------------------------------- ### USP CLI Commands for Sitemap Exploration Source: https://context7.com/gatenlp/ultimate-sitemap-parser/llms.txt Utilize the USP command-line interface for quick sitemap exploration without writing Python code. Supports listing pages, filtering output, and controlling discovery behavior. ```bash # List all sitemap pages in tree format usp ls https://www.example.org/ # Output: # https://www.example.org/ # https://www.example.org/robots.txt # https://www.example.org/sitemap.xml # https://www.example.org/page1 # https://www.example.org/page2 # List pages only (flat format, one URL per line) usp ls https://www.example.org/ -f pages # Strip the base URL from output usp ls https://www.example.org/ -u # Output: /page1, /page2, etc. # Skip robots.txt discovery usp ls https://www.example.org/ --no-robots # Skip well-known path discovery usp ls https://www.example.org/ --no-known # Enable verbose logging usp ls https://www.example.org/ -v # INFO level usp ls https://www.example.org/ -vv # DEBUG level # Write logs to file usp ls https://www.example.org/ -v --log-file sitemap.log # Show version usp --version ``` -------------------------------- ### Download Integration Test Cassettes Source: https://github.com/gatenlp/ultimate-sitemap-parser/blob/main/docs/contributing.rst Download cassette files for integration tests from a separate repository. This is necessary to run tests against real-world sitemaps. ```bash uv run tests/integration/download.py uv run pytest --integration tests/integration ``` -------------------------------- ### Run Ruff Linting and Formatting Source: https://github.com/gatenlp/ultimate-sitemap-parser/blob/main/docs/contributing.rst Execute Ruff for code linting and formatting. Ensure code quality and consistency. ```bash uv run ruff check --fix uv run ruff format ``` -------------------------------- ### Load Sitemap Tree from Pickle File Source: https://github.com/gatenlp/ultimate-sitemap-parser/blob/main/docs/guides/saving.rst Load a sitemap tree object from a Pickle file. This process requires the tree object to be loaded into memory, so be cautious with very large sitemaps. Ensure the Pickle file is from a trusted source and compatible with the current USP version. ```python # Later, to load the tree back with open('sitemap.pickle', 'rb') as f: tree = pickle.load(f) for page in tree.all_pages(): print(page.url) ``` -------------------------------- ### Implement Custom Web Client in Python Source: https://context7.com/gatenlp/ultimate-sitemap-parser/llms.txt Create a custom web client by inheriting from AbstractWebClient for specialized HTTP handling. This allows for custom authentication, proxies, rate limiting, or headers. ```python from usp.tree import sitemap_tree_for_homepage from usp.web_client.abstract_client import ( AbstractWebClient, AbstractWebClientResponse, AbstractWebClientSuccessResponse, WebClientErrorResponse ) import requests class CustomWebClientResponse(AbstractWebClientSuccessResponse): def __init__(self, response: requests.Response): self._response = response def status_code(self) -> int: return self._response.status_code def status_message(self) -> str: return self._response.reason def header(self, name: str) -> str | None: return self._response.headers.get(name.lower()) def raw_data(self) -> bytes: return self._response.content def url(self) -> str: return self._response.url class CustomWebClient(AbstractWebClient): def __init__(self, api_key: str = None): self._session = requests.Session() self._max_length = None if api_key: self._session.headers['Authorization'] = f'Bearer {api_key}' self._session.headers['User-Agent'] = 'CustomSitemapBot/1.0' def set_max_response_data_length(self, max_length: int | None) -> None: self._max_length = max_length def get(self, url: str) -> AbstractWebClientResponse: try: response = self._session.get(url, timeout=30) if 200 <= response.status_code < 300: return CustomWebClientResponse(response) else: return WebClientErrorResponse( message=f"{response.status_code} {response.reason}", retryable=response.status_code >= 500 ) except requests.RequestException as e: return WebClientErrorResponse(message=str(e), retryable=True) # Use the custom web client client = CustomWebClient(api_key='your-api-key') tree = sitemap_tree_for_homepage('https://api.example.org/', web_client=client) ``` -------------------------------- ### Profile Performance During Integration Tests Source: https://github.com/gatenlp/ultimate-sitemap-parser/blob/main/docs/contributing.rst Profile performance during integration tests using pyinstrument. This helps in identifying performance bottlenecks. Reports can be generated as interactive HTML. ```bash uv run pyinstrument -m pytest --integration tests/integration ``` -------------------------------- ### LocalWebClient Source: https://github.com/gatenlp/ultimate-sitemap-parser/blob/main/docs/reference/api/usp.web_client.abstract_client.rst A web client implementation that reads content from the local file system. ```APIDOC ## LocalWebClient ### Description A web client implementation that reads content from the local file system. ### Inheritance Inherits from `AbstractWebClient`. ``` -------------------------------- ### Parse Google News Sitemap Source: https://context7.com/gatenlp/ultimate-sitemap-parser/llms.txt Parses a Google News sitemap from an XML string. This example demonstrates how to extract news-specific information like title, publication date, and keywords from news articles within the sitemap. ```python news_sitemap = """ https://www.example.org/news/article1 Example News en 2024-01-15T10:30:00Z Breaking: Important News Story breaking, news, important """ sitemap = sitemap_from_str(news_sitemap) for page in sitemap.all_pages(): if page.news_story: print(f"Title: {page.news_story.title}") print(f"Published: {page.news_story.publish_date}") print(f"Keywords: {page.news_story.keywords}") ``` -------------------------------- ### NoWebClientException Source: https://github.com/gatenlp/ultimate-sitemap-parser/blob/main/docs/reference/api/usp.web_client.abstract_client.rst Exception raised when no suitable web client is found. ```APIDOC ## NoWebClientException ### Description Exception raised when no suitable web client is found. ### Inheritance Inherits from `Exception`. ``` -------------------------------- ### Iterate Sitemap Pages Directly (Good Practice) Source: https://github.com/gatenlp/ultimate-sitemap-parser/blob/main/docs/guides/performance.rst This is the recommended way to iterate through sitemap pages. It loads one sitemap's pages into memory at a time, making it memory-efficient for large sitemaps. ```python for page in tree.all_pages(): print(page.url) ``` -------------------------------- ### AbstractWebClient Source: https://github.com/gatenlp/ultimate-sitemap-parser/blob/main/docs/reference/api/usp.web_client.abstract_client.rst Abstract base class for web clients, defining the interface for fetching web content. ```APIDOC ## AbstractWebClient ### Description Abstract base class for web clients. Defines the interface for fetching web content. ### Members This class has the following members: - `__init__` - `get` - `post` - `delete` - `put` - `close` ``` -------------------------------- ### Configure Built-in RequestsWebClient in Python Source: https://context7.com/gatenlp/ultimate-sitemap-parser/llms.txt Configure the default RequestsWebClient for options like proxies, SSL verification, and rate limiting. A custom requests.Session can be provided for further customization. ```python from usp.tree import sitemap_tree_for_homepage from usp.web_client.requests_client import RequestsWebClient import requests # Create a custom session with specific settings session = requests.Session() session.headers['Accept-Language'] = 'en-US,en;q=0.9' # Configure the built-in web client client = RequestsWebClient( verify=True, # Verify SSL certificates (default: True) wait=1.0, # Wait 1 second between requests random_wait=True, # Randomize wait time (0.5x to 1.5x) session=session # Use custom session ) # Set a proxy client.set_proxies({ 'http': 'http://proxy.example.com:8080', 'https': 'http://proxy.example.com:8080' }) # Set custom timeout (connect_timeout, read_timeout) client.set_timeout((5.0, 30.0)) # Use with sitemap parser tree = sitemap_tree_for_homepage('https://www.example.org/', web_client=client) ``` -------------------------------- ### List Sitemap Pages Command Usage Source: https://github.com/gatenlp/ultimate-sitemap-parser/blob/main/docs/reference/cli.rst Usage for the 'usp ls' command to download, parse, and list sitemap structure. Specify the URL of the site. ```none usage: usp ls [-h] [-f FORMAT] [-r] [-k] [-u] [-v/-vv] [-l LOG_FILE] url download, parse and list the sitemap structure positional arguments: url URL of the site including protocol options: -h, --help show this help message and exit -f FORMAT, --format FORMAT set output format (default: tabtree) choices: tabtree: Sitemaps and pages, nested with tab indentation pages: Flat list of pages, one per line -r, --no-robots don't discover sitemaps through robots.txt -k, --no-known don't discover sitemaps through well-known URLs -u, --strip-url strip the supplied URL from each page and sitemap URL -v, --verbose increase output verbosity (-v=INFO, -vv=DEBUG) -l LOG_FILE, --log-file LOG_FILE write log to this file and suppress console output ``` -------------------------------- ### Core API: sitemap_tree_for_homepage Source: https://context7.com/gatenlp/ultimate-sitemap-parser/llms.txt The main entry point for fetching and parsing a website's complete sitemap tree. It discovers sitemaps from robots.txt and known paths, recursively fetches all sub-sitemaps, and returns a tree structure of sitemaps and pages. ```APIDOC ## Core API: sitemap_tree_for_homepage ### Description The main entry point for fetching and parsing a website's complete sitemap tree. It discovers sitemaps from robots.txt and known paths, recursively fetches all sub-sitemaps, and returns a tree structure of sitemaps and pages. ### Method ```python from usp.tree import sitemap_tree_for_homepage ``` ### Parameters #### Path Parameters - **homepage_url** (str) - Required - The URL of the homepage for which to parse the sitemap. #### Query Parameters - **use_robots** (bool) - Optional - Discover sitemaps from robots.txt (default: True). - **use_known_paths** (bool) - Optional - Check well-known sitemap paths (default: True). - **extra_known_paths** (set[str]) - Optional - Additional paths to check for sitemaps. - **normalize_homepage_url** (bool) - Optional - Normalize URL to domain root (default: True). ### Request Example ```python # Basic usage tree = sitemap_tree_for_homepage('https://www.example.org/') # Advanced usage with options tree = sitemap_tree_for_homepage( 'https://www.example.org/', use_robots=True, use_known_paths=True, extra_known_paths={'custom-sitemap.xml'}, normalize_homepage_url=True ) ``` ### Response #### Success Response (200) - **tree** (SitemapTree) - A tree structure representing the sitemaps and pages found. #### Response Example ```python # Iterating over pages for page in tree.all_pages(): print(f"URL: {page.url}") print(f"Last Modified: {page.last_modified}") print(f"Priority: {page.priority}") print(f"Change Frequency: {page.change_frequency}") # Google News story data if page.news_story: print(f"News Title: {page.news_story.title}") print(f"Publish Date: {page.news_story.publish_date}") # Image sitemap data if page.images: for image in page.images: print(f"Image URL: {image.loc}") # Get total count of pages page_count = sum(1 for _ in tree.all_pages()) print(f"Total pages found: {page_count}") ``` ``` -------------------------------- ### Pickle Sitemap Tree for Caching Source: https://context7.com/gatenlp/ultimate-sitemap-parser/llms.txt Save a fetched sitemap tree to a pickle file for later retrieval, enabling caching and offline analysis. Load the tree using `pickle.load()`. Ensure the same USP version is used for pickling and unpickling. ```python import pickle from usp.tree import sitemap_tree_for_homepage # Fetch and save the sitemap tree tree = sitemap_tree_for_homepage('https://www.example.org/') # Save to pickle file with open('sitemap_cache.pickle', 'wb') as f: pickle.dump(tree, f) # Delete the tree to free memory (removes temporary page files) del tree # Later: load the tree from cache with open('sitemap_cache.pickle', 'rb') as f: tree = pickle.load(f) # Use the restored tree normally for page in tree.all_pages(): print(page.url) # Warning: Pickle files are version-dependent # Only load pickle files created with the same USP version ``` -------------------------------- ### Avoid Loading All Pages into Memory (Bad Practice) Source: https://github.com/gatenlp/ultimate-sitemap-parser/blob/main/docs/guides/performance.rst Avoid converting the result of `all_pages()` to a list if memory efficiency is a concern. This loads all sitemap pages into memory at once, which can be problematic for very large sitemaps. ```python for page in list(tree.all_pages()): print(page.url) ``` -------------------------------- ### Abstract Sitemap Classes Source: https://github.com/gatenlp/ultimate-sitemap-parser/blob/main/docs/reference/api/usp.objects.sitemap.rst Documentation for abstract base classes related to sitemaps. ```APIDOC ## AbstractSitemap ### Description Abstract base class for all sitemap objects. ### Members - `members`: Includes all public members of the class. - `show-inheritance`: Indicates that inherited members are shown. ``` ```APIDOC ## InvalidSitemap ### Description Represents an invalid sitemap. ### Members - `members`: Includes all public members of the class. - `show-inheritance`: Indicates that inherited members are shown. ``` -------------------------------- ### Add Proxy Support with RequestsWebClient Source: https://github.com/gatenlp/ultimate-sitemap-parser/blob/main/docs/changelog.rst Use this method to configure proxy settings for the RequestsWebClient. This is useful when making requests from environments with network restrictions. ```python RequestsWebClient.set_proxies() ``` -------------------------------- ### Index Sitemap Classes Source: https://github.com/gatenlp/ultimate-sitemap-parser/blob/main/docs/reference/api/usp.objects.sitemap.rst Documentation for classes representing index sitemaps. ```APIDOC ## AbstractIndexSitemap ### Description Abstract base class for index sitemap objects. ### Members - `members`: Includes all public members of the class. - `show-inheritance`: Indicates that inherited members are shown. - `inherited-members`: Includes members inherited from base classes. ``` ```APIDOC ## IndexWebsiteSitemap ### Description Represents an index sitemap for a website. ### Members - `members`: Includes all public members of the class. - `show-inheritance`: Indicates that inherited members are shown. ``` ```APIDOC ## IndexXMLSitemap ### Description Represents an index sitemap parsed from an XML file. ### Members - `members`: Includes all public members of the class. - `show-inheritance`: Indicates that inherited members are shown. ``` ```APIDOC ## IndexRobotsTxtSitemap ### Description Represents an index sitemap derived from a robots.txt file. ### Members - `members`: Includes all public members of the class. - `show-inheritance`: Indicates that inherited members are shown. ``` -------------------------------- ### AbstractWebClientResponse Source: https://github.com/gatenlp/ultimate-sitemap-parser/blob/main/docs/reference/api/usp.web_client.abstract_client.rst Abstract base class for web client responses. ```APIDOC ## AbstractWebClientResponse ### Description Abstract base class for web client responses. ### Members This class has the following members: - `status_code` - `headers` - `text` - `json` - `content` ``` -------------------------------- ### Sitemap String Parsers Source: https://github.com/gatenlp/ultimate-sitemap-parser/blob/main/docs/reference/api/usp.fetch_parse.rst Classes for parsing sitemap content from strings. ```APIDOC ## SitemapStrParser ### Description An abstract base class for parsers that operate on sitemap content provided as a string. ### Methods - **parse(sitemap_content: str)**: Parses the sitemap content string. ### Parameters - **sitemap_content** (str) - Required - The sitemap content to parse. ### Request Example ```python from usp.fetch_parse import SitemapStrParser class MyParser(SitemapStrParser): def parse(self, sitemap_content: str): # Custom parsing logic pass parser = MyParser() parser.parse("...") ``` ### Response - The `parse` method's return value depends on the specific implementation. ### Response Example ```json { "parsed_data": "..." } ``` ``` ```APIDOC ## AbstractSitemapParser ### Description Abstract base class defining the interface for sitemap parsers. ### Methods - **parse(sitemap_content: str)**: Abstract method to parse sitemap content. ### Parameters - **sitemap_content** (str) - Required - The sitemap content to parse. ### Request Example ```python from usp.fetch_parse import AbstractSitemapParser # This is an abstract class and cannot be instantiated directly. # Subclasses must implement the parse method. ``` ### Response - The `parse` method's return value is not defined for the abstract class. ``` ```APIDOC ## IndexRobotsTxtSitemapParser ### Description Parses sitemaps referenced in a robots.txt file. ### Inheritance Inherits from `SitemapStrParser`. ### Methods - **parse(robots_txt_content: str)**: Parses the robots.txt content to find sitemap URLs. ### Parameters - **robots_txt_content** (str) - Required - The content of the robots.txt file. ### Request Example ```python from usp.fetch_parse import IndexRobotsTxtSitemapParser parser = IndexRobotsTxtSitemapParser() sitemap_urls = parser.parse("Sitemap: http://example.com/sitemap.xml\n") ``` ### Response - **parse**: Returns a list of sitemap URLs found in the robots.txt. ### Response Example ```json { "sitemap_urls": ["http://example.com/sitemap.xml"] } ``` ``` ```APIDOC ## PlainTextSitemapParser ### Description Parses a sitemap that is in plain text format, typically a list of URLs. ### Inheritance Inherits from `SitemapStrParser`. ### Methods - **parse(sitemap_content: str)**: Parses the plain text sitemap content. ### Parameters - **sitemap_content** (str) - Required - The plain text sitemap content. ### Request Example ```python from usp.fetch_parse import PlainTextSitemapParser parser = PlainTextSitemapParser() urls = parser.parse("http://example.com/page1\nhttp://example.com/page2") ``` ### Response - **parse**: Returns a list of URLs from the plain text sitemap. ### Response Example ```json { "urls": ["http://example.com/page1", "http://example.com/page2"] } ``` ``` ```APIDOC ## XMLSitemapParser ### Description An abstract base class for parsing XML-based sitemaps. ### Inheritance Inherits from `SitemapStrParser`. ### Methods - **parse(sitemap_content: str)**: Abstract method to parse XML sitemap content. ### Parameters - **sitemap_content** (str) - Required - The XML sitemap content. ### Request Example ```python from usp.fetch_parse import XMLSitemapParser # This is an abstract class and cannot be instantiated directly. # Subclasses must implement the parse method. ``` ### Response - The `parse` method's return value is not defined for the abstract class. ``` ```APIDOC ## AbstractXMLSitemapParser ### Description Abstract base class for specific XML sitemap format parsers. ### Inheritance Inherits from `XMLSitemapParser`. ### Methods - **parse(sitemap_content: str)**: Abstract method to parse XML sitemap content. ### Parameters - **sitemap_content** (str) - Required - The XML sitemap content. ### Request Example ```python from usp.fetch_parse import AbstractXMLSitemapParser # This is an abstract class and cannot be instantiated directly. # Subclasses must implement the parse method. ``` ### Response - The `parse` method's return value is not defined for the abstract class. ``` ```APIDOC ## IndexXMLSitemapParser ### Description Parses an XML sitemap index file. ### Inheritance Inherits from `AbstractXMLSitemapParser`. ### Methods - **parse(sitemap_content: str)**: Parses the XML sitemap index. ### Parameters - **sitemap_content** (str) - Required - The XML sitemap index content. ### Request Example ```python from usp.fetch_parse import IndexXMLSitemapParser parser = IndexXMLSitemapParser() sitemap_index_data = parser.parse("...") ``` ### Response - **parse**: Returns data representing the sitemap index, likely containing URLs of other sitemaps. ### Response Example ```json { "sitemap_index": [ {"loc": "http://example.com/sitemap1.xml", "lastmod": "2023-01-01"} ] } ``` ``` ```APIDOC ## PagesXMLSitemapParser ### Description Parses a standard XML sitemap containing page URLs. ### Inheritance Inherits from `AbstractXMLSitemapParser`. ### Methods - **parse(sitemap_content: str)**: Parses the XML sitemap content for pages. ### Parameters - **sitemap_content** (str) - Required - The XML sitemap content. ### Request Example ```python from usp.fetch_parse import PagesXMLSitemapParser parser = PagesXMLSitemapParser() page_data = parser.parse("...") ``` ### Response - **parse**: Returns data representing the pages listed in the sitemap. ### Response Example ```json { "pages": [ {"loc": "http://example.com/page1", "lastmod": "2023-01-01", "changefreq": "daily", "priority": 0.8} ] } ``` ``` ```APIDOC ## PagesRSSSitemapParser ### Description Parses a sitemap in RSS format, typically containing page information. ### Inheritance Inherits from `AbstractXMLSitemapParser`. ### Methods - **parse(sitemap_content: str)**: Parses the RSS sitemap content. ### Parameters - **sitemap_content** (str) - Required - The RSS sitemap content. ### Request Example ```python from usp.fetch_parse import PagesRSSSitemapParser parser = PagesRSSSitemapParser() page_data = parser.parse("...") ``` ### Response - **parse**: Returns data representing the pages listed in the RSS sitemap. ### Response Example ```json { "pages": [ {"loc": "http://example.com/article1", "pubDate": "Mon, 01 Jan 2023 12:00:00 GMT"} ] } ``` ``` ```APIDOC ## PagesAtomSitemapParser ### Description Parses a sitemap in Atom format, typically containing page information. ### Inheritance Inherits from `AbstractXMLSitemapParser`. ### Methods - **parse(sitemap_content: str)**: Parses the Atom sitemap content. ### Parameters - **sitemap_content** (str) - Required - The Atom sitemap content. ### Request Example ```python from usp.fetch_parse import PagesAtomSitemapParser parser = PagesAtomSitemapParser() page_data = parser.parse("...") ``` ### Response - **parse**: Returns data representing the pages listed in the Atom sitemap. ### Response Example ```json { "pages": [ {"loc": "http://example.com/entry1", "updated": "2023-01-01T12:00:00Z"} ] } ``` ``` -------------------------------- ### Accessing SitemapPage Properties Source: https://context7.com/gatenlp/ultimate-sitemap-parser/llms.txt Iterate through all pages in a sitemap tree and access core, optional, and extension-specific properties like URL, priority, last modified date, change frequency, Google News/Image extensions, and hreflang alternates. Ensure the sitemap tree is fetched first. ```python from usp.tree import sitemap_tree_for_homepage from usp.objects.page import SitemapPageChangeFrequency tree = sitemap_tree_for_homepage('https://www.example.org/') for page in tree.all_pages(): # Core properties (always available) url = page.url # str: The page URL priority = page.priority # Decimal: Priority 0.0-1.0, default 0.5 # Optional standard sitemap properties last_modified = page.last_modified # datetime or None change_frequency = page.change_frequency # SitemapPageChangeFrequency enum or None if change_frequency: # Possible values: ALWAYS, HOURLY, DAILY, WEEKLY, MONTHLY, YEARLY, NEVER print(f"Change freq: {change_frequency.value}") # Google News extension (from news sitemaps) news = page.news_story if news: print(f"Title: {news.title}") print(f"Publish date: {news.publish_date}") print(f"Publication: {news.publication_name}") print(f"Language: {news.publication_language}") print(f"Access: {news.access}") print(f"Genres: {news.genres}") # List of strings print(f"Keywords: {news.keywords}") # List of strings print(f"Stock tickers: {news.stock_tickers}") # List of strings # Google Image extension (from image sitemaps) images = page.images if images: for img in images: print(f"Image URL: {img.loc}") print(f"Caption: {img.caption}") print(f"Geo location: {img.geo_location}") print(f"Title: {img.title}") print(f"License: {img.license}") # hreflang alternates (internationalization) alternates = page.alternates if alternates: for lang_code, alt_url in alternates: print(f"Alternate ({lang_code}): {alt_url}") ``` -------------------------------- ### RequestsWebClient Class Source: https://github.com/gatenlp/ultimate-sitemap-parser/blob/main/docs/reference/api/usp.web_client.requests_client.rst The main client class for making web requests using the requests library. ```APIDOC ## RequestsWebClient Class ### Description Provides methods for making HTTP requests. ### Members - **__init__(self, base_url: str, timeout: int = 10, **kwargs)**: Initializes the RequestsWebClient. - **get(self, url: str, **kwargs)**: Performs an HTTP GET request. - **post(self, url: str, data: Optional[Dict] = None, **kwargs)**: Performs an HTTP POST request. - **put(self, url: str, data: Optional[Dict] = None, **kwargs)**: Performs an HTTP PUT request. - **delete(self, url: str, **kwargs)**: Performs an HTTP DELETE request. - **request(self, method: str, url: str, **kwargs)**: Performs a generic HTTP request. ### Inheritance Inherits from [BaseWebClient](?module=usp.web_client.base_web_client&class=BaseWebClient) ### Inherited Members - **__init__(self, base_url: str, timeout: int = 10, **kwargs)** - **get(self, url: str, **kwargs)** - **post(self, url: str, data: Optional[Dict] = None, **kwargs)** - **put(self, url: str, data: Optional[Dict] = None, **kwargs)** - **delete(self, url: str, **kwargs)** - **request(self, method: str, url: str, **kwargs)** ``` -------------------------------- ### Profile Specific Integration Test with Pyinstrument Source: https://github.com/gatenlp/ultimate-sitemap-parser/blob/main/docs/contributing.rst Filter and profile a specific integration test using pyinstrument and pytest's -k flag. This is useful for focusing performance analysis on a particular test case. ```bash uv run pyinstrument -m pytest --integration -k bbc tests/integration ``` -------------------------------- ### Fetch and Parse Sitemap Tree Source: https://context7.com/gatenlp/ultimate-sitemap-parser/llms.txt Use `sitemap_tree_for_homepage` to fetch and parse all sitemaps for a given website. Iterate through all found pages and access their properties like URL, last modified date, priority, and change frequency. Optionally, check for Google News story data and image sitemap data. ```python from usp.tree import sitemap_tree_for_homepage # Basic usage - fetch all sitemaps for a website tree = sitemap_tree_for_homepage('https://www.example.org/') # Iterate over all pages found in the sitemap tree for page in tree.all_pages(): print(f"URL: {page.url}") print(f"Last Modified: {page.last_modified}") print(f"Priority: {page.priority}") print(f"Change Frequency: {page.change_frequency}") # Check for Google News story data if page.news_story: print(f"News Title: {page.news_story.title}") print(f"Publish Date: {page.news_story.publish_date}") # Check for image sitemap data if page.images: for image in page.images: print(f"Image URL: {image.loc}") ``` ```python # Advanced usage with options tree = sitemap_tree_for_homepage( 'https://www.example.org/', use_robots=True, # Discover sitemaps from robots.txt (default: True) use_known_paths=True, # Check well-known sitemap paths (default: True) extra_known_paths={'custom-sitemap.xml'}, normalize_homepage_url=True # Normalize URL to domain root (default: True) ) # Get total count of pages page_count = sum(1 for _ in tree.all_pages()) print(f"Total pages found: {page_count}") ``` -------------------------------- ### AbstractWebClientSuccessResponse Source: https://github.com/gatenlp/ultimate-sitemap-parser/blob/main/docs/reference/api/usp.web_client.abstract_client.rst Represents a successful response from a web client. ```APIDOC ## AbstractWebClientSuccessResponse ### Description Represents a successful response from a web client. ### Inheritance Inherits from `AbstractWebClientResponse`. ``` -------------------------------- ### Gatenlp Ultimate Sitemap Parser - API Modules Source: https://github.com/gatenlp/ultimate-sitemap-parser/blob/main/docs/reference/api/index.rst Overview of the Python modules available in the Gatenlp Ultimate Sitemap Parser library. ```APIDOC ## Gatenlp Ultimate Sitemap Parser API Modules ### Description This section outlines the primary modules available within the Gatenlp Ultimate Sitemap Parser Python library, providing access to various functionalities for parsing sitemaps. ### Modules - **usp.exceptions**: Contains custom exception classes used throughout the library. - **usp.fetch_parse**: Provides functionalities for fetching and parsing sitemap files. - **usp.helpers**: Includes utility and helper functions. - **usp.objects**: Defines the data structures and objects used to represent sitemap data. - **usp.tree**: Offers tools for working with the sitemap as a tree structure. - **usp.web_client**: Contains components for interacting with sitemaps via a web client. ``` -------------------------------- ### Save Sitemap Tree to Pickle File Source: https://github.com/gatenlp/ultimate-sitemap-parser/blob/main/docs/guides/saving.rst Save the sitemap tree object to a Pickle file for later loading. Be aware that loading untrusted Pickle data can be unsafe and that pickling relies on internal APIs which may change. ```python import pickle from usp.tree import sitemap_tree_for_homepage tree = sitemap_tree_for_homepage('https://www.example.org/') with open('sitemap.pickle', 'wb') as f: pickle.dump(tree, f) # This will delete the temporary files used to store the pages of the tree del tree ``` -------------------------------- ### usp.tree Module Functions Source: https://github.com/gatenlp/ultimate-sitemap-parser/blob/main/docs/reference/api/usp.tree.rst Documentation for functions within the usp.tree module. ```APIDOC ## usp.tree Module ### Description This module provides functionalities for parsing sitemaps and generating tree structures. ### Functions #### sitemap_tree_for_homepage ##### Description Generates a sitemap tree structure, likely for a homepage. ##### Parameters (No specific parameters detailed in the provided text) #### sitemap_from_str ##### Description Parses a sitemap from a string representation. ##### Parameters (No specific parameters detailed in the provided text) ### Data #### _UNPUBLISHED_SITEMAP_PATHS ##### Description Represents paths for unpublished sitemaps. ``` -------------------------------- ### Save Sitemap Pages to CSV using Pandas Source: https://github.com/gatenlp/ultimate-sitemap-parser/blob/main/docs/guides/saving.rst Convert sitemap pages to a dictionary, normalize the data using Pandas, and save it to a CSV file. This flattens nested keys for easier data analysis. ```python data = [page.to_dict() for page in tree.all_pages()] # pd.json_normalize() flattens the nested key for news stories # to dot-separated keys pages_df = pd.DataFrame(pd.json_normalize(data)) pages_df.to_csv('sitemap-pages.csv', index=False) ``` -------------------------------- ### Handle Sitemap Parsing Errors Source: https://context7.com/gatenlp/ultimate-sitemap-parser/llms.txt Use a try-except block to catch and report SitemapXMLParsingException when parsing invalid XML strings. ```python try: sitemap = sitemap_from_str("xml") except SitemapXMLParsingException as e: print(f"XML parsing error: {e}") ``` -------------------------------- ### RequestsWebClientSuccessResponse Class Source: https://github.com/gatenlp/ultimate-sitemap-parser/blob/main/docs/reference/api/usp.web_client.requests_client.rst Represents a successful response from the RequestsWebClient. ```APIDOC ## RequestsWebClientSuccessResponse Class ### Description Represents a successful response from the RequestsWebClient, containing the returned data. ### Members - **status_code (int)**: The HTTP status code of the successful response. - **data (Any)**: The data returned in the response body. ### Inheritance Inherits from [BaseWebClientSuccessResponse](?module=usp.web_client.base_web_client&class=BaseWebClientSuccessResponse) ### Inherited Members - **status_code (int)** - **data (Any)** ```