### Host Command Example for Single IP
Source: https://github.com/eliasdabbas/advertools/blob/master/docs/advertools.reverse_dns_lookup.md
This shows a basic example of using the `host` command on the command line for a single domain name.
```bash
$ host advertools.readthedocs.io
advertools.readthedocs.io has address 104.17.32.82
```
--------------------------------
### Install advertools
Source: https://github.com/eliasdabbas/advertools/blob/master/README.rst
Install the advertools package using pip.
```bash
python3 -m pip install advertools
```
--------------------------------
### Install advertools with Claude support
Source: https://github.com/eliasdabbas/advertools/blob/master/docs/advertools.serp_claude.md
Install the `advertools` library with the optional Claude dependency. Alternatively, install the `anthropic` library directly.
```bash
pip install "advertools[claude]" # or: pip install anthropic
```
--------------------------------
### Start Discovery Crawl with advertools
Source: https://github.com/eliasdabbas/advertools/blob/master/docs/advertools.spider.md
Initiates a website crawl starting from the provided URL, following all links and saving the output to a jsonlines file. Ensure the output file name is descriptive and unique for each crawl to avoid duplicate data.
```python
import advertools as adv
adv.crawl('https://example.com', 'my_output_file.jl', follow_links=True)
```
--------------------------------
### Command Line Host Command Example
Source: https://github.com/eliasdabbas/advertools/blob/master/docs/advertools.reverse_dns_lookup.md
This demonstrates the equivalent of the `host` command from the command line, which the `reverse_dns_lookup` function emulates on a large scale.
```bash
$ host 66.249.80.0
0.80.249.66.in-addr.arpa domain name pointer google-proxy-66-249-80-0.google.com.
```
--------------------------------
### Install scrapy-rotating-proxies
Source: https://github.com/eliasdabbas/advertools/blob/master/docs/advertools.code_recipes.spider_strategies.md
Install the necessary third-party package for proxy rotation. This package handles proxy rotation and retries automatically.
```bash
pip install scrapy-rotating-proxies
```
--------------------------------
### Initialize text list for tokenization
Source: https://github.com/eliasdabbas/advertools/blob/master/docs/advertools.word_tokenize.md
Define a list of strings to be tokenized. This includes examples with punctuation and internal delimiters.
```python
>>> t = ['split me into length-n-words',
... 'commas, (parentheses) get removed!',
... 'commas within text remain $1,000, but not the trailing commas.']
```
--------------------------------
### Partitioning with Delimiter at Start and End
Source: https://github.com/eliasdabbas/advertools/blob/master/docs/advertools.partition.md
Shows how partition works when the delimiter appears at both the beginning and the end of the string. The output includes the delimiter and the text between them.
```python
partition("delimtextdelim", r"delim")
```
--------------------------------
### Run Scrapy Spider with Advertools
Source: https://github.com/eliasdabbas/advertools/blob/master/docs/advertools.crawlytics.md
Example of running a Scrapy spider using Advertools with various parameters for URL lists, domains, link following, and output file.
```bash
/opt/tljh/user/bin/python /opt/tljh/user/bin/scrapy runspider /opt/tljh/user/lib/python3.10/site-packages/advertools/spider.py -a url_list=https://nytimes.com -a allowed_domains=nytimes.com -a follow_links=True -a exclude_url_params=None -a include_url_params=None -a exclude_url_regex=None -a include_url_regex=None -a css_selectors=None -a xpath_selectors=None -o nyt.jl -s CLOSESPIDER_PAGECOUNT=200
```
--------------------------------
### guide_categories_list
Source: https://github.com/eliasdabbas/advertools/blob/master/docs/advertools.youtube.md
Fetches a list of guide categories that can be associated with YouTube channels. Allows filtering by category ID or region code.
```APIDOC
## guide_categories_list(key, part, id=None, regionCode=None, hl=None)
### Description
Returns a list of categories that can be associated with YouTube channels.
### Parameters
#### Required Parameters
* **key** (string) - Your Google API key.
* **part** (string) - Specifies the guideCategory resource properties to include (must be 'snippet').
#### Filters (specify exactly one)
* **id** (string) - The YouTube channel category ID(s) to retrieve.
* **regionCode** (string) - An ISO 3166-1 alpha-2 country code to filter categories by region.
#### Optional Parameters
* **hl** (string) - Specifies the language for text values in the API response. Defaults to 'en-US'.
```
--------------------------------
### Import advertools and define description text
Source: https://github.com/eliasdabbas/advertools/blob/master/docs/advertools.ad_from_string.md
This snippet shows how to import the advertools library and define a long description string that will be used in subsequent examples. It also demonstrates how to check the length of the description string.
```python
import advertools as adv
desc_text = "Get the latest gadget online. The GX12 model comes with 13 things that do a lot of good stuff for your health. Start shopping now."
len(desc_text) # 130
```
--------------------------------
### Count Articles per Language (Output)
Source: https://github.com/eliasdabbas/advertools/blob/master/docs/advertools.sitemaps.md
Example output of value_counts() showing the number of articles for each language. This helps understand content distribution by language.
```python
russian 14022
persian 10968
portuguese 5403
urdu 5068
mundo 5065
vietnamese 3561
arabic 2984
hindi 1677
turkce 706
ukchina 545
Name: dir_1, dtype: int64
```
--------------------------------
### Facebook Instant Article Ad Example
Source: https://github.com/eliasdabbas/advertools/blob/master/docs/advertools.ad_from_string.md
Demonstrates ad_from_string when the input text exceeds the provided slot lengths for Facebook instant article ads. The excess text is placed in the last slot.
```python
adv.ad_from_string(desc_text, [25, 30]) # Facebook instant article ad
```
```text
['Get the latest gadget',
'online. The GX12 model comes',
'with 13 things that do a lot of good stuff for your health. Start shopping now.']
```
--------------------------------
### Add device information as meta data
Source: https://github.com/eliasdabbas/advertools/blob/master/docs/advertools.spider.md
Use meta data to record the device used for crawling, which is easier than parsing the user agent string. This example also shows how to set a custom user agent.
```python
adv.crawl(
"https://example.com",
"output.jsonl",
custom_settings={
"USER_AGENT": "Mozilla/5.0 (iPhone; CPU iPhone OS 14_7_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.2 Mobile/15E148 Safari/604.1"
},
meta={"device": "Apple iPhone 12 Pro (Safari)"},
)
```
--------------------------------
### Google Text Ads Example
Source: https://github.com/eliasdabbas/advertools/blob/master/docs/advertools.ad_from_string.md
Demonstrates using ad_from_string with default values for Google text ads. The output shows extra empty slots when the input string is shorter than the default slot lengths.
```python
adv.ad_from_string(desc_text) # default values (Google text ads)
```
```text
['Get the latest gadget online.',
'The GX12 model comes with 13',
'things that do a lot of good',
'stuff for your health. Start shopping now.',
'',
'',
'',
'']
```
--------------------------------
### Partitioning Markdown by Headings
Source: https://github.com/eliasdabbas/advertools/blob/master/docs/advertools.partition.md
Demonstrates how to use advertools.partition to split a Markdown document into chunks based on heading levels. Requires importing advertools and re.
```python
import advertools as adv
import re
markdown_text = '''
# Document Title
Some introductory text.
## Section 1
Content for section 1.
### Subsection 1.1
Details for subsection 1.1.
## Section 2
Content for section 2.
'''
heading_regex = r"^#+ .*?$"
# Partition the markdown text by headings
# Note: This is a simplified example. A robust markdown parser would be more complex.
chunks = adv.partition(markdown_text, heading_regex, flags=re.MULTILINE)
# The 'chunks' list would contain alternating text blocks and the matched headings,
# allowing further processing of each part of the document.
print(*chunks, sep="\n----\n")
```
--------------------------------
### Get Twitter Rate Limit Status
Source: https://github.com/eliasdabbas/advertools/blob/master/docs/advertools.twitter.md
Retrieve the current rate limits for Twitter API endpoints. By default, it returns only the consumed rate limits. Set consumed_only=False to get the full list of rate limits.
```python
adv.twitter.get_application_rate_limit_status(consumed_only=True)
```
--------------------------------
### Extract Exclamations Summary
Source: https://github.com/eliasdabbas/advertools/blob/master/docs/advertools.extract.md
Get a summary of exclamation marks in a list of texts. Returns a dictionary with various statistics.
```python
>>> posts = ['Who are you!', 'What is this!', 'No exclamation here?']
>>> exclamation_summary = extract_exclamations(posts)
>>> exclamation_summary.keys()
dict_keys(['exclamation_marks', 'exclamation_marks_flat',
'exclamation_mark_counts', 'exclamation_mark_freq',
'top_exclamation_marks', 'overview', 'exclamation_mark_names',
'exclamation_text'])
```
```python
>>> exclamation_summary['exclamation_marks']
[['!'], ['!'], []]
```
```python
>>> exclamation_summary['exclamation_marks_flat']
['!', '!']
```
```python
>>> exclamation_summary['exclamation_mark_counts']
[1, 1, 0]
```
```python
>>> exclamation_summary['exclamation_mark_freq']
[(0, 1), (1, 2)]
```
```python
>>> exclamation_summary['top_exclamation_marks']
[('!', 2)]
```
```python
>>> exclamation_summary['exclamation_mark_names']
[['exclamation mark'], ['exclamation mark'], []]
```
```python
>>> exclamation_summary['overview']
{'num_posts': 3,
'num_exclamation_marks': 2,
'exclamation_marks_per_post': 0.6666666666666666,
'unique_exclamation_marks': 1}
```
--------------------------------
### Extract User-Agents from robots.txt DataFrame
Source: https://github.com/eliasdabbas/advertools/blob/master/docs/advertools.robotstxt.md
Filter the DataFrame to get unique user-agents from the 'User-agent' directives. This list is then used for testing.
```python
fb_useragents = (
fb_robots
[fb_robots['directive']=='User-agent']
['content'].drop_duplicates()
.tolist()
)
fb_useragents
```
--------------------------------
### Basic String Partitioning by Numbers
Source: https://github.com/eliasdabbas/advertools/blob/master/docs/advertools.partition.md
Demonstrates basic partitioning of a string using a regex pattern that matches digits. The output includes the text segments and the matched numbers.
```python
import advertools as adv
text = "abc123def456ghi"
regex = r"\\d+"
adv.partition(text, regex)
```
--------------------------------
### Get Stopwords for a Language
Source: https://github.com/eliasdabbas/advertools/blob/master/docs/advertools.cli_docs.cli.md
Retrieves a list of stopwords for a specified language. Use this to filter out common words from text analysis.
```console
advertools stopwords [-h] {arabic,azerbaijani,bengali,catalan,chinese,croatian,danish,dutch,english,finnish,french,german,greek,hebrew,hindi,hungarian,indonesian,irish,italian,japanese,kazakh,nepali,norwegian,persian,polish,portuguese,romanian,russian,sinhala,spanish,swedish,tagalog,tamil,tatar,telugu,thai,turkish,ukrainian,urdu,vietnamese}
```
--------------------------------
### Basic Ad Creation with Replacements
Source: https://github.com/eliasdabbas/advertools/blob/master/docs/advertools.ad_create.md
Generates ads by inserting product names into a template. Ensure the template and replacements are within the specified `max_len`.
```python
import advertools as adv
products = ['Dubai', 'Tokyo', 'Singapore']
adv.ad_create(template='5-star Hotels in {}',
replacements=products,
max_len=30,
fallback='Great Cities')
```
```python
['5-star Hotels In Dubai',
'5-star Hotels In Tokyo',
'5-star Hotels In Singapore']
```
--------------------------------
### Advertools Package Initialization
Source: https://github.com/eliasdabbas/advertools/blob/master/docs/include_changelog.md
Configuration for launching a Python kernel in a Jupyter environment, often used for interactive documentation or tutorials.
```json
{
"requestKernel": true,
"binderOptions": {
"repo": "eliasdabbas/adv_docs_thebe",
"ref": "main",
},
"codeMirrorConfig": {
"theme": "abcdef",
"mode": "python"
},
"kernelOptions": {
"name": "python3",
"path": "./."
},
"predefinedOutput": true
}
```
--------------------------------
### Extract Hashtags Summary
Source: https://github.com/eliasdabbas/advertools/blob/master/docs/advertools.extract.md
Get a summary of hashtags in a list of texts. Returns a dictionary with statistics on counts, frequency, and top hashtags.
```python
>>> posts = ['i like #blue', 'i like #green and #blue', 'i like all']
>>> hashtag_summary = extract_hashtags(posts)
>>> hashtag_summary.keys()
dict_keys(['hashtags', 'hashtags_flat', 'hashtag_counts', 'hashtag_freq',
'top_hashtags', 'overview'])
```
```python
>>> hashtag_summary['hashtags']
[['#blue'], ['#green', '#blue'], []]
```
```python
>>> hashtag_summary['hashtags_flat']
['#blue', '#green', '#blue']
```
```python
>>> hashtag_summary['hashtag_counts']
[1, 2, 0]
```
```python
>>> hashtag_summary['hashtag_freq']
[(0, 1), (1, 1), (2, 1)]
```
```python
>>> hashtag_summary['top_hashtags']
[('#blue', 2), ('#green', 1)]
```
```python
>>> hashtag_summary['overview']
{'num_posts': 3,
'num_hashtags': 3,
'hashtags_per_post': 1.0,
'unique_hashtags': 2}
```
--------------------------------
### main() Function
Source: https://github.com/eliasdabbas/advertools/blob/master/docs/advertools.cli.md
The main entry point for the advertools CLI. This function is designed to be executed when the advertools command-line tool is invoked.
```APIDOC
## main()
### Description
Main function for the advertools command-line interface.
### Usage
This function is typically called when running advertools from the command line.
### Thebe Configuration
```json
{
"requestKernel": true,
"binderOptions": {
"repo": "eliasdabbas/adv_docs_thebe",
"ref": "main"
},
"codeMirrorConfig": {
"theme": "abcdef",
"mode": "python"
},
"kernelOptions": {
"name": "python3",
"path": "./."
},
"predefinedOutput": true
}
```
```javascript
kernelName = 'python3'
```
```
--------------------------------
### get_application_rate_limit_status
Source: https://github.com/eliasdabbas/advertools/blob/master/docs/advertools.twitter.md
Returns the current rate limits for methods belonging to the specified resource families. You can choose to get only consumed limits or the full list.
```APIDOC
## get_application_rate_limit_status(consumed_only=True)
### Description
Returns the current rate limits for methods belonging to the specified resource families. You can choose to get only consumed limits or the full list.
### Parameters
#### Query Parameters
- **consumed_only** (bool) - Optional - Whether or not to return only items that have been consumed. Defaults to True.
### Endpoint
[https://developer.twitter.com/en/docs/developer-utilities/rate-limit-status/api-reference/get-application-rate_limit_status](https://developer.twitter.com/en/docs/developer-utilities/rate-limit-status/api-reference/get-application-rate_limit_status)
```
--------------------------------
### Thebe Configuration for Advertools
Source: https://github.com/eliasdabbas/advertools/blob/master/docs/advertools.crawlytics.md
Configuration script for Thebe to enable interactive Python kernels, specify binder options, and set up CodeMirror for Python development.
```html
```
--------------------------------
### Extract Hashtags with Statistics
Source: https://github.com/eliasdabbas/advertools/blob/master/docs/advertools.extract.md
Use this function to extract hashtags from a list of texts and get detailed statistics. It requires importing the advertools library.
```python
import advertools as adv
text_list = ['This is the first #text.', 'Second #sentence is here.',
'Hello, how are you?', 'This #sentence is the last #sentence']
hashtag_summary = adv.extract_hashtags(text_list)
hashtag_summary.keys()
```
```python
hashtag_summary
```
```python
hashtag_summary['overview']
```
```python
hashtag_summary['hashtags']
```
```python
hashtag_summary['hashtags_flat']
```
```python
hashtag_summary['hashtag_counts']
```
```python
hashtag_summary['hashtag_freq']
```
```python
hashtag_summary['top_hashtags']
```
--------------------------------
### Partitioning with Consecutive Delimiters
Source: https://github.com/eliasdabbas/advertools/blob/master/docs/advertools.partition.md
Demonstrates how partition handles consecutive delimiters and delimiters at the beginning or end of the string. Empty strings are included in the output to represent zero-length parts.
```python
partition(",a,,b,", r",")
```
--------------------------------
### Resample Articles by Month (Output)
Source: https://github.com/eliasdabbas/advertools/blob/master/docs/advertools.sitemaps.md
Example output of resampling sitemap data by month for a specific language. It shows the count of articles published each month.
```python
lastmod
2009-04-30 00:00:00+00:00 1506
2009-05-31 00:00:00+00:00 2910
2009-06-30 00:00:00+00:00 3021
2009-07-31 00:00:00+00:00 3250
2009-08-31 00:00:00+00:00 2769
...
2019-09-30 00:00:00+00:00 8
2019-10-31 00:00:00+00:00 17
2019-11-30 00:00:00+00:00 11
2019-12-31 00:00:00+00:00 24
2020-01-31 00:00:00+00:00 6
Freq: M, Name: loc, Length: 130, dtype: int64
```
--------------------------------
### Case-Insensitive String Partitioning with Regex Flags
Source: https://github.com/eliasdabbas/advertools/blob/master/docs/advertools.partition.md
Demonstrates how to perform case-insensitive partitioning using the flags parameter with re.IGNORECASE. This allows matching the delimiter regardless of its case.
```python
import advertools as adv
import re
text = "TestData"
regex = r"t"
adv.partition(text, regex, flags=re.IGNORECASE)
```
--------------------------------
### Get Available Twitter Trends Locations
Source: https://github.com/eliasdabbas/advertools/blob/master/docs/advertools.twitter.md
Fetch a list of geographical locations for which Twitter provides trending topic information. This function does not require authentication.
```python
adv.twitter.get_available_trends()
```
--------------------------------
### crawl_images
Source: https://github.com/eliasdabbas/advertools/blob/master/docs/advertools.image_spider.md
Downloads all images available on the provided start URLs and saves them to a specified output directory. This function is experimental and may change in future versions.
```APIDOC
## crawl_images(start_urls, output_dir, min_width=0, min_height=0, include_img_regex=None, custom_settings=None)
### Description
Download all images available on start_urls and save them to output_dir.
THIS FUNCTION IS STILL EXPERIMENTAL. Expect many changes.
### Parameters
* **start_urls** (*list*) -- A list of URLs from which you want to download available images.
* **output_dir** (*str*) -- The directory where you want the images to be saved.
* **min_width** (*int*) -- The minimum width in pixels for an image to be downloaded.
* **min_height** (*int*) -- The minimum height in pixels for an image to be downloaded.
* **include_img_regex** (*str*) -- A regular expression to select image src URLs. Use this to restrict image
files that match this regex.
* **custom_settings** (*dict*) -- Additional settings to customize the crawling behaviour.
```
--------------------------------
### Thebe Configuration for Interactive Python
Source: https://github.com/eliasdabbas/advertools/blob/master/docs/readme.md
This script configures Thebe to run Python code interactively, specifying the repository, kernel name, and CodeMirror settings.
```html
```
--------------------------------
### Basic SERP query with advertools.serp_claude
Source: https://github.com/eliasdabbas/advertools/blob/master/docs/advertools.serp_claude.md
Perform a simple query using a literal string and an empty variables dictionary. The `anthropic` SDK is imported lazily.
```python
import advertools as adv
df = adv.serp_claude("best running shoes 2026", {})
```
--------------------------------
### serp_goog
Source: https://github.com/eliasdabbas/advertools/blob/master/docs/advertools.serp.md
Query Google's search API and get search results in a DataFrame. Supports single or multiple arguments for parameters, generating all possible combinations.
```APIDOC
## serp_goog(q, cx, key, c2coff=None, cr=None, dateRestrict=None, exactTerms=None, excludeTerms=None, fileType=None, filter=None, gl=None, highRange=None, hl=None, hq=None, imgColorType=None, imgDominantColor=None, imgSize=None, imgType=None, linkSite=None, lowRange=None, lr=None, num=None, orTerms=None, rights=None, safe=None, searchType=None, siteSearch=None, siteSearchFilter=None, sort=None, start=None)
### Description
Queries Google's search API and returns results as a pandas DataFrame. This function can handle multiple values for parameters, creating a Cartesian product of all possible queries.
### Parameters
* **q** (*str*) - Required - The search expression.
* **cx** (*str*) - Required - The custom search engine ID to use for this request.
* **key** (*str*) - Required - The API key of your custom search engine.
* **c2coff** (*str*) - Optional - Enables or disables Simplified and Traditional Chinese Search. Default is 0 (enabled). Supported values: 1 (Disabled), 0 (Enabled).
* **cr** (*str*) - Optional - Restricts search results to documents originating in a particular country. Boolean operators can be used. See Google's Country Parameter Values page for a list of valid values.
* **dateRestrict** (*str*) - Optional - Restricts results to URLs based on date. Supported formats: d[number] (days), w[number] (weeks), m[number] (months), y[number] (years).
* **exactTerms** (*str*) - Optional - Identifies a phrase that all documents in the search results must contain.
* **excludeTerms** (*str*) - Optional - Identifies a word or phrase that should not appear in any documents in the search results.
* **fileType** (*str*) - Optional - Restricts results to files of a specified extension.
* **filter** (*str*) - Optional - Controls the duplicate content filter. Acceptable values: "0" (Turns off), "1" (Turns on). Default is to apply filtering.
* **gl** (*str*) - Optional - Geolocation of end user (two-letter country code). Boosts search results originating from the specified country.
* **highRange** (*str*) - Optional - Specifies the ending value for a search range. Use with lowRange for an inclusive range.
* **hl** (*str*) - Optional - Sets the user interface language. Explicitly setting this improves search performance and quality.
* **hq** (*str*) - Optional - Appends the specified query terms to the query, treated as if combined with a logical AND operator.
* **imgColorType** (*str*) - Optional - Returns images of a specific color type. Acceptable values: "color", "gray", "mono".
* **imgDominantColor** (*str*) - Optional - Returns images of a specific dominant color. Acceptable values: "black", "blue", "brown", "gray", "green", "orange", "pink", "purple", "red", "teal", "white", "yellow".
* **imgSize** (*str*) - Optional - Returns images of a specified size. Acceptable values: "huge", "icon", "large", "medium", "small", "xlarge", "xxlarge".
* **imgType** (*str*) - Optional - Returns images of a specific type. Acceptable values: "clipart", "face", "lineart", "news", "photo".
* **linkSite** (*str*) - Optional - Specifies that all search results should contain a link to a particular URL.
* **lowRange** (*str*) - Optional - Specifies the starting value for a search range. Use with highRange for an inclusive range.
* **lr** (*str*) - Optional - Specifies one or more language restrictions.
* **num** (*str*) - Optional - Number of search results to return.
* **orTerms** (*str*) - Optional - Specifies a list of search terms that should be treated as OR.
* **rights** (*str*) - Optional - Returns results that are licensed or have explicit usage rights.
* **safe** (*str*) - Optional - Enables or disables SafeSearch.
* **searchType** (*str*) - Optional - Specifies the type of search to perform (e.g., "images").
* **siteSearch** (*str*) - Optional - Specifies a domain to search within.
* **siteSearchFilter** (*str*) - Optional - Specifies whether to include or exclude results from the siteSearch domain.
* **sort** (*str*) - Optional - Specifies the order of search results.
* **start** (*str*) - Optional - The zero-based index of the first result to return.
```
--------------------------------
### Get Redirect DataFrame with advertools
Source: https://github.com/eliasdabbas/advertools/blob/master/docs/advertools.crawlytics.md
Use the `redirects` function from the `advertools.crawlytics` module to obtain a DataFrame containing redirect information. This requires a crawl DataFrame as input.
```python
>>> redirect_df = adv.crawlytics.redirects(crawldf)
>>> redirect_df
```
--------------------------------
### Partitioning with Consecutive Delimiters and Edge Matches
Source: https://github.com/eliasdabbas/advertools/blob/master/docs/advertools.partition.md
Shows how advertools.partition handles strings with consecutive delimiters or delimiters at the beginning and end of the string. The delimiters are included in the output list.
```python
import advertools as adv
text = ",a,,b,"
regex = r",""
adv.partition(text, regex)
```