### Install iocextract Source: https://context7.com/inquest/iocextract/llms.txt Install the iocextract library using pip. On Ubuntu/Debian, Python dev headers might be required first. ```bash pip install iocextract # On Ubuntu/Debian, Python dev headers may be required first: sudo apt-get install python-dev && pip install iocextract ``` -------------------------------- ### Install iocextract on Windows Source: https://github.com/inquest/iocextract/blob/master/docs/README.md Installs the iocextract package on Windows by first downloading the appropriate 'regex' wheel file from PyPI and then installing it via pip. ```bash pip install regex-2018.06.21-cp27-none-win_amd64.whl ``` -------------------------------- ### Install Python Development Headers (Ubuntu/Debian) Source: https://github.com/inquest/iocextract/blob/master/docs/index.md Installs the Python development headers, which may be required for installing the `regex` dependency on Ubuntu/Debian-based systems. ```bash sudo apt-get install python-dev ``` -------------------------------- ### Install iocextract on Ubuntu/Debian Source: https://github.com/inquest/iocextract/blob/master/docs/README.md Installs the Python development headers required for the 'regex' dependency, followed by installing the iocextract package using pip. ```bash sudo apt-get install python-dev pip install iocextract ``` -------------------------------- ### Install iocextract via Pip Source: https://github.com/inquest/iocextract/blob/master/docs/index.md Installs the iocextract package using pip, the standard Python package installer. ```bash pip install iocextract ``` -------------------------------- ### Custom Regex Example in Python Source: https://github.com/inquest/iocextract/blob/master/README.md Provides Python examples demonstrating how to define custom regex patterns for IOC extraction. It highlights the behavior with single and multiple capture groups. ```python [ r'(my regex)', # This yields 'my regex' if the pattern matches r'my (re)gex', # This yields 're' if the pattern matches ] ``` ```python [ r'my regex', # This doesn't yield anything r'(my) (re)gex', # This yields 'my' if the pattern matches ] ``` ```python [ r'(?:my|your) (re)gex', # This yields 're' if the pattern matches ] ``` -------------------------------- ### Custom Regex File Examples Source: https://github.com/inquest/iocextract/blob/master/docs/README.md Create a plain text file with one regex string per line. Each regex must include exactly one capture group. The `(?: )` syntax denotes a non-capture group. ```text http://(example\.com)/ ``` ```text (?:https|ftp)://(example\.com)/ ``` -------------------------------- ### Example of Defanged IP Address Source: https://github.com/inquest/iocextract/blob/master/docs/README.md Illustrates a common defanging technique where periods in an IP address are replaced with brackets. Standard IP regex tools may miss this format. ```default 127[.]0[.]0[.]1 ``` -------------------------------- ### Custom Regex Error Messages Source: https://github.com/inquest/iocextract/blob/master/README.md Shows example error messages encountered when using invalid custom regex patterns or regex patterns without a capture group. ```text Error in custom regex: missing ) at position 5 ``` ```text Error in custom regex: no such group ``` -------------------------------- ### Error Message for Invalid Regex Source: https://github.com/inquest/iocextract/blob/master/docs/index.md This is an example of an error message displayed when a custom regex is syntactically invalid. ```text Error in custom regex: missing ) at position 5 ``` -------------------------------- ### iocextract CLI Help Source: https://github.com/inquest/iocextract/blob/master/docs/README.md Displays the help message for the iocextract command-line interface, outlining available arguments for extracting various IOCs and configuring the extraction process. ```bash $ iocextract -h usage: iocextract [-h] [--input INPUT] [--output OUTPUT] [--extract-emails] [--extract-ips] [--extract-ipv4s] [--extract-ipv6s] [--extract-urls] [--extract-yara-rules] [--extract-hashes] [--custom-regex REGEX_FILE] [--refang] [--strip-urls] [--wide] Advanced Indicator of Compromise (IOC) extractor. If no arguments are specified, the default behavior is to extract all IOCs. optional arguments: -h, --help show this help message and exit --input INPUT default: stdin --output OUTPUT default: stdout --extract-emails --extract-ips --extract-ipv4s --extract-ipv6s --extract-urls --extract-yara-rules --extract-hashes --custom-regex REGEX_FILE file with custom regex strings, one per line, with one capture group each --refang default: no --strip-urls remove possible garbage from the end of urls. default: no --wide preprocess input to allow wide-encoded character matches. default: no ``` -------------------------------- ### Defang URL Techniques Source: https://github.com/inquest/iocextract/blob/master/README.md Demonstrates various techniques for defanging URLs, making them safe to share. These include replacing characters like '.' and '/' with bracketed equivalents or using hex, URL, and Base64 encoding. ```text hxxp://example.com/bad/url ``` ```text tcp://example[.]com:8989/bad ``` ```text example[.]com ``` ```text tcp://example[.]com:8989/bad ``` ```text me@} example[.com ``` ```text me@example.com ``` ```text me@example [.] com ``` ```text me@example.com ``` ```text me @example [.)com ``` ```text me@example.com ``` ```text example[.]com/path ``` ```text http://example.com/path ``` ```text example(.)com/path ``` ```text http://example.com/path ``` ```text example\.com/path ``` ```text http://example.com/path ``` ```text http://example[.com/path ``` ```text http://example.com/path ``` ```text http://example.com[/]path ``` ```text http://example.com/path ``` ```text http:// example .com /path ``` ```text http://example.com/path ``` ```text http__example.com/path ``` ```text http://example.com/path ``` ```text http:\\example.com/path ``` ```text http://example.com/path ``` ```text http[:]//example.com/path ``` ```text http://example.com/path ``` ```text hxxp://example.com/path ``` ```text http://example.com/path ``` ```text hxxp__ example( .com[/]path ``` ```text http://example.com/path ``` ```text 687474703a2f2f6578616d706c652e636f6d2f70617468 ``` ```text http://example.com/path ``` ```text http%3A%2F%2fexample%2Ecom%2Fpath ``` ```text http://example.com/path ``` ```text aHR0cDovL2V4YW1wbGUuY29tL3BhdGgK ``` ```text http://example.com/path ``` -------------------------------- ### IOCTract CLI Help Source: https://github.com/inquest/iocextract/blob/master/README.md Displays the help message for the IOCTract command-line interface, outlining available arguments and their default behaviors. ```bash $ iocextract -h usage: iocextract [-h] [--input INPUT] [--output OUTPUT] [--extract-emails] [--extract-ips] [--extract-ipv4s] [--extract-ipv6s] [--extract-urls] [--extract-yara-rules] [--extract-hashes] [--custom-regex REGEX_FILE] [--refang] [--strip-urls] [--wide] Advanced Indicator of Compromise (IOC) extractor. If no arguments are specified, the default behavior is to extract all IOCs. optional arguments: -h, --help show this help message and exit --input INPUT default: stdin --output OUTPUT default: stdout --extract-emails --extract-ips --extract-ipv4s --extract-ipv6s --extract-urls --extract-yara-rules --extract-hashes --custom-regex REGEX_FILE file with custom regex strings, one per line, with one capture group each --refang default: no --strip-urls remove possible garbage from the end of urls. default: no --wide preprocess input to allow wide-encoded character matches. default: no ``` -------------------------------- ### Command Line Interface - Custom Regex and Options Source: https://context7.com/inquest/iocextract/llms.txt Use custom regular expressions for specific patterns and apply options like stripping URL garbage or wide-character mode. ```APIDOC ## Command Line Interface ### Use custom regex to extract specific patterns ```bash cat > custom.txt << 'EOF' campaign_id=([A-Z0-9]{8}) (APT\d+) EOF iocextract --input report.txt --custom-regex custom.txt ``` ### Strip trailing garbage from URLs in HTML output ```bash curl https://threatintel.example.com/report | iocextract --extract-urls --strip-urls --refang ``` ### Wide-character mode for Windows memory dumps ```bash iocextract --input memory_dump.txt --wide --extract-urls ``` ### Remove URL schemes from output ```bash iocextract --input report.txt --extract-urls --refang --rm_scheme ``` ``` -------------------------------- ### Command Line Interface - Filtering and Formatting Source: https://context7.com/inquest/iocextract/llms.txt Control the types of IOCs extracted, refang them, and specify output formats like JSON. ```APIDOC ## Command Line Interface ### Extract only URLs and IPs, refanging them ```bash iocextract --input report.txt --extract-urls --extract-ips --refang ``` ### Extract from stdin, output as JSON ```bash echo "C2: hxxp://evil[.]com and 1.2.3[.]4" | iocextract --json ``` ``` -------------------------------- ### Command Line Interface - Directory and Remote Input Source: https://context7.com/inquest/iocextract/llms.txt Process multiple files within a directory or extract IOCs from remote URLs. ```APIDOC ## Command Line Interface ### Extract all IOC types from every .txt file in a directory, output JSON ```bash iocextract --dir --dirname /path/to/reports/ --json --output results.json ``` ### Extract IOCs from a remote page ```bash iocextract --remote-input --url https://pastebin.com/raw/somepaste --extract-urls --refang ``` ``` -------------------------------- ### Command Line Interface - Basic Usage Source: https://context7.com/inquest/iocextract/llms.txt Extract IOCs from stdin, files, directories, or remote URLs using the `iocextract` CLI. Default behavior extracts all IOC types. ```APIDOC ## Command Line Interface ### Extract all IOCs from a file ```bash iocextract --input threat_report.txt ``` ### Extract from stdin, output as JSON ```bash echo "C2: hxxp://evil[.]com and 1.2.3[.]4" | iocextract --json ``` ``` -------------------------------- ### Batch Extract IOCs from Directory to JSON using CLI Source: https://context7.com/inquest/iocextract/llms.txt Process all `.txt` files within a specified directory, extract all IOC types, and save the results as a JSON file. This is ideal for processing large collections of reports. ```bash # Extract all IOC types from every .txt file in a directory, output JSON iocextract --dir --dirname /path/to/reports/ --json --output results.json ``` -------------------------------- ### Extract IOCs from Remote URL using CLI Source: https://context7.com/inquest/iocextract/llms.txt Fetch content from a remote URL and extract specific IOC types, applying refanging. This command is useful for processing IOCs from online pastebins or reports. ```bash # Extract IOCs from a remote page iocextract --remote-input --url https://pastebin.com/raw/somepaste --extract-urls --refang ``` -------------------------------- ### Extract All IOCs from a File using CLI Source: https://context7.com/inquest/iocextract/llms.txt Use the `iocextract` CLI to extract all types of IOCs from a specified input file. The default output is one IOC per line. ```bash # Extract all IOCs from a file iocextract --input threat_report.txt ``` -------------------------------- ### Extract Specific IOCs and Refang using CLI Source: https://context7.com/inquest/iocextract/llms.txt Extract only URLs and IP addresses from a report file and apply refanging. This command allows targeted extraction and prepares the output for safe sharing. ```bash # Extract only URLs and IPs, refanging them iocextract --input report.txt --extract-urls --extract-ips --refang ``` -------------------------------- ### Refang Obfuscated URLs with iocextract Source: https://github.com/inquest/iocextract/blob/master/README.md Shows how to use the `refang=True` option to automatically remove common obfuscation methods from extracted URLs, converting them back to their standard format. ```python import iocextract content = \ """ I really love example[.]com! All the bots are on hxxp://example.com/bad/url these days. C2: tcp://example[.]com:8989/bad """ for url in iocextract.extract_urls(content, refang=True): print(url) # Output # http://example.com/bad/url # http://example.com:8989/bad # http://example.com # http://example.com:8989/bad ``` -------------------------------- ### Defang IOCs with Python Source: https://context7.com/inquest/iocextract/llms.txt Use the `defang_data` function to replace dots with `[.]` in domains and transform URL schemes for safe sharing. This is useful for preparing IOCs before sharing them in reports or communications. ```python import iocextract live_iocs = [ 'http://example.com/some/long/path.ext/', 'https://example.com/', 'ftp://example.com/', 'example.com', 'example.com/some/path.ext/', '127.0.0.1', ] for ioc in live_iocs: print(iocextract.defang_data(ioc)) ``` -------------------------------- ### Extracting URLs from Text Source: https://github.com/inquest/iocextract/blob/master/docs/README.md Demonstrates how to use iocextract to pull URLs from a given text corpus. The output shows both standard and defanged URLs extracted. ```default https://researchcenter.paloaltonetworks.com/2018/02/unit42-sofacy-attacks-multiple-government-entities/ hotfixmsupload[.]com cdnverify[.]net ``` -------------------------------- ### Extract Custom IOC Patterns using CLI Source: https://context7.com/inquest/iocextract/llms.txt Use a custom regex file to extract specific patterns from an input report. This allows for the extraction of organization-specific indicator formats. ```bash # Use custom regex to extract specific patterns cat > custom.txt << 'EOF' campaign_id=([A-Z0-9]{8}) (APT\d+) EOF iocextract --input report.txt --custom-regex custom.txt ``` -------------------------------- ### Custom Regex for IOC Extraction Source: https://github.com/inquest/iocextract/blob/master/README.md Illustrates how to use custom regular expressions with the CLI for extracting Indicators of Compromise (IOCs). It emphasizes the use of a single capture group for desired output. ```regex http://(example\.com)/ ``` ```regex (?:https|ftp)://(example\.com)/ ``` ```regex (https?://.*?.com) ``` -------------------------------- ### Remove URL Schemes from Output using CLI Source: https://context7.com/inquest/iocextract/llms.txt Extract URLs, refang them, and remove the URL schemes (e.g., http, https) from the output. This provides cleaner domain-focused IOCs. ```bash # Remove URL schemes from output iocextract --input report.txt --extract-urls --refang --rm_scheme ``` -------------------------------- ### Extract All IOCs with iocextract Source: https://context7.com/inquest/iocextract/llms.txt Extract all supported IOC types from a text string in a single pass. The `refang=True` option de-obfuscates IOCs. ```python import iocextract report = """ Threat report 2024-01-15: C2 server at hxxp://malicious[.]example[.]com:4444/beacon Secondary host: 192[.]168[.]1[.]100 Contact: attacker at evil-domain[.]org MD5: 68b329da9893e34099c7d8ad5cb9c940 SHA256: 01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b """ # Extract everything, refanging obfuscation in the output iocs = list(iocextract.extract_iocs(report, refang=True)) for ioc in iocs: print(ioc) # Output (order: urls, ips, emails, hashes, yara, phones): # http://malicious.example.com:4444/beacon # 192.168.1.100 # attacker@evil-domain.org # 68b329da9893e34099c7d8ad5cb9c940 # 01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b ``` -------------------------------- ### Extract IOCs from Stdin as JSON using CLI Source: https://context7.com/inquest/iocextract/llms.txt Process IOCs piped from standard input and output the results in JSON format. This is useful for integrating with other command-line tools or scripts. ```bash # Extract from stdin, output as JSON echo "C2: hxxp://evil[.]com and 1.2.3[.]4" | iocextract --json ``` -------------------------------- ### Strip Trailing Garbage from URLs in HTML using CLI Source: https://context7.com/inquest/iocextract/llms.txt Extract URLs from HTML content, strip trailing garbage, and refang them. This command is useful for cleaning up URLs found in web-based reports. ```bash # Strip trailing garbage from URLs in HTML output curl https://threatintel.example.com/report | iocextract --extract-urls --strip-urls --refang ``` -------------------------------- ### Extract Defanged URLs with iocextract Source: https://github.com/inquest/iocextract/blob/master/README.md Demonstrates extracting URLs, including those with defanged elements like '[.]', from a given text content. Note that some URLs might appear twice if matched by multiple patterns. ```python import iocextract content = \ """ I really love example[.]com! All the bots are on hxxp://example.com/bad/url these days. C2: tcp://example[.]com:8989/bad """ for url in iocextract.extract_urls(content): print(url) # Output # hxxp://example.com/bad/url # tcp://example[.]com:8989/bad # example[.]com # tcp://example[.]com:8989/bad ``` -------------------------------- ### Convert Extracted URLs Iterator to List Source: https://github.com/inquest/iocextract/blob/master/docs/README.md Demonstrates how to convert the iterator returned by `extract_urls` into a list. This is necessary if you need to iterate over the IOCs more than once. ```python import iocextract content = \ """ I really love example[.]com! All the bots are on hxxp://example.com/bad/url these days. C2: tcp://example[.]com:8989/bad """ print(list(iocextract.extract_urls(content))) # ['hxxp://example.com/bad/url', 'tcp://example[.]com:8989/bad', 'example[.]com', 'tcp://example[.]com:8989/bad'] ``` -------------------------------- ### Extract IP Addresses (IPv4 and IPv6) Source: https://context7.com/inquest/iocextract/llms.txt Extracts both IPv4 and IPv6 addresses from text. Supports various defanged formats for IPv4 (e.g., `[.]`, `(.)`, `\.`) and requires a minimum length for IPv6. Use `refang=True` to reconstruct defanged IPs. ```python import iocextract text = """ Infected hosts: 10[.]0[.]0[.]1 192(.)168(.)1(.)100 10\10\10\10 2001:0db8:85a3:0000:0000:8a2e:0370:7334 fe80::1ff:fe23:4567:890a """ # Extract with refanging for ip in iocextract.extract_ips(text, refang=True): print(ip) # 10.0.0.1 # 192.168.1.100 # 10.10.10.10 # 2001:0db8:85a3:0000:0000:8a2e:0370:7334 # fe80::1ff:fe23:4567:890a # IPv4 only, complex bracket combinations tricky = "10[[[[.]]]]10[[[.]]10[.10" print(list(iocextract.extract_ipv4s(tricky, refang=True))) # ['10.10.10.10'] ``` -------------------------------- ### Extract URLs with iocextract Source: https://context7.com/inquest/iocextract/llms.txt Extract URLs, including defanged and encoded variants. Use `refang=True` to de-obfuscate, `strip=True` to remove trailing punctuation, and `no_scheme=True` to remove the URL scheme. ```python import iocextract content = """ Observed IOCs from malware sample: hxxps://evil[.]example[.]com/payload.exe tcp://c2[.]domain[.]net:8080/cmd http__backup-c2.net/stage2 687474703a2f2f6578616d706c652e636f6d2f706174682f6d616c77617265 http%3A%2F%2Fexample%2Ecom%2Fwhite%2Ehta """ # Keep defanged form (default) for url in iocextract.extract_urls(content): print(url) # hxxps://evil[.]example[.]com/payload.exe # tcp://c2[.]domain[.]net:8080/cmd # http__backup-c2.net/stage2 # 687474703a2f2f6578616d706c652e636f6d2f706174682f6d616c77617265 # http%3A%2F%2Fexample%2Ecom%2Fwhite%2Ehta print("---") for url in iocextract.extract_urls(content, refang=True): print(url) # https://evil.example.com/payload.exe # http://c2.domain.net:8080/cmd # http://backup-c2.net/stage2 # http://example.com/path/malware # http://example.com/white.hta # Strip trailing garbage from URLs (useful for HTML/XML input) messy = "Check out https://example.com/path?q=1#anchor!!!! for details" print(list(iocextract.extract_urls(messy, strip=True))) # ['https://example.com/path?q=1#anchor'] # Remove scheme from output print(list(iocextract.extract_urls("https://example.com/test", refang=True, no_scheme=True))) # ['example.com'] ``` -------------------------------- ### Extract Custom IOCs with Regex Source: https://context7.com/inquest/iocextract/llms.txt Extracts Indicators of Compromise (IOCs) from data using a provided list of regular expressions. Each regex must contain exactly one capture group. This function is useful for extracting specific patterns like campaign IDs or C2 paths. ```python import iocextract # Extract specific threat actor campaign IDs and custom C2 paths regex_list = [ r'campaign_id=([A-Z0-9]{8})', # Campaign identifiers r'c2_path=(/[a-z/]+\.php)', # C2 endpoint paths r'(?:APT|TA)(\d+)', # Threat actor numbers ] report = """ Observed campaign_id=AB12CD34 using c2_path=/gate/check.php Attributed to APT28 and TA505 actors. Secondary c2_path=/update/beacon.php seen. """ for match in iocextract.extract_custom_iocs(report, regex_list): print(match) # AB12CD34 # /gate/check.php # 28 # 505 # /update/beacon.php # Error handling for bad regex import regex as re try: list(iocextract.extract_custom_iocs("data", [r'(mismatched'])) except re.error as e: print(f"Regex error: {e}") try: list(iocextract.extract_custom_iocs("data", [r'no capture group'])) except IndexError: print("No capture group defined in regex") ``` -------------------------------- ### Extract URLs and Convert to List Source: https://github.com/inquest/iocextract/blob/master/README.md Use this snippet when you need to iterate over extracted URLs more than once. All extract_* functions return iterators by default for memory efficiency. ```python import iocextract content = """ I really love example[.]com! All the bots are on hxxp://example.com/bad/url these days. C2: tcp://example[.]com:8989/bad """ print(list(iocextract.extract_urls(content))) ``` -------------------------------- ### Extracting URLs from Tweet Text Source: https://github.com/inquest/iocextract/blob/master/docs/index.md Demonstrates how the iocextract package can pull URLs from a block of text, including those that are 'defanged'. Passing `refang=True` during extraction would remove the obfuscation. ```text Recommended reading and great work from @unit42_intel: https://researchcenter.paloaltonetworks.com/2018/02/unit42-sofacy-attacks-multiple-government-entities/ ... InQuest customers have had detection for threats delivered from hotfixmsupload[.]com since 6/3/2017 and cdnverify[.]net since 2/1/18. ``` ```text https://researchcenter.paloaltonetworks.com/2018/02/unit42-sofacy-attacks-multiple-government-entities/ hotfixmsupload[.]com cdnverify[.]net ``` -------------------------------- ### defang_data(ioc) Source: https://context7.com/inquest/iocextract/llms.txt Defangs a URL, domain, or IPv4 address for safe sharing by replacing '.' with '[.]' in the netloc/domain and transforming URL schemes. ```APIDOC ## defang_data(ioc) ### Description Defang a URL, domain, or IPv4 address for safe sharing. Replaces `.` with `[.]` in the netloc/domain portion and transforms the URL scheme (`http` → `hxxp`, `https` → `hxxps`, `ftp` → `fxp`). Path components are left unmodified. ### Parameters - **ioc** (string) - The IOC string to defang. ### Returns - (string) The defanged IOC string. ### Example ```python import iocextract print(iocextract.defang_data('http://example.com')) # Output: hxxp://example[.]com ``` ``` -------------------------------- ### Disable Defanging During URL Extraction Source: https://github.com/inquest/iocextract/blob/master/README.md Illustrates how to disable the default defanging behavior during URL extraction using the `defang=False` option. This is useful when the input content already contains standard, non-obfuscated URLs. ```python import iocextract content = \ """ http://example.com/bad/url http://example.com:8989/bad http://example.com http://example.com:8989/bad """ for url in iocextract.extract_urls(content, defang=False): print(url) # Output # http://example.com/bad/url # http://example.com:8989/bad # http://example.com ``` -------------------------------- ### Extract Plain/Defanged URLs Source: https://context7.com/inquest/iocextract/llms.txt Extract only plain or defanged URLs, skipping encoded variants. Use `refang=True` to de-obfuscate and `defang=True` to re-obfuscate the output. ```python import iocextract text = "C2: hxxp://attacker[.]com/gate.php and also example\.com/path" plain_urls = list(iocextract.extract_unencoded_urls(text, refang=True)) print(plain_urls) # ['http://attacker.com/gate.php', 'http://example.com/path'] # Defang a live URL as you extract it live_text = "Visit http://example.com/test for details" defanged = list(iocextract.extract_unencoded_urls(live_text, defang=True)) print(defanged) # ['hxxp://example[.]com/test'] ``` -------------------------------- ### extract_iocs Source: https://context7.com/inquest/iocextract/llms.txt Extract all supported IOC types from a text string in a single pass. Returns a lazy iterator yielding results from various extraction functions. ```APIDOC ## extract_iocs(data, refang=False, strip=False) ### Description Extract all supported IOC types from a text string in a single pass. Returns an `itertools.chain` iterator that lazily yields results from `extract_urls`, `extract_ips`, `extract_emails`, `extract_hashes`, `extract_yara_rules`, and `extract_telephone_nums`. ### Parameters - **data** (str) - The input text data to extract IOCs from. - **refang** (bool, optional) - If True, de-obfuscates defanged IOCs in the output. Defaults to False. - **strip** (bool, optional) - If True, strips trailing garbage characters from extracted IOCs. Defaults to False. ### Request Example ```python import iocextract report = """ Threat report 2024-01-15: C2 server at hxxp://malicious[.]example[.]com:4444/beacon Secondary host: 192[.]168[.]1[.]100 Contact: attacker at evil-domain[.]org MD5: 68b329da9893e34099c7d8ad5cb9c940 SHA256: 01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b """ # Extract everything, refanging obfuscation in the output iocs = list(iocextract.extract_iocs(report, refang=True)) for ioc in iocs: print(ioc) ``` ### Response An iterator yielding extracted IOCs. The order of extraction is generally URLs, IPs, emails, hashes, YARA rules, and telephone numbers. ``` -------------------------------- ### refang_ipv4 Source: https://context7.com/inquest/iocextract/llms.txt Refangs a single defanged IPv4 address string into a standard IPv4 format. ```APIDOC ## refang_ipv4(ip_address) ### Description Refang a single defanged IPv4 address string. ### Parameters #### Path Parameters - **ip_address** (string) - Required - The defanged IPv4 address string to refang. ### Request Example ```python import iocextract defanged_ips = [ '192[.]168[.]1[.]1', '10(.)0(.)0(.)1', '172\.16\.0\.1', '10[[[[.]]]]0[[[.]]0[.1', ] for ip in defanged_ips: print(iocextract.refang_ipv4(ip)) ``` ### Response #### Success Response - **refanged_ip** (string) - The refanged IPv4 address. ### Response Example ``` 192.168.1.1 10.0.0.1 172.16.0.1 10.0.0.1 ``` ``` -------------------------------- ### extract_hashes, extract_md5_hashes, extract_sha1_hashes, extract_sha256_hashes, extract_sha512_hashes Source: https://context7.com/inquest/iocextract/llms.txt Extracts cryptographic file hashes (MD5, SHA1, SHA256, SHA512) from text. Specific functions allow for type-specific extraction. ```APIDOC ## extract_hashes(data) / extract_md5_hashes / extract_sha1_hashes / extract_sha256_hashes / extract_sha512_hashes Extract cryptographic file hashes (MD5, SHA1, SHA256, SHA512) from text. Each function matches the exact hex character count for its hash type (32/40/64/128 chars respectively), preventing shorter hashes from being incorrectly matched inside longer ones. ### Parameters - **data** (str): The input string to extract hashes from. ### Request Example ```python import iocextract report = """ Malware analysis results: MD5: 68b329da9893e34099c7d8ad5cb9c940 SHA1: adc83b19e793491b1c6ea0fd8b46cd9f32e592fc SHA256: 01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b SHA512: be688838ca8686e5c90689bf2ab585cef1137c999b48c70b92f67a5c34dc15697b5d11c982ed6d71be1e1e7f7b4e0733884aa97c3f7a339a8ed03577cf74be09 """ # Extract all types of hashes all_hashes = list(iocextract.extract_hashes(report)) print(f"Found {len(all_hashes)} hashes") # Type-specific extraction print(list(iocextract.extract_md5_hashes(report))) print(list(iocextract.extract_sha256_hashes(report))) # SHA1 content does NOT appear in MD5 results (length filtering works correctly) sha1 = 'adc83b19e793491b1c6ea0fd8b46cd9f32e592fc' print(list(iocextract.extract_md5_hashes(sha1))) ``` ### Response Example ``` Found 4 ['68b329da9893e34099c7d8ad5cb9c940'] ['01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b'] [] ``` ``` -------------------------------- ### Extract Email Addresses Source: https://context7.com/inquest/iocextract/llms.txt Extracts email addresses, handling numerous defanged formats using `@` or `at` and various separators like `[.]`, `(.)`, `{.}`, `_dot_`, `[@]`, `(@)`, `{@}`, `_at_`, and spaces. Use `refang=True` to reconstruct standard email formats. ```python import iocextract text = """ Contact the attacker at: admin[@]evil-domain[.]com phishing at victim-corp [.] org exfil{@}c2[.]net spammer AT example DOT com """ for email in iocextract.extract_emails(text, refang=True): print(email) # admin@evil-domain.com # phishing@victim-corp.org # exfil@c2.net # spammer@example.com # Without refanging, original defanged form is preserved for email in iocextract.extract_emails(text): print(email) # admin[@]evil-domain[.]com # phishing at victim-corp [.] org # exfil{@}c2[.]net # spammer AT example DOT com ``` -------------------------------- ### Refang Extracted URLs Source: https://github.com/inquest/iocextract/blob/master/docs/README.md Extracts URLs from a string and applies refanging to remove common obfuscation methods. This is useful for normalizing URLs. ```python import iocextract for url in iocextract.extract_urls(content, refang=True): print(url) # Output # http://example.com/bad/url # http://example.com:8989/bad # http://example.com # http://example.com:8989/bad ``` -------------------------------- ### extract_ips, extract_ipv4s, extract_ipv6s Source: https://context7.com/inquest/iocextract/llms.txt Extracts IPv4 and IPv6 addresses from text, including their defanged forms. `extract_ips` combines both IPv4 and IPv6 extraction. ```APIDOC ## extract_ips(data, refang=False) / extract_ipv4s / extract_ipv6s Extract IPv4 and IPv6 addresses, including defanged forms. `extract_ips` chains both IPv4 and IPv6 extractors. IPv4 supports `[.]`, `(.)`, `\.`, and mixed/partial defangs. IPv6 extraction requires at least 15 characters (minimum for a valid collapsed address like `2001:db8::1`). ### Parameters - **data** (str): The input string to extract IP addresses from. - **refang** (bool, optional): If True, refangs the extracted IP addresses (e.g., replaces `[.]` with `.`). Defaults to False. ### Request Example ```python import iocextract text = """ Infected hosts: 10[.]0[.]0[.]1 192(.)168(.)1(.)100 10\.10\.10\.10 2001:0db8:85a3:0000:0000:8a2e:0370:7334 fe80::1ff:fe23:4567:890a """ # Extract with refanging for ip in iocextract.extract_ips(text, refang=True): print(ip) # IPv4 only, complex bracket combinations tricky = "10[[[[.]]]]10[[[.]]10[.10" print(list(iocextract.extract_ipv4s(tricky, refang=True))) ``` ### Response Example ``` 10.0.0.1 192.168.1.100 10.10.10.10 2001:0db8:85a3:0000:0000:8a2e:0370:7334 fe80::1ff:fe23:4567:890a ['10.10.10.10'] ``` ``` -------------------------------- ### Error Message for Missing Capture Group Source: https://github.com/inquest/iocextract/blob/master/docs/index.md This error message indicates that the provided custom regex does not contain the required single capture group. ```text Error in custom regex: no such group ``` -------------------------------- ### refang_email Source: https://context7.com/inquest/iocextract/llms.txt Refangs a single defanged email address string into a standard email format. ```APIDOC ## refang_email(email) ### Description Refang a single defanged email address string. ### Parameters #### Path Parameters - **email** (string) - Required - The defanged email address string to refang. ### Request Example ```python import iocextract defanged_emails = [ 'user[@]example[.]com', 'user at example dot com', 'user{@}example(.)com', 'user [ at ] example [ dot ] com', ] for email in defanged_emails: print(iocextract.refang_email(email)) ``` ### Response #### Success Response - **refanged_email** (string) - The refanged email address. ### Response Example ``` user@example.com user@example.com user@example.com user@example.com ``` ``` -------------------------------- ### Extract Cryptographic Hashes (MD5, SHA1, SHA256, SHA512) Source: https://context7.com/inquest/iocextract/llms.txt Extracts cryptographic file hashes including MD5, SHA1, SHA256, and SHA512. Each function precisely matches the character count for its hash type, preventing incorrect matches of shorter hashes within longer ones. ```python import iocextract report = """ Malware analysis results: MD5: 68b329da9893e34099c7d8ad5cb9c940 SHA1: adc83b19e793491b1c6ea0fd8b46cd9f32e592fc SHA256: 01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b SHA512: be688838ca8686e5c90689bf2ab585cef1137c999b48c70b92f67a5c34dc15697b5d11c982ed6d71be1e1e7f7b4e0733884aa97c3f7a339a8ed03577cf74be09 """ all_hashes = list(iocextract.extract_hashes(report)) print(f"Found {len(all_hashes)} hashes") # Found 4 # Type-specific extraction print(list(iocextract.extract_md5_hashes(report))) # ['68b329da9893e34099c7d8ad5cb9c940'] print(list(iocextract.extract_sha256_hashes(report))) # ['01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b'] # SHA1 content does NOT appear in MD5 results (length filtering works correctly) sha1 = 'adc83b19e793491b1c6ea0fd8b46cd9f32e592fc' print(list(iocextract.extract_md5_hashes(sha1))) # [] ``` -------------------------------- ### Refang Defanged URL Source: https://context7.com/inquest/iocextract/llms.txt Normalizes a defanged URL string into a standard URL format. It handles various defanging techniques and corrects scheme normalization. Optionally, the scheme can be stripped from the output. ```python import iocextract defanged_urls = [ 'hxxp://example[.]com/path', 'hxxps://example[dot]com/path', 'http__example.com/path', 'http:\\example.com/path', 'http[:]//example.com/path', 'http:// example .com /path', # Cisco ESA style 'ftx://example.com/path', # ftx -> ftp ] for url in defanged_urls: print(iocextract.refang_data(url)) # http://example.com/path # https://example.com/path # http://example.com/path # http://example.com/path # http://example.com/path # http://example.com/path # ftp://example.com/path # Strip scheme from output print(iocextract.refang_data('https://example.com/test', no_scheme=True)) # example.com ``` -------------------------------- ### refang_data Source: https://context7.com/inquest/iocextract/llms.txt Refangs a single defanged URL string, returning a normalized URL. Handles various defang techniques and corrects scheme normalization. ```APIDOC ## refang_data(url, no_scheme=False) ### Description Refang a single defanged URL string, returning a normalized URL. Handles all documented defang techniques: `[.]`, `(.)`, `hxxp`, `[:]//`, `__`, `\`, Cisco ESA spaces, `[/]`, and backslash-escaped characters. Corrects scheme normalization (`ftx` → `ftp`, `hxxps` → `https`). Safe against `urlparse` exceptions on malformed input. ### Parameters #### Path Parameters - **url** (string) - Required - The defanged URL string to refang. - **no_scheme** (boolean) - Optional - If True, the scheme will be stripped from the output URL. ### Request Example ```python import iocextract defanged_urls = [ 'hxxp://example[.]com/path', 'hxxps://example[dot]com/path', 'http__example.com/path', 'http:\\example.com/path', 'http[:]//example.com/path', 'http:// example .com /path', # Cisco ESA style 'ftx://example.com/path', # ftx -> ftp ] for url in defanged_urls: print(iocextract.refang_data(url)) # Strip scheme from output print(iocextract.refang_data('https://example.com/test', no_scheme=True)) ``` ### Response #### Success Response - **normalized_url** (string) - The refanged and normalized URL. ### Response Example ``` http://example.com/path https://example.com/path http://example.com/path http://example.com/path http://example.com/path http://example.com/path ftp://example.com/path example.com ``` ``` -------------------------------- ### Extract Encoded URLs (Hex, URL, Base64) Source: https://context7.com/inquest/iocextract/llms.txt Extracts hex-encoded, URL-encoded, and Base64-encoded URLs. Handles various encoding schemes and can recursively parse JSON values when `parse_json=True`. Use `refang=True` to reconstruct defanged URLs. ```python import iocextract import base64 import binascii # Hex-encoded URL hex_url = binascii.hexlify(b'http://example.com/payload').decode('ascii') print(hex_url) # 687474703a2f2f6578616d706c652e636f6d2f7061796c6f6164 print(list(iocextract.extract_encoded_urls(hex_url, refang=True))) # ['http://example.com/payload'] # URL-encoded url_encoded = "target=http%3A%2F%2Fevil%2Ecom%2Fmalware.exe" print(list(iocextract.extract_encoded_urls(url_encoded, refang=True))) # ['http://evil.com/malware.exe'] # Base64-encoded b64_url = base64.b64encode(b'http://example.com/stage2').decode('ascii') print(b64_url) # aHR0cDovL2V4YW1wbGUuY29tL3N0YWdlMg== print(list(iocextract.extract_encoded_urls(b64_url, refang=True))) # ['http://example.com/stage2'] # Whitespace-split Base64 (as seen in some obfuscated samples) spaced = 'aHR0c DovL2V4YW1w bGUuY29t' print(list(iocextract.extract_encoded_urls(spaced, refang=True))) # ['http://example.com'] ``` -------------------------------- ### Wide-character Mode for Memory Dumps using CLI Source: https://context7.com/inquest/iocextract/llms.txt Process memory dump files using wide-character mode to extract IOCs, such as URLs. This is specifically for handling memory dumps that may contain wide characters. ```bash # Wide-character mode for Windows memory dumps iocextract --input memory_dump.txt --wide --extract-urls ``` -------------------------------- ### Extracting Entire Match with Custom Regex Source: https://github.com/inquest/iocextract/blob/master/docs/README.md To extract the entire matched string, enclose the entire regex pattern within a single capture group. ```text (https?://.*?.com) ```