iocextract

https://iocextract.readthedocs.io/en/latest/
Admin
iocextract is a Python library and CLI tool for extracting Indicators of Compromise (IOCs) like...
Tokens:4,821
Snippets:47
Trust Score:9.6
Update:5 months ago
Show doc for...
Tokens:
Raw
### Install regex dependency on Windows

Source: https://iocextract.readthedocs.io

Offers guidance for installing the `regex` dependency on Windows by downloading a wheel file from PyPI and installing it via pip.

```bash
pip install regex-2018.06.21-cp27-none-win_amd64.whl

```

--------------------------------

### Install iocextract

Source: https://iocextract.readthedocs.io

Provides installation instructions for the iocextract package using pip. It also mentions the potential need for Python development headers on Ubuntu/Debian-based systems.

```bash
sudo apt-get install python-dev

pip install iocextract

```

--------------------------------

### Extracted URLs from Example Tweet

Source: https://iocextract.readthedocs.io

Shows the output of extracting URLs from the example tweet, highlighting the ability to capture defanged IOCs.

```text
https://researchcenter.paloaltonetworks.com/2018/02/unit42-sofacy-attacks-multiple-government-entities/
hotfixmsupload[.]com
cdnverify[.]net

```

--------------------------------

### Example Tweet with Defanged URLs

Source: https://iocextract.readthedocs.io

Presents a sample tweet containing defanged URLs, demonstrating a real-world scenario where IOC extraction is valuable.

```text
Recommended reading and great work from @unit42_intel:
https://researchcenter.paloaltonetworks.com/2018/02/unit42-sofacy-attacks-multiple-government-entities/ ...
InQuest customers have had detection for threats delivered from hotfixmsupload[.]com
since 6/3/2017 and cdnverify[.]net since 2/1/18.

```

--------------------------------

### Example of Defanged IP Address

Source: https://iocextract.readthedocs.io

Illustrates a common technique for defanging IP addresses by enclosing periods in brackets, making them unparseable by standard IP address regex.

```text
127[.]0[.]0[.]1

```

--------------------------------

### Python Regex with Combined Capture and Non-Capture Groups

Source: https://iocextract.readthedocs.io

For complex regex, combine non-capture groups `(?: )` with capture groups `( )` to precisely control what is extracted. This example extracts 're'.

```python
[
    r'(?:my|your) (re)gex',  # This yields 're' if the pattern matches
]
```

--------------------------------

### Python Regex with Single Capture Group

Source: https://iocextract.readthedocs.io

When using Python lists for custom regex, ensure each pattern has exactly one capture group. This example shows valid patterns yielding specific substrings.

```python
[
    r'(my regex)',  # This yields 'my regex' if the pattern matches
    r'my (re)gex',  # This yields 're' if the pattern matches
]
```

--------------------------------

### Display IOCTextract CLI Help

Source: https://iocextract.readthedocs.io

Use the -h flag to display the help message for the IOCTextract command-line interface, showing all available options and their descriptions.

```bash
$ iocextract -h
```

--------------------------------

### Run IOCTextract CLI with Options

Source: https://iocextract.readthedocs.io

The IOCTextract CLI can extract various IOC types. Specify input and output files, or enable specific extraction types like emails, IPs, URLs, Yara rules, and hashes. Custom regex patterns can be loaded from a file. Options like --refang and --strip-urls modify the extraction behavior.

```bash
usage: iocextract [-h] [--input INPUT] [--output OUTPUT] [--extract-emails]
              [--extract-ips] [--extract-ipv4s] [--extract-ipv6s]
              [--extract-urls] [--extract-yara-rules] [--extract-hashes]
              [--custom-regex REGEX_FILE] [--refang] [--strip-urls]
              [--wide]

Advanced Indicator of Compromise (IOC) extractor. If no arguments are
specified, the default behavior is to extract all IOCs.

optional arguments:
  -h, --help            show this help message and exit
  --input INPUT         default: stdin
  --output OUTPUT       default: stdout
  --extract-emails
  --extract-ips
  --extract-ipv4s
  --extract-ipv6s
  --extract-urls
  --extract-yara-rules
  --extract-hashes
  --custom-regex REGEX_FILE
                        file with custom regex strings, one per line, with one
                        capture group each
  --refang              default: no
  --strip-urls          remove possible garbage from the end of urls. default:
                        no
  --wide                preprocess input to allow wide-encoded character
                        matches. default: no
```

--------------------------------

### Refang Obfuscated URLs with iocextract

Source: https://iocextract.readthedocs.io

Shows how to extract URLs and simultaneously remove common obfuscation methods (refanging) by setting `refang=True`.

```python
import iocextract

content = \
"""
I really love example[.]com!
All the bots are on hxxp://example.com/bad/url these days.
C2: tcp://example[.]com:8989/bad
"""

for url in iocextract.extract_urls(content, refang=True):
    print(url)

    # Output

    # http://example.com/bad/url
    # http://example.com:8989/bad
    # http://example.com
    # http://example.com:8989/bad

```

--------------------------------

### Define Custom Regex Patterns

Source: https://iocextract.readthedocs.io

Create a plain text file with one regex string per line. Each regex must include exactly one capture group to extract specific IOCs.

```regex
http://(example\.com)/
```

```regex
(?:https|ftp)://(example\.com)/
```

--------------------------------

### Convert Iterator to List

Source: https://iocextract.readthedocs.io

Demonstrates how to convert the iterator returned by extraction functions into a list, useful for iterating multiple times.

```APIDOC
## Converting Iterator to List

### Description
All extraction functions in `iocextract` return iterators for memory efficiency. If you need to iterate over the results multiple times, you can convert the iterator to a list.

### Request Example
```python
import iocextract

content = \
"""
I really love example[.]com!
All the bots are on hxxp://example.com/bad/url these days.
C2: tcp://example[.]com:8989/bad
"""

urls_list = list(iocextract.extract_urls(content))
print(urls_list)
```

### Response Example
```
# Output:
# ['hxxp://example.com/bad/url', 'tcp://example[.]com:8989/bad', 'example[.]com', 'tcp://example[.]com:8989/bad']
```
```

--------------------------------

### Convert iocextract Iterator to List

Source: https://iocextract.readthedocs.io

Demonstrates how to convert the iterator returned by `extract_urls` into a list. This is necessary if you need to iterate over the extracted IOCs multiple times.

```python
import iocextract

content = \
"""
I really love example[.]com!
All the bots are on hxxp://example.com/bad/url these days.
C2: tcp://example[.]com:8989/bad
"""

print(list(iocextract.extract_urls(content)))

```

--------------------------------

### Extract Defanged URLs with iocextract

Source: https://iocextract.readthedocs.io

Demonstrates basic extraction of defanged URLs from a given text content. Note that some URLs might appear twice if matched by multiple regex patterns.

```python
import iocextract

content = \
"""
I really love example[.]com!
All the bots are on hxxp://example.com/bad/url these days.
C2: tcp://example[.]com:8989/bad
"""

for url in iocextract.extract_urls(content):
    print(url)

    # Output

    # hxxp://example.com/bad/url
    # tcp://example[.]com:8989/bad
    # example[.]com
    # tcp://example[.]com:8989/bad

```

--------------------------------

### Disable Defanging during Extraction

Source: https://iocextract.readthedocs.io

Illustrates how to disable the automatic defanging of IOCs during extraction by setting `defang=False`. This is useful when the input content already contains non-defanged IOCs.

```python
import iocextract

content = \
"""
http://example.com/bad/url
http://example.com:8989/bad
http://example.com
http://example.com:8989/bad
"""

for url in iocextract.extract_urls(content, defang=False):
    print(url)

    # Output

    # http://example.com/bad/url
    # http://example.com:8989/bad
    # http://example.com
    # http://example.com:8989/bad

```

--------------------------------

### Extract YARA Rules

Source: https://iocextract.readthedocs.io

Extracts YARA rules from a given text.

```APIDOC
## extract_yara(content)

### Description
Extracts YARA rules from the provided text content.

### Parameters
- **content** (string) - Required - The text content to extract YARA rules from.

### Returns
An iterator yielding extracted YARA rules.

### Request Example
```python
import iocextract

content = \
"""
rule example_rule {
  strings:
    $a = "bad_stuff"
  condition:
    $a
}
"""

for rule in iocextract.extract_yara(content):
    print(rule)
```

### Response Example
```
# Output for extract_yara(content):
# rule example_rule {
#   strings:
#     $a = "bad_stuff"
#   condition:
#     $a
# }
```
```

--------------------------------

### Extract Hashes

Source: https://iocextract.readthedocs.io

Extracts MD5 and SHA hashes from a given text. Supports options to disable defanging during extraction.

```APIDOC
## extract_hashes(content, defang=True)

### Description
Extracts MD5 and SHA hashes from the provided text content. It can optionally control whether defanging is applied during extraction.

### Parameters
- **content** (string) - Required - The text content to extract hashes from.
- **defang** (boolean) - Optional - If False, defanging will not be applied during extraction.

### Returns
An iterator yielding extracted hashes.

### Request Example
```python
import iocextract

content = \
"""
MD5: d41d8cd98f00b204e9800998ecf8427e
SHA1: a94a8fe5ccb19ba61c4c0873d391e987982fbbd3
SHA256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
"""

for hash_val in iocextract.extract_hashes(content):
    print(hash_val)
```

### Response Example
```
# Output for extract_hashes(content):
# d41d8cd98f00b204e9800998ecf8427e
# a94a8fe5ccb19ba61c4c0873d391e987982fbbd3
# e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
```
```

--------------------------------

### Handle Invalid Regex Errors

Source: https://iocextract.readthedocs.io

If a custom regex is invalid (e.g., missing a closing parenthesis), the CLI will report an error indicating the position of the issue.

```text
Error in custom regex: missing ) at position 5
```

--------------------------------

### Extract URLs

Source: https://iocextract.readthedocs.io

Extracts URLs from a given text. Supports options to refang (decode) obfuscated URLs and to disable defanging during extraction.

```APIDOC
## extract_urls(content, refang=False, defang=True)

### Description
Extracts URLs from the provided text content. It can optionally refang (decode) obfuscated URLs and control whether defanging is applied during extraction.

### Parameters
- **content** (string) - Required - The text content to extract URLs from.
- **refang** (boolean) - Optional - If True, obfuscated URLs will be decoded.
- **defang** (boolean) - Optional - If False, defanging will not be applied during extraction.

### Returns
An iterator yielding extracted URLs.

### Request Example
```python
import iocextract

content = \
"""
I really love example[.]com!
All the bots are on hxxp://example.com/bad/url these days.
C2: tcp://example[.]com:8989/bad
"""

for url in iocextract.extract_urls(content):
    print(url)

# Example with refang=True
for url in iocextract.extract_urls(content, refang=True):
    print(url)

# Example with defang=False
for url in iocextract.extract_urls(content, defang=False):
    print(url)
```

### Response Example
```
# Output for extract_urls(content):
# hxxp://example.com/bad/url
# tcp://example[.]com:8989/bad
# example[.]com
# tcp://example[.]com:8989/bad

# Output for extract_urls(content, refang=True):
# http://example.com/bad/url
# http://example.com:8989/bad
# http://example.com
# http://example.com:8989/bad

# Output for extract_urls(content, defang=False):
# http://example.com/bad/url
# http://example.com:8989/bad
# http://example.com
# http://example.com:8989/bad
```
```

--------------------------------

### Extract IPs

Source: https://iocextract.readthedocs.io

Extracts IP addresses from a given text. Supports options to refang (decode) obfuscated IPs and to disable defanging during extraction.

```APIDOC
## extract_ips(content, refang=False, defang=True)

### Description
Extracts IP addresses from the provided text content. It can optionally refang (decode) obfuscated IPs and control whether defanging is applied during extraction.

### Parameters
- **content** (string) - Required - The text content to extract IPs from.
- **refang** (boolean) - Optional - If True, obfuscated IPs will be decoded.
- **defang** (boolean) - Optional - If False, defanging will not be applied during extraction.

### Returns
An iterator yielding extracted IP addresses.

### Request Example
```python
import iocextract

content = \
"""
This is a test with an IP: 127[.]0[.]0[.]1 and another one 192.168.1.1
"""

for ip in iocextract.extract_ips(content):
    print(ip)

# Example with refang=True
for ip in iocextract.extract_ips(content, refang=True):
    print(ip)
```

### Response Example
```
# Output for extract_ips(content):
# 127[.]0[.]0[.]1
# 192.168.1.1

# Output for extract_ips(content, refang=True):
# 127.0.0.1
# 192.168.1.1
```
```

--------------------------------

### Handle Regex Without Capture Group Errors

Source: https://iocextract.readthedocs.io

If a custom regex does not contain a capture group, an error will be raised indicating that no group was found.

```text
Error in custom regex: no such group
```

--------------------------------

### Extract Entire Match with Custom Regex

Source: https://iocextract.readthedocs.io

To extract the entire matched string, enclose your entire regex pattern within a single capture group.

```regex
(https?://.*?.com)
```

--------------------------------

### Python Regex with Multiple Capture Groups (Unexpected Results)

Source: https://iocextract.readthedocs.io

Using more than one capture group in a Python regex pattern can lead to unexpected results, as only the first group's match is yielded.

```python
[
    r'my regex',  # This doesn't yield anything
    r'(my) (re)gex',  # This yields 'my' if the pattern matches
]
```

--------------------------------

### Extract Emails

Source: https://iocextract.readthedocs.io

Extracts email addresses from a given text. Supports options to refang (decode) obfuscated emails and to disable defanging during extraction.

```APIDOC
## extract_emails(content, refang=False, defang=True)

### Description
Extracts email addresses from the provided text content. It can optionally refang (decode) obfuscated emails and control whether defanging is applied during extraction.

### Parameters
- **content** (string) - Required - The text content to extract emails from.
- **refang** (boolean) - Optional - If True, obfuscated emails will be decoded.
- **defang** (boolean) - Optional - If False, defanging will not be applied during extraction.

### Returns
An iterator yielding extracted email addresses.

### Request Example
```python
import iocextract

content = \
"""
Contact us at support@example.com or sales[.]example[.]org.
"""

for email in iocextract.extract_emails(content):
    print(email)

# Example with refang=True
for email in iocextract.extract_emails(content, refang=True):
    print(email)
```

### Response Example
```
# Output for extract_emails(content):
# support@example.com
# sales[.]example[.]org

# Output for extract_emails(content, refang=True):
# support@example.com
# sales.example.org
```
```
Tokens:
Raw