CyberIntegrations

https://pypi.org/project/cyberintegrations/
Admin
CyberIntegrations is a Python library for communicating with TI and DRP systems via API, simplifying...
Tokens:6,020
Snippets:63
Trust Score:10
Update:1 month ago
Show doc for...
Tokens:
Raw
### Install CyberIntegrations

Source: https://pypi.org/project/cyberintegrations

Install the library via PyPI or from a local wheel file.

```bash
pip install cyberintegrations
```

```bash
pip install ./cyberintegrations-X.X.X-py3-none-any.whl
```

--------------------------------

### Retrieve threat data with custom limit

Source: https://pypi.org/project/cyberintegrations

Example of a GET request to the threat update endpoint with a specified record limit.

```bash
curl -X 'GET' 'https://<base URL>/api/v2/apt/threat/updated?limit=500&seqUpdate=16172928022293'
```

--------------------------------

### Example API Response Structure

Source: https://pypi.org/project/cyberintegrations

A sample JSON response showing the items list and the top-level seqUpdate field.

```json
{
    "count": 1761,
    "items": [
        {"id": "fake286ca753feed3476649438e4e4488"...},
        {"id": "fake51d29357b22b80564a1d2f9fc8751"...},
        {
            "author": null,
            "companyId": [],
            "id": "fake4f16300296d20ef9b909dc0d354fb",
            ......,
            "indicators": [
                {
                    "dateFirstSeen": null,
                    "dateLastSeen": null,
                    "deleted": false,
                    "description": null,
                    "domain": "fake-fakesop.net",
                    "id": "fakebe483bb82759fbee7038235e0f52d0",
                    .....
                }
            ],
            "indicatorsIds": [
                "fakebe483bb82759fbee7038235e0f52d0"
            ],
            "isPublished": true,
            "isTailored": false,
            "labels": [],
            "langs": [
                "en"
            ],
            "malwareList": [],
            ......,
            "seqUpdate": 16172928022293
        },
    ],
    "seqUpdate": 16172928022293
}
```

--------------------------------

### GET /api/v2/sequnce_list

Source: https://pypi.org/project/cyberintegrations

Retrieve the starting sequence update number for a specific collection.

```APIDOC
## GET /api/v2/sequnce_list

### Description
Get the latest sequence update number for a collection to begin iterating through events.

### Method
GET

### Endpoint
/api/v2/sequnce_list

### Parameters
#### Query Parameters
- **date** (string) - Optional - The date to filter updates.
- **collection** (string) - Optional - The specific collection name (e.g., apt/threat).
```

--------------------------------

### Global Search Response Structure

Source: https://pypi.org/project/cyberintegrations

Example JSON response from the /api/v2/search endpoint showing collections containing the search term.

```json
[
    {
        "apiPath": "suspicious_ip/open_proxy",
        "label": "Suspicious IP :: Open Proxy",
        "link": "https://<base-url>/api/v2/suspicious_ip/open_proxy?q=ip:8.8.8.8",
        "count": 14,
        "time": 0.304644684,
        "detailedLinks": null
    },
    {
        "apiPath": "attacks/ddos",
        "label": "Attack :: DDoS",
        "link": "https://<base-url>/api/v2/attacks/ddos?q=ip:8.8.8.8",
        "count": 1490,
        "time": 0.389418291,
        "detailedLinks": null
    },
    {"apiPath": "attacks/deface"...},
    {"apiPath": "malware/config"...},
    {"apiPath": "suspicious_ip/scanner"...}
]
```

--------------------------------

### Define API Response Structure

Source: https://pypi.org/project/cyberintegrations

Example structure of a raw API response received from a collection.

```python
api_response = [
    {
        'iocs': {
            'network': [
                {
                    'ip': [1, 2],
                    'url': 'url.com'
                },
                {
                    'ip': [3],
                    'url': ''
                }
            ]
        }
    },
    {
        'iocs': {
            'network': [
                {
                    'ip': [4, 5],
                    'url': 'new_url.com'
                }
            ]
        }
    }
]
```

--------------------------------

### Get IoCs Output

Source: https://pypi.org/project/cyberintegrations

Resulting structure after extracting IoCs using the defined mapping.

```python
parsing_result = {
    'ips': [1, 2, 3, 4, 5], 
    'url': ['url.com', 'new_url.com']
}
```

--------------------------------

### Collection Data Response

Source: https://pypi.org/project/cyberintegrations

Example JSON response from a specific collection endpoint containing items and the seqUpdate field for pagination.

```json
{
  "count": 1490,
  "items": [
    {
      "body": null,
      "cnc": {"cnc": "http://ex-ex.net/drv/"...},
      "company": null,
      "companyId": null,
      "dateBegin": null,
      "dateEnd": null,
      "dateReg": "2017-08-16T00:00:00+00:00",
      "evaluation": {},
      "favouriteForCompanies": [],
      "headers": [],
      "hideForCompanies": [],
      "id": "examplec58903baddc84b8c51eaef1f904374025d",
      "isFavourite": false,
      ...
    }
  ],
  ...,
  "seqUpdate": 1673373011294
}
```

--------------------------------

### GET /api/v2/search

Source: https://pypi.org/project/cyberintegrations

Search for IPs, domains, hashes, or emails to retrieve relevant collections.

```APIDOC
## GET /api/v2/search

### Description
Search for specific indicators like IPs, domains, or hashes to identify collections containing the search result.

### Method
GET

### Endpoint
/api/v2/search

### Parameters
#### Query Parameters
- **q** (string) - Required - The search query (e.g., an IP address or domain).
```

--------------------------------

### GET /api/v2/apt/threat/updated

Source: https://pypi.org/project/cyberintegrations

Retrieves updated threat records. Users can control the number of records returned by specifying a limit parameter.

```APIDOC
## GET /api/v2/apt/threat/updated

### Description
Retrieves updated threat records from the API. The default limit is 100 records per request, but this can be adjusted using the limit query parameter.

### Method
GET

### Endpoint
https://<base URL>/api/v2/apt/threat/updated

### Parameters
#### Query Parameters
- **limit** (integer) - Optional - The number of records to return in the response.
- **seqUpdate** (string) - Optional - Sequence update timestamp for filtering records.

### Request Example
curl -X 'GET' 'https://<base URL>/api/v2/apt/threat/updated?limit=500&seqUpdate=16172928022293'
```

--------------------------------

### GET /api/v2/{collection}/updated

Source: https://pypi.org/project/cyberintegrations

Retrieve a portion of events from a collection using a sequence update number.

```APIDOC
## GET /api/v2/{collection}/updated

### Description
Fetch a batch of events from a specific collection starting from the provided sequence update number.

### Method
GET

### Endpoint
/api/v2/{collection}/updated

### Parameters
#### Path Parameters
- **collection** (string) - Required - The name of the collection (e.g., apt/threat).

#### Query Parameters
- **seqUpdate** (number) - Required - The sequence update number retrieved from a previous response or sequence_list.
```

--------------------------------

### GET /api/v2/{collection}/updated

Source: https://pypi.org/project/cyberintegrations

Retrieves updated items from a specific collection using a sequence update token to paginate through results.

```APIDOC
## GET /api/v2/{collection}/updated

### Description
Retrieves a portion of data from a collection using a sequence update token to ensure incremental data fetching.

### Method
GET

### Endpoint
/api/v2/{collection}/updated

### Parameters
#### Path Parameters
- **collection** (string) - Required - The name of the collection (e.g., attacks/ddos).

#### Query Parameters
- **q** (string) - Optional - The search query filter.
- **seqUpdate** (integer) - Required - The sequence update token retrieved from the previous response.
- **limit** (integer) - Optional - The maximum number of items to return.
```

--------------------------------

### GET /api/v2/search

Source: https://pypi.org/project/cyberintegrations

Performs a global search across the threat intelligence database to identify collections containing the specified search query.

```APIDOC
## GET /api/v2/search

### Description
Finds events related to a specific indicator (IP, domain, hash, email) and returns a list of collections containing the data.

### Method
GET

### Endpoint
/api/v2/search

### Parameters
#### Query Parameters
- **q** (string) - Required - The search query, optionally prefixed with a type (e.g., ip:8.8.8.8, domain:google.com, email:example@gmail.com).

### Response
#### Success Response (200)
- **apiPath** (string) - The path to the collection.
- **label** (string) - Human-readable label for the collection.
- **link** (string) - Full URL to the collection.
- **count** (integer) - Number of items in the collection.
- **time** (float) - Execution time.
```

--------------------------------

### Full TIPoller Implementation

Source: https://pypi.org/project/cyberintegrations

Demonstrates the complete lifecycle of a TIPoller instance, including proxy configuration, key management, and error handling for data retrieval.

```python
import logging
from cyberintegrations import TIPoller
from cyberintegrations.exception import InputException, ConnectionException, ParserException

logging.basicConfig(level=logging.DEBUG, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s')
...

try:
   poller = TIPoller(username=username, api_key=api_key, api_url=api_url)
   poller.set_proxies(proxy_protocol=PROXY_PROTOCOL,
                      proxy_port=PROXY_PORT,
                      proxy_ip=PROXY_ADDRESS,
                      proxy_password=PROXY_PASSWORD,
                      proxy_username=PROXY_USERNAME)
   poller.set_verify(True)
   for collection, keys in keys_config.items():
   poller.set_keys(collection, keys)	
   for collection, state in update_generator_config.items():
        if state.get("sequpdate"):
        generator = poller.create_update_generator(collection_name=collection, sequpdate=state.get("sequpdate"))
    elif state.get("date_from"):
        sequpdate = poller.get_seq_update_dict(date=state.get('date_from'), collection_name=collection).get(collection)
        generator = poller.create_update_generator(collection_name=collection, sequpdate=sequpdate)
   else:
       continue
   for portion in generator:
       parsed_portion = portion.parse_portion()
           save_portion(parsed_portion)
       update_generator_config[collection]["sequpdate"] = portion.sequpdate
       
except InputException as e:
   logging.exception("Wrong input: {0}".format(e))
except ConnectionException as e:
   logging.exception("Something wrong with connection: {0}".format(e))
except ParserException as e:
   logging.exception("Exception occured during parsing: {0}".format(e))
finally:
   poller.close_session()
```

--------------------------------

### Initialize Poller

Source: https://pypi.org/project/cyberintegrations

Configure the poller with credentials, proxy settings, and SSL verification preferences.

```python
from cyberintegrations import TIPoller, DRPPoller

poller = TIPoller(username='example@gmail.com', api_key='API_KEY', api_url="API_URL")
poller.set_proxies(
                proxy_protocol=PROXY_PROTOCOL,
                proxy_port=PROXY_PORT,
                proxy_ip=PROXY_ADDRESS,
                proxy_password=PROXY_PASSWORD,
                proxy_username=PROXY_USERNAME
            )
poller.set_verify(True)
```

--------------------------------

### Create Update Generator

Source: https://pypi.org/project/cyberintegrations

Initialize a generator to fetch feed portions based on sequence updates or search criteria.

```python
generator = poller.create_update_generator(
    collection_name='compromised/account_group', 
    date_from='2021-01-30', 
    date_to='2021-02-03', 
    query='8.8.8.8', 
    sequpdate=20000000, 
    limit=200
)
```

--------------------------------

### Sequential Update API Requests

Source: https://pypi.org/project/cyberintegrations

cURL commands demonstrating the initial search and subsequent iterative requests using the seqUpdate parameter.

```bash
curl -X 'GET' 'https://<base URL>/api/v2/search?q=ip:8.8.8.8'
```

```bash
curl -X 'GET' 'https://<base URL>/api/v2/apt/threat/updated?seqUpdate=1673373011294'
```

```bash
curl -X 'GET' 'https://<base URL>/api/v2/apt/threat/updated?seqUpdate=1673375930599'
```

--------------------------------

### API Iteration Requests

Source: https://pypi.org/project/cyberintegrations

Commands to retrieve sequence lists and iterate through collection updates using seqUpdate.

```bash
curl -X 'GET' 'https://<base URL>/api/v2/sequnce_list'
```

```bash
curl -X 'GET' 'https://<base URL>/api/v2/apt/threat/updated?seqUpdate=16172928022293'
```

```bash
curl -X 'GET' 'https://<base URL>/api/v2/apt/threat/updated?seqUpdate=16172928536227'
```

--------------------------------

### TIPoller.search_file_in_threats()

Source: https://pypi.org/project/cyberintegrations

Downloads a binary file associated with a threat report.

```APIDOC
## TIPoller.search_file_in_threats()

### Description
Retrieves a binary file from threat reports based on the collection, feed, and file ID.

### Method
SDK Method

### Parameters
- **collection_name** (string) - Required - The name of the collection.
- **feed_id** (string) - Required - The unique identifier of the feed.
- **file_id** (string) - Required - The unique identifier of the file within the feed.

### Returns
- **binary_file** (bytes) - The binary content of the requested file.
```

--------------------------------

### set_verify

Source: https://pypi.org/project/cyberintegrations

Configures SSL certificate verification.

```APIDOC
## set_verify(verify)

### Description
Configures SSL verification settings. If set to True, verifies the TLS certificate. If False, accepts any certificate. Can also accept a path string to a custom certificate.

### Parameters
- **verify** (bool/str) - Required - Boolean flag or path to certificate file.
```

--------------------------------

### TIPoller / DRPPoller Initialization

Source: https://pypi.org/project/cyberintegrations

Initializes the poller instance with credentials and API URL.

```APIDOC
## TIPoller(username, api_key, api_url)

### Description
Initializes a new instance of the threat intelligence or DRP poller.

### Parameters
- **username** (str) - Required - The username for authentication.
- **api_key** (str) - Required - The API key for authentication.
- **api_url** (str) - Required - The base URL for the API service.
```

--------------------------------

### Configure Parsing Mapping

Source: https://pypi.org/project/cyberintegrations

Define mapping keys to transform raw API data into a structured format using parse_portion or bulk_parse_portion.

```python
mapping_keys = {
    'network': {'ips': 'iocs.network.ip'},
    'url': 'iocs.network.url',
    'type': '*custom_network'
}
```

--------------------------------

### Download Binary File

Source: https://pypi.org/project/cyberintegrations

Retrieve binary files associated with threat reports using collection, feed, and file identifiers.

```python
binary_file = poller.search_file_in_threats(collection_name='hi/threat', feed_id='some_id', file_id='some_file_id_inside_feed')
```

--------------------------------

### Configure IoC Mapping

Source: https://pypi.org/project/cyberintegrations

Mapping dictionary used specifically for the get_iocs method.

```python
mapping_keys = {
    'ips': 'iocs.network.ip',
    'url': 'iocs.network.url'
}
```

--------------------------------

### Manage Session Lifecycle

Source: https://pypi.org/project/cyberintegrations

Ensure sessions are closed properly using a try-except-finally block or a context manager.

```python
from cyberintegrations import TIPoller
from cyberintegrations.exception import InputException

...

try:
    poller = TIPoller(username='example@gmail.com', api_key='API_KEY', api_url="API_URL")
   ...
except InputException as e:
   logger.info("Wrong input: {0}".format(e))
finally:
   poller.close_session()
```

--------------------------------

### Retrieve Available Collections and Updates

Source: https://pypi.org/project/cyberintegrations

Use get_available_collections to verify access before querying. The seq_update_dict provides access to specific account updates.

```python
collection_list = poller.get_available_collections()  
seq_update_dict = poller.get_seq_update_dict(date='2020-12-12')  
compromised_account_sequpdate = seq_update_dict.get('compromised/account')
```

--------------------------------

### TIPoller.get_available_collections()

Source: https://pypi.org/project/cyberintegrations

Retrieves a list of available collections that the user has access to.

```APIDOC
## TIPoller.get_available_collections()

### Description
Returns a list of collections available to the authenticated user. It is recommended to call this before accessing collections to avoid errors.

### Method
SDK Method

### Returns
- **collection_list** (list) - A list of available collection names.
```

--------------------------------

### Override Mapping Keys

Source: https://pypi.org/project/cyberintegrations

Apply custom mapping keys during the parsing process to override default configurations.

```python
parsed_json = portion.parse_portion(as_json=False)  
iocs = portion.get_iocs(as_json=False, keys=mapping_override_keys)
```

--------------------------------

### set_keys / set_iocs_keys

Source: https://pypi.org/project/cyberintegrations

Defines mapping keys for parsing collection data.

```APIDOC
## set_keys(collection_name, keys)
## set_iocs_keys(collection_name, keys)

### Description
Sets the mapping keys used to extract and parse data from API responses.

### Parameters
- **collection_name** (str) - Required - The name of the collection.
- **keys** (dict) - Required - A dictionary mapping result names to dot-notation paths in the API response.
```

--------------------------------

### Parse Portion Output

Source: https://pypi.org/project/cyberintegrations

Resulting structure after applying the mapping to the API response.

```python
parsing_result = [
    {
        'network': {'ips': [[1, 2], [3]]},
        'url': ['url.com', ''],
        'type': 'custom_network'
    },
    {
        'network': {'ips': [[4, 5]]},
        'url': ['new_url.com'],
        'type': 'custom_network'
    }
]
```

--------------------------------

### Calculate seqUpdate Timestamp

Source: https://pypi.org/project/cyberintegrations

Formula for calculating the seqUpdate value based on UTC timestamp.

```text
UTC timestamp * 1000 * 1000. 
```

--------------------------------

### create_update_generator / create_search_generator

Source: https://pypi.org/project/cyberintegrations

Creates a generator to iterate over feed portions.

```APIDOC
## create_update_generator(collection_name, date_from, date_to, query, sequpdate, limit)
## create_search_generator(collection_name, date_from, date_to, query, limit)

### Description
Creates a generator object to fetch portions of feeds. Update generator iterates by seqUpdate; Search generator iterates by resultId.

### Parameters
- **collection_name** (str) - Required - The target collection.
- **date_from** (str) - Optional - Start date filter.
- **date_to** (str) - Optional - End date filter.
- **query** (str) - Optional - Search query string.
- **sequpdate** (int) - Optional - Sequence update starting point.
- **limit** (int) - Optional - Maximum number of items.
```

--------------------------------

### Bulk Parse Portion Output

Source: https://pypi.org/project/cyberintegrations

Resulting structure after applying bulk parsing to the API response.

```python
parsing_result = [
    [
        {
            'network': {'ips': [[1, 2], [3]]}, 
            'url': ['url.com', ''], 
            'type': 'custom_network'}
    ],
    [
        {
            'network': {'ips': [[4, 5]]},
            'url': ['new_url.com'], 
            'type': 'custom_network'}
    ]
]
```

--------------------------------

### set_proxies

Source: https://pypi.org/project/cyberintegrations

Configures proxy settings for the poller instance.

```APIDOC
## set_proxies(proxy_protocol, proxy_port, proxy_ip, proxy_password, proxy_username)

### Description
Sets the proxy configuration for network requests.

### Parameters
- **proxy_protocol** (str) - Required - The protocol (e.g., http, https).
- **proxy_port** (int) - Required - The port number.
- **proxy_ip** (str) - Required - The proxy IP address.
- **proxy_password** (str) - Optional - Password for proxy authentication.
- **proxy_username** (str) - Optional - Username for proxy authentication.
```

--------------------------------

### Bulk Parse Portions

Source: https://pypi.org/project/cyberintegrations

Extract multiple parsed dictionaries from a single feed portion using bulk processing.

```python
new_parsed_json = portion.bulk_parse_portion(keys_list=[{"ips": "indicators.params.ip"}, {"url": 'indicators.params.url'}], as_json=False)
```

--------------------------------

### Process Generator Portions

Source: https://pypi.org/project/cyberintegrations

Iterate through generator portions to parse data, extract IoCs, or access raw attributes.

```python
for portion in generator:  
    parsed_json = portion.parse_portion(as_json=False)  
    iocs = portion.get_iocs(as_json=False) 
    sequpdate = portion.sequpdate  
    count = portion.count  
    raw_json = portion.raw_json  
    raw_dict = portion.raw_dict
    new_parsed_json = portion.bulk_parse_portion(keys_list=[{"ips": "indicators.params.ip"}, {"url": 'indicators.params.url'}], as_json=False)
```

--------------------------------

### Define Collection Mapping

Source: https://pypi.org/project/cyberintegrations

Map API response fields to custom keys using dot-notation or static values.

```python
mapping_keys = {"result_name": "searchable_key_1.searchable_key_2"}
```

```python
mapping_keys = {
	"network": "indicators.params.ip", 
	"result_name": "*My_Value"
}
```

```python
mapping_keys = {
	'network': {
		'ips': 'indicators.params.ip'
	}, 
	'url': 'indicators.params.url', 
	'type': '*network'
}
poller.set_keys(collection_name="apt/threat", keys=mapping_keys)
poller.set_iocs_keys(collection_name="apt/threat", keys={"ips": "indicators.params.ip"})
```

--------------------------------

### Search Feed by ID

Source: https://pypi.org/project/cyberintegrations

Locate a specific feed by ID to obtain a Parser object for further processing.

```python
feed = poller.search_feed_by_id(collection_name='compromised/account', feed_id='some_id')  
parsed_feed = feed.parse_portion()
```

--------------------------------

### Access Generator Attributes

Source: https://pypi.org/project/cyberintegrations

Retrieve metadata like sequence update numbers or remaining feed counts from the generator.

```python
sequpdate = portion.sequpdate
```

```python
count = portion.count
```

--------------------------------

### TIPoller.search_feed_by_id()

Source: https://pypi.org/project/cyberintegrations

Searches for a specific feed within a collection by its ID.

```APIDOC
## TIPoller.search_feed_by_id()

### Description
Finds a specific feed by its ID within a given collection and returns a Parser object.

### Method
SDK Method

### Parameters
- **collection_name** (string) - Required - The name of the collection.
- **feed_id** (string) - Required - The unique identifier of the feed.

### Returns
- **feed** (Parser) - A Parser object used to process the feed data.
```
Tokens:
Raw