### Install Windows Prerequisites for Building NFStream Source: https://www.nfstream.org/docs Installs necessary tools and libraries on Windows using MSYS2 for building NFStream from source. Ensure npcap is also installed. ```bash pacman -S git unzip mingw-w64-x86_64-toolchain automake1.16 automake-wrapper autoconf libtool make mingw-w64-x86_64-json-c mingw-w64-x86_64-crt-git ``` -------------------------------- ### Install Linux Prerequisites for Building NFStream Source: https://www.nfstream.org/docs Installs necessary development packages on Debian/Ubuntu-based Linux systems for building NFStream from source. ```bash sudo apt-get update sudo apt-get install python3-dev autoconf automake libtool pkg-config flex bison gettext libjson-c-dev sudo apt-get install libusb-1.0-0-dev libdbus-glib-1-dev libbluetooth-dev libnl-genl-3-dev ``` -------------------------------- ### Build NFStream from Source Source: https://www.nfstream.org/docs Clones the NFStream repository and installs it from source, including development dependencies. Ensure submodules are initialized. ```bash git clone --recurse-submodules https://github.com/nfstream/nfstream.git cd nfstream python3 -m pip install --upgrade pip python3 -m pip install -r dev_requirements.txt python3 -m pip install . ``` -------------------------------- ### Install NFStream using pip Source: https://www.nfstream.org/docs Install the latest released version of NFStream using the Python package manager pip. ```bash pip install nfstream ``` -------------------------------- ### Install macOS Prerequisites for Building NFStream Source: https://www.nfstream.org/docs Installs required build tools on macOS using Homebrew for building NFStream from source. ```bash brew install autoconf automake libtool pkg-config gettext json-c ``` -------------------------------- ### Create Custom Flow Features with NFPlugin Source: https://www.nfstream.org Extend NFStream's capabilities by creating custom plugins. This example defines a plugin to check if the first packet of a UDP stream has the SYN flag set. Instantiate NFStreamer with the custom plugin to process flows with the new feature. ```python from nfstream import NFPlugin, NFStreamer class FirstPacketIsSyn(NFPlugin): def on_init(self, packet, flow): flow.udps.first_pkt_is_syn = packet.syn streamer = NFStreamer(source='facebook.pcap', udps=FirstPacketIsSyn()) for flow in extended_streamer: print(flow.udps.first_pkt_is_syn) ``` -------------------------------- ### Initialize NFStreamer Source: https://www.nfstream.org/docs/api Instantiate NFStreamer with various configuration options for network traffic analysis. Adjust parameters like source, tunnel decoding, BPF filters, and timeouts based on your needs. ```python from nfstream import NFStreamer my_streamer = NFStreamer(source="facebook.pcap", # or network interface decode_tunnels=True, bpf_filter=None, promiscuous_mode=True, snapshot_length=1536, idle_timeout=120, active_timeout=1800, accounting_mode=0, udps=None, n_dissections=20, statistical_analysis=False, splt_analysis=0, n_meters=0, max_nflows=0, performance_report=0, system_visibility_mode=0, system_visibility_poll_ms=100) ``` -------------------------------- ### NFStreamer Initialization Source: https://www.nfstream.org/docs/api Instantiate NFStreamer to capture and analyze network traffic from a specified source. ```APIDOC ## NFStreamer Initialization ### Description Instantiate NFStreamer to capture and analyze network traffic from a specified source. ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Request Example ```python from nfstream import NFStreamer my_streamer = NFStreamer(source="facebook.pcap", decode_tunnels=True, bpf_filter=None, promiscuous_mode=True, snapshot_length=1536, idle_timeout=120, active_timeout=1800, accounting_mode=0, udps=None, n_dissections=20, statistical_analysis=False, splt_analysis=0, n_meters=0, max_nflows=0, performance_report=0, system_visibility_mode=0, system_visibility_poll_ms=100) ``` ### Response #### Success Response (200) NFStreamer object #### Response Example None ``` -------------------------------- ### Implement Flow Slicer Plugin Source: https://www.nfstream.org/docs/api Create a custom NFPlugin to expire flows based on packet count. Use this to limit flow duration or manage resources. ```python from nfstream import NFStreamer, NFPlugin class FlowSlicer(NFPlugin): def on_init(self, packet, flow): if self.limit == 1: flow.expiration_id = -1 def on_update(self, packet, flow): if self.limit == flow.bidirectional_packets: flow.expiration_id = -1 # -1 value force expiration streamer = NFStreamer(source="eth0", udps=FlowSlicer(limit=7)) # Iterate, convert it to pandas or csv. ``` -------------------------------- ### Deploy ML Model in NFStreamer Source: https://www.nfstream.org/docs/api Integrate a trained machine learning model into NFStreamer for live traffic analysis. This plugin predicts flow classification at the flow expiration stage using bidirectional packet and byte counts. ```python import numpy class ModelPrediction(NFPlugin): def on_init(self, packet, flow): flow.udps.model_prediction = 0 def on_expire(self, flow): # You can do the same in on_update entrypoint and force expiration with custom id. to_predict = numpy.array([flow.bidirectional_packets, flow.bidirectional_bytes]).reshape((1,-1)) flow.udps.model_prediction = self.my_model.predict(to_predict) ml_streamer = NFStreamer(source="eth0", udps=ModelPrediction(my_model=model)) for flow in ml_streamer: print(flow.udps.model_prediction) ``` -------------------------------- ### NFStreamer Attributes Source: https://www.nfstream.org/docs/api Configuration options for NFStreamer. ```APIDOC ## NFStreamer Attributes ### Description Configuration options for NFStreamer. ### Attributes - **source** (string) - Packet capture source. Pcap file path, List of pcap files path (considered as a single file) or network interface name. - **decode_tunnels** (boolean) - Enable/Disable GTP/CAPWAP/TZSP tunnels decoding. - **bpf_filter** (string) - Specify a BPF filter filter for filtering selected traffic. - **promiscuous_mode** (boolean) - Enable/Disable promiscuous capture mode. - **snapshot_length** (integer) - Control packet slicing size (truncation) in bytes. - **idle_timeout** (integer) - Flows that are idle (no packets received) for more than this value in seconds are expired. - **active_timeout** (integer) - Flows that are active for more than this value in seconds are expired. - **accounting_mode** (integer) - Specify the accounting mode that will be used to report bytes related features (0: Link layer, 1: IP layer, 2: Transport layer, 3: Payload). - **udps** (list) - Specify user defined NFPlugins used to extend NFStreamer. - **n_dissections** (integer) - Number of per flow packets to dissect for L7 visibility feature. When set to 0, L7 visibility feature is disabled. - **statistical_analysis** (boolean) - Enable/Disable post-mortem flow statistical analysis. - **splt_analysis** (integer) - Specify the sequence of first packets length for early statistical analysis. When set to 0, splt_analysis is disabled. - **max_nflows** (integer) - Specify the number of maximum flows to capture before returning. Unset when equal to 0. - **n_meters** (integer) - Specify the number of parallel metering processes. When set to 0, NFStreamer will automatically scale metering according to available physical cores on the running host. - **performance_report** (integer) - Performance report interval in seconds. Disabled when set to 0. Ignored for offline capture. - **system_visibility_mode** (integer) - Enable system process mapping by probing the host machine. - **system_visibility_poll_ms** (integer) - Set the polling interval in milliseconds for system process mapping feature (0 is the maximum achievable rate). ``` -------------------------------- ### Reimplement SPLT Analysis Plugin Source: https://www.nfstream.org/docs/api Reimplement the SPLT analysis as an NFPlugin for demonstration. This plugin analyzes packet sequences, lengths, and timings, generating metrics for packet direction, size, and inter-packet arrival time. ```python from nfstream import NFStreamer, NFPlugin class SPLT(NFPlugin): """ Reimplementation of SPLT native analysis as NFPlugin: For demo purposes. SPLT: Sequence of packet length and time analyzer. This plugin will take 2 arguments: - sequence_length: determines the maximum sequence length (number of packets to analyze) - accounting_mode: Set how packet size will be reported (0: raw_size, 1: ip_size, 2: transport_size, 3: payload_size) Plugin will generate 3 new metrics as follows: - splt_directions: Array with direction of each packet (0: src_to_dst, 1:dst_to_src) - splt_ps: Array with packet size in bytes according to accounting_mode value. - splt_ipt: Array with inter packet arrival time in milliseconds. Note: Tail will be set with default value -1. """ @staticmethod def _get_packet_size(packet, accounting_mode): if accounting_mode == 0: return packet.raw_size elif accounting_mode == 1: return packet.ip_size elif accounting_mode == 2: return packet.transport_size else: return packet.payload_size def on_init(self, packet, flow): flow.udps.splt_direction = [-1] * self.sequence_length flow.udps.splt_direction[0] = 0 # First packet so src->dst flow.udps.splt_ps = [-1] * self.sequence_length flow.udps.splt_ps[0] = self._get_packet_size(packet, self.accounting_mode) flow.udps.splt_piat_ms = [-1] * self.sequence_length flow.udps.splt_piat_ms[0] = packet.delta_time def on_update(self, packet, flow): if flow.bidirectional_packets <= self.sequence_length: packet_index = flow.bidirectional_packets - 1 flow.udps.splt_direction[packet_index] = packet.direction flow.udps.splt_ps[packet_index] = self._get_packet_size(packet, self.accounting_mode) flow.udps.splt_piat_ms[packet_index] = packet.delta_time streamer = NFStreamer(source="facebook.pcap", udps=SPLT(sequence_length=7, accouncting_mode=1)) for flow in streamer: # Work also with to_pandas, to_csv print(flow.udps.splt_direction) ``` -------------------------------- ### Flow Iteration Source: https://www.nfstream.org/docs/api Iterate over captured network flows. ```APIDOC ## Flow Iteration ### Description Iterate over captured network flows. ### Method Iteration ### Endpoint N/A ### Parameters None ### Request Example ```python for flow in my_streamer: print(flow) # or whatever ``` ### Response #### Success Response (200) Flow objects #### Response Example None ``` -------------------------------- ### Iterate Through Flows Source: https://www.nfstream.org/docs/api Process network flows one by one using a simple for loop. This method is suitable for real-time or sequential flow analysis. ```python for flow in my_streamer: print(flow) # or whatever ``` -------------------------------- ### NFPlugin Class Prototype Source: https://www.nfstream.org/docs/api The base class for creating custom NFStream plugins. Implement methods like on_init, on_update, and on_expire to customize flow processing and expiration logic. User-defined arguments are stored as plugin attributes. ```python class NFPlugin(object): """ NFPlugin class: Main entry point to extend NFStream """ def __init__(self, **kwargs): """ NFPlugin Parameters: kwargs : user defined named arguments that will be stored as Plugin attributes """ for key, value in kwargs.items(): setattr(self, key, value) def on_init(self, packet, flow): """ on_init(self, packet, flow): Method called at flow creation. You must initiate your udps values if you plan to compute ones. Example: ------------------------------------------------------- flow.udps.magic_message = "NO" if packet.raw_size == 40: flow.udps.packet_40_count = 1 else: flow.udps.packet_40_count = 0 ---------------------------------------------------------------- """ def on_update(self, packet, flow): """ on_update(self, packet, flow): Method called to update each flow with its belonging packet. Example: ------------------------------------------------------- if packet.raw_size == 40: flow.udps.packet_40_count += 1 ---------------------------------------------------------------- """ def on_expire(self, flow): """ on_expire(self, flow): Method called at flow expiration. Example: ------------------------------------------------------- if flow.udps.packet_40_count >= 10: flow.udps.magic_message = "YES" ---------------------------------------------------------------- """ pass def cleanup(self): """ cleanup(self): Method called for plugin cleanup. Example: ------------------------------------------------------- del self.large_dict_passed_as_plugin_attribute ---------------------------------------------------------------- """ pass ``` -------------------------------- ### Convert to Pandas DataFrame Source: https://www.nfstream.org/docs/api Convert captured network flows into a Pandas DataFrame for easier data manipulation and analysis. Specify columns to anonymize if needed. ```python my_dataframe = my_streamer.to_pandas(columns_to_anonymize=[]) my_dataframe.head() ``` -------------------------------- ### Pandas DataFrame Conversion Source: https://www.nfstream.org/docs/api Convert captured flows into a Pandas DataFrame for further analysis. ```APIDOC ## Pandas DataFrame Conversion ### Description Convert captured flows into a Pandas DataFrame for further analysis. ### Method `to_pandas()` ### Endpoint N/A ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body - **columns_to_anonymize** (list) - Optional - List of columns names to anonymize. Anonymization is based on a random secret key generation at each start of NFStreamer. The generated key is used to anonymize configured values using blake2b algorithm. ### Request Example ```python my_dataframe = my_streamer.to_pandas(columns_to_anonymize=[]) my_dataframe.head() ``` ### Response #### Success Response (200) Pandas DataFrame #### Response Example None ``` -------------------------------- ### NFPlugin Class Source: https://www.nfstream.org/docs/api The NFPlugin class is the base class for creating custom extensions in NFStream. It provides methods that are called at different stages of the packet and flow lifecycle. ```APIDOC ## NFPlugin Class ### Description The NFPlugin class is the main entry point for extending NFStream. Users must inherit from this class to implement custom logic for changing expiration, creating new features, or deploying machine learning models. ### Methods #### `__init__(self, **kwargs)` - **Description**: Constructor for the NFPlugin class. User-defined named arguments are stored as plugin attributes. - **Parameters**: - `kwargs` (dict): User-defined named arguments. #### `on_init(self, packet, flow)` - **Description**: Method called at flow creation. Used to initialize custom values for the flow. - **Parameters**: - `packet` (NFPacket): The current packet being processed. - `flow` (Flow): The flow object associated with the packet. #### `on_update(self, packet, flow)` - **Description**: Method called to update each flow with its belonging packet. This method is called for every packet belonging to a flow. - **Parameters**: - `packet` (NFPacket): The current packet being processed. - `flow` (Flow): The flow object associated with the packet. #### `on_expire(self, flow)` - **Description**: Method called at flow expiration. Used to perform actions when a flow is considered expired. - **Parameters**: - `flow` (Flow): The flow object that has expired. #### `cleanup(self)` - **Description**: Method called for plugin cleanup. Used to release resources or perform final actions before the plugin is unloaded. ``` -------------------------------- ### Train Machine Learning Model for Flow Classification Source: https://www.nfstream.org/docs/api Train a RandomForestClassifier to classify social network flows based on bidirectional packets and bytes. The model is trained on a pcap file and predicts a binary outcome (SocialNetwork or not). ```python from nfstream import NFPlugin, NFStreamer from sklearn.ensemble import RandomForestClassifier df = NFStreamer(source="training_traffic.pcap").to_pandas() X = df[["bidirectional_packets", "bidirectional_bytes"]] y = df["application_category_name"].apply(lambda x: 1 if 'SocialNetwork' in x else 0) model = RandomForestClassifier() model.fit(X, y) ``` -------------------------------- ### Convert to CSV Source: https://www.nfstream.org/docs/api Converts network flow data to CSV format. Specify output path, columns to anonymize, flows per file, and file rotation settings. When path is None, NFStream uses the source path with a '.csv' extension. ```python total_flows_count = my_streamer.to_csv(path=None, columns_to_anonymize=[], flows_per_file=0, rotate_files=0) ``` -------------------------------- ### Process Network Traffic Online and Offline Source: https://www.nfstream.org Use NFStreamer to process network traffic either from a live interface (e.g., 'eth0') or from a pcap file. For offline processing, enable statistical analysis and flow splitting. The results can be converted to a Pandas DataFrame or a CSV file. ```python from nfstream import NFStreamer online_streamer = NFStreamer(source="eth0") for flow in online_streamer: print(flow) # print it. offline_streamer = NFStreamer(source="tor.pcap", statistical_analysis=True, splt_analysis=10) df = offline_streamer.to_pandas(columns_to_anonymize=()) n_flows = offline_streamer.to_csv(flows_per_file=10000, columns_to_anonymize=()) ``` -------------------------------- ### NFPacket Object Attributes Source: https://www.nfstream.org/docs/api The NFPacket object provides detailed information about a network packet after it has been matched to a flow and preprocessed. ```APIDOC ## NFPacket Object ### Description The NFPacket object contains preprocessed information about a network packet that has been associated with a flow. It is passed to the `on_init` and `on_update` methods of NFPlugin. ### Attributes - **`time`** (`int`): Packet timestamp in milliseconds. - **`delta_time`** (`int`): Delta time in milliseconds with the previous flow packet. - **`raw_size`** (`int`): Link layer packet size. - **`ip_size`** (`int`): IP packet size. - **`transport_size`** (`int`): Transport layer packet size. - **`payload_size`** (`int`): Packet payload size. - **`src_ip`** (`str`): Source IP address string representation. - **`src_mac`** (`str`): Source MAC address string representation. - **`src_oui`** (`str`): Source Organizationally Unique Identifier string representation. - **`dst_ip`** (`str`): Destination IP address string representation. - **`dst_mac`** (`str`): Destination MAC address string representation. - **`dst_oui`** (`str`): Destination Organizationally Unique Identifier string representation. - **`src_port`** (`int`): Transport layer source port. - **`dst_port`** (`int`): Transport layer destination port. - **`protocol`** (`int`): Transport layer protocol. - **`vlan_id`** (`int`): Virtual LAN identifier. - **`ip_version`** (`int`): IP version. - **`ip_packet`** (`bytes`): Raw content starting from the IP Header. - **`direction`** (`int`): Packet direction (0 for src_to_dst, 1 for dst_to_src). - **`syn`** (`bool`): TCP SYN Flag present. - **`cwr`** (`bool`): TCP CWR Flag present. - **`ece`** (`bool`): TCP ECE Flag present. - **`urg`** (`bool`): TCP URG Flag present. - **`ack`** (`bool`): TCP ACK Flag present. - **`psh`** (`bool`): TCP PSH Flag present. - **`rst`** (`bool`): TCP RST Flag present. - **`fin`** (`bool`): TCP FIN Flag present. - **`tunnel_id`** (`int`): Tunnel identifier (0: No Tunnel, 1: GTP, 2: CAPWAP, 3: TZSP). ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.