### Install NetworkML and Dependencies Source: https://github.com/faucetsdn/networkml/blob/main/notebooks/networkml_exploration.ipynb Before using NetworkML, install the library and the nest_asyncio package. nest_asyncio is a workaround for running asynchronous code in Jupyter environments. ```bash pip3 install . pip3 install nest_asyncio ``` -------------------------------- ### Install NetworkML Package Source: https://github.com/faucetsdn/networkml/blob/main/README.md Install the NetworkML package using pip after making changes to the project. This updates the installed package for testing or debugging. ```bash pip3 install . ``` -------------------------------- ### Activate NetworkML Development Environment Source: https://github.com/faucetsdn/networkml/blob/main/README.md Activate the 'posml-dev' Conda environment to start working on the project. This makes the development environment available. ```bash conda activate posml-dev ``` -------------------------------- ### Flow Featurizer Class Structure Source: https://github.com/faucetsdn/networkml/blob/main/notebooks/networkml_exploration.ipynb Illustrates the structure of a featurizer class, showing how to define functions that process rows and subclass the 'Features' class. Includes examples of 'default_tcp_5tuple' and 'default_udp_5tuple' methods. ```python from networkml.featurizers.features import Features class Flow(Features): def default_tcp_5tuple(self, rows): fields = ['ip.src_host', 'ip.dst_host', 'tcp.dstport', 'tcp.srcport', 'frame.protocols'] return self.get_columns(fields, rows) def default_udp_5tuple(self, rows): fields = ['ip.src_host', 'ip.dst_host', 'udp.dstport', 'udp.srcport', 'frame.protocols'] return self.get_columns(fields, rows) ``` -------------------------------- ### Setup for PCAP to CSV Conversion Source: https://github.com/faucetsdn/networkml/blob/main/notebooks/networkml_exploration.ipynb This Python script sets up the necessary paths and arguments for the PCAP to CSV conversion process. It specifies the input PCAP file and the output CSV file name. ```python # some initial setup import sys # let's set a path to a pcap (we'll use one included in the tests) path = '../tests/test_data/trace_ab12_2001-01-01_02_03-client-ip-1-2-3-4.pcap' # let's change the output so it's easy to find output = 'trace_ab12_2001-01-01_02_03-client-ip-1-2-3-4.pcap.csv.gz' # set arguments for arg parse sys.argv = ['pcap_to_csv.py', f'-o{output}', path] ``` -------------------------------- ### Set PYTHONPATH for NetworkML Source: https://github.com/faucetsdn/networkml/blob/main/README.md Modify your PYTHONPATH to include the networkml directory for absolute path imports. This is necessary when working with the project outside of an installed package. ```bash export PYTHONPATH=$PWD/networkml:$PYTHONPATH ``` -------------------------------- ### Set up NetworkML Development Environment with Conda Source: https://github.com/faucetsdn/networkml/blob/main/README.md Use 'make dev' to set up the development environment for NetworkML. This command handles the necessary configurations for Conda. ```bash make dev ``` -------------------------------- ### Instantiating CSVToFeatures and Reading a CSV Source: https://github.com/faucetsdn/networkml/blob/main/notebooks/networkml_exploration.ipynb Demonstrates how to instantiate the CSVToFeatures class and read the first row from a CSV file, showing the structure of the returned rows. ```python # So simply use an existing python file and class already in the `funcs` directory # or create a new one, and make sure the class subclasses `Features` # and the function signatures take in rows and return rows # the above example as a helper function `get_columns` that lets you provide a list of fields and the rows # and returns the rows with only those fields # You don't need to use the helper function, but the returns rows should be a list of dictionaries # just like the input of rows is (the same thing you get back from a CSV DictReader) # Here's a sample to test: from networkml.featurizers.csv_to_features import CSVToFeatures import csv instance = CSVToFeatures() rows = list(csv.DictReader(instance.get_reader('trace_ab12_2001-01-01_02_03-client-ip-1-2-3-4.pcap.csv.gz', True))) print(rows[0]) ``` -------------------------------- ### Train Host Footprint Model Source: https://github.com/faucetsdn/networkml/blob/main/networkml/trained_models/README.md Trains the 'host_footprint' model using specified data and saves the model, label encoder, and scaler. Optionally evaluates the model against provided test data. ```bash networkml --kfolds=5 --first_stage=algorithm --trained_model=networkml/trained_models/host_footprint.json --label_encoder=networkml/trained_models/host_footprint_le.json --scaler=networkml/trained_models/host_footprint_scaler.mod --operation train [--eval_data=/tmp/test_host.csv] /tmp/train_host.csv ``` -------------------------------- ### Run PCAP to CSV Conversion Source: https://github.com/faucetsdn/networkml/blob/main/notebooks/networkml_exploration.ipynb This code snippet applies the nest_asyncio hack for Jupyter environments and then uses the PCAPToCSV class to convert a PCAP file to a gzipped CSV file. The instance.main() call executes the conversion using the previously set arguments. ```python # hack for jupyterlab since pyshark tries to take the main run loop import nest_asyncio nest_asyncio.apply() # import the class for converting PCAPs to CSVs from networkml.parsers.pcap_to_csv import PCAPToCSV instance = PCAPToCSV() # this will parse the args we specified above in sys.argv and run using them instance.main() # the will take a minute or so to run ``` -------------------------------- ### Extracting Specific Fields from Network Rows Source: https://github.com/faucetsdn/networkml/blob/main/notebooks/networkml_exploration.ipynb Use the `example_simple` method to extract specified fields like 'eth.src' and 'eth.dst' from network packet data. This method is part of the `Flow` class, which inherits from `Features`. ```python from networkml.featurizers.features import Features class Flow(Features): def example_simple(self, rows): fields = ['eth.src', 'eth.dst'] return self.get_columns(fields, rows) flow = Flow() new_rows = flow.example_simple(rows) print(new_rows[0]) ``` -------------------------------- ### Predict from Featurizer Output using Trained Model Source: https://github.com/faucetsdn/networkml/blob/main/networkml/trained_models/README.md Performs a prediction on featurizer output (CSV) using an existing trained model. The output directory must exist and be empty. ```bash networkml --kfolds=5 --first_stage=algorithm -o /tmp/out --operation predict /tmp/combined.csv ``` -------------------------------- ### Predict from PCAP using Trained Model Source: https://github.com/faucetsdn/networkml/blob/main/networkml/trained_models/README.md Performs a prediction on a pcap file using an existing trained model. The output directory must exist and be empty. ```bash networkml --kfolds=5 --first_stage=parser -o /tmp/out--operation predict /tmp/test.pcap ``` -------------------------------- ### Inspecting Available Fields in Network Rows Source: https://github.com/faucetsdn/networkml/blob/main/notebooks/networkml_exploration.ipynb To understand which fields are available for extraction, you can call the `.keys()` method on a row object. This is useful for identifying fields to use with featurizer methods. ```python # great, but how did we know the fields? rows[0].keys() ``` -------------------------------- ### Reading Flow Featurizer Functions Source: https://github.com/faucetsdn/networkml/blob/main/notebooks/networkml_exploration.ipynb Reads and prints the content of the existing flow featurizer functions from the 'flow.py' file. ```python with open('../networkml/featurizers/funcs/flow.py', 'r') as f: for line in f: print(line) ``` -------------------------------- ### Load and Inspect CSV Data Source: https://github.com/faucetsdn/networkml/blob/main/notebooks/networkml_exploration.ipynb After conversion, this script uses CSVToFeatures to load the gzipped CSV file. It then reads the data using csv.DictReader, allowing access to each packet as a dictionary. The first row is printed to show the structure. ```python # the output is a gzipped csv, so let's quickly pop that open and see what we have # we're going to use DictReader from the CSV lib, so we'll get back a list of dictionaries, where the keys in the dicts are the fieldnames # each dictionary in the list is a record (packet) import csv from networkml.featurizers.csv_to_features import CSVToFeatures instance = CSVToFeatures() rows = list(csv.DictReader(instance.get_reader('trace_ab12_2001-01-01_02_03-client-ip-1-2-3-4.pcap.csv.gz', True))) print(rows[0]) ``` -------------------------------- ### Set CONDA_EXE Environment Variable Source: https://github.com/faucetsdn/networkml/blob/main/README.md Ensure the CONDA_EXE environment variable is set. If it's empty, export it using the provided command. ```bash export CONDA_EXE=$_CONDA_EXE ``` -------------------------------- ### Evaluate Existing Trained Model Source: https://github.com/faucetsdn/networkml/blob/main/networkml/trained_models/README.md Evaluates an existing trained model against a test dataset without retraining. Requires the test data CSV. ```bash networkml --first_stage=algorithm --final_stage=algorithm --operation eval ~/tmp/test_host.csv ``` -------------------------------- ### Convert PCAP to Featurized CSV Source: https://github.com/faucetsdn/networkml/blob/main/notebooks/networkml_exploration.ipynb This script converts a gzipped CSV file containing network packet data into a featurized CSV format using NetworkML's CSVToFeatures class. It requires specifying input and output paths, as well as the directory containing featurizer functions. ```python # we have a gzipped csv with all of the fields we could extract at the 'packet' level # (we could have supplied an arg above to do a different level, like 'flow') # we can now take that file and reduce or change or add which fields should be included using the featurizer # let's set a path to a gzipped csv (we'll use one we just made) path = 'trace_ab12_2001-01-01_02_03-client-ip-1-2-3-4.pcap.csv.gz' # let's change the output so it's easy to find output = 'trace_ab12_2001-01-01_02_03-client-ip-1-2-3-4.features.csv.gz' # we need to specify where the featurizer functions are features_path = '../networkml/featurizers/funcs' import sys # set arguments for arg parse sys.argv = ['csv_to_features.py', f'-o{output}', f'-p{features_path}', path] from networkml.featurizers.csv_to_features import CSVToFeatures instance = CSVToFeatures() instance.main() ``` -------------------------------- ### Inspect Featurized CSV Data Source: https://github.com/faucetsdn/networkml/blob/main/notebooks/networkml_exploration.ipynb This code snippet demonstrates how to read the featurized CSV output generated by NetworkML's CSVToFeatures class. It uses `csv.DictReader` to load the data into a list of dictionaries, allowing easy access to individual packet features. ```python # the output is a gzipped csv again, so let's quickly pop that open and see what we have # we're going to use DictReader from the CSV lib, so we'll get back a list of dictionaries, where the keys in the dicts are the fieldnames # each dictionary in the list is a record (packet) from networkml.featurizers.csv_to_features import CSVToFeatures import csv instance = CSVToFeatures() rows = list(csv.DictReader(instance.get_reader('trace_ab12_2001-01-01_02_03-client-ip-1-2-3-4.features.csv.gz', True))) print(rows[0]) ``` -------------------------------- ### Create New Columns and Deduplicate Records Source: https://github.com/faucetsdn/networkml/blob/main/notebooks/networkml_exploration.ipynb This snippet demonstrates how to add new fields (columns) to network traffic records and remove duplicate entries. It's useful when you need to derive new information or clean up your dataset. Ensure the necessary fields like 'layers', 'ip.src', and 'ip.dst' exist in your input rows. ```python from networkml.featurizers.features import Features class NewColumn(Features): def example_modify(self, rows): # reduce rows first as needed fields = ['layers', 'eth.src', 'eth.dst', 'ip.src', 'ip.dst'] rows = self.get_columns(fields, rows) # create new columns using existing column info last_layer = 'Last Layer' combined_ips = 'Combined IPs' # each row is a dict for row in rows: # not all rows are guaranteed to have 'layers' if 'layers' in row: # get the last element in the stringified list and clean it up row[last_layer] = row['layers'].split('<')[-1][:-2].split()[0] # not all rows are guaranteed to have 'ip.src' and ip.dst if 'ip.src' in row and 'ip.dst' in row: # combine two fields with a colon, making a new field row[combined_ips] = row['ip.src']+':'+row['ip.dst'] # remove ip.src and ip.dst now that we have them del row['ip.src'] del row['ip.dst'] # remove duplicate rows rows = [dict(t) for t in {tuple(d.items()) for d in rows}] return rows print(f'Number of original records: {len(rows)}\n') ncol = NewColumn() new_rows = ncol.example_modify(rows) print(f'Fields/Values in first record (row): {new_rows[0]}\n') print(f'Number of records after deduplication: {len(new_rows)}\n') ``` -------------------------------- ### Remove NetworkML Conda Development Environment Source: https://github.com/faucetsdn/networkml/blob/main/README.md Remove the 'posml-dev' Conda environment using the standard conda command. This frees up resources and cleans up the environment. ```bash conda env remove -y -n posml-dev ``` -------------------------------- ### Deactivate Conda Environment Source: https://github.com/faucetsdn/networkml/blob/main/README.md Deactivate the current Conda environment to exit the development environment. ```bash conda deactivate ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.