### Burp Extension Installation and Configuration Source: https://github.com/nccgroup/blackboxprotobuf/blob/master/burp/README.md Steps to install and configure the BlackBox Protobuf Burp Extension within Burp Suite, including Jython setup and dependency management. ```bash git submodule update --init ``` -------------------------------- ### Install BlackBox Protobuf Source: https://github.com/nccgroup/blackboxprotobuf/blob/master/lib/README.md Installs the BlackBox Protobuf library using either Poetry or pip. ```Shell poetry install ``` ```Shell pip install bbpb ``` -------------------------------- ### Alternative Installation of Blackbox Protobuf Addon Source: https://github.com/nccgroup/blackboxprotobuf/blob/master/mitmproxy/README.md Alternative method for installing the Blackbox Protobuf addon by installing the 'bbpb' Python package and downloading the addon script. ```bash pip install bbpb # Download bbpb.py from https://github.com/nccgroup/blackboxprotobuf/blob/master/mitmproxy/bbpb.py mitmproxy -s mitmproxy/bbpb.py ``` -------------------------------- ### Install Blackbox Protobuf Addon Source: https://github.com/nccgroup/blackboxprotobuf/blob/master/mitmproxy/README.md Instructions for installing the Blackbox Protobuf addon for mitmproxy, including cloning the repository, updating submodules, and running mitmproxy with the addon. ```bash git clone https://github.com/nccgroup/blackboxprotobuf.git cd blackboxprotobuf/ git submodule update --init mitmproxy -s mitmproxy/bbpb.py ``` -------------------------------- ### Install Blackbox Protobuf Source: https://github.com/nccgroup/blackboxprotobuf/blob/master/lib/CLI.md Installs the Blackbox Protobuf library using pip. This makes the 'bbpb' command available for use. ```bash pip install bbpb ``` -------------------------------- ### Blackbox Protobuf CLI Decoding Examples Source: https://github.com/nccgroup/blackboxprotobuf/blob/master/lib/CLI.md Provides examples of using the Blackbox Protobuf CLI for decoding protobuf messages. Includes raw decoding, saving type definitions, and decoding with a specified type definition. ```bash cat test_data | bbpb -r ``` ```bash cat test_data | bbpb -ot ./saved_type.json ``` ```bash cat test_data | bbpb -it ./saved_type.json ``` -------------------------------- ### Example Type Definition Format Source: https://github.com/nccgroup/blackboxprotobuf/blob/master/docs/TypeDefs.md Illustrates the structure of a type definition used by Blackbox Protobuf, showing field numbers mapped to their respective metadata like name and type. ```json { "1": { "name": "email", "type": "string", }, "2": { "name": "uid", "type": "int", }, "3": { "type": "string", } } ``` -------------------------------- ### Blackbox Protobuf CLI Decode, Edit, and Re-encode Example Source: https://github.com/nccgroup/blackboxprotobuf/blob/master/lib/CLI.md Illustrates a workflow for decoding a protobuf message, editing it in JSON format, and then re-encoding it back into protobuf format. ```bash cat test_data | bbpb > message.json vim message.json cat message.json | bbpb -e > test_data_out ``` -------------------------------- ### Blackbox Protobuf CLI Encoding Example Source: https://github.com/nccgroup/blackboxprotobuf/blob/master/lib/CLI.md Shows how to use the Blackbox Protobuf CLI in encoding mode. It takes a JSON message and type definition and outputs an encoded protobuf message. ```bash cat message.json | bbpb -e > test_data_out ``` -------------------------------- ### JSON Encode/Decode Functions Source: https://github.com/nccgroup/blackboxprotobuf/blob/master/lib/README.md Convenience functions for encoding and decoding protobuf messages to and from JSON. These functions handle sorting, bytestring encoding for better printing, and annotating example values onto the type definition. ```python from blackboxprotobuf import protobuf_to_json, protobuf_from_json # Example usage: # json_output = protobuf_to_json(message_dict, typedef) # message_dict_output = protobuf_from_json(json_string, typedef) ``` -------------------------------- ### Length Delimited Data Types and Encoding Source: https://github.com/nccgroup/blackboxprotobuf/blob/master/docs/TypeDefs.md Explains the length-delimited wire type, which starts with a varint length. Covers its broad applicability to messages, strings, bytes, and packed fields, including BBPB's decoding fallback strategy and supported packed types. ```protobuf Length Delimited Types: - message: Encoded protobuf message. Type definition in `message_typedef` or referenced by `message_type_name`. - bytes: Raw bytes. - bytes_hex: Bytes encoded as a hex string. - string: UTF-8 or ASCII strings. - packed_*: Efficient encoding for repeated values (e.g., packed_uint, packed_int, packed_float, packed_double). Must be set explicitly by the user. BBPB decoding fallback: message -> string -> bytes. User must explicitly set packed types. ``` -------------------------------- ### Mitmproxy Commands for Blackbox Protobuf Source: https://github.com/nccgroup/blackboxprotobuf/blob/master/mitmproxy/README.md Documentation for mitmproxy commands provided by the Blackbox Protobuf addon for editing protobuf messages and type definitions. ```APIDOC :bbpb.edit Opens a text editor with the protobuf decoded to JSON for editing. Re-encodes the edited JSON payload to protobuf. Supported flow_parts: request-body, response-body, websocket. :bbpb.edit_type Opens a text editor to edit the type definition for protobuf data, allowing field name and type changes. Supported flow_parts: request-body, response-body, websocket-request, websocket-response. :bbpb.new_type Attaches a name to the current type definition and stores it in 'known_types'. :bbpb.apply_type Applies a previously saved named type to the endpoint/protobuf payload. Use '(clear)' to remove the saved type. :bbpb.del_type Removes a named type. Endpoints associated with the named type will also be reset. :bbpb.project.save Manually saves the current Blackbox Protobuf project (type definitions and endpoint mappings) to a JSON file. :bbpb.project.load Manually loads a Blackbox Protobuf project JSON file, applying saved type definitions and endpoint mappings. ``` -------------------------------- ### Mitmproxy Addon for Blackbox Protobuf Source: https://github.com/nccgroup/blackboxprotobuf/blob/master/README.md A mitmproxy addon that enables working with Protocol Buffers without definition files within the mitmproxy environment. It facilitates analysis of network traffic. ```Python # This is a conceptual example of a mitmproxy addon. # Actual implementation would involve mitmproxy's event hooks. import blackboxprotobuf class MitmproxyAddon: def request(self, flow): # Process incoming requests pass def response(self, flow): # Process outgoing responses # Example: Decode protobuf if content type matches if 'protobuf' in flow.response.headers.get('Content-Type', ''): try: # Assuming type definitions are available or can be inferred decoded_data = blackboxprotobuf.decode(flow.response.content) # Process decoded_data as needed except Exception as e: print(f"Error decoding protobuf: {e}") pass addons = [MitmproxyAddon()] ``` -------------------------------- ### Run Blackbox Protobuf CLI Source: https://github.com/nccgroup/blackboxprotobuf/blob/master/lib/CLI.md Demonstrates how to invoke the Blackbox Protobuf CLI. It can be run directly as 'bbpb' or via the python module. ```bash bbpb ``` ```bash python3 -m blackboxprotobuf ``` -------------------------------- ### Protofile Export/Import Source: https://github.com/nccgroup/blackboxprotobuf/blob/master/lib/README.md Functions to convert between protobuf `.proto` files and the blackboxprotobuf type definition structure. Note that these functions do not support 'import' statements within `.proto` files and require manual import of dependencies. ```python from blackboxprotobuf import export_protofile, import_protofile # Example usage: # export_protofile('path/to/message.proto', 'path/to/output_typedef.json') # import_protofile('path/to/typedef.json', 'message_type_name') ``` -------------------------------- ### User Functions for Behavior Customization Source: https://github.com/nccgroup/blackboxprotobuf/blob/master/burp/README.md Details user-configurable functions in `burp/blackboxprotobuf/burp/user_funcs.py` to customize extension behavior. This includes how protobuf messages are detected, data is retrieved, and messages are identified. ```python User Functions in `burp/blackboxprotobuf/burp/user_funcs.py`: * `detect_protobuf`: - Customizes protobuf message detection. - Default: Checks content-type headers. - Customization: Can check other headers, parameters, or always return True. - Return values: - `True`: Message is protobuf. - `False`: Message is not protobuf. - `None`: Fall back to content-type check. * `get_protobuf_data`: - Customizes retrieval of protobuf data from a message. - Default: Retrieves binary data from the message body. - Customization: Get data from headers, parameters, or parse non-default encodings. - Return value: The protobuf data. * `set_protobuf_data`: - Customizes how protobuf data is stored back into a request/response after re-encoding. - Requirement: Only necessary if `get_protobuf_data` is customized. - Purpose: Should mirror the logic in `get_protobuf_data`. * `hash_message`: - Customizes how the extension identifies message types for requests/responses. - Default: Uses a combination of path and request/response status. - Customization: Use application-specific indicators like `MessageType` header or parameter. - Return value: A string value used as a key for message type identification (e.g., for persistence). ``` -------------------------------- ### Python Library for Blackbox Protobuf Source: https://github.com/nccgroup/blackboxprotobuf/blob/master/README.md A Python library that can be used in other applications for working with Protocol Buffers without definition files. It is designed for reverse engineering and forensics tooling. ```Python import blackboxprotobuf # Example usage (assuming you have encoded protobuf data and type definitions) encoded_data = b'...' # Your encoded protobuf data type_defs = { 'message_name': { 'field_name': {'type': 'int32', 'tag': 1} } } decoded_message = blackboxprotobuf.decode(encoded_data, type_defs) print(decoded_message) ``` -------------------------------- ### Jython Burp Extension for Blackbox Protobuf Source: https://github.com/nccgroup/blackboxprotobuf/blob/master/README.md A Jython Burp extension that allows decoding and modifying Protocol Buffers messages during mobile pentests. It integrates with the Burp Suite. ```Jython # This is a conceptual example of how the Burp extension might be structured. # Actual implementation would involve Burp API interactions. from burp import IBurpExtender import blackboxprotobuf class BurpExtender(IBurpExtender): def registerExtenderCallbacks(self, callbacks): self.callbacks = callbacks self.callbacks.setExtensionName("Blackbox Protobuf") self.callbacks.registerHttpListener(self) self.callbacks.registerProxyListener(self) def processHttpMessage(self, toolFlag, messageIsRequest, messageInfo): # Logic to intercept and process HTTP messages using blackboxprotobuf pass def proxyHttpMessage(self, proxyEvent): # Logic to handle proxy events and potentially decode/encode protobuf pass ``` -------------------------------- ### Output Helper Functions Source: https://github.com/nccgroup/blackboxprotobuf/blob/master/lib/README.md Helper functions for creating readable JSON output. `json_safe_transform` handles bytes encoding/decoding, `sort_output` sorts messages based on typedef field numbers, and `sort_typedef` sorts typedef fields for better readability. ```python from blackboxprotobuf.lib.output import json_safe_transform, sort_output, sort_typedef # Example usage: # readable_bytes = json_safe_transform(data, typedef) # sorted_message = sort_output(message_dict, typedef) # sorted_td = sort_typedef(typedef) ``` -------------------------------- ### Protobuf Type Editor Tab Functionality Source: https://github.com/nccgroup/blackboxprotobuf/blob/master/burp/README.md Explains the Protobuf Type Editor tab for managing message definitions globally. It covers creating, renaming, editing, and removing type definitions, as well as exporting and importing types using JSON or `.proto` files. ```APIDOC Protobuf Type Editor Tab: Functionality: - Manage message definitions globally. - Create, rename, edit, and remove type definitions. - Save/Load All Types: Export/import type definitions as JSON files for backup and sharing. - `.proto` file import/export: - Export: Saves all known type definitions into the protobuf type definition format. - Import: Reads a `.proto` file and creates a Blackbox protobuf type definition. - Limitations: Does not support 'import' statements; referenced files should be imported first. - Note: Import/export functionality is experimental and may not work for all message types. ``` -------------------------------- ### Config Object Source: https://github.com/nccgroup/blackboxprotobuf/blob/master/lib/README.md The `Config` class allows modification of encoding/decoding functionality and state management. Key attributes include `known_types`, `default_binary_type`, and `default_types` for customizing behavior with unknown fields. ```python from blackboxprotobuf.lib.config import Config # Example usage: # config = Config() # config.known_types = {'MyMessage': {...}} # config.default_binary_type = 'bytes_hex' # config.default_types[WIRETYPE_VARINT] = 'uint' ``` -------------------------------- ### Protobuf Type Reference Source: https://github.com/nccgroup/blackboxprotobuf/blob/master/burp/README.md Defines the various Protobuf data types and their characteristics, including Varint, Fixed32, Fixed64, Length Delimited, and Group types. This reference is crucial for understanding how data is encoded and decoded. ```protobuf Varint - Variable length integers (up to 8 bytes) - `uint` - unsigned, represents positive numbers efficiently, can't represent negative numbers - `int` - (default) signed, but represents negative numbers inefficiently - `sint` - Zig-zag encoding to map unsigned space to signed Fixed32 - Always 32 bits - `fixed32` - (default) unsigned integer - `sfixed32` - signed integer - `float` - floating point number Fixed64 - Always 64 bits - `fixed64` - (default) unsigned integer - `sfixed64` - signed integer - `double` - floating point number Length Delimited - Prefixed by length representing varint - `bytes` - (default) Plain data, used for strings as well - `message` - (detected) Protobuf message. Can contain a nested type definition ('`message_typedef`') or labeled type name ('`message_type_name`') - `string` - Similar to bytes, but will return a string python type - `bytes_hex` - Output binary data as a string of hex characters rather than an escaped string - `packed_*` - Repeated fields of the same type packed into a buffer. Can be combined with any Varint, or fixed wiretype (eg. `packed_fixed32`) Group (Start/End) - `group` - Deprecated way to group fields. Replaced with nested Protobuf Messages. Not supported ``` -------------------------------- ### Varint Data Types and Encoding Source: https://github.com/nccgroup/blackboxprotobuf/blob/master/docs/TypeDefs.md Explains how varint wire types map to different blackboxprotobuf data types (int, uint, sint) and details their encoding characteristics, including two's complement for negative numbers and Zigzag encoding for sint. ```protobuf Varint Types: - int (default): Represents positive and negative values using two's complement. Inefficient for frequent negative values. Handles most unsigned integers correctly. - uint: Unsigned integers, supports larger values than int. Use when negative numbers are incorrectly decoded. - sint: Signed integers using Zigzag encoding. Efficient for small negative numbers. Use if decoded numbers are off by ~2x or appear positive when negative. Boolean values are encoded as varints: 0 for False, 1 for True. ``` -------------------------------- ### Protobuf Type Definition Editing Source: https://github.com/nccgroup/blackboxprotobuf/blob/master/burp/README.md Details how to modify protobuf type definitions to change decoding behavior, assign names to fields, and improve message readability. Includes notes on field number and type constraints. -------------------------------- ### Protobuf Wire Types Explained Source: https://github.com/nccgroup/blackboxprotobuf/blob/master/docs/TypeDefs.md Details the different wire types used in protobuf encoding, which dictate how field lengths are determined. This is crucial for backward compatibility and parsing unknown fields. ```APIDOC Protobuf Wire Types: - Varint: Variable length integer representation. Uses one bit per byte to signal the last byte. - Fixed 64 bit: Represents 64-bit numbers, which can be integers or floating-point numbers. - Fixed 32 bit: Represents 32-bit numbers, which can be integers or floating-point numbers. - Length Delimited: Field data is prefixed with a varint indicating the data length. - Start/End Group: Used to group fields. Deprecated in favor of sub-messages and not supported by BBPB. ``` -------------------------------- ### Encode and Decode Protobuf Message Source: https://github.com/nccgroup/blackboxprotobuf/blob/master/lib/README.md Demonstrates the process of decoding a protobuf message, modifying a field, and then re-encoding it. This showcases the library's capability to manipulate protobuf data without the original schema. ```Python import blackboxprotobuf import base64 data = base64.b64decode('KglNb2RpZnkgTWU=') message,typedef = blackboxprotobuf.decode_message(data) message[5] = 'Modified Me' data = blackboxprotobuf.encode_message(message,typedef) print(data) ``` -------------------------------- ### Payload Encoding Handling Source: https://github.com/nccgroup/blackboxprotobuf/blob/master/lib/CLI.md Explains how Blackbox Protobuf detects and applies payload encodings such as gzip and gRPC during decoding and encoding. The detected encoding is stored in the `payload_encoding` field. ```python class BlackboxProtobuf: def decode(self, payload): # ... decoding logic ... if payload_encoding_detected: self.payload_encoding = payload_encoding_detected # ... rest of decoding ... def encode(self, data): # ... encoding logic ... if self.payload_encoding == 'gzip': # apply gzip encoding pass elif self.payload_encoding == 'grpc': # apply gRPC encoding pass else: # apply 'none' encoding (plain protobuf) pass # ... rest of encoding ... ``` -------------------------------- ### Variable Length Integers (varint) Encoding Source: https://github.com/nccgroup/blackboxprotobuf/blob/master/lib/README.md Explains the varint wire type for encoding integers, booleans, and enums. Covers uint (unsigned), int (standard, inefficient for negatives), and sint (ZigZag encoding for efficient negative number representation). The default is 'int'. ```python ## Varint Encoding Types: # uint: Varint encoding with no representation of negative numbers. # int: Standard encoding but inefficient for negative numbers (always 10 bytes). # sint: Uses ZigZag encoding to efficiently represent negative numbers. # Default type is 'int' with no ZigZag encoding. ``` -------------------------------- ### Fixed32/64 Wire Types Source: https://github.com/nccgroup/blackboxprotobuf/blob/master/lib/README.md Describes fixed length wire types which have an implicit size. These support fixed size integers (signed/unsigned) or floating point numbers (float/double). The default type is floating point. ```python ## Fixed32/64: # Supports fixed size integers (signed/unsigned) or floating point numbers (float/double). # Default type is floating point. ``` -------------------------------- ### gRPC ALPN Handling in mitmproxy Source: https://github.com/nccgroup/blackboxprotobuf/blob/master/mitmproxy/README.md This snippet describes a workaround for mitmproxy's gRPC interception issues. It involves modifying the proxy to handle the 'grpc-exp' ALPN value in addition to 'h2'. This is relevant for users experiencing problems intercepting gRPC traffic from certain applications. ```python def load(l): # Example modification (conceptual): # if flow.request.headers.get('alt-svc') == 'h2': # flow.request.headers['alt-svc'] = 'grpc-exp' pass ``` -------------------------------- ### Type Definition Structure Source: https://github.com/nccgroup/blackboxprotobuf/blob/master/lib/README.md Describes the structure of a type definition object, which is a Python dictionary representing the type structure of a message. It details the minimum 'type' entry and optional 'message_typedef' or 'message_type_name' for nested messages. ```python { 'field_number': { 'type': 'string_identifier', 'message_typedef': { ... } # For nested messages 'message_type_name': 'message_type_identifier' # For previously known messages } } ``` -------------------------------- ### Fixed 64 Data Types and Encoding Source: https://github.com/nccgroup/blackboxprotobuf/blob/master/docs/TypeDefs.md Details the fixed 64 wire type and its mappings to unsigned 64-bit integers (fixed64), signed 64-bit integers (sfixed64), and 64-bit floating-point numbers (double). ```protobuf Fixed 64 Types: - fixed64 (default): Unsigned 64-bit integer. - sfixed64: Signed 64-bit integer. - double: 64-bit floating-point number. Default value is an integer. Consider 'double' as a potential default with further research. ``` -------------------------------- ### Changing Default Wiretype Source: https://github.com/nccgroup/blackboxprotobuf/blob/master/docs/TypeDefs.md Demonstrates how to change the default type mapping for a specific wire type in blackboxprotobuf. This is useful for customizing how certain data types are interpreted during decoding. ```python config.default_types[wiretypes.FIXED64] = 'double' ``` -------------------------------- ### Overriding Payload Encoding Source: https://github.com/nccgroup/blackboxprotobuf/blob/master/lib/CLI.md Demonstrates how to manually set or override the payload encoding during decoding or encoding using command-line arguments. ```bash # Decode with gzip payload encoding blackboxprotobuf decode --payload-encoding gzip # Encode with gRPC payload encoding blackboxprotobuf encode --payload-encoding grpc ``` -------------------------------- ### Protobuf Message Editing in Burp Source: https://github.com/nccgroup/blackboxprotobuf/blob/master/burp/README.md Explains how to edit protobuf messages within Burp Suite after they have been parsed into JSON. Covers modifying values, selecting type definitions, validating changes, and resetting messages. -------------------------------- ### Decode Protobuf Message to JSON Source: https://github.com/nccgroup/blackboxprotobuf/blob/master/lib/README.md Decodes a protobuf bytestring and converts it to a JSON-like dictionary. It also returns the generated type definition. This function is useful for inspecting and understanding protobuf messages when the schema is unknown. ```Python import blackboxprotobuf import base64 data = base64.b64decode('KglNb2RpZnkgTWU=') message,typedef = blackboxprotobuf.protobuf_to_json(data) print(message) ``` -------------------------------- ### Fixed 32 Data Types and Encoding Source: https://github.com/nccgroup/blackboxprotobuf/blob/master/docs/TypeDefs.md Describes the fixed 32 wire type and its supported data types: unsigned 32-bit integers (fixed32), signed 32-bit integers (sfixed32), and 32-bit floating-point numbers (float). ```protobuf Fixed 32 Types: - fixed32 (default): Unsigned 32-bit integer. - sfixed32: Signed 32-bit integer. - float: 32-bit floating-point number. Default value is an integer. Consider 'float' as a potential default with further research. ``` -------------------------------- ### Adding a New Field to a Typedef Source: https://github.com/nccgroup/blackboxprotobuf/blob/master/docs/TypeDefs.md Illustrates how to add a new field to an existing protobuf type definition. This involves adding a new key (field number) to the typedef dictionary with the field's name and type. ```python { "1": { ... (existing field number) ... }, "5": { "name": "uid", "type": "int" } } ``` ```python typedef["5"] = { "name": "uid", "type": "int" } ``` -------------------------------- ### Length Delimited Wire Types Source: https://github.com/nccgroup/blackboxprotobuf/blob/master/lib/README.md Details length-delimited wire types, prefixed with a varint indicating length. Used for strings, bytestrings, inner messages, and packed repeated fields. Messages can be identified by validating protobuf binary. Default type is string/byte, configurable via `blackboxprotobuf.lib.types.default_binary_type`. Packed repeated fields require parser knowledge of element types. ```python ## Length Delimited: # Prefixed with a varint indicating length. # Used for strings, bytestrings, inner messages, and packed repeated fields. # Default binary type can be changed with `blackboxprotobuf.lib.types.default_binary_type`. # Packed repeated fields require explicit packing declaration in protobuf v2, and are packed by default in v3. ``` -------------------------------- ### Validate Typedef Function Source: https://github.com/nccgroup/blackboxprotobuf/blob/master/lib/README.md A function to perform sanity checks on modified type definitions, ensuring internal consistency and consistency with previous definitions. It helps detect issues like incompatible type changes or duplicate field names. ```python from blackboxprotobuf import validate_typedef # Example usage: # is_valid = validate_typedef(new_typedef, old_typedef=None) ``` -------------------------------- ### Modifying Field Types in Protobuf Type Definitions Source: https://github.com/nccgroup/blackboxprotobuf/blob/master/docs/TypeDefs.md Explains how to correct individual field types within a protobuf type definition. Emphasizes that the modified type must retain the original wiretype for valid re-encoding. ```APIDOC Modifying Field Types: To correct the type of an individual field, modify the 'type' field within the typedef. Use the modified typedef to re-decode the protobuf data. Avoid using the modified typedef directly with the encoder to prevent inconsistent data. Constraints: The modified type must share the same wiretype as the original type in the typedef. For example, an 'int' can be changed to 'sint' (for zigzag encoding), but not to 'float' or 'double'. ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.