### Include pbf_writer Header Source: https://github.com/mapbox/protozero/blob/master/doc/tutorial.md Include the `protozero/pbf_writer.hpp` header to start writing protobuf-encoded messages. The `pbf_writer` class includes asserts to help detect programming errors during development. ```cpp #include ``` -------------------------------- ### Conditional Writer Tests Setup with Protobuf Source: https://github.com/mapbox/protozero/blob/master/test/CMakeLists.txt Enables writer tests only if Protocol Buffers (protobuf) libraries are found. It includes necessary directories and prepares for generating C++ code from .proto files. ```cmake if(PROTOBUF_FOUND) message(STATUS "Found protobuf libraries: Adding writer tests...") include_directories(SYSTEM ${PROTOBUF_INCLUDE_DIRS} ${CMAKE_CURRENT_BINARY_DIR}) set(PROTOBUF_GENERATE_CPP_APPEND_PATH false) foreach(_dir IN LISTS TEST_DIRS) set(_full_src_dir "${CMAKE_CURRENT_SOURCE_DIR}/t/${_dir}") if(EXISTS "${_full_src_dir}/writer_test_cases.cpp") message(STATUS " Adding ${_dir}") set(_full_bin_dir "${CMAKE_CURRENT_BINARY_DIR}/t/${_dir}") set(_proto_file "${_full_src_dir}/${_dir}_testcase.proto") set(_src_file "${_full_bin_dir}/${_dir}_testcase.pb.cc") set(_hdr_file "${_full_bin_dir}/${_dir}_testcase.pb.h") file(MAKE_DIRECTORY ${_full_bin_dir}) list(APPEND SOURCES "${_full_src_dir}/writer_test_cases.cpp") list(APPEND PROTO_FILES "${_proto_file}") list(APPEND PROTO_SRCS "${_src_file}") list(APPEND PROTO_HDRS "${_hdr_file}") set_source_files_properties(${_proto_file} ${_hdr_file} PROPERTIES GENERATED TRUE) add_custom_command( OUTPUT ${_src_file} ${_hdr_file} COMMAND ${PROTOBUF_PROTOC_EXECUTABLE} ARGS --cpp_out=${_full_bin_dir} -I ${_full_src_dir} ${_proto_file} DEPENDS ${_proto_file} VERBATIM) endif() endforeach() add_executable(writer_tests writer_tests.cpp ${SOURCES} ${PROTO_SRCS} ${PROTO_HDRS}) target_compile_features(writer_tests PUBLIC cxx_std_14) target_link_libraries(writer_tests PRIVATE protobuf::libprotobuf-lite) if(NOT MSVC) set_target_properties(writer_tests PROPERTIES COMPILE_FLAGS "-pthread") if(NOT APPLE) set_target_properties(writer_tests PROPERTIES LINK_FLAGS "-pthread") endif() endif() add_test(NAME writer_tests COMMAND writer_tests) else() message(STATUS "Protobuf libraries not found: Disabling writer tests.") endif() ``` -------------------------------- ### Use data_view for GetView() Method Source: https://github.com/mapbox/protozero/blob/master/UPGRADING.md Starting from v1.4.0, use the `get_view()` method on `pbf_reader` which returns a `protozero::data_view`. This provides an interface compatible with `std::string_view`, allowing access to data via `data()` and `size()`. ```cpp auto x = message.get_packed_int32(); for (auto it = x.first; it != x.second; ++it) { .... } ``` -------------------------------- ### Compile and Run pbf-fuzzer with AFL++ Source: https://github.com/mapbox/protozero/blob/master/FUZZING.md Compile the pbf-fuzzer using AFL++'s fast compiler wrapper and then run AFL++ to fuzz the pbf-fuzzer tool. This offers an alternative to the standard AFL setup. ```bash afl-clang-fast++ -O2 -std=c++17 -g -DNDEBUG -Iinclude -fsanitize=address,fuzzer tools/pbf-fuzzer.cpp -o tools/pbf-fuzzer afl-fuzz -i testcase_dir -o findings_dir -- tools/pbf-fuzzer ``` -------------------------------- ### Concatenate Strings with Extra Copy Source: https://github.com/mapbox/protozero/blob/master/doc/advanced.md This example demonstrates the less efficient way of concatenating strings before adding them to a message, which involves an expensive extra copy. ```cpp std::string a{"very long string..."}; std::string b{"another very long string..."}; std::string data; protozero::pbf_writer writer{data}; a.append(b); // expensive extra copy writer.add_string(1, a); ``` -------------------------------- ### Regenerate PBF Test Data Source: https://github.com/mapbox/protozero/blob/master/test/README.md Run this script from the 'test' directory to regenerate PBF data files used in tests. Ensure you have the necessary dependencies installed. ```shell cd test ./create_pbf_test_data.sh ``` -------------------------------- ### Get String or Bytes View Source: https://github.com/mapbox/protozero/blob/master/doc/tutorial.md Use `get_view()` to obtain a `data_view` for string and bytes types, avoiding memory allocation. Access the data pointer with `data()` and the length with `size()`. ```cpp // Example usage of get_view() would go here, demonstrating data() and size() // For instance: // auto view = message.get_view(); // std::cout << "Data: " << view.data() << ", Size: " << view.size() << std::endl; ``` -------------------------------- ### Iterate Using begin() and end() for Packed Fields Source: https://github.com/mapbox/protozero/blob/master/UPGRADING.md When upgrading from v1.3.0 to v1.4.0, change attribute accesses from `first` and `second` to `begin()` and `end()` for `iterator_range` objects returned by `get_packed_*()` functions. This example demonstrates the explicit iterator usage. ```cpp auto x = message.get_packed_int32(); for (auto it = x.begin(); it != x.end(); ++it) { .... } ``` -------------------------------- ### Read Specific Field with pbf_reader Source: https://github.com/mapbox/protozero/blob/master/doc/tutorial.md Efficiently extract a specific field by passing its tag to the `next()` function. This skips all other fields in the message. The example shows how to retrieve all instances of a `fixed64` field with tag 17. ```cpp // same .proto file and initialization as above // get all fields with tag 17, skip all others while (message.next(17)) { auto r = message.get_fixed64(); std::cout << r << "\n"; } ``` -------------------------------- ### Use Range-Based For Loop for Packed Fields Source: https://github.com/mapbox/protozero/blob/master/UPGRADING.md When upgrading from v1.3.0 to v1.4.0, change attribute accesses from `first` and `second` to `begin()` and `end()` for `iterator_range` objects returned by `get_packed_*()` functions. This example shows the preferred range-based for loop syntax. ```cpp auto x = message.get_packed_int32(); for (auto val : x) { .... } ``` -------------------------------- ### Using Fixed Size Buffer Adaptor Source: https://github.com/mapbox/protozero/blob/master/doc/advanced.md Utilize `protozero::fixed_size_buffer_adaptor` with `basic_pbf_writer` for fixed-size buffers. Include `protozero/buffer_fixed.hpp` for this functionality. ```cpp #include your_buffer_class some_buffer; protozero::fixed_size_buffer_adaptor buffer_adaptor{some_buffer.data(), some_buffer.size()}; basic_pbf_writer writer{buffer_adaptor}; ``` ```cpp protozero::fixed_size_buffer_adaptor buffer_adaptor{some_buffer}; ``` -------------------------------- ### Include Directories for Protozero Tests Source: https://github.com/mapbox/protozero/blob/master/test/CMakeLists.txt Includes directories for Catch testing framework and Protozero headers. Use this to make test-related headers available. ```cmake include_directories(SYSTEM "${CMAKE_CURRENT_SOURCE_DIR}/catch") include_directories("${CMAKE_CURRENT_SOURCE_DIR}/include") ``` -------------------------------- ### Protozero Aliases for Basic Writer/Builder Source: https://github.com/mapbox/protozero/blob/master/doc/advanced.md Protozero provides convenient aliases for `basic_pbf_writer` and `basic_pbf_builder` that default to using `std::string` as the buffer. ```cpp using pbf_writer = basic_pbf_writer; template using pbf_builder = basic_pbf_builder; ``` -------------------------------- ### Include Protozero Header Source: https://github.com/mapbox/protozero/blob/master/doc/tutorial.md Include the main Protozero header file in your C++ program. Ensure the `protozero` directory is accessible by your build system. ```cpp #include ``` -------------------------------- ### Get Size of Packed Repeated Field Source: https://github.com/mapbox/protozero/blob/master/doc/advanced.md Use `range.size()` to determine the number of items in a packed repeated field, useful for pre-allocating memory. The performance of `size()` varies by field type. ```cpp protozero::pbf_reader message{...}; message.next(...); const auto range = message.get_packed_sint32(); std::vector myvalues; myvalues.reserve(range.size()); for (auto value : range) { myvalues.push_back(value); } ``` -------------------------------- ### Define PROTOZERO_USE_VIEW to use std::string_view Source: https://github.com/mapbox/protozero/blob/master/doc/advanced.md Set the PROTOZERO_USE_VIEW macro before including Protozero headers to use std::string_view instead of Protozero's internal data_view implementation. ```cpp #define PROTOZERO_USE_VIEW std::string_view ``` -------------------------------- ### Node.js Require for Protozero Headers Source: https://github.com/mapbox/protozero/blob/master/UPGRADING.md In Node.js environments, `require('protozero')` can be used to print the include paths for Protozero headers, simplifying module resolution. ```javascript require('protozero') ``` -------------------------------- ### Run Fuzzing with AFL and pbf-decoder Source: https://github.com/mapbox/protozero/blob/master/FUZZING.md Execute the AFL fuzzer using the prepared test cases and the pbf-decoder tool. Findings will be saved in the specified output directory. ```bash afl-fuzz -i testcase_dir -o findings_dir -- tools/pbf-decoder - ``` -------------------------------- ### Configure Documentation Generation Source: https://github.com/mapbox/protozero/blob/master/doc/CMakeLists.txt This snippet configures Doxygen to generate API documentation. It finds Doxygen and the DOT executable, configures the Doxyfile, and sets up a custom command to generate HTML documentation. ```cmake message(STATUS "Configuring documentation") message(STATUS "Looking for doxygen") find_package(Doxygen) if(DOXYGEN_FOUND AND DOXYGEN_DOT_FOUND) message(STATUS "Looking for doxygen - found") configure_file(Doxyfile.in ${CMAKE_CURRENT_BINARY_DIR}/Doxyfile @ONLY) file(GLOB HEADER_FILES "${CMAKE_SOURCE_DIR}/include/protozero/*.hpp") add_custom_command(OUTPUT html/index.html COMMAND ${DOXYGEN_EXECUTABLE} ARGS ${CMAKE_CURRENT_BINARY_DIR}/Doxyfile DEPENDS Doxyfile.in advanced.md cheatsheet.md tutorial.md ${HEADER_FILES} WORKING_DIRECTORY ${CMAKE_SOURCE_DIR} COMMENT "Generating API documentation with Doxygen" VERBATIM) add_custom_target(doc ALL DEPENDS html/index.html) else() message(STATUS "Looking for doxygen - not found") message(STATUS " Disabled making of documentation.") endif() message(STATUS "Configuring documentation - done") ``` -------------------------------- ### Handling Protozero Parsing Exceptions Source: https://context7.com/mapbox/protozero/llms.txt Illustrates how to catch specific exceptions derived from `protozero::exception` during Protocol Buffer parsing. This includes handling truncated messages, corrupted varints, unknown wire types, invalid tags, and invalid packed field lengths. ```cpp #include #include #include void safe_parse(const std::string& data) { try { protozero::pbf_reader msg{data}; while (msg.next()) { // ... process fields msg.skip(); } } catch (const protozero::end_of_buffer_exception&) { // Buffer too short / truncated message std::cerr << "Truncated protobuf message\n"; } catch (const protozero::varint_too_long_exception&) { // Varint exceeds 10 bytes — corrupted data std::cerr << "Corrupted varint encoding\n"; } catch (const protozero::unknown_pbf_wire_type_exception&) { // Unrecognised wire type — corrupted or unsupported data std::cerr << "Unknown wire type\n"; } catch (const protozero::invalid_tag_exception&) { // Tag value 0 or in reserved range 19000–19999 std::cerr << "Invalid field tag\n"; } catch (const protozero::invalid_length_exception&) { // Packed field length not a multiple of the element size std::cerr << "Invalid packed field length\n"; } catch (const protozero::exception& e) { // Catch-all for any other protozero error std::cerr << "Protozero error: " << e.what() << "\n"; } } ``` -------------------------------- ### Include Protozero Varint Header Source: https://github.com/mapbox/protozero/blob/master/doc/advanced.md Include this header to access low-level varint and zigzag encoding/decoding functions. ```cpp #include ``` -------------------------------- ### Tag New Release Source: https://github.com/mapbox/protozero/blob/master/CONTRIBUTING.md Create a Git tag for the new release version. ```git git tag vX.Y.Z ``` -------------------------------- ### Compile-Time Version Checking Source: https://context7.com/mapbox/protozero/llms.txt Demonstrates using `PROTOZERO_VERSION_MAJOR`, `PROTOZERO_VERSION_MINOR`, `PROTOZERO_VERSION_PATCH`, `PROTOZERO_VERSION_CODE`, and `PROTOZERO_VERSION_STRING` macros for conditional compilation based on the Protozero library version. This is useful for ensuring compatibility or enabling features available only in newer versions. ```cpp #include #include void print_version() { // PROTOZERO_VERSION_MAJOR, _MINOR, _PATCH: individual integer components // PROTOZERO_VERSION_CODE: (major * 10000 + minor * 100 + patch) // PROTOZERO_VERSION_STRING: "1.8.1" #if PROTOZERO_VERSION_CODE >= 10800 std::cout << "Using protozero " << PROTOZERO_VERSION_STRING << " (code=" << PROTOZERO_VERSION_CODE << ")\n"; // Output: Using protozero 1.8.1 (code=10801) #else #error "Protozero >= 1.8.0 required" #endif } ``` -------------------------------- ### Define Protocol Buffer Message Source: https://github.com/mapbox/protozero/blob/master/doc/tutorial.md This is a sample Protocol Buffer message definition in a .proto file. ```protobuf message Example { required uint32 x = 1; optional string s = 2; repeated fixed64 r = 17; } ``` -------------------------------- ### Compile and Run Dedicated pbf-fuzzer Source: https://github.com/mapbox/protozero/blob/master/FUZZING.md Compile the dedicated pbf-fuzzer tool with AddressSanitizer and fuzzer enabled for increased speed. Then, run the compiled fuzzer directly. ```bash clang++ -O2 -std=c++17 -g -DNDEBUG -Iinclude -fsanitize=address,fuzzer tools/pbf-fuzzer.cpp -o tools/pbf-fuzzer ./tools/pbf-fuzzer ``` -------------------------------- ### Using Custom Buffer with Basic PBF Writer Source: https://github.com/mapbox/protozero/blob/master/doc/advanced.md Instantiate `basic_pbf_writer` or `basic_pbf_builder` with a custom buffer type by providing it as a template parameter and passing a reference to the buffer object to the constructor. ```cpp some_buffer_class buffer; basic_pbf_writer writer{buffer}; ``` -------------------------------- ### Writing to a Fixed-Size Buffer Source: https://context7.com/mapbox/protozero/llms.txt Shows how to use `basic_pbf_writer` with a `fixed_size_buffer_adaptor` to write Protocol Buffer data into a pre-allocated `std::array` without heap allocations. This is useful for embedded systems or performance-critical scenarios. ```cpp #include #include #include #include void write_to_fixed_buffer() { std::array raw_buf{}; protozero::fixed_size_buffer_adaptor adaptor{raw_buf.data(), raw_buf.size()}; protozero::basic_pbf_writer writer{adaptor}; writer.add_uint32(1, 42); writer.add_string(2, "hello"); // data is now in raw_buf[0..N]; no heap allocation occurred // Access written bytes // std::size_t written = adaptor.size(); } ``` -------------------------------- ### Parse Protobuf Message with pbf_reader Source: https://github.com/mapbox/protozero/blob/master/doc/tutorial.md Initialize `pbf_reader` with input data and iterate through message fields using `next()`. Use `get_uint32()`, `get_string()`, or `skip()` based on the field tag. Always call `next()` before accessing or skipping a field, and never call `next()` twice in a row. ```cpp #include // get data from somewhere into the input string std::string input = get_input_data(); // initialize pbf message with this data protozero::pbf_reader message{input}; // iterate over fields in the message while (message.next()) { // switch depending on the field tag (the field name is not available) switch (message.tag()) { case 1: // get data for tag 1 (in this case an uint32) auto x = message.get_uint32(); break; case 2: // get data for tag 2 (in this case a string) std::string s = message.get_string(); break; case 17: // ignore data for tag 17 message.skip(); break; default: // ignore data for unknown tags to allow for future extensions message.skip(); } } ``` -------------------------------- ### Compile with AFL Compiler Wrappers Source: https://github.com/mapbox/protozero/blob/master/FUZZING.md Compile the project using AFL's clang wrappers for fuzz testing. Ensure you are in a build directory and have CMake configured. ```bash mkdir build cd build CC=afl-clang CXX=afl-clang++ cmake .. mkdir testcase_dir ``` -------------------------------- ### Create Reader Tests Executable Source: https://github.com/mapbox/protozero/blob/master/test/CMakeLists.txt Builds the 'reader_tests' executable using the main source file and the generated test case source files. This executable contains tests for reading data with Protozero. ```cmake add_executable(reader_tests reader_tests.cpp ${_test_sources}) ``` -------------------------------- ### Write Nested Sub-Messages with Protozero Source: https://context7.com/mapbox/protozero/llms.txt Shows how to write nested sub-messages using protozero. A child pbf_writer is constructed from the parent writer and a tag number, writing into the same buffer. The length prefix is finalized when the child goes out of scope. ```cpp #include #include // Proto: // message Point { required double x = 1; required double y = 2; } // message Polyline { repeated Point points = 10; } std::string build_polyline(const std::vector>& pts) { std::string buffer; protozero.pbf_writer parent{buffer}; for (auto& [x, y] : pts) { protozero.pbf_writer child{parent, 10}; // opens sub-message at tag 10 child.add_double(1, x); child.add_double(2, y); // child destructor finalises the length prefix } return buffer; } ``` -------------------------------- ### Write Packed Repeated Fields with Protozero Source: https://context7.com/mapbox/protozero/llms.txt Demonstrates writing packed repeated fields using protozero. Supports writing from an iterator range, incrementally with a scope object, and an optimized method for known element counts. ```cpp #include #include #include std::string build_packed_demo() { std::string buffer; protozero.pbf_writer pw{buffer}; // Method 1: from iterator range std::vector scores = {10, 20, 30, 40}; pw.add_packed_int32(1, scores.begin(), scores.end()); // Method 2: incremental with rollback support { protozero.packed_field_double field{pw, 2}; field.add_element(3.14); field.add_element(2.71); // field committed on scope exit; call field.rollback() to cancel } // Method 3: known element count optimization (fixed-size types only) { protozero.packed_field_fixed32 fixed_field{pw, 3, 2}; // exactly 2 elements fixed_field.add_element(100U); fixed_field.add_element(200U); } return buffer; } ``` -------------------------------- ### Define Test Directories for Protozero Source: https://github.com/mapbox/protozero/blob/master/test/CMakeLists.txt Sets a list of directories containing test cases for Protozero. These are used to generate source files for tests. ```cmake set(TEST_DIRS alignment bool bytes complex double enum fixed32 fixed64 float int32 int64 message nested repeated repeated_packed_bool repeated_packed_double repeated_packed_enum repeated_packed_fixed32 repeated_packed_fixed64 repeated_packed_float repeated_packed_int32 repeated_packed_int64 repeated_packed_sfixed32 repeated_packed_sfixed64 repeated_packed_sint32 repeated_packed_sint64 repeated_packed_uint32 repeated_packed_uint64 rollback sfixed32 sfixed64 sint32 sint64 skip string tag_and_type tags uint32 uint64 vector_tile wrong_type_access) ``` -------------------------------- ### Handle Mixed Packed/Unpacked Fields with `tag_and_type` Source: https://context7.com/mapbox/protozero/llms.txt Explains how to use `tag_and_type()` with protozero's `pbf_reader` or `pbf_message` to correctly handle repeated fields that can be encoded as either packed or unpacked within the same message. This is crucial for robust decoding. ```cpp #include #include #include enum class MixedMsg : protozero.pbf_tag_type { repeated_uint32_values = 1 }; std::vector decode_mixed(const std::string& data) { protozero.pbf_message msg{data}; std::vector result; while (msg.next()) { switch (msg.tag_and_type()) { case protozero.tag_and_type(MixedMsg::repeated_uint32_values, protozero.pbf_wire_type::length_delimited): { // packed encoding for (auto v : msg.get_packed_uint32()) { result.push_back(v); } break; } case protozero.tag_and_type(MixedMsg::repeated_uint32_values, protozero.pbf_wire_type::varint): { // non-packed encoding result.push_back(msg.get_uint32()); break; } default: msg.skip(); } } return result; } ``` -------------------------------- ### Prepare Fuzzing Test Data Source: https://github.com/mapbox/protozero/blob/master/FUZZING.md Collect all non-empty PBF test messages from the unit tests into a directory for fuzzing input. This step ensures a diverse set of data for the fuzzer. ```bash find ../test/t/ -name data-\*\.pbf -a -not -empty -exec cp {} testcase_dir/ \; ``` -------------------------------- ### Low-Level Varint and ZigZag Encoding Source: https://context7.com/mapbox/protozero/llms.txt Demonstrates ZigZag encoding/decoding for 32-bit and 64-bit integers, decoding varints from a raw buffer, and calculating the encoded length of a varint. Ensure input values are within the expected range for correct encoding and decoding. ```cpp #include #include #include #include void varint_examples() { // ZigZag encoding (maps signed to unsigned for efficient varint storage) int32_t original = -300; uint32_t encoded32 = protozero::encode_zigzag32(original); int32_t decoded32 = protozero::decode_zigzag32(encoded32); assert(decoded32 == original); // -300 → 599 → -300 // ZigZag 64-bit int64_t val64 = -1000000LL; uint64_t enc64 = protozero::encode_zigzag64(val64); assert(protozero::decode_zigzag64(enc64) == val64); // Decode varint from raw buffer std::string buf; buf.push_back('\xac'); buf.push_back('\x02'); // varint for 300 const char* ptr = buf.data(); uint64_t v = protozero::decode_varint(&ptr, buf.data() + buf.size()); assert(v == 300); // Query encoded length without writing int len = protozero::length_of_varint(300); // returns 2 assert(len == 2); } ``` -------------------------------- ### Check specific tag and type with pbf_reader::next() Source: https://github.com/mapbox/protozero/blob/master/doc/advanced.md Efficiently iterate through a message by checking for a specific field tag and wire type using the two-argument version of pbf_reader::next(). ```cpp std::string data = ... pbf_reader message{data}; while (message.next(17, pbf_wire_type::varint)) { auto foo = message.get_int32(); ... } ``` -------------------------------- ### Push Tags Source: https://github.com/mapbox/protozero/blob/master/CONTRIBUTING.md Push all tags, including the newly created release tag, to the remote repository. ```git git push --tags ``` -------------------------------- ### Write Messages with `pbf_builder` (Type-Safe) Source: https://context7.com/mapbox/protozero/llms.txt Utilize the type-safe `pbf_builder` for writing messages, which uses enum values for tags instead of raw integers. This enhances code readability and reduces errors. ```cpp #include #include enum class Location : protozero::pbf_tag_type { required_double_lat = 1, required_double_lon = 2, optional_string_label = 3, }; std::string build_location(double lat, double lon, const std::string& label) { std::string buffer; protozero::pbf_builder builder{buffer}; builder.add_double(Location::required_double_lat, lat); builder.add_double(Location::required_double_lon, lon); if (!label.empty()) { builder.add_string(Location::optional_string_label, label); } return buffer; } ``` -------------------------------- ### Include pbf_reader Header Source: https://github.com/mapbox/protozero/blob/master/doc/tutorial.md Include the `pbf_reader.hpp` header to use the `pbf_reader` class for parsing protobuf messages. Compiling with asserts enabled is recommended for debug builds. ```cpp #include ``` -------------------------------- ### Generate Reader Test Source Files Source: https://github.com/mapbox/protozero/blob/master/test/CMakeLists.txt Generates a list of C++ source files for reader tests by transforming the TEST_DIRS list. Each directory is mapped to a corresponding reader_test_cases.cpp file. ```cmake string(REGEX REPLACE "([^;]+)" "t/\1/reader_test_cases.cpp" _test_sources "${TEST_DIRS}") ``` -------------------------------- ### Extract Timestamps with `next(tag)` Source: https://context7.com/mapbox/protozero/llms.txt Use `next(tag)` to efficiently skip to and extract specific fields, like timestamps, without scanning the entire message. This is useful when only a subset of data is needed. ```cpp #include #include #include // Proto: message Track { repeated fixed64 timestamps = 17 [packed=true]; } std::vector extract_timestamps(const std::string& data) { protozero::pbf_reader msg{data}; std::vector result; if (msg.next(17)) { // skip straight to tag 17 auto range = msg.get_packed_fixed64(); // returns iterator_range result.reserve(range.size()); // O(1) for fixed-size types for (auto ts : range) { result.push_back(ts); } } return result; // e.g. {1700000000, 1700000060, 1700000120} } ``` -------------------------------- ### Parse Protobuf Messages with pbf_reader Source: https://context7.com/mapbox/protozero/llms.txt Use `pbf_reader` for tag-based parsing of protobuf data from various sources. Remember to always skip unknown tags using `skip()` to ensure correct parsing and avoid errors. ```cpp #include #include #include // Proto definition (for reference only — not read at runtime): // message Person { // required uint32 id = 1; // optional string name = 2; // optional bool admin = 3; // } void parse_person(const std::string& data) { protozero::pbf_reader msg{data}; uint32_t id = 0; std::string name; bool admin = false; try { while (msg.next()) { switch (msg.tag()) { case 1: id = msg.get_uint32(); break; case 2: name = msg.get_string(); // allocates std::string break; case 3: admin = msg.get_bool(); break; default: msg.skip(); // always skip unknown tags } } } catch (const protozero::end_of_buffer_exception& e) { std::cerr << "Truncated message: " << e.what() << "\n"; } catch (const protozero::exception& e) { std::cerr << "Parse error: " << e.what() << "\n"; } std::cout << "id=" << id << " name=" << name << " admin=" << admin << "\n"; // Output: id=42 name=Alice admin=1 } ``` -------------------------------- ### Handle repeated fields with tag_and_type() Source: https://github.com/mapbox/protozero/blob/master/doc/advanced.md Use tag_and_type() to correctly parse both packed and unpacked repeated fields, as Protozero does not enforce the protobuf specification's handling of these fields. ```cpp enum class ExampleMsg : protozero::pbf_tag_type { repeated_uint32_x = 1 }; std::string data = ... pbf_message message{data}; while (message.next()) { switch (message.tag_and_type()) { case tag_and_type(ExampleMsg::repeated_uint32_x, pbf_wire_type::length_delimited): { auto xit = message.get_packed_uint32(); ... // handle the repeated field when it is packed } break; case tag_and_type(ExampleMsg::repeated_uint32_x, pbf_wire_type::varint): { auto x = message.get_uint32(); ... // handle the repeated field when it is not packed } break; default: message.skip(); } } ``` -------------------------------- ### Write Basic Protocol Buffer Message Source: https://github.com/mapbox/protozero/blob/master/doc/tutorial.md Use pbf_writer to add scalar fields to a Protocol Buffer message. The buffer can be pre-populated. ```cpp #include std::string data; protozero.pbf_writer pbf_example{data}; pbf_example.add_uint32(1, 27); // uint32_t x pbf_example.add_fixed64(17, 1); // fixed64 r pbf_example.add_fixed64(17, 2); pbf_example.add_fixed64(17, 3); pbf_example.add_string(2, "foobar"); // string s ``` -------------------------------- ### Write Messages with `pbf_writer` Source: https://context7.com/mapbox/protozero/llms.txt Append protobuf-encoded fields to a string buffer using `pbf_writer`. Use `add_TYPE(tag, value)` to write fields in any order. The writer requires the buffer to outlive it. ```cpp #include #include #include // Proto: // message Person { required uint32 id = 1; optional string name = 2; } std::string build_person(uint32_t id, const std::string& name) { std::string buffer; buffer.reserve(64); // optional: hint for expected size protozero::pbf_writer writer{buffer}; writer.add_uint32(1, id); writer.add_string(2, name); // writer goes out of scope — no cleanup needed for top-level message return buffer; // ready to send/store } int main() { auto msg = build_person(42, "Alice"); std::cout << "Encoded " << msg.size() << " bytes\n"; // Output: Encoded 9 bytes } ``` -------------------------------- ### Rollback Sub-Messages and Packed Fields with Protozero Source: https://context7.com/mapbox/protozero/llms.txt Illustrates the rollback functionality for sub-message writers and packed_field_* objects in protozero. Rollback removes partially written fields, preventing them from being included in the final buffer. ```cpp #include #include std::string build_with_rollback(bool include_optional) { std::string buffer; protozero.pbf_writer pw{buffer}; pw.add_uint32(1, 99); // always written { protozero.pbf_writer sub{pw, 2}; sub.add_string(1, "tentative"); if (!include_optional) { sub.rollback(); // removes the entire sub-message from buffer } // if include_optional==true, sub commits on scope exit } return buffer; } ``` -------------------------------- ### Vectored Writes with Protozero `add_bytes_vectored` Source: https://context7.com/mapbox/protozero/llms.txt Demonstrates using `add_bytes_vectored` for scatter-gather writes in protozero. This method concatenates multiple buffers into a single length-delimited field without an intermediate copy, ideal for data spanning several string or data_view objects. ```cpp #include #include std::string build_vectored() { std::string buffer; protozero.pbf_writer writer{buffer}; std::string header{"HEADER:"}; std::string payload{"actual-data-payload"}; protozero.data_view suffix{" [END]", 6}; // Writes all three into tag 1 as one length-delimited field, single copy writer.add_bytes_vectored(1, header, payload, suffix); // Resulting field value: "HEADER:actual-data-payload [END]" return buffer; } ``` -------------------------------- ### Add Fixed Size Packed Field with Exact Element Count Source: https://github.com/mapbox/protozero/blob/master/doc/tutorial.md Optimize writing fixed-size packed fields by specifying the exact number of elements. You must provide the promised number of elements, or the message will be malformed. ```cpp std::string data; protozero.pbf_writer pw{data}; { protozero.packed_field_fixed32 field{pw, 1, 2}; // exactly two elements field.add_element(42); field.add_element(13); } ``` -------------------------------- ### Abandon Sub-message Writing with Rollback Source: https://github.com/mapbox/protozero/blob/master/doc/tutorial.md Allows abandoning the writing of a sub-message by calling `rollback()` on the sub-message writer. The result is as if the sub-message content was never added. ```cpp std::string data; protozero::pbf_writer pbf_parent{data}; // open a new scope { // create new pbf_writer with parent and the tag (field number) // as parameters protozero::pbf_writer pbf_sub{pbf_parent, 1}; // add fields to sub here... pbf_sub.add_...(...); // some problem occurs and you want to abandon the submessage: pbf_sub.rollback(); } // optionally add more fields to parent here pbf_parent.add_...(...); ``` -------------------------------- ### Add Sub-message to Parent Message (In-place) Source: https://github.com/mapbox/protozero/blob/master/doc/tutorial.md Handles nested sub-messages efficiently by creating a sub-message writer that reuses the parent's buffer. This avoids a separate buffer for the sub-message. ```cpp std::string data; protozero::pbf_writer pbf_parent{data}; // optionally add fields to parent here pbf_parent.add_...(...); // open a new scope { // create new pbf_writer with parent and the tag (field number) // as parameters protozero::pbf_writer pbf_sub{pbf_parent, 1}; // add fields to sub here... pbf_sub.add_...(...); } // closing the scope will close the sub-message // optionally add more fields to parent here pbf_parent.add_...(...); ``` -------------------------------- ### Add Sub-message to Parent Message (Separate Buffer) Source: https://github.com/mapbox/protozero/blob/master/doc/tutorial.md Handles nested sub-messages by creating a separate buffer for the sub-message and then adding it to the parent message. Requires a separate std::string buffer for the sub-message. ```cpp std::string buffer_sub; protozero::pbf_writer pbf_sub{buffer_sub}; // add fields to sub-message pbf_sub.add_...(...); // ... // sub-message is finished here std::string buffer_parent; protozero::pbf_writer pbf_parent{buffer_parent}; pbf_parent.add_message(1, buffer_sub); ``` -------------------------------- ### Set Working Directory for Reader Tests Source: https://github.com/mapbox/protozero/blob/master/test/CMakeLists.txt Configures the working directory for the 'reader_tests' executable when it is run. This ensures tests can access necessary files relative to the source directory. ```cmake set_tests_properties(reader_tests PROPERTIES WORKING_DIRECTORY "${CMAKE_SOURCE_DIR}") ``` -------------------------------- ### Commit Release Changes Source: https://github.com/mapbox/protozero/blob/master/CONTRIBUTING.md Commit the version number updates and changelog entries before tagging the release. ```git git commit -m "Release X.Y.Z" include/protozero/version.hpp CMakeLists.txt CHANGELOG.md UPGRADING.md ``` -------------------------------- ### Read Embedded Sub-Messages with `get_message()` Source: https://context7.com/mapbox/protozero/llms.txt Access nested messages using `get_message()`, which provides a zero-copy `pbf_reader` for the sub-message. This allows for arbitrarily deep nesting and efficient parsing. ```cpp #include #include // Proto: // message Point { required double x = 1; required double y = 2; } // message Polyline { repeated Point points = 10; } void parse_polyline(const std::string& data) { protozero::pbf_reader msg{data}; while (msg.next(10)) { protozero::pbf_reader point = msg.get_message(); // no copy double x = 0.0, y = 0.0; while (point.next()) { switch (point.tag()) { case 1: x = point.get_double(); break; case 2: y = point.get_double(); break; default: point.skip(); } } std::cout << "(" << x << ", " << y << ")\n"; } // Output: // (1.0, 2.0) // (3.5, 4.5) } ``` -------------------------------- ### Type-Safe Protobuf Parsing with pbf_message Source: https://context7.com/mapbox/protozero/llms.txt Utilize `pbf_message` for a type-safe reading interface, using an enum class to map field descriptors to getter calls. This approach eliminates magic numbers and is the recommended method for parsing. ```cpp #include #include // Declare enum matching the .proto file: // message Location { // required double lat = 1; // required double lon = 2; // optional string label = 3; // } enum class Location : protozero::pbf_tag_type { required_double_lat = 1, required_double_lon = 2, optional_string_label = 3, }; void parse_location(const std::string& data) { protozero::pbf_message msg{data}; double lat = 0.0, lon = 0.0; protozero::data_view label{}; // zero-copy view, no allocation while (msg.next()) { switch (msg.tag()) { case Location::required_double_lat: lat = msg.get_double(); break; case Location::required_double_lon: lon = msg.get_double(); break; case Location::optional_string_label: label = msg.get_view(); // returns data_view (no copy) break; default: msg.skip(); } } std::cout << "lat=" << lat << " lon=" << lon << " label=" << std::string(label) << "\n"; // Output: lat=48.8566 lon=2.3522 label=Paris } ``` -------------------------------- ### Define Enum for Protobuf Tags Source: https://github.com/mapbox/protozero/blob/master/doc/tutorial.md Define an enum class based on protozero::pbf_tag_type to represent protobuf field tags. This improves code readability and maintainability by replacing magic numbers with meaningful names. ```cpp enum class Example1 : protozero::pbf_tag_type { required_uint32_x = 1, optional_string_s = 2, repeated_fixed64_r = 17 }; ``` -------------------------------- ### Parse Protobuf Messages with pbf_message Source: https://github.com/mapbox/protozero/blob/master/doc/tutorial.md Use the `pbf_message` template class to parse protobuf-encoded messages. It iterates through fields, allowing you to retrieve data based on the defined enum tags. Ensure the input data remains valid for the lifetime of the message object. ```cpp #include // get data from somewhere into the input string std::string input = get_input_data(); // initialize pbf message with this data protozero::pbf_message message{input}; // iterate over fields in the message while (message.next()) { // switch depending on the field tag (the field name is not available) switch (message.tag()) { case Example1::required_uint32_x: auto x = message.get_uint32(); break; case Example1::optional_string_s: std::string s = message.get_string(); break; case Example1::repeated_fixed64_r: message.skip(); break; default: // ignore data for unknown tags to allow for future extensions message.skip(); } } ``` -------------------------------- ### Add Reader Tests to CTest Source: https://github.com/mapbox/protozero/blob/master/test/CMakeLists.txt Registers the 'reader_tests' executable as a test case in CTest. This allows the test to be run using the 'ctest' command. ```cmake add_test(NAME reader_tests COMMAND reader_tests) ``` -------------------------------- ### Iterate Repeated Packed Fields in C++ Source: https://github.com/mapbox/protozero/blob/master/doc/tutorial.md Access data from repeated packed fields using iterator ranges. Ensure the field is marked with `[packed=true]` in your .proto file. ```cpp protozero::pbf_reader message{input.data(), input.size()}; // set current field message.next(1); // get an iterator range auto pi = message.get_packed_sint32(); // iterate to get to all values for (auto it = pi.begin(); it != pi.end(); ++it) { std::cout << *it << '\n'; } ``` ```cpp for (auto value : pi) { std::cout << v << '\n'; } ``` -------------------------------- ### Add Repeated Packed Int32 Field with Elements Source: https://github.com/mapbox/protozero/blob/master/doc/tutorial.md Write a repeated packed int32 field by adding elements within a nested scope. The scope ensures the field length is correctly set upon destruction. ```cpp std::string data; protozero.pbf_writer pw{data}; { protozero.packed_field_int32 field{pw, 1}; field.add_element(1); field.add_element(10); field.add_element(100); } ```