### Install PyETWkit Source: https://github.com/m96-chan/pyetwkit/blob/main/README.md Install the core library using pip. Optional dependencies for dashboard and export features can be installed with extras. ```bash pip install pyetwkit # Optional: Dashboard support pip install pyetwkit[dashboard] # Optional: Export to Parquet/Arrow pip install pyetwkit[export] ``` -------------------------------- ### Install PyETWkit Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/index.md Install the PyETWkit library using pip. This is the first step to using the library. ```bash pip install pyetwkit ``` -------------------------------- ### Run Provider Discovery Example Source: https://github.com/m96-chan/pyetwkit/blob/main/examples/README.md Demonstrates how to run the provider_discovery.py script to discover and search for ETW providers. Administrator privileges are not required. ```bash python provider_discovery.py ``` -------------------------------- ### Run Basic Session Example Source: https://github.com/m96-chan/pyetwkit/blob/main/examples/README.md Demonstrates how to run the basic_session.py script for monitoring DNS client events. Requires administrator privileges. ```bash # Run as administrator python basic_session.py ``` -------------------------------- ### Run Kernel Trace Example Source: https://github.com/m96-chan/pyetwkit/blob/main/examples/README.md Demonstrates how to run the kernel_trace.py script for kernel-level tracing of process events. Requires administrator privileges. ```bash # Run as administrator python kernel_trace.py ``` -------------------------------- ### Run Profiles Example Source: https://github.com/m96-chan/pyetwkit/blob/main/examples/README.md Demonstrates how to run the profiles.py script to utilize pre-configured provider profiles for common monitoring scenarios. Requires administrator privileges. ```bash # Run as administrator python profiles.py ``` -------------------------------- ### Start Trace to File Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/advanced/etw_basics.md This bash command starts an ETW trace session named 'mytrace', capturing events from the Microsoft-Windows-DNS-Client provider and outputting them to 'trace.etl'. ```bash logman start mytrace -p Microsoft-Windows-DNS-Client -o trace.etl -ets ``` -------------------------------- ### Verify PyETWkit Installation Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/quickstart.md Verify the installation by importing the library and printing its version. This confirms that the package is installed correctly and accessible. ```python import pyetwkit print(pyetwkit.__version__) ``` -------------------------------- ### Start Kernel Tracing Session Source: https://github.com/m96-chan/pyetwkit/blob/main/README.md Enables process events and starts a kernel tracing session. It then iterates through events, printing the image file name for process start events. ```python from pyetwkit._core import PyKernelFlags, PyKernelSession flags = PyKernelFlags() flags = flags.with_process() # Enable process events session = PyKernelSession(flags) session.start() for _ in range(10): event = session.next_event_timeout(1000) if event and event.event_id == 1: # Process start props = event.to_dict().get("properties", {}) print(f"Process: {props.get('ImageFileName')}") session.stop() ``` -------------------------------- ### Profile API Examples Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/profiles.md This snippet demonstrates various functions from the Profile API for retrieving, listing, and registering ETW profiles programmatically. ```python from pyetwkit.profiles import ( get_profile, list_profiles, load_profile, register_profile, Profile, ProviderConfig, ) # Get a profile audio = get_profile("audio") # List all profiles for profile in list_profiles(): print(f"{profile.name}: {profile.description}") # Create programmatically custom = Profile( name="custom", description="Custom profile", providers=[ ProviderConfig(name="Microsoft-Windows-DNS-Client"), ] ) register_profile(custom) ``` -------------------------------- ### PyETWkit CLI Commands Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/index.md Examples of using the PyETWkit command-line interface to list providers, list profiles, and monitor events. Note that monitoring events requires administrator privileges. ```bash # List providers pyetwkit providers --search Kernel # List profiles pyetwkit profiles # Monitor events (requires admin) pyetwkit listen Microsoft-Windows-Kernel-Process ``` -------------------------------- ### List Available Profiles Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/api/profiles.md Use `list_profiles` to get a list of all available profile configurations. This helps in understanding the pre-defined options for tracing. ```python from pyetwkit.profiles import list_profiles for profile in list_profiles(): print(f"{profile.name}: {profile.description}") ``` -------------------------------- ### Run Export Events Example Source: https://github.com/m96-chan/pyetwkit/blob/main/examples/README.md Demonstrates how to run the export_events.py script to capture events and export them to CSV, JSON, and JSONL formats. Requires administrator privileges. ```bash # Run as administrator python export_events.py ``` -------------------------------- ### Try Get Next Event Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/api/streamer.md Demonstrates how to attempt to retrieve the next event without blocking. ```python # Try to get event without waiting event = streamer.try_next_event() ``` -------------------------------- ### Basic Python ETW Session Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/tutorial.md Demonstrates basic Python usage for creating an ETW session, adding a provider, and processing events. Requires administrator privileges to start the session. ```python from pyetwkit._core import EtwProvider, EtwSession # Create a session session = EtwSession("MySession") # Add a provider provider = EtwProvider( "Microsoft-Windows-DNS-Client", "Microsoft-Windows-DNS-Client" ) provider = provider.with_level(4) # Info level session.add_provider(provider) # Start and process events session.start() try: while True: event = session.next_event_timeout(1000) if event: print(f"Event {event.event_id}: {event.provider_name}") except KeyboardInterrupt: pass finally: session.stop() ``` -------------------------------- ### Get a Pre-configured Profile Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/api/profiles.md Use `get_profile` to retrieve a named profile configuration. This is useful for quickly setting up tracing for common tasks like audio capture. ```python from pyetwkit.profiles import get_profile audio = get_profile("audio") print(f"Name: {audio.name}") print(f"Description: {audio.description}") for p in audio.providers: print(f" - {p.name} ({p.level})") ``` -------------------------------- ### Configure EtwListener Keywords Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/api/listener.md Filters events by specifying keywords for the EtwListener. The example uses a hexadecimal value for all keywords. ```python listener = EtwListener("provider", keywords=0xFFFFFFFFFFFFFFFF) ``` -------------------------------- ### Get Next Event with Timeout Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/api/streamer.md Illustrates how to retrieve the next event with a specified timeout. ```python # Get next event with timeout event = await streamer.next_event(timeout=1.0) ``` -------------------------------- ### Async Event Processing with EtwStreamer Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/api/streamer.md An example of asynchronously processing events and breaking the loop after a certain count. ```python import asyncio from pyetwkit import EtwStreamer async def process_events(): async with EtwStreamer("Microsoft-Windows-DNS-Client") as streamer: async for event in streamer: # Async processing await save_to_database(event) if streamer.stats().events_received >= 100: break asyncio.run(process_events()) ``` -------------------------------- ### Discover and Get Provider Information Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/advanced/low_level.md List all available ETW providers, search for specific providers by name, and retrieve detailed information about a given provider. ```python from pyetwkit._core import list_providers, search_providers, get_provider_info # List all providers providers = list_providers() print(f"Total: {len(providers)}") # Search results = search_providers("Audio") # Get details info = get_provider_info("Microsoft-Windows-Audio") print(f"GUID: {info.guid}") ``` -------------------------------- ### Output Provider Profiles as JSON Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/cli.md Get a list of available provider profiles in JSON format. This allows for easier parsing and integration with other tools. ```bash pyetwkit profiles --format json ``` -------------------------------- ### Output ETW Providers as JSON Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/cli.md Get a list of ETW providers in JSON format. This is useful for programmatic processing or saving to a file. ```bash pyetwkit providers --format json ``` ```bash # Export provider list to JSON pyetwkit providers --format json > providers.json ``` -------------------------------- ### Getting Statistics Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/api/listener.md Retrieves statistics about the ETW listener, including the number of events received and lost. ```APIDOC ## Getting Statistics ### Description Retrieves statistics about the ETW listener, including the number of events received and lost. ### Method `listener.stats()` ### Returns An object containing statistics: - **events_received** (integer) - The total number of events received. - **events_lost** (integer) - The total number of events lost due to buffer overflows or other issues. ``` -------------------------------- ### Listen to a Specific ETW Provider Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/cli.md Monitor ETW events from a specific provider by its name or GUID. This command requires administrator privileges. ```bash # Listen to a specific provider pyetwkit listen Microsoft-Windows-Kernel-Process ``` ```bash # Monitor DNS queries (requires admin) pyetwkit listen Microsoft-Windows-DNS-Client ``` -------------------------------- ### Discover ETW Providers Source: https://github.com/m96-chan/pyetwkit/blob/main/README.md Lists the first 10 available ETW providers with their names and GUIDs, and then searches for providers whose names contain 'Kernel'. ```python from pyetwkit._core import list_providers, search_providers # List all providers for p in list_providers()[:10]: print(f"{p.name}: {p.guid}") # Search by name for p in search_providers("Kernel"): print(p.name) ``` -------------------------------- ### Using EtwStreamer with Profiles Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/api/streamer.md Demonstrates how to use a predefined profile to configure the EtwStreamer. ```python async with EtwStreamer(profile="network") as streamer: async for event in streamer: print(event) ``` -------------------------------- ### Use Pre-defined Profile with EtwListener Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/api/listener.md Initializes EtwListener using a pre-defined profile, such as 'audio', to simplify configuration. ```python listener = EtwListener(profile="audio") ``` -------------------------------- ### Launch Live Dashboard using CLI Source: https://github.com/m96-chan/pyetwkit/blob/main/README.md Launch the live dashboard for real-time visualization of ETW events. Requires administrator privileges. You can specify a port. ```bash # Launch live dashboard (requires admin) pyetwkit dashboard Microsoft-Windows-Kernel-Process pyetwkit dashboard --profile network --port 8080 ``` -------------------------------- ### Basic EtwStreamer Usage Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/api/streamer.md Demonstrates how to initialize and use EtwStreamer to monitor events from a single provider. ```python import asyncio from pyetwkit import EtwStreamer async def monitor(): async with EtwStreamer("Microsoft-Windows-DNS-Client") as streamer: async for event in streamer: print(event) asyncio.run(monitor()) ``` -------------------------------- ### Kernel Tracing with PyETWkit Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/tutorial.md Demonstrates kernel-level tracing for process, thread, and image load events. Requires administrator privileges. ```python from pyetwkit._core import PyKernelFlags, PyKernelSession # Configure kernel flags flags = PyKernelFlags() flags = flags.with_process() # Process events flags = flags.with_thread() # Thread events flags = flags.with_image_load() # DLL/module loads # Create and start session session = PyKernelSession(flags) session.start() try: while True: event = session.next_event_timeout(1000) if event: if event.event_id == 1: # Process start props = event.to_dict().get("properties", {}) print(f"Process started: {props.get('ImageFileName')}") except KeyboardInterrupt: pass finally: session.stop() ``` -------------------------------- ### Create and Read ETL File Source: https://github.com/m96-chan/pyetwkit/blob/main/examples/README.md Shows how to create an ETL file using logman and then read its events using the read_etl.py script. ```bash # First, create an ETL file logman start mytrace -p Microsoft-Windows-DNS-Client -o trace.etl -ets ping example.com logman stop mytrace -ets # Then read it python read_etl.py trace.etl ``` -------------------------------- ### Display CLI Help Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/cli.md View the help message for the PyETWKit CLI to understand available commands and options. ```bash pyetwkit --help ``` -------------------------------- ### Python API: Live Dashboard Source: https://github.com/m96-chan/pyetwkit/blob/main/README.md Create and launch a live dashboard for real-time visualization. Add providers to monitor. The dashboard will open in your browser. ```python from pyetwkit import Dashboard # Create and launch dashboard dashboard = Dashboard(port=7860) dashboard.add_provider("Microsoft-Windows-Kernel-Process") dashboard.add_provider("Microsoft-Windows-DNS-Client") # Opens browser at http://localhost:7860 dashboard.launch() ``` -------------------------------- ### Basic EtwListener Usage Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/api/listener.md Demonstrates the basic usage of EtwListener to monitor events from a specific provider and iterate through them. ```python from pyetwkit import EtwListener with EtwListener("Microsoft-Windows-DNS-Client") as listener: for event in listener: print(event) ``` -------------------------------- ### Create and Register a Custom Profile from Code Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/api/profiles.md Define and register a custom profile programmatically using `Profile` and `ProviderConfig` classes. This allows for fine-grained control over the providers included in a trace. ```python from pyetwkit.profiles import Profile, ProviderConfig, register_profile custom = Profile( name="my_custom", description="My custom profile", providers=[ ProviderConfig( name="Microsoft-Windows-DNS-Client", level="verbose", ), ProviderConfig( name="Microsoft-Windows-TCPIP", level="information", keywords=0xFFFFFFFF, ), ], ) register_profile(custom) ``` -------------------------------- ### List and Search ETW Providers in Python Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/tutorial.md Discover available ETW providers on the system using Python. Shows listing the first 10 and searching for 'Kernel' providers. ```python from pyetwkit._core import list_providers, search_providers, get_provider_info # List all providers providers = list_providers() for p in providers[:10]: print(f"{p.name}: {p.guid}") # Search by name kernel_providers = search_providers("Kernel") # Get detailed info info = get_provider_info("Microsoft-Windows-DNS-Client") if info: print(f"GUID: {info.guid}") ``` -------------------------------- ### PyETWkit CLI: List Profiles Source: https://github.com/m96-chan/pyetwkit/blob/main/examples/README.md Lists all pre-configured provider profiles available through the PyETWkit command-line interface. ```bash # List profiles pyetwkit profiles ``` -------------------------------- ### CLI: List ETW Profiles Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/quickstart.md Use the PyETWkit CLI to list available pre-configured ETW profiles. This command shows the names and descriptions of available profiles. ```bash pyetwkit profiles ``` -------------------------------- ### Stop an Existing ETW Session Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/tutorial.md Use this command to stop a previously started ETW session if it was not properly terminated. This is useful for cleaning up lingering sessions that might cause conflicts. ```bash logman stop MySession -ets ``` -------------------------------- ### Get Session Statistics Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/advanced/low_level.md Retrieve statistics about an active ETW session, such as the number of events received and lost, and buffers processed. This helps in monitoring session performance and data integrity. ```python stats = session.stats() print(f"Events received: {stats.events_received}") print(f"Events lost: {stats.events_lost}") print(f"Buffers processed: {stats.buffers_processed}") ``` -------------------------------- ### PyETWkit CLI: Listen to Events Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/tutorial.md Use the command-line interface to listen to events from a specific provider. Requires administrator privileges. ```bash pyetwkit listen Microsoft-Windows-DNS-Client ``` -------------------------------- ### Use Built-in Audio Profile Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/profiles.md This snippet demonstrates how to use the pre-configured 'audio' profile to listen for audio-related ETW events. ```python with EtwListener(profile="audio") as listener: for event in listener: print(event) ``` -------------------------------- ### EtwStreamer Initialization with Profile Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/api/streamer.md Initializes EtwStreamer to monitor events based on a predefined profile. ```APIDOC ## EtwStreamer Initialization with Profile ### Description Initializes EtwStreamer to monitor events based on a predefined profile. ### Method Asynchronous Context Manager (`async with`) ### Endpoint N/A (Class method) ### Parameters #### Path Parameters N/A #### Query Parameters N/A #### Request Body N/A ### Request Example ```python async with EtwStreamer(profile="network") as streamer: async for event in streamer: print(event) ``` ### Response #### Success Response (Iteration) - **event** (object) - An ETW event object. #### Response Example ```json { "example": "event object details" } ``` ``` -------------------------------- ### EtwStreamer Initialization and Async Iteration Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/api/streamer.md Initializes EtwStreamer with a single provider and asynchronously iterates through received events. ```APIDOC ## EtwStreamer Initialization and Async Iteration ### Description Initializes EtwStreamer with a single provider and asynchronously iterates through received events. ### Method Asynchronous Context Manager (`async with`) ### Endpoint N/A (Class method) ### Parameters #### Path Parameters N/A #### Query Parameters N/A #### Request Body N/A ### Request Example ```python async with EtwStreamer("Microsoft-Windows-DNS-Client") as streamer: async for event in streamer: print(event) ``` ### Response #### Success Response (Iteration) - **event** (object) - An ETW event object. #### Response Example ```json { "example": "event object details" } ``` ``` -------------------------------- ### Monitoring Multiple Providers Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/api/streamer.md Shows how to configure EtwStreamer to monitor events from a list of providers simultaneously. ```python providers = [ "Microsoft-Windows-Kernel-Process", "Microsoft-Windows-Kernel-File", ] async with EtwStreamer(providers) as streamer: async for event in streamer: print(f"{event.provider_name}: {event.event_id}") ``` -------------------------------- ### EtwStreamer Initialization with Multiple Providers Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/api/streamer.md Initializes EtwStreamer to monitor events from multiple specified providers. ```APIDOC ## EtwStreamer Initialization with Multiple Providers ### Description Initializes EtwStreamer to monitor events from multiple specified providers. ### Method Asynchronous Context Manager (`async with`) ### Endpoint N/A (Class method) ### Parameters #### Path Parameters N/A #### Query Parameters N/A #### Request Body N/A ### Request Example ```python providers = [ "Microsoft-Windows-Kernel-Process", "Microsoft-Windows-Kernel-File", ] async with EtwStreamer(providers) as streamer: async for event in streamer: print(f"{event.provider_name}: {event.event_id}") ``` ### Response #### Success Response (Iteration) - **event** (object) - An ETW event object with provider name and event ID. #### Response Example ```json { "example": "{event.provider_name}: {event.event_id}" } ``` ``` -------------------------------- ### Basic EtwListener Usage Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/index.md Listen to process events using EtwListener. This snippet demonstrates how to instantiate and use the listener for a specific provider. ```python from pyetwkit import EtwListener # Listen to process events with EtwListener("Microsoft-Windows-Kernel-Process") as listener: for event in listener: print(f"Event: {event.event_id}") ``` -------------------------------- ### CLI: Search ETW Providers Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/quickstart.md Use the PyETWkit CLI to search for ETW providers by keyword. This helps in finding specific providers, such as those related to 'Audio'. ```bash pyetwkit providers --search Audio ``` -------------------------------- ### Define and Load Custom Profile from YAML Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/profiles.md This snippet shows how to create a custom ETW profile using a YAML file and then load it into the EtwListener. ```yaml # my_profile.yaml name: my_profile description: My custom monitoring profile providers: - name: Microsoft-Windows-DNS-Client level: verbose keywords: 0xFFFFFFFFFFFFFFFF - name: Microsoft-Windows-TCPIP level: information ``` ```python from pyetwkit.profiles import load_profile profile = load_profile("my_profile.yaml") with EtwListener(profile=profile.name) as listener: for event in listener: print(event) ``` -------------------------------- ### Enable All Keywords Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/advanced/etw_basics.md This Python snippet shows how to enable all keywords for an ETW provider using PyETWkit. ```python provider = EtwProvider("guid", "name") provider = provider.with_keywords(0xFFFFFFFFFFFFFFFF) ``` -------------------------------- ### List All Providers Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/advanced/etw_basics.md This Python snippet uses PyETWkit to list all available ETW providers on the system. ```python from pyetwkit import list_providers # List all providers = list_providers() ``` -------------------------------- ### CLI: List ETW Providers Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/quickstart.md Use the PyETWkit CLI to list available ETW providers. This command provides a quick way to discover providers from the terminal. ```bash pyetwkit providers ``` -------------------------------- ### EtwListener with Profile Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/index.md Use a pre-defined profile with EtwListener to listen for specific types of events. This simplifies configuration for common use cases. ```python from pyetwkit import EtwListener # Use a pre-defined profile with EtwListener(profile="audio") as listener: for event in listener: print(event) ``` -------------------------------- ### CLI: Monitor Kernel Process Events Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/quickstart.md Use the PyETWkit CLI to monitor events from the Microsoft-Windows-Kernel-Process provider. This command requires administrator privileges. ```bash pyetwkit listen Microsoft-Windows-Kernel-Process ``` -------------------------------- ### Listen to ETW Events using CLI Source: https://github.com/m96-chan/pyetwkit/blob/main/README.md Listen to real-time ETW events from specific providers or profiles. Requires administrator privileges. ```bash # Listen to events (requires admin) pyetwkit listen Microsoft-Windows-DNS-Client pyetwkit listen --profile network ``` -------------------------------- ### Listen to ETW Events Using a Profile Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/cli.md Monitor ETW events by using a predefined provider profile. This simplifies listening to a group of related events. Requires administrator privileges. ```bash # Use a profile pyetwkit listen --profile audio ``` ```bash # Monitor network with profile (requires admin) pyetwkit listen --profile network ``` -------------------------------- ### List and Use ETW Profiles Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/quickstart.md List available pre-configured ETW profiles and use a profile (e.g., 'network') to monitor events. Profiles simplify monitoring by grouping common providers. ```python from pyetwkit import EtwListener from pyetwkit.profiles import list_profiles # List available profiles for profile in list_profiles(): print(f"{profile.name}: {profile.description}") # Use a profile with EtwListener(profile="network") as listener: for event in listener: print(event) ``` -------------------------------- ### Create ETW Session in Python Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/tutorial.md Instantiate an ETW session with a unique name. ```python from pyetwkit._core import EtwSession session = EtwSession("UniqueSessionName") ``` -------------------------------- ### Process Monitoring with EtwListener Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/api/listener.md Monitors process events using EtwListener and prints the event ID and image file name if available. ```python from pyetwkit import EtwListener with EtwListener("Microsoft-Windows-Kernel-Process") as listener: for event in listener: print(f"Process event: {event.event_id}") if event.properties.get("ImageFileName"): print(f" Image: {event.properties['ImageFileName']}") ``` -------------------------------- ### PyETWkit CLI: List Providers Source: https://github.com/m96-chan/pyetwkit/blob/main/examples/README.md Lists all available ETW providers or searches for specific providers using the PyETWkit command-line interface. ```bash # List providers pyetwkit providers pyetwkit providers --search Kernel ``` -------------------------------- ### EtwListener Initialization Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/api/listener.md Initializes an EtwListener to monitor ETW events. It can monitor a single provider, multiple providers, or use a pre-defined profile. Configuration options for level and keywords are also available. ```APIDOC ## EtwListener Initialization ### Description Initializes an EtwListener to monitor ETW events. It can monitor a single provider, multiple providers, or use a pre-defined profile. Configuration options for level and keywords are also available. ### Parameters #### Providers - **providers** (string or list[string]) - Required - The ETW provider(s) to listen to. Can be a single provider name or a list of provider names. - **profile** (string) - Optional - Use a pre-defined profile for listening to a group of providers. #### Configuration Options - **level** (string) - Optional - Set the trace level. Possible values: critical, error, warning, information, verbose. - **keywords** (integer) - Optional - Filter events by keywords. Expects a bitmask. ``` -------------------------------- ### PyETWkit CLI: Search Providers Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/tutorial.md Use the command-line interface to search for specific ETW providers by name. ```bash pyetwkit providers --search DNS ``` -------------------------------- ### Enable Stack Trace Capture Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/advanced/low_level.md Configure a provider to capture stack traces for events. Access captured stack traces from the event object and iterate through the addresses. ```python from pyetwkit._core import EtwProvider, EnableProperty provider = EtwProvider("guid", "name") provider = provider.with_enable_property(EnableProperty.STACK_TRACE) # Access stack trace for event in listener: if event.stack_trace: for addr in event.stack_trace: print(f" 0x{addr:016x}") ``` -------------------------------- ### Configure ETW Provider in Python Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/tutorial.md Configure an ETW provider by name, setting its trace level and keywords. ```python from pyetwkit._core import EtwProvider # Using provider name provider = EtwProvider("Microsoft-Windows-DNS-Client", "DNS-Client") # Set trace level (1=Critical, 2=Error, 3=Warning, 4=Info, 5=Verbose) provider = provider.with_level(4) # Enable all keywords (event categories) provider = provider.with_any_keyword(0xFFFFFFFF) ``` -------------------------------- ### list_profiles Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/api/profiles.md Lists all available pre-configured profiles. This function helps in discovering the different provider sets that can be loaded. ```APIDOC ## list_profiles ### Description Lists all available pre-configured profiles. ### Function Signature `list_profiles() -> List[Profile]` ### Parameters None ### Response #### Success Response - **List[Profile]** - A list of Profile objects, each representing an available profile. ``` -------------------------------- ### Capture and Export DNS Client Events Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/api/export.md Demonstrates capturing a limited number of DNS Client events and exporting them to both a Pandas DataFrame for analysis and a Parquet file for storage. ```python from pyetwkit import EtwListener from pyetwkit.export import to_dataframe, to_parquet # Capture events with EtwListener("Microsoft-Windows-DNS-Client") as listener: events = list(listener.events(max_events=1000)) # To DataFrame for analysis df = to_dataframe(events) print(df.describe()) # Save for later to_parquet(events, "dns_events.parquet") ``` -------------------------------- ### PyETWkit CLI: Export ETL to Parquet Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/tutorial.md Export events from an ETL file to a Parquet file using the command-line interface. ```bash pyetwkit export trace.etl -o events.parquet -f parquet ``` -------------------------------- ### Load a Custom Profile from YAML Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/api/profiles.md Use `load_profile` to load a profile configuration from a YAML file. This is a convenient way to manage and share custom trace configurations. ```python from pyetwkit.profiles import load_profile profile = load_profile("my_profile.yaml") ``` -------------------------------- ### Python API: Recording and Replay Source: https://github.com/m96-chan/pyetwkit/blob/main/README.md Record ETW events to a compressed file (.etwpack) using the Recorder, and replay them later using the Player. Supports different compression types. ```python from pyetwkit import Recorder, Player, CompressionType, RecorderConfig # Record events config = RecorderConfig(compression=CompressionType.ZSTD) recorder = Recorder("session.etwpack", config=config) recorder.add_provider("Microsoft-Windows-DNS-Client") recorder.start() # ... capture events ... recorder.stop() # Replay events player = Player("session.etwpack") print(f"Duration: {player.duration:.2f}s, Events: {player.event_count}") for event in player.events(): print(f"Event {event['event_id']}") ``` -------------------------------- ### Async Iteration with EtwStreamer Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/api/streamer.md Shows how to asynchronously iterate over events received by the EtwStreamer. ```python async with EtwStreamer("provider") as streamer: async for event in streamer: await process_event(event) ``` -------------------------------- ### Monitor Multiple Providers with EtwListener Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/api/listener.md Configures EtwListener to monitor events from multiple specified providers simultaneously. ```python providers = [ "Microsoft-Windows-Kernel-Process", "Microsoft-Windows-Kernel-File", ] listener = EtwListener(providers) ``` -------------------------------- ### PyETWkit CLI: Listen to Events Source: https://github.com/m96-chan/pyetwkit/blob/main/examples/README.md Listens to ETW events in real-time using specific providers or pre-configured profiles via the PyETWkit command-line interface. ```bash # Listen to events pyetwkit listen Microsoft-Windows-DNS-Client pyetwkit listen --profile network ``` -------------------------------- ### Configure EtwListener Level Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/api/listener.md Sets the trace level for the EtwListener. Available levels include critical, error, warning, information, and verbose. ```python # Level can be: critical, error, warning, information, verbose listener = EtwListener("provider", level="verbose") ``` -------------------------------- ### Retrieve EtwListener Statistics Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/api/listener.md Demonstrates how to retrieve statistics about received and lost events from an EtwListener instance. ```python with EtwListener("provider") as listener: for event in listener: stats = listener.stats() print(f"Received: {stats.events_received}") print(f"Dropped: {stats.events_lost}") ``` -------------------------------- ### Enable Specific Keywords Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/advanced/etw_basics.md This Python snippet demonstrates enabling a specific keyword for an ETW provider using PyETWkit. ```python provider = provider.with_keywords(0x0000000000000010) ``` -------------------------------- ### Streaming Export of Kernel Process Events Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/api/export.md Illustrates capturing a large stream of Kernel Process events and exporting them in batches to JSON Lines format for efficient handling of high-volume data. ```python from pyetwkit import EtwListener from pyetwkit.export import to_jsonl with EtwListener("Microsoft-Windows-Kernel-Process") as listener: # Batch export batch = [] batch_num = 0 for event in listener: batch.append(event) if len(batch) >= 10000: to_jsonl(batch, f"events_{batch_num}.jsonl") batch = [] batch_num += 1 ``` -------------------------------- ### Concurrent EtwStreamer Monitoring Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/api/streamer.md Shows how to run multiple EtwStreamer instances concurrently using asyncio.gather. ```python import asyncio from pyetwkit import EtwStreamer async def monitor(profile: str): async with EtwStreamer(profile=profile) as streamer: async for event in streamer: print(f"[{profile}] {event.event_id}") async def main(): await asyncio.gather( monitor("network"), monitor("process"), ) asyncio.run(main()) ``` -------------------------------- ### Python Thread Model for EtwListener Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/architecture/overview.md Illustrates the interaction between the main Python thread and the background Rust thread when using the synchronous EtwListener. The Python thread initiates the listener, iterates over events processed by the Rust thread, and stops the listener. ```text Main Thread (Python) Background Thread (Rust) ────────────────────── ───────────────────────── listener.start() ──────────────▶ spawn trace thread │ ▼ for event in listener: ◀───────── ProcessTrace loop process(event) │ ├─▶ callback(record) │ │ │ ▼ │ parse event │ │ │ ▼ │ channel.send(event) │ listener.stop() ──────────────▶ stop trace │ ▼ thread join ``` -------------------------------- ### Manage ETW Profiles in Python Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/tutorial.md List available ETW profiles and retrieve details for a specific profile like 'network'. ```python from pyetwkit.profiles import list_profiles, get_profile # List available profiles for profile in list_profiles(): print(f"{profile.name}: {profile.description}") # Get a specific profile network = get_profile("network") if network: for p in network.providers: print(f" {p.name}") ``` -------------------------------- ### CLI: Monitor Events Using a Profile Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/quickstart.md Use the PyETWkit CLI to monitor events using a specific profile, such as 'network'. This simplifies the process of listening to a common set of ETW providers. ```bash pyetwkit listen --profile network ``` -------------------------------- ### CLI: Monitor Events and Output to JSONL Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/quickstart.md Use the PyETWkit CLI to monitor events using a profile and output them to a JSON Lines file. This command allows for saving monitored events in a structured format. ```bash pyetwkit listen --profile network --output events.jsonl --format jsonl ``` -------------------------------- ### PyETWkit CLI: Limited Export Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/tutorial.md Export a limited number of events from an ETL file to a JSON file using the command-line interface. ```bash pyetwkit export trace.etl -o sample.json -f json --limit 1000 ``` -------------------------------- ### Read ETL File Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/advanced/etw_basics.md This Python snippet demonstrates how to read events from an ETL file using PyETWkit's EtlReader. ```python from pyetwkit import EtlReader with EtlReader("trace.etl") as reader: for event in reader: print(event) ``` -------------------------------- ### PyETWkit CLI: Export ETL to CSV Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/tutorial.md Export events from an ETL file to a CSV file using the command-line interface. ```bash pyetwkit export trace.etl -o events.csv ``` -------------------------------- ### to_parquet Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/api/export.md Exports captured ETW events to Parquet format. Requires the 'pyarrow' library. ```APIDOC ## to_parquet ### Description Exports captured ETW events to Parquet format. Requires the 'pyarrow' library. ### Function Signature ```python to_parquet(events, path, compression=None) ``` ### Parameters #### Arguments - **events**: The list of captured ETW events. - **path** (str): The file path to save the Parquet file. - **compression** (str, optional): The compression method to use (e.g., 'snappy'). Defaults to None. ### Usage Example ```python from pyetwkit.export import to_parquet to_parquet(events, "events.parquet") # With compression to_parquet(events, "events.parquet", compression="snappy") ``` ``` -------------------------------- ### Access Event Structure in Python Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/tutorial.md Demonstrates how to access event details, including event ID and properties, by converting an event object to a dictionary. ```python event_dict = event.to_dict() print(f"Event ID: {event_dict['event_id']}") print(f"Properties: {event_dict.get('properties', {})}") ``` -------------------------------- ### Monitor Kernel Process Events Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/quickstart.md Monitor events from the Microsoft-Windows-Kernel-Process provider using EtwListener. Requires administrator privileges. This snippet demonstrates basic event listening and processing. ```python from pyetwkit import EtwListener # Requires administrator privileges with EtwListener("Microsoft-Windows-Kernel-Process") as listener: for event in listener: print(f"Provider: {event.provider_name}") print(f"Event ID: {event.event_id}") print(f"Properties: {event.properties}") # Limit to 10 events for demo if listener.stats().events_received >= 10: break ``` -------------------------------- ### load_profile Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/api/profiles.md Loads a profile configuration from a YAML file. This allows for defining and managing complex provider sets externally. ```APIDOC ## load_profile ### Description Loads a profile configuration from a YAML file. ### Function Signature `load_profile(file_path: str) -> Profile` ### Parameters #### Path Parameters - **file_path** (string) - Required - The path to the YAML file containing the profile definition. ### Response #### Success Response - **Profile** - A Profile object loaded from the specified YAML file. ``` -------------------------------- ### to_arrow Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/api/export.md Converts captured ETW events into an Apache Arrow Table. Requires the 'pyarrow' library. ```APIDOC ## to_arrow ### Description Converts captured ETW events into an Apache Arrow Table. Requires the 'pyarrow' library. ### Function Signature ```python to_arrow(events) ``` ### Parameters #### Arguments - **events**: The list of captured ETW events. ### Usage Example ```python from pyetwkit.export import to_arrow table = to_arrow(events) print(table.schema) ``` ``` -------------------------------- ### Show More ETW Providers Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/cli.md Increase the number of ETW providers displayed when listing them. Use this to see more than the default limit. ```bash pyetwkit providers --limit 100 ``` -------------------------------- ### List and Search ETW Providers Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/quickstart.md List all available ETW providers or search for specific ones using keywords. This is useful for identifying which providers to monitor. ```python from pyetwkit import list_providers, search_providers # List all providers providers = list_providers() print(f"Found {len(providers)} providers") # Search for specific providers kernel_providers = search_providers("Kernel") for p in kernel_providers[:5]: print(f"{p.name}: {p.guid}") ``` -------------------------------- ### Convert Events to Apache Arrow Table Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/api/export.md Converts captured events into an Apache Arrow Table, useful for interoperability with other data processing tools. Requires the 'pyarrow' library. ```python from pyetwkit.export import to_arrow table = to_arrow(events) print(table.schema) ``` -------------------------------- ### Read ETL Files with PyETWkit Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/tutorial.md Use the EtlReader class to iterate through events stored in an ETL file. ```python from pyetwkit._core import EtlReader reader = EtlReader("trace.etl") for event in reader.events(): print(f"[{event.timestamp}] {event.provider_name} Event {event.event_id}") ``` -------------------------------- ### Export Events to Parquet Format Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/api/export.md Exports captured events to the Parquet file format, which is efficient for large datasets. Requires the 'pyarrow' library. Supports compression options like 'snappy'. ```python from pyetwkit.export import to_parquet to_parquet(events, "events.parquet") # With compression to_parquet(events, "events.parquet", compression="snappy") ``` -------------------------------- ### PyETWkit CLI: Export ETL File Source: https://github.com/m96-chan/pyetwkit/blob/main/examples/README.md Exports events from an ETL file to different formats (CSV, Parquet) using the PyETWkit command-line interface. ```bash # Export ETL file pyetwkit export trace.etl -o events.csv pyetwkit export trace.etl -o events.parquet -f parquet ``` -------------------------------- ### Profile API Reference Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/profiles.md This section outlines the core functions for managing provider profiles programmatically. You can retrieve specific profiles, list all available profiles, load custom profiles from files, and register new profiles. ```APIDOC ## Profile API This API allows for programmatic management of provider profiles. ### Functions - **get_profile(profile_name: str) -> Profile** Retrieves a pre-configured or registered profile by its name. - **list_profiles() -> List[Profile]** Returns a list of all available profile objects. - **load_profile(profile_path: str) -> Profile** Loads a custom profile from a YAML file specified by its path. - **register_profile(profile: Profile)** Registers a custom-defined profile object, making it available for use. ### Data Structures - **Profile** Represents a monitoring profile. - **name** (str): The name of the profile. - **description** (str): A description of the profile's purpose. - **providers** (List[ProviderConfig]): A list of provider configurations included in the profile. - **ProviderConfig** Configuration for a single ETW provider. - **name** (str): The name of the provider (e.g., "Microsoft-Windows-DNS-Client"). - **level** (str, optional): The logging level (e.g., "verbose", "information"). Defaults to "information". - **keywords** (str, optional): Bitmask for event filtering. Defaults to "0xFFFFFFFFFFFFFFFF". ### Example Usage ```python from pyetwkit.profiles import ( get_profile, list_profiles, load_profile, register_profile, Profile, ProviderConfig, ) # Get a specific profile audio_profile = get_profile("audio") print(f"Retrieved profile: {audio_profile.name}") # List all available profiles print("Available profiles:") for profile in list_profiles(): print(f"- {profile.name}: {profile.description}") # Create and register a custom profile programmatically custom_profile = Profile( name="my_custom_net_profile", description="Custom network monitoring profile", providers=[ ProviderConfig(name="Microsoft-Windows-TCPIP", level="verbose"), ProviderConfig(name="Microsoft-Windows-DNS-Client", keywords="0x10") ] ) register_profile(custom_profile) print(f"Registered custom profile: {custom_profile.name}") # Load a profile from a YAML file (assuming 'my_profile.yaml' exists) # try: # loaded_profile = load_profile("my_profile.yaml") # print(f"Loaded profile from file: {loaded_profile.name}") # except FileNotFoundError: # print("my_profile.yaml not found.") ``` ``` -------------------------------- ### Search for Specific ETW Providers Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/cli.md Search for ETW providers by name. This is useful for finding providers related to a specific component or keyword. ```bash pyetwkit providers --search Kernel ``` ```bash # Find audio-related providers pyetwkit providers --search Audio ``` ```bash # Find process-related providers pyetwkit providers --search "Kernel-Process" ``` -------------------------------- ### Python API: Event Correlation Source: https://github.com/m96-chan/pyetwkit/blob/main/README.md Use the CorrelationEngine to correlate ETW events by process ID or thread ID. Add providers and then add events to the engine for analysis. ```python from pyetwkit import CorrelationEngine # Create correlation engine engine = CorrelationEngine() engine.add_provider("Microsoft-Windows-Kernel-Process") engine.add_provider("Microsoft-Windows-Kernel-Network") # Add events from your ETW session for event in events: engine.add_event(event) # Correlate events by process ID correlated = engine.correlate_by_pid(1234) for event in correlated: print(f"Event {event.event_id} from {event.provider_name}") # Export to timeline JSON timeline = engine.to_timeline_json(pid=1234) ``` -------------------------------- ### PyETWkit CLI: Filtered Export Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/tutorial.md Export events from an ETL file, filtering by a specific provider using the command-line interface. ```bash pyetwkit export trace.etl -o filtered.csv -p Microsoft-Windows-DNS-Client ``` -------------------------------- ### Set ETW Trace Level Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/cli.md Configure the trace level for ETW monitoring. Higher levels like 'verbose' capture more detailed events. ```bash # Set trace level pyetwkit listen --profile network --level verbose ``` -------------------------------- ### Monitor DNS Client Events Asynchronously Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/quickstart.md Monitor events from the Microsoft-Windows-DNS-Client provider asynchronously using EtwStreamer. This is suitable for integration with asyncio applications. ```python import asyncio from pyetwkit import EtwStreamer async def monitor_events(): async with EtwStreamer("Microsoft-Windows-DNS-Client") as streamer: async for event in streamer: print(f"Event: {event.event_id}") asyncio.run(monitor_events()) ``` -------------------------------- ### EtwStreamer.try_next_event() Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/api/streamer.md Attempts to retrieve the next ETW event without waiting. ```APIDOC ## EtwStreamer.try_next_event() ### Description Attempts to retrieve the next ETW event without waiting. ### Method `streamer.try_next_event()` ### Endpoint N/A (Method of EtwStreamer instance) ### Parameters #### Path Parameters N/A #### Query Parameters N/A #### Request Body N/A ### Request Example ```python event = streamer.try_next_event() ``` ### Response #### Success Response - **event** (object) - The next ETW event object, or None if no event is immediately available. #### Response Example ```json { "example": "event object details or null" } ``` ``` -------------------------------- ### Search Providers Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/advanced/etw_basics.md This Python snippet uses PyETWkit to search for ETW providers by name. ```python from pyetwkit import search_providers # Search results = search_providers("Kernel") ``` -------------------------------- ### Read Events from ETL File Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/advanced/low_level.md Read events sequentially from a specified ETL file using the EtlReader. This is useful for offline analysis of captured traces. ```python from pyetwkit._core import EtlReader # Read ETL file with EtlReader("trace.etl") as reader: for event in reader: print(f"{event.provider_id}: {event.event_id}") # Read all events reader = EtlReader("trace.etl") events = reader.read_all() ``` -------------------------------- ### Record ETW Events to a File Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/cli.md Save captured ETW events to a file. The output format can be specified, with 'jsonl' being common for line-delimited JSON. ```bash # Save to file pyetwkit listen --profile network --output events.jsonl --format jsonl ``` ```bash # Record events to JSON Lines file pyetwkit listen --profile network --output events.jsonl --format jsonl ``` ```bash # Record limited events pyetwkit listen Microsoft-Windows-DNS-Client --max-events 1000 --output dns.jsonl --format jsonl ``` -------------------------------- ### Listen to ETW Events with JSON Output Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/cli.md Monitor ETW events and specify the output format as JSON. This is useful for structured data processing. ```bash # Specify output format pyetwkit listen --profile network --format json ``` -------------------------------- ### YAML Profile Configuration Structure Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/api/profiles.md Defines the structure for custom profiles saved in YAML format. It includes profile metadata and a list of providers with their configuration details. ```yaml name: my_profile description: Description of the profile providers: - name: Microsoft-Windows-Provider-Name guid: optional-guid-if-needed level: verbose # critical, error, warning, information, verbose keywords: 0xFFFFFFFFFFFFFFFF # optional keyword filter - name: Another-Provider level: information ``` -------------------------------- ### Iterating Through Events Source: https://github.com/m96-chan/pyetwkit/blob/main/docs/api/listener.md Allows iteration over the listener to receive ETW events as they occur. This is a synchronous operation. ```APIDOC ## Iterating Through Events ### Description Allows iteration over the listener to receive ETW events as they occur. This is a synchronous operation. ### Usage Use a `for` loop to iterate through the `EtwListener` object. Each iteration yields an event object. ### Example ```python from pyetwkit import EtwListener with EtwListener("Microsoft-Windows-DNS-Client") as listener: for event in listener: print(event) ``` ```