### Install ecs-logging-python Source: https://github.com/elastic/ecs-logging-python/blob/main/docs/reference/installation.md Install the ecs-logging-python library using pip. ```cmd $ python -m pip install ecs-logging ``` -------------------------------- ### Install ecs-logging-python Source: https://github.com/elastic/ecs-logging-python/blob/main/README.md Install the library using pip. Ensure you are using a compatible Python version. ```bash python -m pip install ecs-logging ``` -------------------------------- ### Normalize dictionary examples Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/api-reference/utility-functions.md Examples demonstrating basic dotted key normalization and handling of nested lists. ```python from ecs_logging._utils import normalize_dict # Input with dotted keys data = { "log.level": "info", "log.logger": "myapp", "message": "Hello", "http.method": "GET" } result = normalize_dict(data) print(result) # Output: # { # "log": {"level": "info", "logger": "myapp"}, # "message": "Hello", # "http": {"method": "GET"} # } ``` ```python from ecs_logging._utils import normalize_dict data = { "items": [ {"id.value": 1, "name": "first"}, {"id.value": 2, "name": "second"} ] } result = normalize_dict(data) print(result) # Output: # { # "items": [ # {"id": {"value": 1}, "name": "first"}, # {"id": {"value": 2}, "name": "second"} # ] # } ``` -------------------------------- ### Basic setup with StreamHandler Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/api-reference/stdlib-formatter.md Configures a standard StreamHandler to output ECS-formatted logs to stdout. ```python import logging import ecs_logging import sys # Configure handler handler = logging.StreamHandler(sys.stdout) handler.setFormatter(ecs_logging.StdlibFormatter()) # Configure logger logger = logging.getLogger("myapp") logger.addHandler(handler) logger.setLevel(logging.INFO) # Emit logs logger.info("Application started") logger.warning("High memory usage") ``` -------------------------------- ### De-dot examples Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/api-reference/utility-functions.md Examples showing conversion of dotted strings into nested dictionaries at various depths. ```python from ecs_logging._utils import de_dot result = de_dot("error", "RuntimeError") print(result) # Output: {"error": "RuntimeError"} ``` ```python from ecs_logging._utils import de_dot result = de_dot("error.stack_trace", "File 'app.py', line 42") print(result) # Output: {"error": {"stack_trace": "File 'app.py', line 42"}} ``` ```python from ecs_logging._utils import de_dot result = de_dot("http.request.headers.content_type", "application/json") print(result) # Output: # { # "http": { # "request": { # "headers": { # "content_type": "application/json" # } # } # } # } ``` -------------------------------- ### Optional Type Usage Examples Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/types.md Illustrates parameters that accept either None or a specific type. ```python stack_trace_limit: Optional[int] # Can be None, 0, positive, or negative extra: Optional[Dict[str, Any]] # Can be None or a dictionary exc_info: Optional[tuple] # In LogRecord, can be None or exception info tuple ``` -------------------------------- ### Merge dictionaries examples Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/api-reference/utility-functions.md Examples demonstrating deep merging of dictionaries and handling of type mismatch errors. ```python from ecs_logging._utils import merge_dicts into = { "log": {"level": "info"}, "message": "Hello" } from_dict = { "log": {"logger": "myapp"}, "service": {"name": "api"} } result = merge_dicts(from_dict, into) print(result) # Output: # { # "log": {"level": "info", "logger": "myapp"}, # "message": "Hello", # "service": {"name": "api"} # } ``` ```python from ecs_logging._utils import merge_dicts into = {"value": "scalar"} from_dict = {"value": {"nested": "dict"}} try: merge_dicts(from_dict, into) except TypeError as e: print(f"Error: {e}") # "Type mismatch at key `value`: merging dicts would replace value `scalar` with `{'nested': 'dict'}`..." ``` -------------------------------- ### Handle Configuration and Field Errors Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/quick-reference.md Examples of catching TypeErrors when providing invalid configuration parameters or conflicting extra fields during logging. ```python # Invalid parameter types try: formatter = ecs_logging.StdlibFormatter( stack_trace_limit="invalid" # Should be int or None ) except TypeError as e: print(f"Error: {e}") # Conflicting extra fields try: logger.info("test", extra={ "tls": "plaintext", # Conflict! "tls.cipher": "AES" # Can't have both }) except TypeError as e: print(f"Error: {e}") ``` -------------------------------- ### Example ECS-Formatted JSON Output Source: https://github.com/elastic/ecs-logging-python/blob/main/docs/reference/installation.md This is an example of a log message formatted according to the Elastic Common Schema (ECS). It includes timestamp, log level, message, and detailed ECS fields for HTTP and URL information. ```json { "@timestamp": "2020-03-26T13:08:11.728Z", "log.level": "info", "message": "Hello, 世界!", "ecs": { "version": "1.6.0" }, "http": { "request": { "bytes": 1337, "method": "get" }, "version": "2" }, "url": { "domain": "example.com", "original": "https://example.com", "path": "/", "port": 443, "registered_domain": "example.com", "scheme": "https", "top_level_domain": "com" } } ``` -------------------------------- ### ECS Log Output Example Source: https://github.com/elastic/ecs-logging-python/blob/main/docs/reference/installation.md Example of the JSON output generated by the ecs-logging library for a debug log message. ```json { "@timestamp": "2020-03-20T18:11:37.895Z", "log.level": "debug", "message": "Example message!", "ecs": { "version": "1.6.0" }, "http": { "request": { "method": "get" } }, "log": { "logger": "app", "origin": { "file": { "line": 14, "name": "test.py" }, "function": "func" }, "original": "Example message!" } } ``` -------------------------------- ### Initialize Formatter with Default Settings Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/configuration.md Illustrates the default configuration for StdlibFormatter and how to attach it to a standard logging handler. ```python import logging import ecs_logging # Minimal setup - all defaults formatter = ecs_logging.StdlibFormatter() # Equivalent to: # formatter = ecs_logging.StdlibFormatter( # fmt=None, # datefmt=None, # style="%", # validate=None, # stack_trace_limit=None, # All frames # extra=None, # No extra fields # exclude_fields=(), # Include all fields # ensure_ascii=True # ASCII-safe output # ) handler = logging.StreamHandler() handler.setFormatter(formatter) logger = logging.getLogger() logger.addHandler(handler) logger.setLevel(logging.INFO) logger.info("Default configuration") # Output includes all ECS fields with no exclusions ``` -------------------------------- ### Configure logging using environment variables Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/configuration.md Read configuration values from environment variables to initialize the StdlibFormatter. ```python import logging import ecs_logging import os # Recommended: read configuration from environment service_name = os.environ.get("SERVICE_NAME", "unknown") environment = os.environ.get("ENVIRONMENT", "development") log_level = os.environ.get("LOG_LEVEL", "INFO").upper() formatter = ecs_logging.StdlibFormatter( extra={ "service.name": service_name, "environment": environment } ) handler = logging.StreamHandler() handler.setFormatter(formatter) logger = logging.getLogger() logger.addHandler(handler) logger.setLevel(log_level) ``` -------------------------------- ### StdlibFormatter.__init__ Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/configuration.md Initializes the StdlibFormatter with configuration options for log formatting and metadata. ```APIDOC ## StdlibFormatter.__init__(fmt=None, datefmt=None, style="%", validate=None, stack_trace_limit=None, extra=None, exclude_fields=(), ensure_ascii=True) ### Description Initializes a new instance of the StdlibFormatter. Configuration is stateless and must be provided at instantiation. ### Parameters - **fmt** (Optional[str]) - Optional - Format string. - **datefmt** (Optional[str]) - Optional - Date format string. - **style** (Union[Literal["%"], Literal["{"], Literal["$"]]) - Optional - Format style. - **validate** (Optional[bool]) - Optional - Only passed to parent class if explicitly set (Python 3.8+). - **stack_trace_limit** (Optional[int]) - Optional - Stack trace frame limit. None = all, 0 = none, positive = from top, negative = last N frames. - **extra** (Optional[Dict[str, Any]]) - Optional - Global metadata fields added to all log records. - **exclude_fields** (Sequence[str]) - Optional - Field names or prefixes to exclude from output. - **ensure_ascii** (bool) - Optional - If True, escape non-ASCII characters. If False, include literal UTF-8. ``` -------------------------------- ### Importing ecs-logging components Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/module-overview.md Demonstrates how to import the main formatter classes and the ECS version constant from the package. ```python from ecs_logging import StdlibFormatter, StructlogFormatter, ECS_VERSION import ecs_logging # All are accessible formatter1 = ecs_logging.StdlibFormatter() formatter2 = ecs_logging.StructlogFormatter() print(ecs_logging.ECS_VERSION) # "1.6.0" ``` -------------------------------- ### Filebeat < 7.16 Log File Configuration Source: https://github.com/elastic/ecs-logging-python/blob/main/docs/reference/installation.md Configure Filebeat versions prior to 7.16 to parse ECS-formatted JSON logs from a file. This setup includes options for handling JSON keys and errors. ```yaml filebeat.inputs: - type: log paths: /path/to/logs.json json_keys_under_root: true json.overwrite_keys: true json.add_error_key: true json.expand_keys: true processors: - add_host_metadata: ~ - add_cloud_metadata: ~ - add_docker_metadata: ~ - add_kubernetes_metadata: ~ ``` -------------------------------- ### Performance tuning configurations Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/configuration.md Optimized formatter settings for different deployment scenarios including development, production, and high-volume logging. ```python import logging import ecs_logging formatter = ecs_logging.StdlibFormatter( stack_trace_limit=None, # Include all frames for debugging ensure_ascii=True ) ``` ```python import logging import ecs_logging formatter = ecs_logging.StdlibFormatter( stack_trace_limit=-20, # Last 20 frames only exclude_fields=[ "process.thread.id", "process.thread.name", "log.original" # Duplicate of message ], extra={ "service.name": "api-server", "environment": "production" }, ensure_ascii=True ) ``` ```python import logging import ecs_logging formatter = ecs_logging.StdlibFormatter( stack_trace_limit=0, # Suppress all stack traces exclude_fields=[ "process", # Exclude all process fields "error.stack_trace", "log.original" ], extra={ "service.name": "high-volume-service", "environment": "production" } ) ``` -------------------------------- ### Configure Structlog for ECS Logging Source: https://github.com/elastic/ecs-logging-python/blob/main/docs/reference/installation.md Configure structlog to use the ECS formatter, preserving non-ASCII characters. This setup is useful for applications that need to log messages with international characters in an ECS-compliant format. ```python structlog.configure( processors=[ecs_logging.StructlogFormatter(ensure_ascii=False)], wrapper_class=structlog.BoundLogger, context_class=dict, logger_factory=structlog.PrintLoggerFactory(), ) logger = structlog.get_logger("app") logger.info("Hello, 世界!") # Non-ASCII characters are preserved in output ``` -------------------------------- ### Project File Structure Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/README.md Visual representation of the project directory layout. ```text output/ ├── README.md (this file) ├── quick-reference.md Quick examples and API summary ├── module-overview.md Architecture and dependencies ├── types.md Type definitions ├── configuration.md Configuration options ├── errors.md Error reference ├── integration-patterns.md Real-world usage patterns └── api-reference/ ├── stdlib-formatter.md StdlibFormatter API ├── structlog-formatter.md StructlogFormatter API └── utility-functions.md Utility functions API ``` -------------------------------- ### Initialize StructlogFormatter Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/quick-reference.md Instantiate the StructlogFormatter with optional parameters and use its primary methods for formatting. ```python formatter = ecs_logging.StructlogFormatter( # Optional parameters: ensure_ascii=True, # ASCII-safe JSON ) # Main methods: json_string = formatter(logger, level_name, event_dict) # → str ecs_dict = formatter.format_to_ecs(event_dict) # → dict ``` -------------------------------- ### Combine utilities for custom ECS formatting Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/api-reference/utility-functions.md Demonstrates manual ECS event construction using de_dot and merge_dicts utilities. The resulting dictionary is serialized using json_dumps. ```python from ecs_logging._utils import de_dot, merge_dicts, json_dumps # Build ECS event manually ecs_event = {} # Add fields using de_dot merge_dicts(de_dot("@timestamp", "2024-08-31T12:34:56.789Z"), ecs_event) merge_dicts(de_dot("log.level", "info"), ecs_event) merge_dicts(de_dot("message", "Custom event"), ecs_event) merge_dicts(de_dot("service.name", "myapp"), ecs_event) # Serialize to JSON json_output = json_dumps(ecs_event) print(json_output) ``` -------------------------------- ### Configure with dictConfig Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/api-reference/stdlib-formatter.md Integrates the formatter into a standard Python logging dictionary configuration. ```python import logging.config import ecs_logging config = { "version": 1, "handlers": { "ecs": { "class": "logging.StreamHandler", "formatter": "ecs", "stream": "ext://sys.stdout" } }, "formatters": { "ecs": { "()": "ecs_logging.StdlibFormatter", "exclude_fields": ["process.thread.id"], "extra": {"service.name": "myapp"} } }, "root": { "handlers": ["ecs"], "level": "INFO" } } logging.config.dictConfig(config) logger = logging.getLogger() logger.info("Configured via dictConfig") ``` -------------------------------- ### Configure StdlibFormatter for Development Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/README.md Use this configuration to capture full stack traces and include all fields during local development. ```python formatter = ecs_logging.StdlibFormatter( stack_trace_limit=None, # All frames exclude_fields=[] # Include all fields ) ``` -------------------------------- ### StructlogFormatter.__init__ Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/api-reference/structlog-formatter.md Initializes the StructlogFormatter instance. ```APIDOC ## Constructor: StructlogFormatter(ensure_ascii=True) ### Description Initializes the formatter to convert structlog events into ECS JSON format. ### Parameters - **ensure_ascii** (bool) - Optional - If True, non-ASCII characters in output are escaped. If False, output contains literal UTF-8. Defaults to True. ``` -------------------------------- ### StdlibFormatter API and Configuration Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/quick-reference.md Initialize the StdlibFormatter with optional parameters and utilize its primary formatting methods. ```python formatter = ecs_logging.StdlibFormatter( # Optional parameters: fmt=None, # Format string (unused) datefmt=None, # Date format (unused) style="%", # Style ("%", "{", or "$") validate=None, # Validation (Python 3.8+) stack_trace_limit=None, # Frame limit (None=all, 0=none) extra={...}, # Global metadata dict exclude_fields=[...], # Fields to suppress ensure_ascii=True, # ASCII-safe JSON ) # Main methods: json_string = formatter.format(record) # → str ecs_dict = formatter.format_to_ecs(record) # → dict ``` -------------------------------- ### Implement general error recovery for StdlibFormatter Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/errors.md Use a try-except block to catch configuration errors during formatter initialization and fall back to default settings. ```python import logging import ecs_logging import sys try: formatter = ecs_logging.StdlibFormatter( stack_trace_limit=int(sys.argv[1]) if len(sys.argv) > 1 else None ) except (TypeError, ValueError) as e: print(f"Configuration error: {e}", file=sys.stderr) # Fall back to defaults formatter = ecs_logging.StdlibFormatter() handler = logging.StreamHandler() handler.setFormatter(formatter) logger = logging.getLogger() logger.addHandler(handler) ``` -------------------------------- ### Configure structlog with ECS formatter Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/api-reference/structlog-formatter.md Basic initialization of structlog using the ecs_logging.StructlogFormatter. ```python import structlog import ecs_logging structlog.configure( processors=[ecs_logging.StructlogFormatter()], wrapper_class=structlog.BoundLogger, context_class=dict, logger_factory=structlog.PrintLoggerFactory(), ) logger = structlog.get_logger() logger.info("Application started") logger.warning("High memory usage", memory_mb=2048) ``` -------------------------------- ### Configure StdlibFormatter via dictConfig Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/configuration.md Sets up the StdlibFormatter using a dictionary configuration, allowing for custom fields, field exclusion, and stack trace limits. ```python import logging.config import ecs_logging config = { "version": 1, "formatters": { "ecs": { "()": "ecs_logging.StdlibFormatter", "extra": { "service.name": "myapp", "environment": "production" }, "exclude_fields": ["process.thread.id"], "stack_trace_limit": -10, "ensure_ascii": False } }, "handlers": { "console": { "class": "logging.StreamHandler", "formatter": "ecs", "stream": "ext://sys.stdout" } }, "root": { "handlers": ["console"], "level": "INFO" } } logging.config.dictConfig(config) logger = logging.getLogger() logger.info("Configured via dictConfig") ``` -------------------------------- ### Create development logger configuration Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/integration-patterns.md Configures a verbose logger for debugging by including full stack traces and all available fields. ```python import logging import ecs_logging def create_development_logger(name: str) -> logging.Logger: """Create detailed logger for development""" formatter = ecs_logging.StdlibFormatter( # Include all stack trace frames for debugging stack_trace_limit=None, # Include all fields for inspection exclude_fields=[], # Add development context extra={ "service.name": "myapp", "environment": "development", "debug": True } ) handler = logging.StreamHandler() handler.setFormatter(formatter) logger = logging.getLogger(name) logger.addHandler(handler) logger.setLevel(logging.DEBUG) return logger # Usage logger = create_development_logger("myapp") logger.debug("Detailed debug info") logger.info("Information message") ``` -------------------------------- ### Configure Stdlib Logging with FileHandler Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/integration-patterns.md Directs ECS-formatted logs to a specific file on the filesystem. ```python import logging import ecs_logging # Write to file handler = logging.FileHandler("app.log") handler.setFormatter(ecs_logging.StdlibFormatter()) logger = logging.getLogger("myapp") logger.addHandler(handler) logger.setLevel(logging.DEBUG) logger.debug("Debug message") logger.info("Info message") ``` -------------------------------- ### Configure Structlog with ECS Formatter Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/api-reference/structlog-formatter.md Demonstrates how to integrate the ECS formatter into structlog configuration and handle custom objects using the __structlog__ method. ```python import structlog import ecs_logging class Duration: def __init__(self, ms): self.ms = ms def __structlog__(self): return f"{self.ms}ms" structlog.configure( processors=[ecs_logging.StructlogFormatter()], wrapper_class=structlog.BoundLogger, context_class=dict, logger_factory=structlog.PrintLoggerFactory(), ) logger = structlog.get_logger() logger.info("Request completed", duration=Duration(45)) # Output includes: "duration":"45ms" ``` -------------------------------- ### Configure ECS Formatters Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/module-overview.md Initialize StdlibFormatter or StructlogFormatter using constructor-based configuration. ```python # StdlibFormatter configuration formatter = StdlibFormatter( stack_trace_limit=None, # Stack trace frames extra={...}, # Global metadata exclude_fields=[...], # Fields to suppress ensure_ascii=True # ASCII encoding ) # StructlogFormatter configuration formatter = StructlogFormatter( ensure_ascii=True # ASCII encoding ) ``` -------------------------------- ### Configure StructlogFormatter Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/configuration.md Initializes the StructlogFormatter with default settings or UTF-8 support and integrates it into the structlog configuration. ```python import structlog import ecs_logging # Default configuration formatter = ecs_logging.StructlogFormatter() # With UTF-8 support formatter = ecs_logging.StructlogFormatter(ensure_ascii=False) structlog.configure( processors=[formatter], wrapper_class=structlog.BoundLogger, context_class=dict, logger_factory=structlog.PrintLoggerFactory(), ) ``` -------------------------------- ### Visualize Module Dependencies Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/API-INDEX.md Displays the internal dependency tree for the ecs-logging package, showing reliance on standard library modules. ```text ecs_logging ├─ logging (stdlib) ├─ time (stdlib) ├─ sys (stdlib) ├─ json (stdlib) ├─ traceback (stdlib) ├─ functools (stdlib) ├─ collections.abc (stdlib) └─ datetime (stdlib) ``` -------------------------------- ### Importing Type Hints for StdlibFormatter Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/types.md Standard type hint imports including compatibility fallbacks for Python versions older than 3.8. ```python from typing import Any, Callable, Dict, Optional, Sequence, Union try: from typing import Literal # Python 3.8+ except ImportError: from typing_extensions import Literal # Fallback for Python <3.8 ``` -------------------------------- ### Configure logging via fileConfig Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/configuration.md Use standard Python logging configuration files to define the ECS formatter. ```ini [loggers] keys=root [handlers] keys=console [formatters] keys=ecs [logger_root] level=INFO handlers=console [handler_console] class=StreamHandler args=(sys.stdout,) formatter=ecs [formatter_ecs] class=ecs_logging.StdlibFormatter kwargs={"extra": {"service.name": "myapp"}, "exclude_fields": ["process"]} ``` ```python import logging.config logging.config.fileConfig("logging.conf") logger = logging.getLogger() logger.info("Configured via fileConfig") ``` -------------------------------- ### Initialize Logger with Factory Function Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/integration-patterns.md Uses a factory function to dynamically configure loggers with environment-specific metadata and field exclusions. ```python import logging import ecs_logging import sys import os def setup_logger( name: str, level: str = "INFO", service_name: str = "myapp", environment: str = "development" ) -> logging.Logger: """Create and configure an ECS logger""" # Determine exclude fields based on environment exclude_fields = [] if environment == "production": exclude_fields = [ "process.thread.id", "process.thread.name", "log.original" ] # Create formatter formatter = ecs_logging.StdlibFormatter( extra={ "service.name": service_name, "environment": environment, "service.version": os.environ.get("SERVICE_VERSION", "unknown") }, exclude_fields=exclude_fields, stack_trace_limit=-20 if environment == "production" else None ) # Create handler handler = logging.StreamHandler() handler.setFormatter(formatter) # Configure logger logger = logging.getLogger(name) logger.addHandler(handler) logger.setLevel(getattr(logging, level.upper(), logging.INFO)) return logger # Usage logger = setup_logger( name="myapp", level="INFO", service_name="api-server", environment="production" ) logger.info("Service started") ``` -------------------------------- ### Configure Logging with dictConfig Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/quick-reference.md Use this configuration to integrate ECS formatting into the standard Python logging system. ```python import logging.config config = { "version": 1, "formatters": { "ecs": { "()": "ecs_logging.StdlibFormatter", "extra": {"service.name": "myapp"}, "exclude_fields": ["process.thread.id"], } }, "handlers": { "console": { "class": "logging.StreamHandler", "formatter": "ecs" } }, "root": { "level": "INFO", "handlers": ["console"] } } logging.config.dictConfig(config) ``` -------------------------------- ### Configure StdlibFormatter for Production Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/quick-reference.md Configure the StdlibFormatter for production environments by setting stack trace limits and excluding specific fields. ```python import logging import ecs_logging formatter = ecs_logging.StdlibFormatter( stack_trace_limit=-20, exclude_fields=["process.thread.id", "log.original"], extra={ "service.name": "my-api", "environment": "production" } ) ``` -------------------------------- ### Configure Centralized Logging with HTTPHandler Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/integration-patterns.md Sets up a logger with both a console stream handler and an HTTP handler for sending ECS-formatted logs to an Elasticsearch endpoint. ```python import logging import logging.handlers import ecs_logging class HTTPHandler(logging.handlers.HTTPHandler): """Send logs to HTTP endpoint""" pass # Configure multiple handlers def setup_logging(): formatter = ecs_logging.StdlibFormatter( extra={"service.name": "myapp"} ) # Console output console_handler = logging.StreamHandler() console_handler.setFormatter(formatter) # HTTP to Elasticsearch http_handler = logging.handlers.HTTPHandler( "localhost:9200", "/logs/myapp/_doc", method="POST" ) http_handler.setFormatter(formatter) logger = logging.getLogger("myapp") logger.addHandler(console_handler) logger.addHandler(http_handler) logger.setLevel(logging.INFO) return logger logger = setup_logging() logger.info("Message sent to both console and Elasticsearch") ``` -------------------------------- ### Create production logger configuration Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/integration-patterns.md Configures a high-throughput logger by disabling stack traces and excluding verbose fields. ```python import logging import ecs_logging import os def create_production_logger(name: str) -> logging.Logger: """Create optimized logger for high-volume production""" formatter = ecs_logging.StdlibFormatter( # Don't include stack traces for performance stack_trace_limit=0, # Exclude verbose fields exclude_fields=[ "process", # All process fields "error.stack_trace", # Stack traces "log.original" # Duplicate of message ], # Include only essential metadata extra={ "service.name": os.environ.get("SERVICE_NAME", "unknown"), "environment": "production", "instance": os.environ.get("POD_NAME", "unknown") } ) handler = logging.StreamHandler() handler.setFormatter(formatter) logger = logging.getLogger(name) logger.addHandler(handler) logger.setLevel(logging.INFO) return logger # Usage logger = create_production_logger("myapp") logger.info("Event 1") logger.info("Event 2") # Minimal JSON output, maximum throughput ``` -------------------------------- ### Basic Standard Library Logging Source: https://github.com/elastic/ecs-logging-python/blob/main/docs/reference/installation.md Configure Python's standard library logging to use ECS formatting. Ensure the logger is set to an appropriate level and an ECS formatter is added to the handler. ```python import logging import ecs_logging # Get the Logger logger = logging.getLogger("app") logger.setLevel(logging.DEBUG) # Add an ECS formatter to the Handler handler = logging.StreamHandler() handler.setFormatter(ecs_logging.StdlibFormatter()) logger.addHandler(handler) # Emit a log! logger.debug("Example message!", extra={"http.request.method": "get"}) ``` -------------------------------- ### Type Hints for StdlibFormatter Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/quick-reference.md Reference for the expected types and signatures used within the StdlibFormatter. ```python from typing import Dict, Any, Optional, Sequence # StdlibFormatter parameters fmt: Optional[str] datefmt: Optional[str] style: Union[Literal["%"], Literal["{"], Literal["$"]] validate: Optional[bool] stack_trace_limit: Optional[int] extra: Optional[Dict[str, Any]] exclude_fields: Sequence[str] ensure_ascii: bool # Return types format(record: LogRecord) -> str format_to_ecs(record: LogRecord) -> Dict[str, Any] ``` -------------------------------- ### Correct stack_trace_limit Usage Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/errors.md Shows valid configurations for the stack_trace_limit parameter in StdlibFormatter. ```python import ecs_logging # All valid formatter = ecs_logging.StdlibFormatter(stack_trace_limit=None) # All frames formatter = ecs_logging.StdlibFormatter(stack_trace_limit=0) # No traces formatter = ecs_logging.StdlibFormatter(stack_trace_limit=10) # First 10 frames formatter = ecs_logging.StdlibFormatter(stack_trace_limit=-10) # Last 10 frames ``` -------------------------------- ### Log exceptions with StdlibFormatter Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/integration-patterns.md Uses exc_info=True to capture exception details, type, and stack trace in ECS format. ```python import logging import ecs_logging handler = logging.StreamHandler() handler.setFormatter(ecs_logging.StdlibFormatter()) logger = logging.getLogger("myapp") logger.addHandler(handler) logger.setLevel(logging.DEBUG) try: result = 1 / 0 except ZeroDivisionError: # Includes exception type, message, and stack trace logger.error("Calculation failed", exc_info=True) ``` -------------------------------- ### Use bound loggers in application classes Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/api-reference/structlog-formatter.md Initialize a logger within a class instance to automatically include specific context like request IDs. ```python import structlog import ecs_logging structlog.configure( processors=[ecs_logging.StructlogFormatter()], wrapper_class=structlog.BoundLogger, context_class=dict, logger_factory=structlog.PrintLoggerFactory(), ) class RequestHandler: def __init__(self, request_id): self.logger = structlog.get_logger().bind(request_id=request_id) def handle(self): self.logger.info("Processing request") # All logs include request_id automatically self.logger.info("Validating input") self.logger.info("Request complete") ``` -------------------------------- ### Configure ECS logging with Elastic APM Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/integration-patterns.md Initializes the Elastic APM client and configures the standard library logger with the ECS formatter to enable automatic APM field inclusion. ```python import logging import ecs_logging from elasticapm import Client from elasticapm.instrumentation.packages.base import get_module # Initialize Elastic APM apm_client = Client( service_name="my-api", server_url="http://localhost:8200" ) # Configure logging with ECS formatter handler = logging.StreamHandler() handler.setFormatter( ecs_logging.StdlibFormatter( extra={ "service.name": "my-api" } ) ) logger = logging.getLogger("myapp") logger.addHandler(handler) logger.setLevel(logging.INFO) # Application code def handle_request(): transaction = apm_client.begin_transaction("request") try: # APM fields are automatically included in logs logger.info( "Processing request", extra={ "http.method": "GET", "http.url": "/api/data" } ) result = process_data() logger.info("Request successful", http_status_code=200) transaction.result = "success" except Exception: logger.error("Request failed", exc_info=True) transaction.result = "error" raise finally: apm_client.end_transaction() def process_data(): # Span-level logging with apm_client.capture_span("data_processing"): logger.info("Starting data processing") # ... processing code ... logger.info("Data processing completed") return "result" ``` -------------------------------- ### Configure StdlibFormatter for High-Volume Logging Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/quick-reference.md Optimize logging for high-volume services by disabling stack traces and excluding heavy fields. ```python formatter = ecs_logging.StdlibFormatter( stack_trace_limit=0, exclude_fields=["process", "error.stack_trace", "log.original"], extra={"service.name": "high-volume-svc"} ) ``` -------------------------------- ### Configure Global Metadata with Extra Fields Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/configuration.md Add global metadata to every log record using the extra parameter. Keys support both nested dictionaries and dot notation. ```python import logging import ecs_logging formatter = ecs_logging.StdlibFormatter( extra={ "service.name": "my-api", "service.version": "1.0.0", "environment": "production", "datacenter": "us-east-1" } ) handler = logging.StreamHandler() handler.setFormatter(formatter) logger = logging.getLogger() logger.addHandler(handler) # All logs now include these fields logger.info("Application started") ``` ```python formatter = ecs_logging.StdlibFormatter( extra={ "service": { "name": "my-api", "version": "1.0.0" }, "environment": "production" } ) ``` ```python formatter = ecs_logging.StdlibFormatter( extra={ "service.name": "my-api", "service.version": "1.0.0", "environment": "production" } ) ``` -------------------------------- ### Integrate structlog with stdlib logging Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/integration-patterns.md Configures structlog to wrap stdlib logging, allowing both systems to share the ECS formatter. ```python import logging import logging.config import structlog import ecs_logging # Configure stdlib logging with ECS logging_config = { "version": 1, "formatters": { "ecs": { "()": "ecs_logging.StdlibFormatter", "extra": {"service.name": "myapp"} } }, "handlers": { "default": { "level": "DEBUG", "class": "logging.StreamHandler", "formatter": "ecs", } }, "root": { "level": "DEBUG", "handlers": ["default"] } } logging.config.dictConfig(logging_config) # Configure structlog to use stdlib logging structlog.configure( processors=[ structlog.stdlib.ProcessorFormatter.wrap_for_formatter, ], context_class=dict, logger_factory=structlog.stdlib.LoggerFactory(), ) log = structlog.get_logger() log.info("Integrated logging setup", feature="both_stdlib_and_structlog") ``` -------------------------------- ### Validate Formatter Parameter Types Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/configuration.md Demonstrates how invalid parameter types for stack_trace_limit and exclude_fields trigger TypeErrors during initialization. ```python import ecs_logging # TypeError: stack_trace_limit must be None or int try: ecs_logging.StdlibFormatter(stack_trace_limit="all") except TypeError as e: print(f"Error: {e}") # TypeError: exclude_fields must be sequence of strings try: ecs_logging.StdlibFormatter(exclude_fields={"process"}) # Set, not sequence except TypeError as e: print(f"Error: {e}") ``` -------------------------------- ### Integrating Structlog with Elastic APM Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/api-reference/structlog-formatter.md Shows how to include Elastic APM trace, transaction, and span IDs in structlog output. ```python import structlog import ecs_logging from elasticapm.instrumentation import get_transaction structlog.configure( processors=[ecs_logging.StructlogFormatter()], wrapper_class=structlog.BoundLogger, context_class=dict, logger_factory=structlog.PrintLoggerFactory(), ) logger = structlog.get_logger() # Get current APM transaction and include its fields transaction = get_transaction() if transaction: logger.info("Processing request", trace_id=transaction.trace_id, transaction_id=transaction.id, span_id=transaction.current_span.id if transaction.current_span else None ) ``` -------------------------------- ### Implement Per-Request Logger with ECS Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/integration-patterns.md Uses a wrapper class to manage request-scoped context and injects it into standard Python logging via ECS formatters. Requires the ecs_logging library and a configured base logger. ```python import logging import ecs_logging from contextlib import contextmanager class RequestContextLogger: """Logger that includes request context in all messages""" def __init__(self, base_logger: logging.Logger): self.base_logger = base_logger self.context = {} def set_context(self, **kwargs): """Set context that applies to all subsequent logs""" self.context.update(kwargs) def clear_context(self): """Clear all context""" self.context.clear() def info(self, msg, **kwargs): extra = {**self.context, **kwargs} self.base_logger.info(msg, extra=extra) def error(self, msg, exc_info=False, **kwargs): extra = {**self.context, **kwargs} self.base_logger.error(msg, exc_info=exc_info, extra=extra) @contextmanager def context_scope(self, **kwargs): """Context manager for request-scoped context""" old_context = self.context.copy() self.context.update(kwargs) try: yield finally: self.context = old_context # Setup base logger base_handler = logging.StreamHandler() base_handler.setFormatter( ecs_logging.StdlibFormatter( extra={"service.name": "api"} ) ) base_logger = logging.getLogger("api") base_logger.addHandler(base_handler) # Usage in request handler request_logger = RequestContextLogger(base_logger) with request_logger.context_scope( request_id="req-123", user_id=456, path="/api/users" ): request_logger.info("Request received") # All logs in this scope include request context request_logger.info("Validating input") request_logger.info("Processing complete") ``` -------------------------------- ### Validate logger configuration Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/errors.md Implement validation logic for configuration parameters before initializing the StdlibFormatter to prevent runtime errors. ```python import logging import ecs_logging def create_logger(config): """Create logger with validated configuration""" try: # Validate parameters if not isinstance(config.get("exclude_fields"), (list, tuple, type(None))): raise TypeError("exclude_fields must be a list or tuple") if config.get("stack_trace_limit") is not None: if not isinstance(config["stack_trace_limit"], int): raise TypeError("stack_trace_limit must be an integer or None") # Create formatter formatter = ecs_logging.StdlibFormatter( exclude_fields=config.get("exclude_fields", []), stack_trace_limit=config.get("stack_trace_limit"), extra=config.get("extra") ) handler = logging.StreamHandler() handler.setFormatter(formatter) logger = logging.getLogger(config.get("name", __name__)) logger.addHandler(handler) logger.setLevel(config.get("level", "INFO")) return logger except TypeError as e: print(f"Configuration error: {e}") raise # Usage logger = create_logger({ "name": "myapp", "level": "DEBUG", "exclude_fields": ["process.thread.id"], "stack_trace_limit": -20, "extra": {"service.name": "api"} }) ``` -------------------------------- ### StdlibFormatter Constructor Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/API-INDEX.md Constructor signature for the StdlibFormatter class, which inherits from logging.Formatter. ```python def __init__( self, fmt: Optional[str] = None, datefmt: Optional[str] = None, style: Union[Literal["%"], Literal["{"], Literal["$"]] = "%", validate: Optional[bool] = None, stack_trace_limit: Optional[int] = None, extra: Optional[Dict[str, Any]] = None, exclude_fields: Sequence[str] = (), ensure_ascii: bool = True, ) -> None ``` -------------------------------- ### Define __call__ method Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/api-reference/structlog-formatter.md The __call__ method serves as the primary entry point for the structlog processor interface. ```python def __call__( self, _: Any, name: str, event_dict: Dict[str, Any] ) -> str: ``` -------------------------------- ### Configure StdlibFormatter for Filebeat Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/README.md Configures the standard library logging handler to output ECS-formatted JSON for Filebeat ingestion. ```python # Configure logging to file handler = logging.FileHandler("app.log") handler.setFormatter(ecs_logging.StdlibFormatter()) logger.addHandler(handler) # Filebeat configuration picks up JSON lines from app.log # and ships them to Elasticsearch automatically ``` -------------------------------- ### Expand Dotted Keys in Python Dictionaries Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/module-overview.md Demonstrates the transformation of flat dictionaries with dotted keys into nested structures using normalize_dict(). ```python { "log.level": "info", "error.type": "ValueError", "custom.nested.field": "value" } ``` ```python { "log": {"level": "info"}, "error": {"type": "ValueError"}, "custom": {"nested": {"field": "value"}} } ``` -------------------------------- ### Handle Invalid Formatter Configuration Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/README.md Demonstrates catching a TypeError when providing an invalid value to the stack_trace_limit parameter. ```python try: formatter = ecs_logging.StdlibFormatter(stack_trace_limit="all") except TypeError as e: print(f"Config error: {e}") # Fix: use None, 0, or integer value ``` -------------------------------- ### ecs_logging.StdlibFormatter(extra=None, exclude_fields=None, stack_trace_limit=None) Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/api-reference/stdlib-formatter.md Initializes the formatter to convert standard library log records into ECS JSON format. ```APIDOC ### Constructor `ecs_logging.StdlibFormatter(extra=None, exclude_fields=None, stack_trace_limit=None)` ### Parameters - **extra** (dict) - Optional - A dictionary of fields to include in every log record. - **exclude_fields** (list) - Optional - A list of field names to exclude from the output. - **stack_trace_limit** (int) - Optional - The maximum number of stack trace frames to include in the log output. ### Usage Example ```python import logging import ecs_logging handler = logging.StreamHandler() handler.setFormatter( ecs_logging.StdlibFormatter( extra={"service.name": "api-server"}, exclude_fields=["process.name"], stack_trace_limit=10 ) ) ``` ``` -------------------------------- ### Bind context variables to logs Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/api-reference/structlog-formatter.md Use logger.bind to attach persistent context variables that appear in all subsequent log entries. ```python import structlog import ecs_logging structlog.configure( processors=[ecs_logging.StructlogFormatter()], wrapper_class=structlog.BoundLogger, context_class=dict, logger_factory=structlog.PrintLoggerFactory(), ) logger = structlog.get_logger() # Bind context that applies to all logs bound_logger = logger.bind( service="api", environment="production", version="1.0.0" ) # Context variables automatically included in all logs bound_logger.info("Request received", request_id="req-123") bound_logger.error("Processing failed", request_id="req-123", error_code="DB_TIMEOUT") ``` -------------------------------- ### Accessing ECS_VERSION Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/module-overview.md Shows the two ways to import the ECS version constant, either directly from the meta module or via the public API. ```python from ecs_logging._meta import ECS_VERSION # Or via public API: from ecs_logging import ECS_VERSION ``` -------------------------------- ### Utility Functions Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/quick-reference.md Helper functions provided by the library for dictionary manipulation and JSON serialization. ```APIDOC ## Utility Functions ### de_dot(key, value) Converts a dotted key string into a nested dictionary structure. ### normalize_dict(data) Expands all dotted keys within a dictionary into nested structures. ### merge_dicts(source, destination) Performs a deep merge of two dictionaries. ### json_dumps(data) Serializes a dictionary to a JSON string, ensuring ECS field ordering (e.g., @timestamp first). ``` -------------------------------- ### StructlogFormatter Class Interface Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/types.md Type hints for the StructlogFormatter class methods and constructor. ```python from typing import Any, Dict class StructlogFormatter: def __init__(self, ensure_ascii: bool = True) -> None: ... def __call__( self, _: Any, name: str, event_dict: Dict[str, Any] ) -> str: ... def format_to_ecs(self, event_dict: Dict[str, Any]) -> Dict[str, Any]: ... ``` -------------------------------- ### Log exceptions with custom context Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/integration-patterns.md Adds custom metadata to exception logs using the extra parameter and configures stack trace limits. ```python import logging import ecs_logging import sys import traceback handler = logging.StreamHandler() handler.setFormatter(ecs_logging.StdlibFormatter(stack_trace_limit=-20)) logger = logging.getLogger("myapp") logger.addHandler(handler) try: # Some operation result = risky_operation() except Exception as e: # Log with custom context logger.error( f"Operation failed: {type(e).__name__}", exc_info=True, extra={ "operation": "risky_operation", "error_code": "OP_FAILED", "retry_count": 3, "attempt": 1 } ) ``` -------------------------------- ### Resolving Key Conflicts with Consistent Nesting Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/errors.md Shows how to avoid TypeErrors by using either fully nested dictionaries or fully flat dotted keys. ```python import logging import ecs_logging formatter = ecs_logging.StdlibFormatter() handler = logging.StreamHandler() handler.setFormatter(formatter) logger = logging.getLogger() logger.addHandler(handler) # Option 1: Use nested dictionaries logger.info("test", extra={ "tls": { "cipher": "AES", "established": True } }) # Option 2: Use flat dotted keys (but don't mix) logger.info("test", extra={ "tls.cipher": "AES", "tls.established": True }) # NOT this - mixing nested and flat # logger.info("test", extra={"tls": {...}, "tls.other": ...}) ``` -------------------------------- ### StructlogFormatter Constructor Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/API-INDEX.md Constructor signature for the StructlogFormatter class. ```python def __init__( self, ensure_ascii: bool = True, ) -> None ``` -------------------------------- ### Configure Filebeat for ECS Logging Source: https://github.com/elastic/ecs-logging-python/blob/main/_autodocs/quick-reference.md Use this configuration to ingest JSON logs from a file into Elasticsearch via Filebeat. ```yaml filebeat.inputs: - type: log paths: - /var/log/myapp.log json.message_key: message output.elasticsearch: hosts: ["localhost:9200"] ```