### Install dev dependencies and run tests Source: https://github.com/babelqueue/babelqueue-python/blob/main/README.md Install the package with development dependencies and run the test suite. The commented alternative runs the tests without external dependencies using Python's built-in unittest module. ```bash pip install -e ".[dev]" pytest # (or, dependency-free) python -m unittest discover -s tests ``` -------------------------------- ### Install BabelQueue with Redis or AMQP extras Source: https://github.com/babelqueue/babelqueue-python/blob/main/README.md Install the babelqueue package with the redis extra for redis:// URLs or the amqp extra for amqp:// (RabbitMQ) URLs. ```bash pip install "babelqueue[redis]" # redis:// pip install "babelqueue[amqp]" # amqp:// (RabbitMQ) ``` -------------------------------- ### Install BabelQueue with pip Source: https://github.com/babelqueue/babelqueue-python/blob/main/README.md Install the babelqueue package. Requires Python >=3.9. ```bash pip install babelqueue ``` -------------------------------- ### Django adapter: publish and handler registration Source: https://github.com/babelqueue/babelqueue-python/blob/main/README.md Use `publish` in a view or signal and register handlers at startup with `get_app().handler`. Requires the Django adapter installed and configured. ```python from babelqueue.django import publish, get_app publish("urn:babel:orders:created", {"order_id": 1042}) # in a view / signal @get_app().handler("urn:babel:orders:created") # register handlers at startup def on_created(data, meta): ... ``` -------------------------------- ### Celery adapter: from_celery and install_worker Source: https://github.com/babelqueue/babelqueue-python/blob/main/README.md Reuse your Celery app's broker for polyglot interop. Install with `pip install "babelqueue[celery]"`. `from_celery` creates a runtime on Celery's broker; `install_worker` makes `celery worker` also drain URN messages. ```python from babelqueue.celery import from_celery, install_worker bq = from_celery(celery_app, queue="orders") # runtime on Celery's broker bq.publish("urn:babel:orders:created", {"order_id": 1042}) @bq.handler("urn:babel:orders:created") def on_created(data, meta): ... install_worker(celery_app, bq) # `celery worker` also drains URN messages ``` -------------------------------- ### BabelQueue end-to-end example with Redis or AMQP Source: https://github.com/babelqueue/babelqueue-python/blob/main/README.md Shows how to create a BabelQueue app, register a handler, publish a message, and run a worker. The handler receives (data, meta) and the example uses a redis:// URL; the commented line shows the amqp:// alternative. The worker consumes forever until Ctrl-C. ```python from babelqueue import BabelQueue app = BabelQueue("redis://localhost:6379/0", queue="orders") # or: BabelQueue("amqp://guest:guest@localhost:5672/", queue="orders") @app.handler("urn:babel:orders:created") def on_order_created(data, meta): # AI/ML, data processing, anything print("order", data["order_id"]) # producer (any service, any language) … app.publish("urn:babel:orders:created", {"order_id": 1042}) # worker app.run() # consume forever (Ctrl-C to stop) ``` -------------------------------- ### Enable OpenTelemetry tracing with otel.wrap_handler and otel.publish Source: https://github.com/babelqueue/babelqueue-python/blob/main/README.md Install the optional 'babelqueue[otel]' extra to use the babelqueue.otel module. The core never imports OpenTelemetry, so it stays zero-dependency. The producer emits a PRODUCER span per publish and the consumer emits a CONSUMER span per handled message, correlated via trace_id (v0.1) and W3C traceparent (v0.2). The traceparent rides the out-of-band transport-header seam (publish_with_headers / headers_from_context), keeping the envelope frozen (schema_version: 1). ```python from opentelemetry import trace from babelqueue import BabelQueue, otel tracer = trace.get_tracer("orders") app = BabelQueue("redis://localhost:6379/0", queue="orders") # consumer: wrap_handler starts a CONSUMER span (child of the producer span when a # traceparent rode along; else in the trace_id-derived trace) app.register("urn:babel:orders:created", otel.wrap_handler(tracer, on_order_created)) # producer: otel.publish starts a PRODUCER span and carries traceparent + trace_id otel.publish(tracer, app, "urn:babel:orders:created", {"order_id": 1042}) ``` -------------------------------- ### Canonical envelope JSON example Source: https://github.com/babelqueue/babelqueue-python/blob/main/README.md Shows the JSON structure of a BabelQueue envelope, identical across all SDKs. ```json { "job": "urn:babel:orders:created", "trace_id": "…", "data": { "order_id": 1042 }, "meta": { "id": "…", "queue": "default", "lang": "python", "schema_version": 1, "created_at": 1749132727000 }, "attempts": 0 } ``` -------------------------------- ### Producer meta with extension metadata Source: https://github.com/babelqueue/babelqueue-python/blob/main/README.md Shows how to add extension metadata (recommended shape: meta.ext.) when creating an envelope. The meta parameter is keyword-only; colliding with core or forbidden keys raises EnvelopeKeyError. ```python # Producer meta: extra keys next to the core meta keys (recommended shape: meta.ext.). envelope = EnvelopeCodec.make( "urn:babel:orders:created", {"order_id": 1042}, queue="orders", meta={"ext": {"acme": {"tier": "gold"}}}, ) ``` -------------------------------- ### Produce and consume messages with EnvelopeCodec Source: https://github.com/babelqueue/babelqueue-python/blob/main/README.md Shows how to build a canonical envelope, encode it to JSON, and decode an incoming message. The decoded envelope provides job, data, and trace_id fields. ```python from babelqueue import EnvelopeCodec # Produce — build the canonical envelope and publish the JSON to your broker. envelope = EnvelopeCodec.make("urn:babel:orders:created", {"order_id": 1042}) body = EnvelopeCodec.encode(envelope) # -> UTF-8 JSON string # redis.rpush("queues:orders", body) / channel.basic_publish(body=body, ...) # Consume — decode a message produced by ANY BabelQueue SDK. incoming = EnvelopeCodec.decode(body) urn = incoming["job"] # "urn:babel:orders:created" data = incoming["data"] # {"order_id": 1042} trace_id = incoming["trace_id"] # correlate across services ``` -------------------------------- ### Django adapter: run consumer with babelqueue_worker Source: https://github.com/babelqueue/babelqueue-python/blob/main/README.md Run the consumer with the Django management command `babelqueue_worker`. Specify the queue with `--queue orders`. ```bash python manage.py babelqueue_worker --queue orders # run the consumer ``` -------------------------------- ### EnvelopeCodec.make Source: https://github.com/babelqueue/babelqueue-python/blob/main/README.md Creates a canonical BabelQueue envelope from a job URN and data payload. Supports optional queue and extension metadata. ```APIDOC ## EnvelopeCodec.make ### Description Creates a canonical BabelQueue envelope from a job URN and data payload. Supports optional queue and extension metadata. ### Method make ### Signature `EnvelopeCodec.make(job: str, data: dict, *, queue: str = "default", meta: dict = None) -> Envelope` ### Parameters - **job** (str) - Required - The job URN, e.g., `urn:babel:orders:created`. - **data** (dict) - Required - The payload data. - **queue** (str) - Optional - Queue name, defaults to `"default"`. - **meta** (dict) - Optional - Keyword-only. Extension metadata, recommended shape `{"ext": {"": {...}}}`. Keys colliding with core meta keys (`id`, `queue`, `lang`, `schema_version`, `created_at`) or forbidden keys (`max_retries`, `attempts`, `source`, `ts`) raise `EnvelopeKeyError`. ### Returns - **Envelope** - The constructed envelope object. ### Example ```python from babelqueue import EnvelopeCodec envelope = EnvelopeCodec.make("urn:babel:orders:created", {"order_id": 1042}) ``` ``` -------------------------------- ### Dead-letter with dead_letter.annotate Source: https://github.com/babelqueue/babelqueue-python/blob/main/README.md Shows how to annotate an envelope for dead-lettering. The resulting envelope can be encoded and published to the 'orders.dlq' queue. ```python from babelqueue import dead_letter dlq = dead_letter.annotate(envelope, "failed", "orders", attempts=3, error="boom") # publish `EnvelopeCodec.encode(dlq)` to the "orders.dlq" queue ``` -------------------------------- ### AES-GCM Cipher Implementation for babelqueue.gdpr Source: https://github.com/babelqueue/babelqueue-python/blob/main/README.md Caller-provided cipher implementing the Cipher Protocol using AES-256-GCM via the optional cryptography library. Requires `pip install "babelqueue[gdpr]"`; the key is the caller's responsibility (16/24/32 bytes). Encrypt returns base64-encoded nonce+ciphertext; decrypt raises on wrong key or tampering. ```python import base64, os from cryptography.hazmat.primitives.ciphers.aead import AESGCM class AesGcmCipher: def __init__(self, key: bytes) -> None: # 16/24/32-byte key — the CALLER's self._aead = AESGCM(key) def encrypt(self, plaintext: bytes) -> str: nonce = os.urandom(12) return base64.b64encode(nonce + self._aead.encrypt(nonce, plaintext, None)).decode("ascii") def decrypt(self, ciphertext: str) -> bytes: raw = base64.b64decode(ciphertext) return self._aead.decrypt(raw[:12], raw[12:], None) # raises on wrong key / tamper ``` -------------------------------- ### EnvelopeCodec.from_message Source: https://github.com/babelqueue/babelqueue-python/blob/main/README.md Creates an envelope from a PolyglotMessage-compatible object. The object must implement `get_babel_urn` and `to_payload` methods. ```APIDOC ## EnvelopeCodec.from_message ### Description Creates an envelope from a PolyglotMessage-compatible object. The object must implement `get_babel_urn` and `to_payload` methods. ### Method from_message ### Signature `EnvelopeCodec.from_message(message: PolyglotMessage, *, queue: str = "default") -> Envelope` ### Parameters - **message** (PolyglotMessage) - Required - An object implementing `get_babel_urn()` and `to_payload()`. - **queue** (str) - Optional - Queue name, defaults to `"default"`. ### Returns - **Envelope** - The constructed envelope. ### Example ```python from babelqueue import EnvelopeCodec, PolyglotMessage class OrderCreated: def __init__(self, order_id: int): self.order_id = order_id def get_babel_urn(self) -> str: return "urn:babel:orders:created" def to_payload(self) -> dict: return {"order_id": self.order_id} envelope = EnvelopeCodec.from_message(OrderCreated(1042), queue="orders") ``` ``` -------------------------------- ### Django adapter: settings.py BABELQUEUE configuration Source: https://github.com/babelqueue/babelqueue-python/blob/main/README.md Add `"babelqueue.django"` to `INSTALLED_APPS` and configure a `BABELQUEUE` dict. This block shows the settings.py configuration with broker URL, queue name, and dead_letter flag. ```python # settings.py BABELQUEUE = {"broker_url": "redis://localhost:6379/0", "queue": "orders", "dead_letter": True} ``` -------------------------------- ### dead_letter.annotate Source: https://github.com/babelqueue/babelqueue-python/blob/main/README.md Annotates an envelope with dead-letter metadata for retry, redrive, or dead-lettering. Returns a new envelope ready for encoding. ```APIDOC ## dead_letter.annotate ### Description Annotates an envelope with dead-letter metadata for retry, redrive, or dead-lettering. Returns a new envelope ready for encoding. ### Method annotate ### Signature `dead_letter.annotate(envelope: Envelope, reason: str, queue: str, *, attempts: int = 0, error: str = None) -> Envelope` ### Parameters - **envelope** (Envelope) - Required - The original envelope. - **reason** (str) - Required - The reason for dead-lettering. - **queue** (str) - Required - The original queue name. - **attempts** (int) - Optional - Number of attempts, defaults to 0. - **error** (str) - Optional - Error message. ### Returns - **Envelope** - The annotated envelope. ### Example ```python from babelqueue import dead_letter dlq = dead_letter.annotate(envelope, "failed", "orders", attempts=3, error="boom") ``` ``` -------------------------------- ### EnvelopeCodec.encode Source: https://github.com/babelqueue/babelqueue-python/blob/main/README.md Encodes an envelope into a UTF-8 JSON string for publishing to a broker. Rejects envelopes with forbidden keys or the inbound-only `urn` alias. ```APIDOC ## EnvelopeCodec.encode ### Description Encodes an envelope into a UTF-8 JSON string for publishing to a broker. Rejects envelopes with forbidden keys or the inbound-only `urn` alias. ### Method encode ### Signature `EnvelopeCodec.encode(envelope: Envelope) -> str` ### Parameters - **envelope** (Envelope) - Required - The envelope to encode. ### Returns - **str** - The UTF-8 JSON string representation. ### Errors - **EnvelopeKeyError** - Raised if the envelope contains a forbidden key (top-level `timestamp`; `meta.max_retries`, `meta.attempts`, `meta.source`, `meta.ts`) or the inbound-only top-level `urn` alias. ### Example ```python body = EnvelopeCodec.encode(envelope) ``` ``` -------------------------------- ### Typed messages with PolyglotMessage Source: https://github.com/babelqueue/babelqueue-python/blob/main/README.md Shows how to define a class that structurally implements PolyglotMessage and use EnvelopeCodec.from_message to create an envelope. You can continue an existing trace by adding get_babel_trace_id or passing trace_id to make. ```python from babelqueue import EnvelopeCodec, PolyglotMessage class OrderCreated: # structurally a PolyglotMessage def __init__(self, order_id: int): self.order_id = order_id def get_babel_urn(self) -> str: return "urn:babel:orders:created" def to_payload(self) -> dict: return {"order_id": self.order_id} envelope = EnvelopeCodec.from_message(OrderCreated(1042), queue="orders") ``` -------------------------------- ### Enable Laravel-compatible Redis mode Source: https://github.com/babelqueue/babelqueue-python/blob/main/README.md Use this to consume a shared Laravel BabelQueue Redis queue. It replicates Laravel's stock Redis reservation exactly, using the same Lua scripts and queue layout. The `laravel=1` query parameter or `laravel_compat=True` constructor argument enables the mode. ```python app = BabelQueue("redis://localhost:6379/0?laravel=1", queue="orders") # or: RedisTransport("redis://localhost:6379/0", laravel_compat=True) ``` -------------------------------- ### Transactional outbox: write and relay with InMemoryOutboxStore Source: https://github.com/babelqueue/babelqueue-python/blob/main/README.md Removes the producer dual write by persisting the message in the same transaction as business data. The core is stdlib-only; `OutboxStore` is an abstract Protocol you bind to your own DB. `Outbox.write` only encodes and calls `OutboxStore.save`; it does not begin or commit anything. `OutboxRelay` marks a row published only after the transport accepts it; a publish that raises is recorded via `mark_failed` with bounded backoff. `InMemoryOutboxStore` is for tests and single-process demos. ```python from babelqueue import BabelQueue, EnvelopeCodec from babelqueue.outbox import Outbox, OutboxRelay, InMemoryOutboxStore store = InMemoryOutboxStore() # production: your own OutboxStore adapter, DB-backed outbox = Outbox(store) # write side — YOU own the transaction boundary (this is the whole point): with db.transaction(): # your own open transaction db.insert_order(order) # the business write envelope = EnvelopeCodec.make("urn:babel:orders:created", {"order_id": 1042}, queue="orders") outbox.write(envelope) # same connection, same tx — both, or neither # read/publish side — run on a short interval, after the business tx commits: app = BabelQueue("redis://localhost:6379/0", queue="orders") relay = OutboxRelay(app.transport, store) relay.drain() # publish all pending rows; flush() does one batch ``` -------------------------------- ### protect(data, schema, cipher) Source: https://github.com/babelqueue/babelqueue-python/blob/main/README.md Encrypts all sensitive fields in the data dictionary according to the provided schema, replacing their values with ciphertext strings. This is the producer-side enforcement step before encoding the envelope. ```APIDOC ## protect(data, schema, cipher) ### Description Encrypts all sensitive fields in the data dictionary according to the provided schema, replacing their values with ciphertext strings. This is the producer-side enforcement step before encoding the envelope. ### Method Function call ### Signature `protect(data, schema, cipher)` ### Parameters - **data** (dict) - Required - The data dictionary containing the fields to be encrypted. - **schema** (dict) - Required - The JSON Schema that defines which fields are marked as `x-gdpr-sensitive`. - **cipher** (Cipher) - Required - A caller-provided object implementing the `Cipher` protocol with `encrypt(plaintext: bytes) -> str` and `decrypt(ciphertext: str) -> bytes` methods. ### Usage Example ```python from babelqueue.gdpr import protect protect(data, schema, cipher) ``` ### Notes - Mutates `data` in place, replacing sensitive leaf values with ciphertext strings. - Absent marked fields are skipped. - The envelope remains frozen (`schema_version: 1`); only values inside `data` change. ``` -------------------------------- ### EnvelopeCodec.decode Source: https://github.com/babelqueue/babelqueue-python/blob/main/README.md Decodes a JSON message from any BabelQueue SDK into an envelope dictionary. Folds the `urn` alias into `job` and warns on unknown keys. ```APIDOC ## EnvelopeCodec.decode ### Description Decodes a JSON message from any BabelQueue SDK into an envelope dictionary. Folds the `urn` alias into `job` and warns on unknown keys. ### Method decode ### Signature `EnvelopeCodec.decode(body: str) -> dict` ### Parameters - **body** (str) - Required - The UTF-8 JSON string to decode. ### Returns - **dict** - The decoded envelope with keys `job`, `data`, `trace_id`, `meta`, `attempts`, etc. ### Behavior - If `job` is missing, empty, or only ASCII whitespace, falls back to `urn`. - If both `job` and `urn` carry a value, `job` wins and `urn` is dropped with a warning. - Unknown top-level and `meta` keys are preserved. - The decoded envelope never keeps a `urn` key. ### Example ```python incoming = EnvelopeCodec.decode(body) urn = incoming["job"] data = incoming["data"] trace_id = incoming["trace_id"] ``` ``` -------------------------------- ### Configure Kafka transport with retry tiers and run RetryReinjector Source: https://github.com/babelqueue/babelqueue-python/blob/main/README.md Sets up a Kafka transport with opt-in retry tiers (DEFAULT_RETRY_TIERS_MS = 5 s, 30 s, 5 min, 30 min) and a BabelQueue app with max_attempts and dead_letter enabled. The RetryReinjector must run alongside workers to re-inject delayed messages; without retry_tiers_ms, publish_delayed and retries raise BabelQueueError. ```python from babelqueue import BabelQueue from babelqueue.kafka_transport import KafkaTransport, DEFAULT_RETRY_TIERS_MS from babelqueue.kafka_retry import RetryReinjector transport = KafkaTransport("kafka://localhost:9092", retry_tiers_ms=DEFAULT_RETRY_TIERS_MS) app = BabelQueue(transport=transport, queue="orders", max_attempts=5, dead_letter=True) # Run one next to the workers, otherwise a parked message is never delivered again: RetryReinjector("kafka://localhost:9092", work_topics=["orders"], retry_tiers_ms=DEFAULT_RETRY_TIERS_MS).run() ``` -------------------------------- ### unprotect(data, schema, cipher) Source: https://github.com/babelqueue/babelqueue-python/blob/main/README.md Decrypts all sensitive fields in the data dictionary according to the provided schema, restoring the original cleartext values. This is the consumer-side enforcement step after decoding the envelope. ```APIDOC ## unprotect(data, schema, cipher) ### Description Decrypts all sensitive fields in the data dictionary according to the provided schema, restoring the original cleartext values. This is the consumer-side enforcement step after decoding the envelope. ### Method Function call ### Signature `unprotect(data, schema, cipher)` ### Parameters - **data** (dict) - Required - The data dictionary containing the encrypted fields. - **schema** (dict) - Required - The JSON Schema that defines which fields are marked as `x-gdpr-sensitive`. - **cipher** (Cipher) - Required - A caller-provided object implementing the `Cipher` protocol with `encrypt(plaintext: bytes) -> str` and `decrypt(ciphertext: str) -> bytes` methods. ### Usage Example ```python from babelqueue.gdpr import unprotect unprotect(envelope["data"], schema, cipher) ``` ### Notes - Mutates `data` in place, replacing ciphertext strings with cleartext values. - Re-running `unprotect` on cleartext is a no-op (non-string leaves are left alone). - A wrong key or tampered ciphertext raises `babelqueue.DecryptError`. ``` -------------------------------- ### babelqueue.schema.sensitive_paths Source: https://github.com/babelqueue/babelqueue-python/blob/main/README.md Locates every `x-gdpr-sensitive` leaf in a JSON Schema, including nested objects, array items, container or root marks. This is used internally by `protect` and `unprotect` to determine which fields to encrypt or decrypt. ```APIDOC ## babelqueue.schema.sensitive_paths ### Description Locates every `x-gdpr-sensitive` leaf in a JSON Schema, including nested objects, array items, container or root marks. This is used internally by `protect` and `unprotect` to determine which fields to encrypt or decrypt. ### Method Function call ### Signature `sensitive_paths(schema)` ### Parameters - **schema** (dict) - Required - The JSON Schema to scan for `x-gdpr-sensitive` markers. ### Returns - A list of paths (strings or lists) pointing to each sensitive leaf. ### Usage Example ```python from babelqueue.schema import sensitive_paths paths = sensitive_paths(schema) ``` ``` -------------------------------- ### Protect and Unprotect Around EnvelopeCodec Source: https://github.com/babelqueue/babelqueue-python/blob/main/README.md Producer encrypts marked leaves with `protect` before encoding; consumer decrypts with `unprotect` after decoding. Validate cleartext before protect and after unprotect because a schema constraining a sensitive field would reject the ciphertext string. `schema` is the same per-URN JSON Schema used by the validator. ```python from babelqueue import EnvelopeCodec from babelqueue.gdpr import protect, unprotect # producer: encrypt marked leaves in place, then encode protect(data, schema, cipher) body = EnvelopeCodec.encode(EnvelopeCodec.make("urn:babel:orders:created", data, queue="orders")) # consumer: decode, then decrypt marked leaves in place before the handler reads data envelope = EnvelopeCodec.decode(body) unprotect(envelope["data"], schema, cipher) ```