### Initial Carbonite Migration Example Source: https://github.com/bitcrowd/carbonite/blob/main/README.md An example of an initial Ecto migration for Carbonite, including setting up triggers and outbox for a 'rabbits' table. ```elixir defmodule MyApp.Repo.Migrations.InstallCarbonite do use Ecto.Migration def up do Carbonite.Migrations.up(1..11) # For each table that you want to capture changes of, you need to install the trigger. Carbonite.Migrations.create_trigger(:rabbits) # Optionally you may configure the trigger inside the migration. Carbonite.Migrations.put_trigger_config(:rabbits, :excluded_columns, ["age"]) # Optional outbox to process transactions later. Carbonite.Migrations.create_outbox("rabbit_holes") end def down do # Remove outbox again. Carbonite.Migrations.drop_outbox("rabbit_holes") # Remove all triggers before dropping the schema. Carbonite.Migrations.drop_trigger(:rabbits) # Drop the Carbonite tables. Carbonite.Migrations.down(11..1) end end ``` -------------------------------- ### Install Carbonite Migrations Source: https://context7.com/bitcrowd/carbonite/llms.txt Runs Carbonite's database migrations to install the audit trail schema. Use this to set up the necessary tables, functions, and triggers. Supports patch levels from 1 to the current version. ```elixir defmodule MyApp.Repo.Migrations.InstallCarbonite do use Ecto.Migration def up do # Install all Carbonite migrations (patches 1 through 11) Carbonite.Migrations.up(1..11) end def down do # Rollback all Carbonite migrations Carbonite.Migrations.down(11..1) end end ``` -------------------------------- ### Update Carbonite Migration Example Source: https://github.com/bitcrowd/carbonite/blob/main/README.md An example Ecto migration for updating Carbonite to a new version, applying specific migration steps. ```elixir defmodule MyApp.Repo.Migrations.UpdateCarbonite do use Ecto.Migration def up do Carbonite.Migrations.up(3) end def down do Carbonite.Migrations.down(3) end end ``` -------------------------------- ### Migration Transaction Insertion Source: https://context7.com/bitcrowd/carbonite/llms.txt Demonstrates how to insert migration transaction records before executing SQL modifications on audited tables. Includes examples for direct insertion and using a macro for automatic insertion. ```APIDOC ## Carbonite.Migrations.insert_migration_transaction/2 ### Description Inserts a transaction record for data migrations. Required before executing any SQL that modifies audited tables within a migration. Automatically populates metadata with migration direction and name. ### Method Not applicable (function call within Ecto migration) ### Endpoint Not applicable ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Request Example ```elixir defmodule MyApp.Repo.Migrations.BackfillUserNames do use Ecto.Migration import Carbonite.Migrations def change do # Insert transaction before any data modifications insert_migration_transaction("backfill-user-names") execute( "UPDATE users SET display_name = CONCAT(first_name, ' ', last_name) WHERE display_name IS NULL", "UPDATE users SET display_name = NULL WHERE display_name = CONCAT(first_name, ' ', last_name)" ) end end ``` ### Response None (function call within Ecto migration) ## Alternative: Automatic Transaction Insertion ### Description Uses a macro to automatically insert a transaction record before the `change` function is executed in an Ecto migration. ### Method Not applicable (macro usage within Ecto migration) ### Endpoint Not applicable ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Request Example ```elixir # Alternative: Use macro for automatic transaction insertion defmodule MyApp.Repo.Migrations.UpdatePrices do use Ecto.Migration import Carbonite.Migrations insert_migration_transaction_after_begin() def change do execute("UPDATE products SET price = price * 1.1", "UPDATE products SET price = price / 1.1") end end ``` ### Response None (macro usage within Ecto migration) ``` -------------------------------- ### Partition Carbonite Audit Trail with Prefixes Source: https://github.com/bitcrowd/carbonite/blob/main/README.md Use `carbonite_prefix` to install Carbonite tables into multiple database schemas, effectively partitioning the captured data. All Carbonite functions accept this option. ```elixir Carbonite.Migrations.up(1, carbonite_prefix: "carbonite_animals") ``` ```elixir Carbonite.Migrations.create_trigger(:rabbits, carbonite_prefix: "carbonite_animals") ``` -------------------------------- ### Carbonite.Migrations.up/2 and Carbonite.Migrations.down/2 Source: https://context7.com/bitcrowd/carbonite/llms.txt Runs or rolls back Carbonite's database migrations to install or uninstall the audit trail schema, tables, functions, and triggers. Supports patch levels and custom prefixes. ```APIDOC ## Carbonite.Migrations.up/2 and Carbonite.Migrations.down/2 ### Description Runs Carbonite's database migrations to install the audit trail schema, tables, functions, and triggers. Must be called with patch levels ranging from 1 to the current patch. Supports the `carbonite_prefix` option to create multiple audit trail partitions. ### Method Elixir Function Call ### Endpoint N/A (Elixir Function) ### Parameters #### Path Parameters N/A #### Query Parameters N/A #### Request Body N/A ### Request Example ```elixir defmodule MyApp.Repo.Migrations.InstallCarbonite do use Ecto.Migration def up do # Install all Carbonite migrations (patches 1 through 11) Carbonite.Migrations.up(1..11) end def down do # Rollback all Carbonite migrations Carbonite.Migrations.down(11..1) end end ``` ### Response #### Success Response (200) N/A (Elixir Function) #### Response Example N/A ``` -------------------------------- ### Insert Carbonite Transaction without Ecto.Multi Source: https://github.com/bitcrowd/carbonite/blob/main/README.md Use `Carbonite.insert_transaction/3` to insert a transaction when not using `Ecto.Multi`. Ensure a transaction is started with `MyApp.Repo.transaction/1`. ```elixir MyApp.Repo.transaction(fn -> {:ok, _} = Carbonite.insert_transaction(MyApp.Repo, %{meta: %{type: "rabbit_inserted"}}) params |> MyApp.Rabbit.create_changeset() |> MyApp.Repo.insert!() end) ``` -------------------------------- ### Create Carbonite Database Triggers Source: https://context7.com/bitcrowd/carbonite/llms.txt Installs a change capture trigger on a specified database table. The trigger automatically records INSERT, UPDATE, and DELETE statements to the audit trail. Supports custom table prefixes and deferred triggers. ```elixir defmodule MyApp.Repo.Migrations.InstallCarbonite do use Ecto.Migration def up do Carbonite.Migrations.up(1..11) # Install trigger on the rabbits table Carbonite.Migrations.create_trigger(:rabbits) # Install trigger with custom table prefix Carbonite.Migrations.create_trigger(:users, table_prefix: "accounts") # Install deferred trigger (runs at transaction end) Carbonite.Migrations.create_trigger(:orders, initially: :deferred) end def down do Carbonite.Migrations.drop_trigger(:orders) Carbonite.Migrations.drop_trigger(:users, table_prefix: "accounts") Carbonite.Migrations.drop_trigger(:rabbits) Carbonite.Migrations.down(11..1) end end ``` -------------------------------- ### Carbonite.Migrations.create_trigger/2 and Carbonite.Migrations.drop_trigger/2 Source: https://context7.com/bitcrowd/carbonite/llms.txt Installs or removes a change capture trigger on a database table. The trigger fires after INSERT, UPDATE, and DELETE statements, automatically recording mutations to the audit trail. Supports deferred triggers and custom table prefixes. ```APIDOC ## Carbonite.Migrations.create_trigger/2 and Carbonite.Migrations.drop_trigger/2 ### Description Installs a change capture trigger on a database table. The trigger fires after INSERT, UPDATE, and DELETE statements, automatically recording mutations to the audit trail. Supports deferred triggers for conditional transaction insertion. ### Method Elixir Function Call ### Endpoint N/A (Elixir Function) ### Parameters #### Path Parameters N/A #### Query Parameters N/A #### Request Body N/A ### Request Example ```elixir defmodule MyApp.Repo.Migrations.InstallCarbonite do use Ecto.Migration def up do Carbonite.Migrations.up(1..11) # Install trigger on the rabbits table Carbonite.Migrations.create_trigger(:rabbits) # Install trigger with custom table prefix Carbonite.Migrations.create_trigger(:users, table_prefix: "accounts") # Install deferred trigger (runs at transaction end) Carbonite.Migrations.create_trigger(:orders, initially: :deferred) end def down do Carbonite.Migrations.drop_trigger(:orders) Carbonite.Migrations.drop_trigger(:users, table_prefix: "accounts") Carbonite.Migrations.drop_trigger(:rabbits) Carbonite.Migrations.down(11..1) end end ``` ### Response #### Success Response (200) N/A (Elixir Function) #### Response Example N/A ``` -------------------------------- ### Fetch Changes for a Specific Record Source: https://context7.com/bitcrowd/carbonite/llms.txt Use Carbonite.Query.changes/2 to get all changes for a given Ecto schema struct. Changes are ordered by ID ascending by default. Each change includes operation, table name, primary key, data, and changed fields. ```elixir # Fetch change history for a specific rabbit rabbit = MyApp.Repo.get!(Rabbit, 42) changes = rabbit |> Carbonite.Query.changes() |> MyApp.Repo.all() # Each change contains: op, table_name, table_pk, data, changed, changed_from Enum.each(changes, fn change -> IO.puts("Operation: #{change.op}") IO.puts("Data: #{inspect(change.data)}") IO.puts("Changed fields: #{inspect(change.changed)}") if change.changed_from, do: IO.puts("Previous values: #{inspect(change.changed_from)}") end) ``` ```elixir # With preloaded transaction for metadata rabbit |> Carbonite.Query.changes(preload: true) |> MyApp.Repo.all() |> Enum.each(fn change -> IO.puts("Changed by user: #{change.transaction.meta["user_id"]}") end) ``` ```elixir # Custom ordering or no ordering rabbit |> Carbonite.Query.changes(order_by: {:desc, :id}) |> MyApp.Repo.all() rabbit |> Carbonite.Query.changes(order_by: false) # No ordering |> MyApp.Repo.all() ``` -------------------------------- ### Generate Initial Carbonite Migration Source: https://github.com/bitcrowd/carbonite/blob/main/README.md Use the Carbonite Mix task to generate the initial migration file for setting up Carbonite in your Ecto project. ```sh mix carbonite.gen.initial_migration -r MyApp.Repo ``` -------------------------------- ### Process an Outbox with a Callback Function Source: https://github.com/bitcrowd/carbonite/blob/main/README.md This code shows how to process an outbox using `Carbonite.process/4`. It takes the repository, outbox name, and a processor callback function that handles batches of transactions. ```elixir Carbonite.process(MyApp.Repo, "rabbit_holes", fn transactions, _memo -> for transaction <- transactions do send_to_external_database(transaction) end :cont end) ``` -------------------------------- ### Build Changeset for Manual Transaction Insertion Source: https://github.com/bitcrowd/carbonite/blob/main/README.md Create a changeset for a `Carbonite.Transaction` using `Carbonite.Transaction.changeset/1` for manual insertion within a repository transaction. ```elixir MyApp.Repo.transaction(fn -> %{meta: %{type: "rabbit_inserted"}} |> Carbonite.Transaction.changeset() |> MyApp.Repo.insert!() # ... end) ``` -------------------------------- ### Automatic Migration Transaction Insertion Source: https://context7.com/bitcrowd/carbonite/llms.txt Leverage `insert_migration_transaction_after_begin/0` macro to automatically insert a transaction record at the beginning of a migration's `change` function. This simplifies migrations that always require transaction logging. ```elixir defmodule MyApp.Repo.Migrations.UpdatePrices do use Ecto.Migration import Carbonite.Migrations insert_migration_transaction_after_begin() def change do execute("UPDATE products SET price = price * 1.1", "UPDATE products SET price = price / 1.1") end end ``` -------------------------------- ### Configure Oban for Periodic Outbox Processing Source: https://github.com/bitcrowd/carbonite/blob/main/README.md This configuration snippet sets up Oban to periodically process an outbox. It defines a cron job that runs `MyApp.PeriodicOutboxWorker` every day at midnight, passing the outbox name as an argument. ```elixir config :my_app, :oban, repo: MyApp.Repo, plugins: [ {Oban.Plugins.Cron, crontab: [ {"0 0 * * *", MyApp.PeriodicOutboxWorker, args: %{outbox: "rabbit_holes"}} ]} ] ``` -------------------------------- ### Fetch All Transactions with Carbonite.Query Source: https://context7.com/bitcrowd/carbonite/llms.txt Retrieve all transactions from the audit trail using Carbonite.Query.transactions(). Supports preloading associated changes. ```elixir # Fetch all transactions Carbonite.Query.transactions() |> MyApp.Repo.all() ``` ```elixir # Fetch transactions with preloaded changes Carbonite.Query.transactions(preload: true) |> MyApp.Repo.all() ``` ```elixir # From custom carbonite prefix Carbonite.Query.transactions(carbonite_prefix: "carbonite_orders", preload: true) |> MyApp.Repo.all() ``` -------------------------------- ### Using Ecto.Multi with Carbonite override modes Source: https://context7.com/bitcrowd/carbonite/llms.txt Shows how to integrate Carbonite's `override_mode` with Ecto.Multi to selectively disable and re-enable transaction capturing within a multi-step database operation. ```elixir # With Ecto.Multi Ecto.Multi.new() |> Carbonite.Multi.override_mode(to: :ignore) |> Ecto.Multi.insert(:fixture, Rabbit.changeset(%Rabbit{}, %{name: "Test"})) |> Carbonite.Multi.override_mode(to: :capture) |> MyApp.Repo.transaction() ``` -------------------------------- ### Process transactions with chunking and filtering Source: https://context7.com/bitcrowd/carbonite/llms.txt Processes transactions with specified chunk size, fetch limit, minimum age, and a dynamic filter. The callback function processes each chunk and returns a continuation or halt signal with updated memo. ```elixir import Ecto.Query {status, outbox} = Carbonite.process( MyApp.Repo, "analytics", [ chunk: 50, # Process 50 transactions per callback limit: 1000, # Fetch max 1000 per query min_age: 600, # Only process transactions older than 10 minutes filter: dynamic([t], fragment("?->>'type' = ?", t.meta, "order_placed")) ], fn transactions, memo -> processed_count = Map.get(memo, "count", 0) for txn <- transactions do send_to_analytics(txn) end new_count = processed_count + length(transactions) {:cont, memo: %{"count" => new_count}} end ) IO.puts("Final status: #{status}, processed: #{outbox.memo["count"]}") ``` -------------------------------- ### Create an Outbox for Event Processing Source: https://github.com/bitcrowd/carbonite/blob/main/README.md This code creates a named outbox in a migration using `Carbonite.Migrations.create_outbox/1`. Outboxes are used to process and export captured transactions to external data stores. ```elixir Carbonite.Migrations.create_outbox("rabbit_holes") ``` -------------------------------- ### Create Trigger with Initially Deferred Source: https://github.com/bitcrowd/carbonite/blob/main/README.md This code snippet demonstrates how to create a trigger with the `initially: :deferred` option in a migration. This is used to prevent storing Carbonite.Transactions when no changes were made. ```elixir Carbonite.Migrations.create_trigger(:rabbits, initially: :deferred) ``` -------------------------------- ### Process Transactions via Outbox Source: https://context7.com/bitcrowd/carbonite/llms.txt Use Carbonite.process/4 to handle transactions through an outbox for external system integration. It fetches batches of unprocessed transactions and applies a callback function for processing. ```elixir # Basic outbox processing {:ok, outbox} = Carbonite.process(MyApp.Repo, "event_store", fn transactions, _memo -> for transaction <- transactions do event = %{ id: transaction.id, type: transaction.meta["type"], timestamp: transaction.inserted_at, changes: Enum.map(transaction.changes, &serialize_change/1) } :ok = EventStore.append(event) end :cont end) ``` -------------------------------- ### Configure Carbonite Mode by Environment Source: https://github.com/bitcrowd/carbonite/blob/main/README.md This configuration sets the default Carbonite trigger mode based on the Mix environment. In the test environment, it's set to `:ignore` by default, requiring explicit enabling when needed. This approach is generally cheaper as it avoids overhead in tests that don't require capture. ```elixir config :my_app, carbonite_mode: :capture ``` ```elixir config :my_app, carbonite_mode: :ignore ``` ```elixir defmodule MyApp.Repo.Migrations.InstallCarbonite do @mode Application.compile_env!(:my_app, :carbonite_mode) def up do # ... Carbonite.Migrations.create_trigger(:rabbits) Carbonite.Migrations.put_trigger_config(:rabbits, :mode, @mode) end end ``` ```elixir defmodule MyApp.CarboniteHelpers do def enable_carbonite do Carbonite.override_mode(MyApp.Repo, to: :capture) end end ``` ```elixir test "my_operation/0" do rabbit = insert_rabbit() enable_carbonite() my_operation() # Assert on the Carbonite.Transaction... end ``` ```elixir test "my_other_operation/0" do rabbit = insert_rabbit() # this will pass regardless of whether you inserted a transaction in my_other_operation/0 (!) my_operation() end ``` -------------------------------- ### Fetch Transactions with Ecto.Query Source: https://github.com/bitcrowd/carbonite/blob/main/README.md This code constructs an Ecto.Query to fetch Carbonite.Transaction records from the database, with an option to filter by insertion time. It uses `Carbonite.Query.transactions/1` and `Ecto.Query.where/3`. ```elixir Carbonite.Query.transactions() |> Ecto.Query.where([t], t.inserted_at > ^earliest) |> MyApp.Repo.all() ``` -------------------------------- ### Test updating a record and asserting on captured transaction Source: https://context7.com/bitcrowd/carbonite/llms.txt Demonstrates how to use `with_carbonite_disabled` to insert test data without triggering Carbonite, then perform an operation that is captured, and finally assert on the captured transaction details. ```elixir defmodule MyApp.RabbitTest do use MyApp.DataCase import MyApp.CarboniteHelpers test "updating a rabbit records changes" do # Insert test data without triggering Carbonite rabbit = with_carbonite_disabled(fn -> insert(:rabbit, name: "Bugs", age: 5) end) # This operation will be captured {:ok, updated} = Rabbits.update_rabbit(rabbit, %{age: 6}) # Assert on the captured transaction transaction = Carbonite.Query.current_transaction(preload: true) |> MyApp.Repo.one!() assert [change] = transaction.changes assert change.op == :update assert change.changed == ["age"] assert change.data["age"] == 6 end) end ``` -------------------------------- ### Outbox Processing API Source: https://context7.com/bitcrowd/carbonite/llms.txt APIs for processing transactions through an outbox for external systems. ```APIDOC ## Carbonite.process/4 ### Description Processes transactions through an outbox for export to external systems. Fetches batches of unprocessed transactions and passes them to a callback function. Supports chunking, filtering, and memo passing between batches. ### Method POST (Conceptual, function call) ### Endpoint N/A (Function call) ### Parameters #### Path Parameters - **repo** (Ecto.Repo) - Required - The Ecto repository to use. - **outbox_name** (string) - Required - The name of the outbox to process. - **callback_function** (function) - Required - A function that receives a list of transactions and a memo, and returns `:cont` to continue or a result. - **memo** (any) - Optional - Initial memo value passed to the callback function. ### Request Example ```elixir # Basic outbox processing {:ok, outbox} = Carbonite.process(MyApp.Repo, "event_store", fn transactions, _memo -> for transaction <- transactions do event = %{ id: transaction.id, type: transaction.meta["type"], timestamp: transaction.inserted_at, changes: Enum.map(transaction.changes, &serialize_change/1) } :ok = EventStore.append(event) end :cont end) ``` ### Response #### Success Response (200) - **outbox** (any) - The result of the outbox processing, typically the final memo or a status. #### Response Example ```elixir # Example response structure (depends on callback and memo) :ok # or %{processed_count: 100, last_processed_id: 500} ``` ``` -------------------------------- ### Configure Oban PeriodicOutboxWorker Source: https://github.com/bitcrowd/carbonite/blob/main/README.md Configures a unique Oban worker to run every 24 hours, with a 12-hour uniqueness period to prevent concurrent runs. Retries are disabled as the next scheduled run will handle failures. ```elixir defmodule MyApp.PeriodicOutboxWorker do use Oban.Worker, queue: :default, unique: [period: 43_200], retry: false def perform(%Oban.Job{args: %{"outbox" => outbox}}) do Carbonite.process(MyApp.Repo, outbox, fn transactions, _memo -> for transaction <- transactions do send_to_external_database(transaction) end :cont end) end end ``` -------------------------------- ### Insert and Delete Transactions with Ecto.Multi Source: https://context7.com/bitcrowd/carbonite/llms.txt Use Ecto.Multi to chain Carbonite transaction operations like inserting and deleting. Ensure to run the transaction using your Ecto repository. ```elixir Ecto.Multi.new() |> Carbonite.Multi.insert_transaction(%{meta: %{type: "sync"}}) |> Ecto.Multi.run(:sync, fn _, _ -> maybe_sync_data() end) |> Carbonite.Multi.delete_transaction_if_empty() |> MyApp.Repo.transaction() ``` -------------------------------- ### Configure Carbonite Trigger: Primary Key Columns (Composite) Source: https://github.com/bitcrowd/carbonite/blob/main/README.md Configure Carbonite to use a composite primary key consisting of 'house' and 'apartment_no' for change tracking. ```elixir Carbonite.Migrations.put_trigger_config(:rabbits, :primary_key_columns, ["house", "apartment_no"]) ``` -------------------------------- ### Store Transaction Metadata in Process Dictionary Source: https://github.com/bitcrowd/carbonite/blob/main/README.md Use `Carbonite.Transaction.put_meta/2` to store metadata in the process dictionary when it's not available at the code site of transaction creation. This metadata is merged into the transaction. ```elixir # e.g., in a controller or plug Carbonite.Transaction.put_meta(:user_id, ...) ``` -------------------------------- ### Persist Replaced Data on Updates Source: https://github.com/bitcrowd/carbonite/blob/main/README.md Enable `store_changed_from` to store the previous version of data (the diff) in the `changed_from` field for `UPDATE` statements. Defaults to `false`. ```elixir Carbonite.Migrations.put_trigger_config(:rabbits, :store_changed_from, true) ``` -------------------------------- ### Carbonite.Migrations.create_outbox/2 and Carbonite.Migrations.drop_outbox/2 Source: https://context7.com/bitcrowd/carbonite/llms.txt Creates or removes an outbox record for processing and exporting audit trail data. Outboxes track the last processed transaction ID, enabling incremental batch processing of changes for external systems. ```APIDOC ## Carbonite.Migrations.create_outbox/2 and Carbonite.Migrations.drop_outbox/2 ### Description Creates an outbox record for processing and exporting audit trail data. Outboxes track the last processed transaction ID, enabling incremental batch processing of changes for external systems. ### Method Elixir Function Call ### Endpoint N/A (Elixir Function) ### Parameters #### Path Parameters N/A #### Query Parameters N/A #### Request Body N/A ### Request Example ```elixir defmodule MyApp.Repo.Migrations.CreateOutboxes do use Ecto.Migration def up do # Create outbox for processing transactions Carbonite.Migrations.create_outbox("event_store") # Create outbox starting after a specific transaction (skip historical data) Carbonite.Migrations.create_outbox("analytics", last_transaction_id: 1000) # Create outbox in custom carbonite schema Carbonite.Migrations.create_outbox("backup", carbonite_prefix: "carbonite_archive") end def down do Carbonite.Migrations.drop_outbox("backup", carbonite_prefix: "carbonite_archive") Carbonite.Migrations.drop_outbox("analytics") Carbonite.Migrations.drop_outbox("event_store") end end ``` ### Response #### Success Response (200) N/A (Elixir Function) #### Response Example N/A ``` -------------------------------- ### Insert Migration Transaction for Data Migrations Source: https://github.com/bitcrowd/carbonite/blob/main/README.md Use `Carbonite.Migrations.insert_migration_transaction/1` within data migrations to insert a `Carbonite.Transaction` with metadata populated from the migration module before executing other statements. ```elixir import Carbonite.Migrations def change do insert_migration_transaction() execute("UPDATE ...") end ``` -------------------------------- ### Add Transaction Insertion to Ecto.Multi Source: https://context7.com/bitcrowd/carbonite/llms.txt Use `Carbonite.Multi.insert_transaction/3` to add a transaction insertion step to an `Ecto.Multi` pipeline. This is the recommended approach for integrating Carbonite with `Ecto.Multi` workflows. ```elixir # Standard Ecto.Multi usage Ecto.Multi.new() |> Carbonite.Multi.insert_transaction(%{meta: %{type: "create_rabbit", actor_id: current_user.id}}) |> Ecto.Multi.insert(:rabbit, Rabbit.changeset(%Rabbit{}, %{name: "Bugs", age: 5})) |> Ecto.Multi.update(:owner, fn %{rabbit: rabbit} -> User.changeset(current_user, %{rabbit_count: current_user.rabbit_count + 1}) end) |> MyApp.Repo.transaction() # => {:ok, %{carbonite_transaction: %Carbonite.Transaction{}, rabbit: %Rabbit{}, owner: %User{}}} ``` ```elixir # Multiple audit trails in one transaction Ecto.Multi.new() |> Carbonite.Multi.insert_transaction(%{meta: %{type: "sync"}}, carbonite_prefix: "carbonite_orders") |> Carbonite.Multi.insert_transaction(%{meta: %{type: "sync"}}, carbonite_prefix: "carbonite_inventory") |> Ecto.Multi.run(:sync, fn repo, _ -> perform_sync(repo) end) |> MyApp.Repo.transaction() ``` -------------------------------- ### Query API Source: https://context7.com/bitcrowd/carbonite/llms.txt APIs for querying transaction and change data from Carbonite. ```APIDOC ## Carbonite.Query.transactions/1 ### Description Returns an Ecto query for selecting transactions from the audit trail. Supports preloading associated changes and filtering with standard Ecto query composition. ### Method GET (Implicit via Ecto Query) ### Endpoint N/A (Ecto Query) ### Parameters #### Query Parameters - **preload** (boolean) - Optional - Whether to preload associated changes. - **carbonite_prefix** (string) - Optional - Custom prefix for Carbonite tables. ### Request Example ```elixir # Fetch all transactions Carbonite.Query.transactions() |> MyApp.Repo.all() # Fetch transactions with preloaded changes Carbonite.Query.transactions(preload: true) |> MyApp.Repo.all() # Fetch transactions from a specific time range import Ecto.Query Carbonite.Query.transactions(preload: true) |> where([t], t.inserted_at >= ^DateTime.add(DateTime.utc_now(), -24, :hour)) |> where([t], fragment("?->>'type' = ?", t.meta, "user_registration")) |> order_by([t], desc: t.id) |> limit(100) |> MyApp.Repo.all() # From custom carbonite prefix Carbonite.Query.transactions(carbonite_prefix: "carbonite_orders", preload: true) |> MyApp.Repo.all() ``` ### Response #### Success Response (200) - **transactions** (list) - A list of transaction structs. #### Response Example ```elixir # Example response structure (actual fields depend on Ecto schema) [ %{id: 1, meta: %{"type" => "sync"}, inserted_at: ~U[2023-10-27 10:00:00Z], changes: []}, %{id: 2, meta: %{"type" => "user_registration"}, inserted_at: ~U[2023-10-27 10:05:00Z], changes: [...]} ] ``` ``` ```APIDOC ## Carbonite.Query.changes/2 ### Description Returns an Ecto query for selecting all changes recorded for a specific record. Takes an Ecto schema struct and returns changes ordered by ID ascending. ### Method GET (Implicit via Ecto Query) ### Endpoint N/A (Ecto Query) ### Parameters #### Path Parameters - **record** (Ecto.Schema.struct) - Required - The Ecto schema struct for which to fetch changes. #### Query Parameters - **preload** (boolean) - Optional - Whether to preload the associated transaction. - **order_by** (tuple or boolean) - Optional - Custom ordering for the changes. Example: `{:desc, :id}` or `false` for no ordering. ### Request Example ```elixir # Fetch change history for a specific rabbit rabbit = MyApp.Repo.get!(Rabbit, 42) changes = rabbit |> Carbonite.Query.changes() |> MyApp.Repo.all() # Each change contains: op, table_name, table_pk, data, changed, changed_from Enum.each(changes, fn change -> IO.puts("Operation: #{change.op}") IO.puts("Data: #{inspect(change.data)}") IO.puts("Changed fields: #{inspect(change.changed)}") if change.changed_from, do: IO.puts("Previous values: #{inspect(change.changed_from)}") end) # With preloaded transaction for metadata rabbit |> Carbonite.Query.changes(preload: true) |> MyApp.Repo.all() |> Enum.each(fn change -> IO.puts("Changed by user: #{change.transaction.meta["user_id"]}") end) # Custom ordering or no ordering rabbit |> Carbonite.Query.changes(order_by: {:desc, :id}) |> MyApp.Repo.all() rabbit |> Carbonite.Query.changes(order_by: false) # No ordering |> MyApp.Repo.all() ``` ### Response #### Success Response (200) - **changes** (list) - A list of change structs. #### Response Example ```elixir # Example response structure (actual fields depend on Ecto schema) [ %{id: 1, op: :insert, table_name: "rabbits", table_pk: 42, data: %{"name" => "Fluffy"}, changed: ["name"], changed_from: nil, transaction_id: 10}, %{id: 2, op: :update, table_name: "rabbits", table_pk: 42, data: %{"name" => "Fluffykins"}, changed: ["name"], changed_from: %{"name" => "Fluffy"}, transaction_id: 11} ] ``` ``` ```APIDOC ## Carbonite.Query.current_transaction/1 ### Description Returns an Ecto query for the current transaction within a database transaction. Essential for testing to assert on transaction metadata without returning it from business logic. ### Method GET (Implicit via Ecto Query) ### Endpoint N/A (Ecto Query) ### Parameters #### Query Parameters - **preload** (boolean) - Optional - Whether to preload associated changes. ### Request Example ```elixir # In a test using Ecto SQL sandbox defmodule MyApp.OrderTest do use MyApp.DataCase test "placing an order records audit metadata" user = insert(:user) Carbonite.Transaction.put_meta(:user_id, user.id) {:ok, order} = Orders.place_order(%{product_id: 1, quantity: 2}) # Assert on the carbonite transaction transaction = Carbonite.Query.current_transaction(preload: true) |> MyApp.Repo.one!() assert transaction.meta["type"] == "order_placed" assert transaction.meta["user_id"] == user.id # Assert on recorded changes assert [change] = transaction.changes assert change.op == :insert assert change.table_name == "orders" assert change.data["product_id"] == 1 end end ``` ### Response #### Success Response (200) - **transaction** (Carbonite.Transaction) - The current transaction struct. #### Response Example ```elixir # Example response structure (actual fields depend on Ecto schema) %{id: 12, meta: %{"type" => "order_placed", "user_id" => 5}, inserted_at: ~U[2023-10-27 11:00:00Z], changes: [...]} ``` ``` -------------------------------- ### Insert Transaction Conditionally with Ecto.Multi Source: https://github.com/bitcrowd/carbonite/blob/main/README.md This code shows how to conditionally insert a Carbonite.Transaction using Ecto.Multi. It first runs the main application logic and then conditionally inserts the transaction, allowing for "empty" transactions to be avoided. ```elixir Ecto.Multi.new() |> Ecto.Multi.run(:rabbit, &maybe_insert_rabbit/2) |> Ecto.Multi.run(:carbonite_transaction, &maybe_insert_carbonite_transaction/2) |> MyApp.Repo.transaction() ``` -------------------------------- ### Fetch Changes for a Record Source: https://github.com/bitcrowd/carbonite/blob/main/README.md This code snippet demonstrates how to fetch all changes associated with a specific record using `Carbonite.Query.changes/2`. It takes a schema struct as input and returns an Ecto.Query. ```elixir %MyApp.Rabbit{id: 1} |> Carbonite.Query.changes() |> MyApp.Repo.all() ``` -------------------------------- ### Handle errors during processing with partial progress Source: https://context7.com/bitcrowd/carbonite/llms.txt Handles errors during Carbonite processing, allowing for partial progress recording in case of connection failures. Stops processing without recording progress on other errors. ```elixir Carbonite.process(MyApp.Repo, "backup", [chunk: 10], fn transactions, _memo -> case send_batch_to_backup(transactions) do {:ok, last_successful} -> :cont {:error, :connection_failed, last_successful} -> # Record progress up to last successful transaction {:halt, last_transaction_id: last_successful.id} {:error, _reason} -> # Stop without recording progress :halt end end) ``` -------------------------------- ### Create Carbonite Outbox Records Source: https://context7.com/bitcrowd/carbonite/llms.txt Creates an outbox record for processing and exporting audit trail data. Outboxes track the last processed transaction ID, enabling incremental batch processing of changes for external systems. Supports custom transaction IDs and carbonite schemas. ```elixir defmodule MyApp.Repo.Migrations.CreateOutboxes do use Ecto.Migration def up do # Create outbox for processing transactions Carbonite.Migrations.create_outbox("event_store") # Create outbox starting after a specific transaction (skip historical data) Carbonite.Migrations.create_outbox("analytics", last_transaction_id: 1000) # Create outbox in custom carbonite schema Carbonite.Migrations.create_outbox("backup", carbonite_prefix: "carbonite_archive") end def down do Carbonite.Migrations.drop_outbox("backup", carbonite_prefix: "carbonite_archive") Carbonite.Migrations.drop_outbox("analytics") Carbonite.Migrations.drop_outbox("event_store") end end ``` -------------------------------- ### Fetch Filtered and Ordered Transactions Source: https://context7.com/bitcrowd/carbonite/llms.txt Compose Ecto queries with Carbonite.Query.transactions() to filter by time range, meta fields, and order results. Use standard Ecto query functions like where/3 and order_by/2. ```elixir # Fetch transactions from a specific time range import Ecto.Query Carbonite.Query.transactions(preload: true) |> where([t], t.inserted_at >= ^DateTime.add(DateTime.utc_now(), -24, :hour)) |> where([t], fragment("?->>'type' = ?", t.meta, "user_registration")) |> order_by([t], desc: t.id) |> limit(100) |> MyApp.Repo.all() ``` -------------------------------- ### Transaction Management API Source: https://context7.com/bitcrowd/carbonite/llms.txt APIs for managing and fetching transactions. ```APIDOC ## Carbonite.insert_transaction/2 ### Description Inserts a new transaction into the audit trail. Typically used within a database transaction. ### Method POST ### Endpoint N/A (Function call) ### Parameters #### Path Parameters - **repo** (Ecto.Repo) - Required - The Ecto repository to use. #### Request Body - **transaction_data** (map) - Required - A map containing transaction metadata. Example: `%{meta: %{type: "sync"}}`. ### Request Example ```elixir # Inside a Repo.transaction block {:ok, _} = Carbonite.insert_transaction(MyApp.Repo, %{meta: %{type: "bulk_update"}}) ``` ### Response #### Success Response (200) - **transaction** (Carbonite.Transaction) - The newly inserted transaction struct. #### Response Example ```elixir # Example response structure %{id: 15, meta: %{"type" => "bulk_update"}, inserted_at: ~U[2023-10-27 11:15:00Z]} ``` ``` ```APIDOC ## Carbonite.fetch_changes/2 ### Description Fetches all changes recorded in the current transaction. Useful for returning audit information to callers or for conditional logic based on what was changed. ### Method GET (Implicit within transaction) ### Endpoint N/A (Function call within transaction) ### Parameters #### Path Parameters - **repo** (Ecto.Repo) - Required - The Ecto repository to use. ### Request Example ```elixir MyApp.Repo.transaction(fn -> {:ok, _} = Carbonite.insert_transaction(MyApp.Repo, %{meta: %{type: "bulk_update"}}) update_multiple_rabbits(params) {:ok, changes} = Carbonite.fetch_changes(MyApp.Repo) # Log or return change summary change_summary = Enum.map(changes, fn c -> %{op: c.op, table: c.table_name, pk: c.table_pk} end) {:ok, change_summary} end) # With Ecto.Multi Ecto.Multi.new() |> Carbonite.Multi.insert_transaction(%{meta: %{type: "import"}}) |> Ecto.Multi.run(:import, fn _, _ -> import_data(csv_rows) end) |> Carbonite.Multi.fetch_changes() |> MyApp.Repo.transaction() # => {:ok, %{carbonite_transaction: %Transaction{}, import: :ok, carbonite_changes: [%Change{}, ...]}} ``` ### Response #### Success Response (200) - **changes** (list) - A list of change structs recorded in the current transaction. #### Response Example ```elixir # Example response structure [ %{id: 1, op: :update, table_name: "rabbits", table_pk: 42, data: %{"name" => "Fluffykins"}, changed: ["name"], changed_from: %{"name" => "Fluffy"}, transaction_id: 15}, %{id: 2, op: :insert, table_name: "owners", table_pk: 101, data: %{"name" => "Alice"}, changed: ["name"], changed_from: nil, transaction_id: 15} ] ``` ``` -------------------------------- ### Carbonite.process/4 Source: https://context7.com/bitcrowd/carbonite/llms.txt Processes transactions with configurable chunking, limiting, and filtering. It accepts a callback function to handle batches of transactions and update memoized state. ```APIDOC ## POST /api/process ### Description Processes transactions with chunking and filtering. ### Method POST ### Endpoint /api/process ### Parameters #### Query Parameters - **repo** (Ecto.Repo) - Required - The Ecto repository to use. - **outbox** (string) - Required - The name of the outbox to process. - **options** (map) - Optional - Configuration options for processing. - **chunk** (integer) - Optional - Number of transactions to process per callback. - **limit** (integer) - Optional - Maximum number of transactions to fetch per query. - **min_age** (integer) - Optional - Minimum age of transactions to process in seconds. - **filter** (dynamic) - Optional - A dynamic Ecto filter to apply. #### Request Body - **callback** (function) - Required - A function that receives a list of transactions and a memo map, and returns `{:cont, memo}` or `{:halt, memo}`. ### Request Example ```elixir Carbonite.process(MyApp.Repo, "analytics", [chunk: 50, limit: 1000, min_age: 600, filter: dynamic([t], fragment("?->>'type' = ?", t.meta, "order_placed"))], fn transactions, memo -> processed_count = Map.get(memo, "count", 0) for txn <- transactions do send_to_analytics(txn) end new_count = processed_count + length(transactions) {:cont, memo: %{"count" => new_count}} end) ``` ### Response #### Success Response (200) - **status** (atom) - The final status of the processing. - **outbox** (map) - The memo map containing processed counts or other state. #### Response Example ```json { "status": :ok, "outbox": {"count": 123} } ``` ``` -------------------------------- ### Add Carbonite Hex Dependency Source: https://github.com/bitcrowd/carbonite/blob/main/README.md Add the Carbonite dependency to your project's mix.exs file to include it in your Elixir application. ```elixir def deps do [ {:carbonite, "~> 0.16.1"} ] end ``` -------------------------------- ### Insert Carbonite Transaction with Ecto.Multi Source: https://github.com/bitcrowd/carbonite/blob/main/README.md Use `Carbonite.Multi.insert_transaction/2` within an `Ecto.Multi` operation to easily create a `Carbonite.Transaction` record. This is useful for storing metadata like transaction type or user ID. ```elixir Ecto.Multi.new() |> Carbonite.Multi.insert_transaction(%{meta: %{type: "rabbit_inserted"}}) |> Ecto.Multi.insert(:rabbit, &MyApp.Rabbit.create_changeset(&1.params)) |> MyApp.Repo.transaction() ``` -------------------------------- ### Configure Carbonite Trigger: Exclude Columns Source: https://github.com/bitcrowd/carbonite/blob/main/README.md Configure Carbonite trigger settings within a migration to exclude specific columns from being captured. ```elixir Carbonite.Migrations.put_trigger_config(:rabbits, :excluded_columns, ["age"]) ``` -------------------------------- ### Configure Carbonite Trigger: Primary Key Columns (None) Source: https://github.com/bitcrowd/carbonite/blob/main/README.md Disable the copying of primary key columns to the 'changes' table by setting the 'primary_key_columns' option to an empty list. ```elixir Carbonite.Migrations.put_trigger_config(:rabbits, :primary_key_columns, []) ``` -------------------------------- ### Configure Carbonite Trigger: Primary Key Columns (Single) Source: https://github.com/bitcrowd/carbonite/blob/main/README.md Override the default primary key column name to 'identifier' for Carbonite's change tracking. ```elixir Carbonite.Migrations.put_trigger_config(:rabbits, :primary_key_columns, ["identifier"]) ``` -------------------------------- ### Insert Carbonite Transaction with Metadata Source: https://context7.com/bitcrowd/carbonite/llms.txt Insert a `Carbonite.Transaction` record within a `Repo.transaction` block. This function requires the repository and a map of metadata. Subsequent calls within the same transaction are ignored. ```elixir # Basic usage within Repo.transaction MyApp.Repo.transaction(fn -> {:ok, transaction} = Carbonite.insert_transaction(MyApp.Repo, %{ meta: %{ type: "user_registration", user_id: user.id, ip_address: conn.remote_ip |> :inet.ntoa() |> to_string() } }) %User{ மாதிரி: User.registration_changeset(params) } |> MyApp.Repo.insert!() end) ``` ```elixir # With custom carbonite prefix (for partitioned audit trails) MyApp.Repo.transaction(fn -> {:ok, _} = Carbonite.insert_transaction( MyApp.Repo, %{meta: %{type: "order_placed"}}, carbonite_prefix: "carbonite_orders" ) create_order(params) end) ``` -------------------------------- ### Configure Carbonite Triggers Source: https://context7.com/bitcrowd/carbonite/llms.txt Updates trigger configuration options for a specific table. Allows customization of primary key columns, exclusion of sensitive columns, filtering of column values, and enabling storage of previous values on updates. ```elixir defmodule MyApp.Repo.Migrations.ConfigureTriggers do use Ecto.Migration def change do # Exclude sensitive columns from audit trail Carbonite.Migrations.put_trigger_config(:users, :excluded_columns, ["password_hash", "api_key"]) # Filter columns (appear as '[FILTERED]' in data) Carbonite.Migrations.put_trigger_config(:users, :filtered_columns, ["email"]) # Custom primary key columns Carbonite.Migrations.put_trigger_config(:order_items, :primary_key_columns, ["order_id", "product_id"]) # Disable PK copying for tables without standard primary keys Carbonite.Migrations.put_trigger_config(:events, :primary_key_columns, []) # Store previous values on UPDATE (fills changed_from field) Carbonite.Migrations.put_trigger_config(:rabbits, :store_changed_from, true) # Set trigger mode to ignore (useful for test environments) Carbonite.Migrations.put_trigger_config(:rabbits, :mode, :ignore) end end ``` -------------------------------- ### Insert Transaction with Ecto.Multi Source: https://github.com/bitcrowd/carbonite/blob/main/README.md This code inserts a Carbonite.Transaction record, which may be empty if no changes are recorded in the database transaction. It uses Ecto.Multi for managing multiple operations. ```elixir Ecto.Multi.new() |> Carbonite.Multi.insert_transaction(%{meta: %{type: "rabbit_inserted"}}) |> Ecto.Multi.run(:rabbit, &maybe_insert_rabbit/2) |> MyApp.Repo.transaction() ``` -------------------------------- ### Insert Migration Transaction with Ecto.Migration Source: https://context7.com/bitcrowd/carbonite/llms.txt Use `insert_migration_transaction/1` within an Ecto migration to record a data migration event before executing SQL. This ensures that any modifications to audited tables are properly logged. ```elixir defmodule MyApp.Repo.Migrations.BackfillUserNames do use Ecto.Migration import Carbonite.Migrations def change do # Insert transaction before any data modifications insert_migration_transaction("backfill-user-names") execute( "UPDATE users SET display_name = CONCAT(first_name, ' ', last_name) WHERE display_name IS NULL", "UPDATE users SET display_name = NULL WHERE display_name = CONCAT(first_name, ' ', last_name)" ) end end ```