### Setup Development Environment with Docker Source: https://github.com/ambitioneng/django-pgbulk/blob/main/CONTRIBUTING.md Clones the repository and sets up a development environment using Docker. Requires Docker to be installed and running. ```bash git clone git@github.com:AmbitionEng/django-pgbulk.git cd django-pgbulk make docker-setup ``` -------------------------------- ### Setup Development Environment with Conda Source: https://github.com/ambitioneng/django-pgbulk/blob/main/CONTRIBUTING.md Sets up a development environment using Conda. Dependent services like databases need to be managed manually. ```bash git clone git@github.com:AmbitionEng/django-pgbulk.git cd django-pgbulk make conda-setup ``` -------------------------------- ### Build and Serve Documentation with Make Source: https://github.com/ambitioneng/django-pgbulk/blob/main/CONTRIBUTING.md Generates and serves the project documentation using Mkdocs Material. 'make docs' builds the documentation, and 'make docs-serve' builds and serves it locally. ```bash make docs ``` ```bash make docs-serve ``` -------------------------------- ### Run Tests with Make Source: https://github.com/ambitioneng/django-pgbulk/blob/main/CONTRIBUTING.md Executes the test suite for the project. 'make test' runs tests on a single Python version, while 'make full-test-suite' runs against all supported Python versions. ```bash make test ``` ```bash make full-test-suite ``` -------------------------------- ### Bulk Insert Model Instances with Django-pgbulk Source: https://github.com/ambitioneng/django-pgbulk/blob/main/docs/guide.md Demonstrates the basic usage of pgbulk.copy to insert a list of Django model instances into the database. This method provides a high-performance alternative to standard ORM bulk creation. ```python import pgbulk pgbulk.copy( models.TestModel, [ models.TestModel(int_field=5, float_field=1), models.TestModel(int_field=6, float_field=2), models.TestModel(int_field=7, float_field=3), ], ) ``` -------------------------------- ### pgbulk.copy - Fast Bulk Insert (COPY FROM) Source: https://context7.com/ambitioneng/django-pgbulk/llms.txt Performs bulk inserts using PostgreSQL's `COPY FROM` protocol, offering significant speed advantages over regular INSERT statements or Django's `bulk_create` for large datasets. This functionality requires psycopg3. It supports copying specific fields, excluding fields, and utilizing binary mode for enhanced performance with large data volumes. ```python import pgbulk from django.db import models class LogEntry(models.Model): timestamp = models.DateTimeField(auto_now_add=True) level = models.CharField(max_length=20) message = models.TextField() source = models.CharField(max_length=100) # Basic copy - insert many rows efficiently pgbulk.copy( LogEntry, [ LogEntry(level="INFO", message="Application started", source="main"), LogEntry(level="DEBUG", message="Config loaded", source="config"), LogEntry(level="INFO", message="Ready to serve", source="main"), # ... potentially thousands more rows ], ) # Copy specific fields only pgbulk.copy( LogEntry, [ LogEntry(level="ERROR", message="Connection failed", source="db"), ], ["level", "message"] # Only copy these fields (others use defaults) ) # Copy with exclude - copy all fields except specified ones pgbulk.copy( LogEntry, [ LogEntry(level="WARN", message="Slow query detected", source="monitor"), ], exclude=["timestamp"] # Let database generate timestamp ) # Binary mode for improved performance with large datasets pgbulk.copy( LogEntry, [ LogEntry(level="INFO", message=f"Batch entry {i}", source="batch") for i in range(10000) ], binary=True # Use binary format for faster copying ) ``` -------------------------------- ### Perform Bulk Copy with Django-PG-Bulk Source: https://github.com/ambitioneng/django-pgbulk/blob/main/docs/guide.md Explains the usage of pgbulk.copy for performing bulk data insertion using the `COPY ... FROM STDIN` statement. This method is noted for its speed advantage over bulk INSERT statements and Django's `bulk_create`, although it does not return inserted results. -------------------------------- ### Perform Bulk Upserts with pgbulk.upsert Source: https://context7.com/ambitioneng/django-pgbulk/llms.txt Demonstrates how to perform atomic bulk upserts using pgbulk.upsert. It covers basic upserts, returning categorized results (created vs updated), ignoring unchanged data, and performing insert-only operations. ```python import pgbulk from django.db import models class Product(models.Model): sku = models.CharField(max_length=50, unique=True) name = models.CharField(max_length=200) price = models.DecimalField(max_digits=10, decimal_places=2) stock = models.IntegerField(default=0) # Basic upsert pgbulk.upsert( Product, [ Product(sku="SKU001", name="Widget", price=19.99, stock=100), Product(sku="SKU002", name="Gadget", price=29.99, stock=50), ], ["sku"], ["name", "price"] ) # Upsert with returning results results = pgbulk.upsert( Product, [ Product(sku="SKU001", name="Widget Pro", price=24.99, stock=100), Product(sku="SKU003", name="New Item", price=9.99, stock=200), ], ["sku"], ["name", "price"], returning=True ) # Access categorized results print(f"Created: {len(results.created)}") print(f"Updated: {len(results.updated)}") # Upsert with ignore_unchanged pgbulk.upsert( Product, [Product(sku="SKU001", name="Widget Pro", price=24.99, stock=100)], ["sku"], ["name", "price"], ignore_unchanged=True ) # Bulk insert only pgbulk.upsert( Product, [Product(sku="SKU004", name="Another Item", price=15.99, stock=75)], ["sku"], [] ) ``` -------------------------------- ### Async Bulk Operations (aupsert, aupdate, acopy) Source: https://context7.com/ambitioneng/django-pgbulk/llms.txt Provides asynchronous variants for bulk operations (`aupsert`, `aupdate`, `acopy`) designed for use in async Django views and other asynchronous contexts. These functions wrap the synchronous `pgbulk` operations using `sync_to_async`, enabling non-blocking execution. ```python import pgbulk from django.db import models class Session(models.Model): session_id = models.CharField(max_length=64, unique=True) user_id = models.IntegerField() last_activity = models.DateTimeField(auto_now=True) data = models.JSONField(default=dict) # Async upsert async def update_sessions(session_data): results = await pgbulk.aupsert( Session, [ Session(session_id=s["id"], user_id=s["user"], data=s["data"]) for s in session_data ], ["session_id"], ["user_id", "data"], returning=True ) return {"created": len(results.created), "updated": len(results.updated)} # Async update async def refresh_sessions(session_ids, user_id): await pgbulk.aupdate( Session, [Session(session_id=sid, user_id=user_id) for sid in session_ids], ["user_id"] ) ``` -------------------------------- ### Configure Columns for Bulk Insertion Source: https://github.com/ambitioneng/django-pgbulk/blob/main/docs/guide.md Shows how to restrict or exclude specific model fields during the bulk copy process. Excluded columns must be database-generated, nullable, or have default values to avoid integrity errors. ```python pgbulk.copy( models.TestModel, [ models.TestModel(int_field=5, float_field=1), models.TestModel(int_field=6, float_field=2), models.TestModel(int_field=7, float_field=3), ], ["int_field"] # Only copy the int_field ) ``` ```python pgbulk.copy( models.TestModel, [...], exclude=["generated_field"] # Exclude only this field ) ``` -------------------------------- ### Copy Rows into Table Source: https://github.com/ambitioneng/django-pgbulk/blob/main/README.md Efficiently inserts multiple rows into a table using PostgreSQL's `COPY FROM` command, which can be significantly faster than `bulk_create`. ```APIDOC ## Copy Rows into Table ### Description Efficiently inserts multiple rows into a table using PostgreSQL's `COPY FROM` command, which can be significantly faster than `bulk_create`. ### Method `pgbulk.copy` ### Parameters #### Model - **model** (Model) - Required - The Django model class to copy data into. #### Instances - **instances** (list) - Required - A list of model instances to be copied. ### Request Example ```python import pgbulk pgbulk.copy( MyModel, # Insert these rows using COPY FROM [ MyModel(id=1, some_attr='some_val1'), MyModel(id=2, some_attr='some_val2') ], ) ``` ### Response This function does not return a value directly but performs the database operation. ``` -------------------------------- ### Async Copy Operation with pgbulk Source: https://context7.com/ambitioneng/django-pgbulk/llms.txt Demonstrates how to perform an asynchronous copy operation using pgbulk.acopy. This function is suitable for importing large datasets efficiently into a Django model asynchronously. ```python async def import_sessions(sessions): await pgbulk.acopy( Session, [ Session(session_id=s.id, user_id=s.user_id, data=s.data) for s in sessions ], ) ``` -------------------------------- ### Lint and Fix Code with Make Source: https://github.com/ambitioneng/django-pgbulk/blob/main/CONTRIBUTING.md Validates code quality using linters. 'make lint' checks for issues, and 'make lint-fix' attempts to automatically resolve common linting errors. ```bash make lint ``` ```bash make lint-fix ``` -------------------------------- ### Upsert Operation with Returning Results Source: https://context7.com/ambitioneng/django-pgbulk/llms.txt Illustrates how to perform an upsert operation using pgbulk.upsert and retrieve detailed results. The `returning=True` argument allows access to created and updated rows separately via the UpsertResult object. ```python import pgbulk from django.db import models class Inventory(models.Model): product_code = models.CharField(max_length=50, unique=True) quantity = models.IntegerField() warehouse = models.CharField(max_length=100) # Perform upsert with returning results = pgbulk.upsert( Inventory, [ Inventory(product_code="PROD-A", quantity=100, warehouse="Main"), Inventory(product_code="PROD-B", quantity=50, warehouse="Main"), Inventory(product_code="PROD-C", quantity=200, warehouse="Secondary"), ], ["product_code"], ["quantity", "warehouse"], returning=True ) # UpsertResult is a list-like object print(f"Total affected rows: {len(results)}") # Access all results (both created and updated) for row in results: print(f"Product: {row.product_code}, Qty: {row.quantity}") # Access only created rows for row in results.created: print(f"New inventory: {row.product_code}") # Access only updated rows for row in results.updated: print(f"Updated inventory: {row.product_code}") # Check status of individual rows using status_ attribute for row in results: status = "created" if row.status_ == "c" else "updated" print(f"{row.product_code}: {status}") ``` -------------------------------- ### Perform Bulk Copy with pgbulk Source: https://github.com/ambitioneng/django-pgbulk/blob/main/docs/index.md Uses the pgbulk.copy method to insert rows into a table using the PostgreSQL COPY FROM command. This is significantly faster than standard INSERT statements. ```python import pgbulk pgbulk.copy( MyModel, [ MyModel(id=1, some_attr='some_val1'), MyModel(id=2, some_attr='some_val2') ], ) ``` -------------------------------- ### pgbulk.copy Source: https://context7.com/ambitioneng/django-pgbulk/llms.txt Performs high-speed bulk inserts using PostgreSQL's COPY FROM protocol. ```APIDOC ## pgbulk.copy ### Description Performs bulk inserts using PostgreSQL's `COPY FROM` protocol, significantly faster than standard INSERT statements. Requires psycopg3. ### Parameters - **model** (Model) - Required - The Django model class to insert into. - **objs** (List) - Required - List of model instances to insert. - **fields** (List) - Optional - Specific fields to include in the copy. - **exclude** (List) - Optional - Fields to exclude from the copy. - **binary** (bool) - Optional - Use binary format for improved performance with large datasets. ### Request Example pgbulk.copy(LogEntry, [LogEntry(level="INFO", message="Test")], binary=True) ``` -------------------------------- ### Perform Expression-Based Updates with pgbulk.UpdateField Source: https://context7.com/ambitioneng/django-pgbulk/llms.txt Shows how to use pgbulk.UpdateField to apply SQL expressions during bulk operations. This allows for atomic increments or complex calculations directly within the database update statement. ```python import pgbulk from django.db import models from django.db.models import F class Counter(models.Model): name = models.CharField(max_length=100, unique=True) count = models.IntegerField(default=0) last_value = models.CharField(max_length=200, default="") # Increment counter atomically pgbulk.upsert( Counter, [ Counter(name="page_views", count=0, last_value="initial"), Counter(name="api_calls", count=0, last_value="initial"), ], ["name"], [ pgbulk.UpdateField("count", expression=F("count") + 1), "last_value" ], ) # Multiple expression-based updates pgbulk.upsert( Counter, [Counter(name="page_views", count=5, last_value="batch update")], ["name"], [ pgbulk.UpdateField("count", expression=F("count") + F("count")), "last_value" ], ) ``` -------------------------------- ### Perform Bulk Update with Django-PG-Bulk Source: https://github.com/ambitioneng/django-pgbulk/blob/main/docs/guide.md Illustrates how to perform a bulk update operation using pgbulk.update. This function executes a bulk UPDATE SET ... FROM VALUES statement. It allows specifying fields to update, returning results, and ignoring unchanged rows. ```python import pgbulk pgbulk.update( MyModel, [ MyModel(id=1, some_attr='some_val1'), MyModel(id=2, some_attr='some_val2') ], # These are the fields that will be updated. If not provided, # all fields will be updated ['some_attr'] ) ``` ```python pgbulk.update( MyModel, [ MyModel(some_int_field=0, some_key="a"), MyModel(some_int_field=0, some_key="b") ], [ # Use UpdateField to specify an expression for the update. pgbulk.UpdateField( "some_int_field", expression=models.F("some_int_field") + 1 ) ], ) ``` ```python results = pgbulk.upsert( MyModel, [ MyModel(int_field=1, some_attr="some_val1"), MyModel(int_field=2, some_attr="some_val2"), ], ["int_field", "some_attr"], # `True` will return all columns. One can also explicitly # list which columns will be returned. returning=True ) # Results can be accessed as a tuple print(results[0].int_field) ``` ```python pgbulk.upsert( MyModel, [ MyModel(int_field=1, some_attr="some_val1"), MyModel(int_field=2, some_attr="some_val2"), ], ["int_field"], ignore_unchanged=True ) ``` -------------------------------- ### pgbulk.update - True Bulk Update Source: https://context7.com/ambitioneng/django-pgbulk/llms.txt Performs a native bulk update using PostgreSQL's `UPDATE SET ... FROM VALUES` syntax. This executes as a single SQL statement, improving performance over Django's default `bulk_update`. Objects are sorted by primary key to minimize deadlock risks. Supports updating specific fields, returning updated rows, excluding fields, using SQL expressions, and ignoring unchanged rows. ```python import pgbulk from django.db import models from django.db.models import F class Employee(models.Model): employee_id = models.CharField(max_length=20, primary_key=True) name = models.CharField(max_length=200) department = models.CharField(max_length=100) salary = models.DecimalField(max_digits=10, decimal_places=2) # Basic bulk update - update specific fields for existing records pgbulk.update( Employee, [ Employee(employee_id="EMP001", department="Engineering", salary=85000), Employee(employee_id="EMP002", department="Sales", salary=72000), Employee(employee_id="EMP003", department="Marketing", salary=68000), ], ["department", "salary"] # Only update these fields (None = all fields) ) # Update with returning - get updated rows back results = pgbulk.update( Employee, [ Employee(employee_id="EMP001", salary=90000), Employee(employee_id="EMP002", salary=75000), ], ["salary"], returning=True # Return all fields (or pass list of specific fields) ) for row in results: print(f"Updated {row.employee_id}: new salary = {row.salary}") # Update with exclude - update all fields except specified ones pgbulk.update( Employee, [ Employee(employee_id="EMP001", name="John Doe", department="HR", salary=95000), ], None, # Update all fields... exclude=["salary"] # ...except salary ) # Update with expression - increment values atomically pgbulk.update( Employee, [ Employee(employee_id="EMP001", salary=0), # Value ignored due to expression Employee(employee_id="EMP002", salary=0), ], [ pgbulk.UpdateField("salary", expression=F("salary") * 1.1) # 10% raise ], ) # Update with ignore_unchanged - skip rows where values haven't changed pgbulk.update( Employee, [ Employee(employee_id="EMP001", department="Engineering"), ], ["department"], ignore_unchanged=True # Won't update if value is already "Engineering" ) ``` -------------------------------- ### Async Operations Source: https://context7.com/ambitioneng/django-pgbulk/llms.txt Asynchronous variants of bulk operations for use in async Django contexts. ```APIDOC ## Async Operations (aupsert, aupdate, acopy) ### Description All bulk operations have async variants that wrap the synchronous versions using `sync_to_async` for compatibility with async views. ### Methods - **aupsert** - Async version of upsert. - **aupdate** - Async version of update. - **acopy** - Async version of copy. ### Request Example await pgbulk.aupdate(Session, [Session(session_id="S1")], ["user_id"]) ``` -------------------------------- ### Perform Bulk Upsert with Django-PG-Bulk Source: https://github.com/ambitioneng/django-pgbulk/blob/main/docs/guide.md Demonstrates how to perform a bulk upsert operation using pgbulk.upsert. This function allows for atomic INSERT ON CONFLICT statements, enabling efficient updates or insertions of rows. It supports specifying unique constraint fields, fields to update, and returning results. ```python import pgbulk pgbulk.upsert( MyModel, [ MyModel(int_field=1, some_attr="some_val1"), MyModel(int_field=2, some_attr="some_val2"), ], # These are the fields that identify the uniqueness constraint. ["int_field"], # These are the fields that will be updated if the row already # exists. If not provided, all fields will be updated ["some_attr"] ) ``` ```python results = pgbulk.upsert( MyModel, [ MyModel(int_field=1, some_attr="some_val1"), MyModel(int_field=2, some_attr="some_val2"), ], ["int_field"], ["some_attr"], # `True` will return all columns. One can also explicitly # list which columns will be returned returning=True ) # Print which results were created print(results.created) # Print which results were updated. # By default, if an update results in no changes, it will not # be updated and will not be returned. print(results.updated) ``` ```python pgbulk.upsert( MyModel, [ MyModel(some_int_field=0, some_key="a"), MyModel(some_int_field=0, some_key="b") ], ["some_key"], [ # Use UpdateField to specify an expression for the update. pgbulk.UpdateField( "some_int_field", expression=models.F("some_int_field") + 1 ) ], ) ``` ```python pgbulk.upsert( MyModel, [ MyModel(int_field=1, some_attr="some_val1"), MyModel(int_field=2, some_attr="some_val2"), ], ["int_field"], ["some_attr"], ignore_unchanged=True ) ``` -------------------------------- ### pgbulk.update Source: https://context7.com/ambitioneng/django-pgbulk/llms.txt Performs a native bulk update using UPDATE SET ... FROM VALUES syntax to update multiple records in a single SQL statement. ```APIDOC ## pgbulk.update ### Description Performs a native bulk update using `UPDATE SET ... FROM VALUES` syntax. This executes as a single SQL statement regardless of the number of rows. ### Parameters - **model** (Model) - Required - The Django model class to update. - **objs** (List) - Required - List of model instances to update. - **fields** (List) - Optional - List of field names to update. If None, all fields are updated. - **returning** (bool) - Optional - If True, returns the updated rows. - **exclude** (List) - Optional - List of fields to exclude from the update. - **ignore_unchanged** (bool) - Optional - If True, skips rows where values have not changed. ### Request Example pgbulk.update(Employee, [Employee(employee_id="EMP001", salary=85000)], ["salary"]) ``` -------------------------------- ### Perform Bulk Update with pgbulk Source: https://github.com/ambitioneng/django-pgbulk/blob/main/docs/index.md Uses the pgbulk.update method to efficiently update a list of model instances in bulk. This is optimized to be faster than standard Django bulk_update operations. ```python import pgbulk pgbulk.update( MyModel, [ MyModel(id=1, some_attr='some_val1'), MyModel(id=2, some_attr='some_val2') ], ['some_attr'] ) ``` -------------------------------- ### Bulk Update Rows Source: https://github.com/ambitioneng/django-pgbulk/blob/main/README.md Performs true bulk updates on existing rows in the database, offering better performance and locking scenarios than Django's default `bulk_update`. ```APIDOC ## Bulk Update Rows ### Description Performs true bulk updates on existing rows in the database, offering better performance and locking scenarios than Django's default `bulk_update`. ### Method `pgbulk.update` ### Parameters #### Model - **model** (Model) - Required - The Django model class to perform the update on. #### Instances - **instances** (list) - Required - A list of model instances with updated values. #### Update Fields - **update_fields** (list of strings) - Optional - Fields to be updated. If not provided, all fields will be updated. ### Request Example ```python import pgbulk pgbulk.update( MyModel, [ MyModel(id=1, some_attr='some_val1'), MyModel(id=2, some_attr='some_val2') ], # These are the fields that will be updated. If not provided, # all fields will be updated ['some_attr'] ) ``` ### Response This function does not return a value directly but performs the database operation. ``` -------------------------------- ### Bulk Upsert Rows Source: https://github.com/ambitioneng/django-pgbulk/blob/main/README.md Performs bulk upserts (INSERT ON CONFLICT) for a list of model instances. It can distinguish between updated/created rows and ignore unchanged updates. ```APIDOC ## Bulk Upsert Rows ### Description Performs bulk upserts (INSERT ON CONFLICT) for a list of model instances. It can distinguish between updated/created rows and ignore unchanged updates. ### Method `pgbulk.upsert` ### Parameters #### Model - **model** (Model) - Required - The Django model class to perform the upsert on. #### Instances - **instances** (list) - Required - A list of model instances to upsert. #### Unique Fields - **unique_fields** (list of strings) - Required - Fields that identify the uniqueness constraint. #### Update Fields - **update_fields** (list of strings) - Optional - Fields to be updated if the row already exists. If not provided, all fields will be updated. ### Request Example ```python import pgbulk pgbulk.upsert( MyModel, [ MyModel(int_field=1, some_attr="some_val1"), MyModel(int_field=2, some_attr="some_val2"), ], # These are the fields that identify the uniqueness constraint. ["int_field"], # These are the fields that will be updated if the row already # exists. If not provided, all fields will be updated ["some_attr"] ) ``` ### Response This function does not return a value directly but performs the database operation. ``` -------------------------------- ### Perform Bulk Upsert with pgbulk Source: https://github.com/ambitioneng/django-pgbulk/blob/main/docs/index.md Uses the pgbulk.upsert method to perform an atomic update or insert operation based on a uniqueness constraint. It allows specifying which fields to update if a conflict occurs. ```python import pgbulk pgbulk.upsert( MyModel, [ MyModel(int_field=1, some_attr="some_val1"), MyModel(int_field=2, some_attr="some_val2"), ], ["int_field"], ["some_attr"] ) ``` -------------------------------- ### pgbulk.upsert Source: https://context7.com/ambitioneng/django-pgbulk/llms.txt Performs atomic bulk insert or update operations using PostgreSQL's INSERT ON CONFLICT syntax, with support for returning categorized results. ```APIDOC ## pgbulk.upsert ### Description Performs atomic bulk insert or update operations using PostgreSQL's `INSERT ... ON CONFLICT` syntax. Rows are automatically sorted by unique fields to reduce deadlock risk. ### Method Python Function Call ### Parameters - **model** (Model) - Required - The Django model class to perform the operation on. - **objs** (List[Model]) - Required - A list of model instances to upsert. - **unique_fields** (List[str]) - Required - The fields that define uniqueness (must have a unique constraint). - **update_fields** (List[str]) - Optional - Fields to update if the row already exists. Pass an empty list to perform insert-only (do nothing on conflict). - **returning** (bool) - Optional - If True, returns an object containing lists of created and updated instances. - **ignore_unchanged** (bool) - Optional - If True, skips updates if the values are identical to existing data. ### Request Example pgbulk.upsert(Product, [Product(sku="SKU001", name="Widget", price=19.99)], ["sku"], ["name", "price"]) ### Response #### Success Response - **results** (UpsertResult) - An object containing `.created` and `.updated` lists of model instances if `returning=True`. ``` -------------------------------- ### pgbulk.UpdateField Source: https://context7.com/ambitioneng/django-pgbulk/llms.txt Allows specifying SQL expressions for field updates during upsert or update operations, enabling atomic increments and complex calculations. ```APIDOC ## pgbulk.UpdateField ### Description Used within the `update_fields` parameter of `pgbulk.upsert` to apply SQL expressions (e.g., F() objects) to specific fields during an update. ### Method Python Class/Object ### Parameters - **field_name** (str) - Required - The name of the model field to update. - **expression** (Expression) - Required - A Django F() expression or other SQL expression to apply. ### Request Example pgbulk.upsert(Counter, [Counter(name="page_views")], ["name"], [pgbulk.UpdateField("count", expression=F("count") + 1)]) ### Response #### Success Response - **None** - Performs the update atomically in the database. ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.