Activerecord Import

https://github.com/zdennis/activerecord-import
Admin
A library for bulk insertion of data into your database using ActiveRecord.
Tokens:4,975
Snippets:40
Trust Score:8.9
License:MIT
Update:1 year ago
Show doc for...
Tokens:
Raw
### Installing Docker and Docker Compose via Homebrew

Source: https://github.com/zdennis/activerecord-import/blob/master/README.markdown

This Bash command demonstrates how to install both Docker and Docker Compose on macOS using Homebrew. This is a prerequisite for setting up the development environment using Docker containers for `activerecord-import`.

```bash
brew install docker && brew install docker-compose
```

--------------------------------

### Running Activerecord-Import Tests for Specific ActiveRecord Version

Source: https://github.com/zdennis/activerecord-import/blob/master/README.markdown

This Bash snippet provides an example command sequence for running the `activerecord-import` tests against a specific `ActiveRecord` version (e.g., 7.0). It first removes the `Gemfile.lock`, then installs dependencies for the specified `AR_VERSION`, and finally executes Rake tests for PostgreSQL, SQLite3, and MySQL2 databases.

```bash
rm Gemfile.lock
AR_VERSION=7.0 bundle install
AR_VERSION=7.0 bundle exec rake test:postgresql test:sqlite3 test:mysql2
```

--------------------------------

### Setting Up and Running Tests with Docker Compose

Source: https://github.com/zdennis/activerecord-import/blob/master/README.markdown

This Bash snippet outlines the steps to set up the development environment and run tests using Docker Compose. It includes commands to build and start the Docker services, access the application container's shell, and then execute the MySQL2 tests within that container.

```bash
docker-compose up --build
docker-compose exec app bash
bundle exec rake test:mysql2
```

--------------------------------

### Bulk Inserting Records with Activerecord-Import

Source: https://github.com/zdennis/activerecord-import/blob/master/README.markdown

This example shows how to use `activerecord-import` to efficiently insert multiple records. By building an array of `Book` objects and then calling `Book.import`, only a single SQL insert statement is executed, significantly improving performance compared to individual `create!` calls.

```Ruby
books = []
10.times do |i|
  books << Book.new(name: "book #{i}")
end
Book.import books    # or use import!
```

--------------------------------

### Importing Data with Columns and Arrays (With Validation)

Source: https://github.com/zdennis/activerecord-import/blob/master/README.markdown

This example illustrates importing data using column names and value arrays while explicitly enabling model validations. The `validate: true` option ensures that ActiveRecord validations are run on each record before insertion, providing data integrity at a slight performance cost.

```Ruby
columns = [ :title, :author ]
values = [ ['Book1', 'George Orwell'], ['Book2', 'Bob Jones'] ]

# Import with model validations
Book.import columns, values, validate: true
```

--------------------------------

### Importing Data with Hashes (With Validation)

Source: https://github.com/zdennis/activerecord-import/blob/master/README.markdown

This example shows how to import data using an array of hashes while enforcing model validations. The `validate: true` option ensures that each record conforms to the model's validation rules before being inserted into the database.

```Ruby
values = [{ title: 'Book1', author: 'George Orwell' }, { title: 'Book2', author: 'Bob Jones'}]

# Import with model validations
Book.import values, validate: true
```

--------------------------------

### Importing Specific Columns from Hashes (With Validation)

Source: https://github.com/zdennis/activerecord-import/blob/master/README.markdown

This example shows how to import data from an array of hashes, specifying a subset of columns to import (e.g., `title`). The `validate: true` option ensures that model validations are applied to the imported data, even when only specific columns are targeted.

```Ruby
books = [
  { title: "Book 1", author: "George Orwell" },
  { title: "Book 2", author: "Bob Jones" }
]
columns = [ :title ]

# with validations
Book.import columns, books, validate: true
```

--------------------------------

### Updating Column from Another Column Value on Duplicate Key

Source: https://github.com/zdennis/activerecord-import/blob/master/README.markdown

Illustrates how to update one column (`author`) with the value from another column (`title`) during a duplicate key import. It provides examples for both MySQL and PostgreSQL, highlighting that PostgreSQL requires specifying `conflict_target`. The example shows `author` being updated from `title`'s new value, while `title` itself remains unchanged.

```Ruby
book = Book.create! title: "Book1", author: "George Orwell"
book.title = "Updated Book Title"

# MySQL version
Book.import [book], on_duplicate_key_update: {author: :title}

# PostgreSQL version (no shorthand version)
Book.import [book], on_duplicate_key_update: {
  conflict_target: [:id], columns: {author: :title}
}
```

--------------------------------

### Resetting ActiveRecord Model Sequence Name in Multi-Tenant Environment

Source: https://github.com/zdennis/activerecord-import/blob/master/README.markdown

This Ruby snippet provides an alternative workaround for the `apartment` gem conflict, demonstrating how to call `.reset_sequence_name` on the `Post` model within a transaction for each tenant schema. This ensures that the `sequence_name` is correctly re-evaluated before performing `Post.import` operations in a multi-tenant setup.

```ruby
schemas.all.each do |schema|
  Apartment::Tenant.switch! schema.name
  ActiveRecord::Base.transaction do
    Post.reset_sequence_name

    Post.import posts
  end
end
```

--------------------------------

### Manually Loading in Scripts Establishing Connection (Ruby)

Source: https://github.com/zdennis/activerecord-import/blob/master/README.markdown

This snippet illustrates the manual loading process for activerecord-import in a Ruby script where gem dependencies are not autoloaded and the script itself establishes the database connection. It first requires active_record to set up the ORM and then activerecord-import, ensuring the import functionality is available after the database connection is ready.

```Ruby
require 'active_record'
require 'activerecord-import'
```

--------------------------------

### Defining Adapter Folder Structure for RubyGems Load Path

Source: https://github.com/zdennis/activerecord-import/blob/master/README.markdown

This snippet illustrates the recommended folder structure for an `activerecord-import` adapter gem (e.g., `activerecord-import-fake_name`). This structure ensures that RubyGems can correctly locate and load the `fake_name_adapter.rb` file when `require` is called, by pushing the `lib` folder onto the `$LOAD_PATH`.

```bash
activerecord-import-fake_name/
|-- activerecord-import-fake_name.gemspec
|-- lib
|   |-- activerecord-import-fake_name.rb
|   |-- activerecord-import-fake_name
|   |   |-- version.rb
|   |-- activerecord-import
|   |   |-- active_record
|   |   |   |-- adapters
|   |   |       |-- fake_name_adapter.rb
```

--------------------------------

### Creating Multiple Records with Standard ActiveRecord

Source: https://github.com/zdennis/activerecord-import/blob/master/README.markdown

This snippet demonstrates the traditional way of creating multiple records using ActiveRecord's `create!` method. Each call to `create!` results in a separate SQL insert statement, leading to N+1 insert problems for bulk operations and significantly impacting performance.

```Ruby
10.times do |i|
  Book.create! name: "book #{i}"
end
```

--------------------------------

### Running Callbacks Before Bulk Import (Ruby)

Source: https://github.com/zdennis/activerecord-import/blob/master/README.markdown

This snippet demonstrates how to manually run before_save and before_create callbacks on a collection of in-memory ActiveRecord objects (books) before performing a bulk import. The false argument prevents after_save from running, which is crucial as the actual save operation hasn't occurred yet. This approach ensures certain pre-import logic is executed.

```Ruby
books.each do |book|
  book.run_callbacks(:save) { false }
  book.run_callbacks(:create) { false }
end
Book.import(books)
```

--------------------------------

### Manually Loading in Rails with Active Connection (Ruby)

Source: https://github.com/zdennis/activerecord-import/blob/master/README.markdown

This snippet demonstrates how to manually load activerecord-import within a Rails environment where an ActiveRecord database connection has already been established. It requires the base functionality and then explicitly loads the specific database adapter (e.g., postgresql_adapter) to ensure activerecord-import is properly initialized and compatible with the active database connection.

```Ruby
require 'activerecord-import/base'
# load the appropriate database adapter (postgresql, mysql2, sqlite3, etc)
require 'activerecord-import/active_record/adapters/postgresql_adapter'
```

--------------------------------

### Importing Data with Columns and Arrays (No Validation)

Source: https://github.com/zdennis/activerecord-import/blob/master/README.markdown

This snippet demonstrates the fastest import mechanism using `activerecord-import` by providing an array of column names and an array of value arrays. The `validate: false` option skips model validations, offering maximum performance for trusted data where validation is not required.

```Ruby
columns = [ :title, :author ]
values = [ ['Book1', 'George Orwell'], ['Book2', 'Bob Jones'] ]

# Importing without model validations
Book.import columns, values, validate: false
```

--------------------------------

### Configuring Gemfile for Manual Loading (Ruby)

Source: https://github.com/zdennis/activerecord-import/blob/master/README.markdown

This Gemfile entry configures activerecord-import to not be automatically required by Bundler. Setting require: false allows developers to manually load the gem or specific parts of it only when needed, providing more granular control over the loading process and potentially optimizing application startup time.

```Ruby
gem 'activerecord-import', require: false
```

--------------------------------

### Accessing Import Result Information

Source: https://github.com/zdennis/activerecord-import/blob/master/README.markdown

Illustrates how to capture and inspect the `Result` object returned by the `import` method. It shows how to initialize an array of `Article` objects, perform an import with the `returning: :title` option, and then access properties like `failed_instances`, `num_inserts`, `ids` (for Postgres), and `results` (for Postgres) to understand the outcome of the bulk import.

```Ruby
articles = [
  Article.new(author_id: 1, title: 'First Article', content: 'This is the first article'),
  Article.new(author_id: 2, title: 'Second Article', content: ''),
  Article.new(author_id: 3, content: '')
]

demo = Article.import(articles, returning: :title)
```

--------------------------------

### Monitoring Batch Import Progress with Callback in Ruby

Source: https://github.com/zdennis/activerecord-import/blob/master/README.markdown

Demonstrates how to use the `batch_progress` option to provide a callable (e.g., a Proc) that will be invoked after each batch import. This allows for real-time progress reporting, enabling actions like sending notifications or updating UI elements during long-running imports.

```Ruby
my_proc = ->(rows_size, num_batches, current_batch_number, batch_duration_in_secs) {
  # Using the arguments provided to the callable, you can
  # send an email, post to a websocket,
  # update slack, alert if import is taking too long, etc.
}

Book.import columns, books, batch_size: 2, batch_progress: my_proc
```

--------------------------------

### Autoloading ActiveRecord-Import via Bundler (Ruby)

Source: https://github.com/zdennis/activerecord-import/blob/master/README.markdown

This snippet shows the standard way to include the activerecord-import gem in a Ruby project using Bundler. By adding this line to the Gemfile, Bundler will automatically manage the gem's dependencies and make it available for use, especially in Rails applications where dependencies are typically autoloaded.

```Ruby
gem 'activerecord-import'
```

--------------------------------

### Validating and Importing Valid Models (Ruby)

Source: https://github.com/zdennis/activerecord-import/blob/master/README.markdown

This code snippet shows an alternative strategy for handling callbacks and validations with activerecord-import. It first iterates through a collection of books to separate them into valid_books and invalid_books based on book.valid?. Then, it runs before_save and before_create callbacks only on the valid_books before performing a bulk import with validate: false, ensuring only validated records are imported and callbacks are triggered appropriately.

```Ruby
valid_books = []
invalid_books = []

books.each do |book|
  if book.valid?
    valid_books << book
  else
    invalid_books << book
  end
end

valid_books.each do |book|
  book.run_callbacks(:save) { false }
  book.run_callbacks(:create) { false }
end

Book.import valid_books, validate: false
```

--------------------------------

### Updating with PostgreSQL Partial Indexes on Duplicate Key

Source: https://github.com/zdennis/activerecord-import/blob/master/README.markdown

Explains how to leverage PostgreSQL partial indexes for `on_duplicate_key_update`. It includes the SQL for creating a partial index and the Ruby code to specify `index_predicate` along with `conflict_target` and `columns` for the import operation. This ensures updates only occur when the index predicate is met.

```Ruby
book = Book.create! title: "Book1", author: "George Orwell", published_at: Time.now
book.author = "Bob Barker"

# in migration
execute <<-SQL
      CREATE INDEX books_published_at_index ON books (published_at) WHERE published_at IS NOT NULL;
    SQL

# PostgreSQL version
Book.import [book], on_duplicate_key_update: {
  conflict_target: [:id],
  index_predicate: "published_at IS NOT NULL",
  columns: [:author]
}
```

--------------------------------

### Importing Specific Columns from ActiveRecord Objects in Ruby

Source: https://github.com/zdennis/activerecord-import/blob/master/README.markdown

Demonstrates importing an array of ActiveRecord models while explicitly specifying which columns to import. This allows for partial imports, where only the listed columns are populated, leaving others as `NULL` or their default values. Validation options can also be applied.

```Ruby
books = [
  Book.new(title: "Book 1", author: "George Orwell"),
  Book.new(title: "Book 2", author: "Bob Jones")
]
columns = [ :title ]

# without validations
Book.import columns, books, validate: false

# with validations
Book.import columns, books, validate: true

# when not specified :validate defaults to true
Book.import columns, books

# result in table books
# title  | author
#--------|--------
# Book 1 | NULL
# Book 2 | NULL
```

--------------------------------

### Dynamically Requiring External Adapters (Ruby)

Source: https://github.com/zdennis/activerecord-import/blob/master/README.markdown

This snippet illustrates the naming convention activerecord-import uses to dynamically load external database adapters. When ActiveRecord::Import.require_adapter("fake_name") is called, it attempts to require a file following this pattern. This mechanism allows external gems to provide new adapter support without modifying the core activerecord-import gem.

```Ruby
require 'activerecord-import/active_record/adapters/fake_name_adapter'
```

--------------------------------

### Importing Specific Columns from Hashes (No Validation)

Source: https://github.com/zdennis/activerecord-import/blob/master/README.markdown

This snippet demonstrates importing data from an array of hashes, explicitly specifying which columns to import using the `columns` array. Only the `title` field will be imported, and `validate: false` is used to skip model validations for faster insertion.

```Ruby
books = [
  { title: "Book 1", author: "George Orwell" },
  { title: "Book 2", author: "Bob Jones" }
]
columns = [ :title ]

# without validations
Book.import columns, books, validate: false
```

--------------------------------

### Importing Data with Hashes (No Validation)

Source: https://github.com/zdennis/activerecord-import/blob/master/README.markdown

This snippet demonstrates importing data using an array of hash objects, where hash keys map to column names. The `validate: false` option is used to bypass model validations, providing a faster import for data that is already known to be valid.

```Ruby
values = [{ title: 'Book1', author: 'George Orwell' }, { title: 'Book2', author: 'Bob Jones'}]

# Importing without model validations
Book.import values, validate: false
```

--------------------------------

### Importing ActiveRecord Objects with Validations in Ruby

Source: https://github.com/zdennis/activerecord-import/blob/master/README.markdown

Illustrates how to import an array of ActiveRecord model instances using `activerecord-import`. It shows how to control validation behavior during the import process by setting the `validate` option to `true` or `false`. By default, `validate` is `true`.

```Ruby
books = [
  Book.new(title: "Book 1", author: "George Orwell"),
  Book.new(title: "Book 2", author: "Bob Jones")
]

# without validations
Book.import books, validate: false

# with validations
Book.import books, validate: true

# when not specified :validate defaults to true
Book.import books
```

--------------------------------

### Updating with PostgreSQL Constraints on Duplicate Key

Source: https://github.com/zdennis/activerecord-import/blob/master/README.markdown

Demonstrates using a named unique constraint for `on_duplicate_key_update` in PostgreSQL. It provides the SQL for adding a unique constraint and the Ruby code to specify `constraint_name` and the columns to update (`:published_at`). This allows for more precise conflict resolution based on defined database constraints.

```Ruby
book = Book.create! title: "Book1", author: "George Orwell", edition: 3, published_at: nil
book.published_at = Time.now

# in migration
execute <<-SQL
      ALTER TABLE books
        ADD CONSTRAINT for_upsert UNIQUE (title, author, edition);
    SQL

# PostgreSQL version
Book.import [book], on_duplicate_key_update: {constraint_name: :for_upsert, columns: [:published_at]}
```

--------------------------------

### Importing Specific Columns from Hashes (Default Validation)

Source: https://github.com/zdennis/activerecord-import/blob/master/README.markdown

This snippet demonstrates importing data from an array of hashes, explicitly specifying columns to import. When `validate` is not specified, it defaults to `true`, meaning model validations will be performed on the records before insertion, ensuring data integrity for the specified columns.

```Ruby
books = [
  { title: "Book 1", author: "George Orwell" },
  { title: "Book 2", author: "Bob Jones" }
]
columns = [ :title ]

# when not specified :validate defaults to true
Book.import columns, books
```

--------------------------------

### Updating with Custom SQL on Duplicate Key

Source: https://github.com/zdennis/activerecord-import/blob/master/README.markdown

Shows how to use custom SQL expressions for `on_duplicate_key_update` in both MySQL and PostgreSQL. It demonstrates updating the `author` column using `values(author)` for MySQL and `excluded.author` for PostgreSQL, including a shorthand version for PostgreSQL when the primary key is the conflict target.

```Ruby
book = Book.create! title: "Book1", author: "George Orwell"
book.author = "Bob Barker"

# MySQL version
Book.import [book], on_duplicate_key_update: "author = values(author)"

# PostgreSQL version
Book.import [book], on_duplicate_key_update: {
  conflict_target: [:id], columns: "author = excluded.author"
}

# PostgreSQL shorthand version (conflict target must be primary key)
Book.import [book], on_duplicate_key_update: "author = excluded.author"
```

--------------------------------

### Importing Data with Inconsistent Hashes in Ruby

Source: https://github.com/zdennis/activerecord-import/blob/master/README.markdown

Demonstrates how `activerecord-import` handles arrays of hashes with inconsistent keys. Directly importing such an array will raise an exception. Provides two workarounds: converting hashes to ActiveRecord objects or grouping hashes by their keys before importing.

```Ruby
arr = [
  { bar: 'abc' },
  { baz: 'xyz' },
  { bar: '123', baz: '456' }
]

# An exception will be raised
Foo.import arr

# better
arr.map! { |args| Foo.new(args) }
Foo.import arr

# better
arr.group_by(&:keys).each_value do |v|
 Foo.import v
end
```

--------------------------------

### Updating Title with PostgreSQL Shorthand (Primary Key Conflict)

Source: https://github.com/zdennis/activerecord-import/blob/master/README.markdown

Demonstrates a shorthand method for updating a single column (`:title`) on duplicate key conflict in PostgreSQL. This method assumes the conflict target is the primary key. It shows how the `title` attribute is updated while `author` remains unchanged.

```Ruby
Book.import [book], on_duplicate_key_update: [:title]
```

--------------------------------

### Batch Importing Records with ActiveRecord-Import in Ruby

Source: https://github.com/zdennis/activerecord-import/blob/master/README.markdown

Explains how to use the `batch_size` option with `activerecord-import` to control the number of records inserted per SQL `INSERT` statement. This can optimize performance for large datasets by breaking down a single large insert into multiple smaller ones.

```Ruby
books = [
  Book.new(title: "Book 1", author: "George Orwell"),
  Book.new(title: "Book 2", author: "Bob Jones"),
  Book.new(title: "Book 1", author: "John Doe"),
  Book.new(title: "Book 2", author: "Richard Wright")
]
columns = [ :title ]

# 2 INSERT statements for 4 records
Book.import columns, books, batch_size: 2
```

--------------------------------

### Importing Data with Columns and Arrays (Default Validation)

Source: https://github.com/zdennis/activerecord-import/blob/master/README.markdown

This snippet shows the default behavior of `activerecord-import` when using columns and arrays: validations are enabled by default. This is equivalent to explicitly setting `validate: true`, ensuring data integrity without needing to specify the option.

```Ruby
columns = [ :title, :author ]
values = [ ['Book1', 'George Orwell'], ['Book2', 'Bob Jones'] ]

# when not specified :validate defaults to true
Book.import columns, values
```

--------------------------------

### Performing Recursive Imports for Associations in Ruby

Source: https://github.com/zdennis/activerecord-import/blob/master/README.markdown

Illustrates the `recursive` import option, which allows importing parent records and their associated child records (e.g., `has_many` associations) in a single operation. This feature is specific to PostgreSQL and requires ActiveRecord objects, not hashes or arrays, as input.

```Ruby
books = []
10.times do |i|
  book = Book.new(name: "book #{i}")
  book.reviews.build(title: "Excellent")
  books << book
end
Book.import books, recursive: true
```

--------------------------------

### Importing with Uniqueness Validation

Source: https://github.com/zdennis/activerecord-import/blob/master/README.markdown

Shows how to enable client-side uniqueness validation during an import operation using the `validate_uniqueness: true` option. This can prevent duplicate records from being inserted based on model validations before hitting the database.

```Ruby
Book.import books, validate_uniqueness: true
```

--------------------------------

### Upserting Records with Duplicate Key Update (PostgreSQL) in Ruby

Source: https://github.com/zdennis/activerecord-import/blob/master/README.markdown

This snippet illustrates the PostgreSQL-specific syntax for performing an upsert operation with `activerecord-import`. It requires specifying both the `conflict_target` (the unique index or primary key causing the conflict) and the `columns` to be updated when a duplicate key is encountered.

```Ruby
book = Book.create! title: "Book1", author: "George Orwell"
book.title = "Updated Book Title"
book.author = "Bob Barker"

# PostgreSQL version
Book.import [book], on_duplicate_key_update: {conflict_target: [:id], columns: [:title]}
```

--------------------------------

### Importing Data with Hashes (Default Validation)

Source: https://github.com/zdennis/activerecord-import/blob/master/README.markdown

This snippet illustrates the default behavior of `activerecord-import` when importing an array of hashes: model validations are performed automatically. This is equivalent to explicitly setting `validate: true`, ensuring data integrity by default.

```Ruby
values = [{ title: 'Book1', author: 'George Orwell' }, { title: 'Book2', author: 'Bob Jones'}]

# when not specified :validate defaults to true
Book.import values
```

--------------------------------

### Explicitly Setting ActiveRecord Model Sequence Name in Ruby

Source: https://github.com/zdennis/activerecord-import/blob/master/README.markdown

This Ruby snippet demonstrates how to explicitly set the `sequence_name` for an `ActiveRecord` model, such as `Post`. This workaround addresses a caching issue that can occur when `activerecord-import` interacts with the `apartment` gem, ensuring the correct sequence is used for primary key generation.

```ruby
class Post < ActiveRecord::Base
  self.sequence_name = "posts_seq"
end
```

--------------------------------

### Importing Records with Duplicate Key Ignore in Ruby

Source: https://github.com/zdennis/activerecord-import/blob/master/README.markdown

This snippet demonstrates how to use the `:on_duplicate_key_ignore` option with `activerecord-import`. It shows that when a record with a duplicate key is imported, the existing record's attributes remain unchanged, effectively skipping the update for the duplicate. This is useful for preventing overwrites of existing data.

```Ruby
book = Book.create! title: "Book1", author: "George Orwell"
book.title = "Updated Book Title"
book.author = "Bob Barker"

Book.import [book], on_duplicate_key_ignore: true

book.reload.title  # => "Book1"     (stayed the same)
book.reload.author # => "George Orwell" (stayed the same)
```

--------------------------------

### Upserting Records with Duplicate Key Update (MySQL) in Ruby

Source: https://github.com/zdennis/activerecord-import/blob/master/README.markdown

This snippet demonstrates how to perform an upsert operation using `activerecord-import` for MySQL databases. It updates the `title` column of an existing record if a primary or unique key conflict occurs, allowing specific fields to be updated on duplicate keys.

```Ruby
book = Book.create! title: "Book1", author: "George Orwell"
book.title = "Updated Book Title"
book.author = "Bob Barker"

# MySQL version
Book.import [book], on_duplicate_key_update: [:title]
```
Tokens:
Raw