Embulk

https://www.embulk.org/docs
Admin
Embulk is an open-source bulk data loader that helps you easily transfer data between various data...
Tokens:4,591
Snippets:27
Trust Score:8.5
Update:1 year ago
Show doc for...
Tokens:
Raw
### Embulk Rename: Combined Rules Example

Source: https://www.embulk.org/docs/built-in

An example demonstrating the application of multiple renaming rules in sequence: converting to lower case, enforcing character types, and ensuring uniqueness.

```yaml
filters:
  ...
  - type: rename
    rules:
    - rule: upper_to_lower        # All upper-case are converted to lower-case.
    - rule: character_types       # Only lower-case, digits and "_" are allowed. (No upper-case by the rule ahove.)
      pass_types: [ "a-z", "0-9" ]
      pass_characters: "_"
    - rule: unique_number_suffix  # Ensure all column names are unique.
```

--------------------------------

### Embulk File Input Plugin Example

Source: https://www.embulk.org/docs/built-in

An example configuration for the Embulk 'file' input plugin. It shows how to specify the path prefix for input files and use the 'last_path' option to skip previously processed files.

```yaml
in:
  type: file
  path_prefix: /path/to/files/sample_
  last_path: /path/to/files/sample_02.csv
  parser:
    ...

```

--------------------------------

### Embulk JSON Parser Example

Source: https://www.embulk.org/docs/built-in

Shows an example of a JSON file containing multiple JSON objects, which the Embulk JSON parser plugin can process. Each line is treated as a separate JSON record.

```json
{"time":1455829282,"ip":"93.184.216.34","name":"frsyuki"}
{"time":1455829282,"ip":"172.36.8.109", "name":"sadayuki"}
{"time":1455829284,"ip":"example.com","name":"Treasure Data"}
{"time":1455829282,"ip":"10.98.43.1","name":"MessagePack"}
```

--------------------------------

### Embulk Preview Executor Configuration

Source: https://www.embulk.org/docs/built-in

Example configuration for Embulk's 'preview' executor, which reads sample data for console output. This configuration demonstrates adjusting the 'preview_sample_buffer_bytes' to control the amount of data read.

```yaml
exec:
  preview_sample_buffer_bytes: 65536 # 64KB
in:
  type: ...
  ...
out:
  type: ...
  ...

```

--------------------------------

### Embulk CSV Parser Configuration Example

Source: https://www.embulk.org/docs/built-in

This example demonstrates a typical configuration for the Embulk CSV parser plugin. It specifies the character set, newline format, delimiter, quote character, escape character, null string handling, header skipping, comment line marker, and detailed column definitions including types and timestamp formats.

```yaml
in:
  ...
  parser:
    type: csv
    charset: UTF-8
    newline: CRLF
    delimiter: "\t"
    quote: '"'
    escape: '"'
    null_string: 'NULL'
    skip_header_lines: 1
    comment_line_marker: '#'
    columns:
    - {name: id, type: long}
    - {name: account, type: long}
    - {name: time, type: timestamp, format: '%Y-%m-%d %H:%M:%S'}
    - {name: purchase, type: timestamp, format: '%Y%m%d'}
    - {name: comment, type: string}

```

--------------------------------

### Embulk Basic YAML Configuration

Source: https://www.embulk.org/docs/built-in

An example of a basic Embulk configuration file in YAML format. It specifies input from files with gzip decoding and CSV parsing, filters using a speedometer, and outputs to stdout.

```yaml
in:
  type: file
  path_prefix: ./mydata/csv/
  decoders:
  - {type: gzip}
  parser:
    charset: UTF-8
    newline: CRLF
    type: csv
    delimiter: ','
    quote: '"'
    escape: '"'
    null_string: 'NULL'
    skip_header_lines: 1
    columns:
    - {name: id, type: long}
    - {name: account, type: long}
    - {name: time, type: timestamp, format: '%Y-%m-%d %H:%M:%S'}
    - {name: purchase, type: timestamp, format: '%Y%m%d'}
    - {name: comment, type: string}
filters:
  - type: speedometer
    speed_limit: 250000
out:
  type: stdout

```

--------------------------------

### Embulk Rename: Deprecated Columns Option Example

Source: https://www.embulk.org/docs/built-in

An example showing the deprecated 'columns' option for renaming columns directly, which is applied before 'rules' if used together (though this combination is discouraged).

```yaml
filters:
  ...
  - type: rename
    columns:
      my_existing_column1: new_column1
      my_existing_column2: new_column2
```

--------------------------------

### Embulk Gzip Decoder Configuration

Source: https://www.embulk.org/docs/built-in

Example of how to apply the Gzip decoder plugin in Embulk to decompress gzip files before they are read by input plugins. This configuration is added within the 'decoders' section.

```yaml
in:
  ...
  decoders:
  - {type: gzip}
```

--------------------------------

### Embulk Guess Executor Configuration

Source: https://www.embulk.org/docs/built-in

Configuration example for Embulk's 'guess' executor, used by the 'guess' command. This snippet illustrates how to specify custom guess plugins using 'guess_plugins' and exclude default ones with 'exclude_guess_plugins'.

```yaml
exec:
  guess_plugins: ['csv_all_strings']
  exclude_guess_plugins: ['csv']
in:
  type: ...
  ...
out:
  type: ...
  ...

```

--------------------------------

### Embulk remove_columns Filter Plugin Configuration

Source: https://www.embulk.org/docs/built-in

Configuration example for the Embulk 'remove_columns' filter plugin, which removes specified columns from the schema. It demonstrates how to use the 'remove' option to list columns to be excluded.

```yaml
filters:
  ...
  - type: remove_columns
    remove: ["_c0", "_c1"]

```

--------------------------------

### Embulk Local Executor Plugin Configuration

Source: https://www.embulk.org/docs/built-in

Example configuration for Embulk's 'local' executor plugin, which manages task execution using local threads. It shows how to set 'max_threads' for concurrency and 'min_output_tasks' to control page scattering.

```yaml
exec:
  max_threads: 8         # run at most 8 tasks concurrently
  min_output_tasks: 1    # disable page scattering
in:
  type: ...
  ...
out:
  type: ...
  ...

```

--------------------------------

### Embulk Rename: first_character_types Rule

Source: https://www.embulk.org/docs/built-in

The first_character_types rule prefixes or replaces a restricted character at the beginning of a column name. It allows specifying characters or types to pass and defines a prefix or replacement character for disallowed starting characters.

```yaml
filters:
  ...
  - type: rename
    rules:
    - rule: first_character_types
      pass_characters: "_"
      pass_types: [ "a-z" ]
      prefix: "_"
```

--------------------------------

### Embulk Rename Filter with Character Type Rule

Source: https://www.embulk.org/docs/built-in

Applies the Embulk rename filter to modify column names. This example demonstrates the 'character_types' rule, which replaces disallowed characters with an underscore, allowing only underscores, lowercase alphabets, and digits.

```yaml
filters:
  ...
  - type: rename
    rules:
    - rule: character_types
      pass_characters: "_"
      pass_types: [ "a-z", "0-9" ]
```

--------------------------------

### Embulk File Output Plugin Configuration

Source: https://www.embulk.org/docs/built-in

Demonstrates the configuration for the Embulk File output plugin, specifying the path prefix for output files and the file extension. It also mentions the sequence format for naming files.

```yaml
out:
  type: file
  path_prefix: /path/to/output/sample_
  file_ext: csv
  formatter:
    ...
```

--------------------------------

### Embulk Configuration with Liquid Templates

Source: https://www.embulk.org/docs/built-in

Demonstrates using Liquid templates in Embulk configuration files for embedding environment variables and including other configuration files. This allows for dynamic configuration and modularity.

```yaml
in:
  type: file
  path_prefix: 
  decoders:
  - {type: gzip}
  parser:
    ...
out:
  type: postgresql
  host: 
  port: 
  user: 
  password: ""
  database: embulk_load
  mode: insert
  table: 

```

```yaml
# config.yml.liquid
{% include 'in_mysql' %}
out:
  type: stdout

```

```yaml
# _in_mysql.yml.liquid
in:
  type: mysql

```

```yaml
# Resulting configuration
in:
  type: mysql
out:
  type: stdout

```

--------------------------------

### Running Embulk Configuration

Source: https://www.embulk.org/docs/built-in

A bash command to execute an Embulk configuration file. This command initiates the data loading process defined in the specified YAML file.

```bash
$ embulk run config.yml.liquid

```

--------------------------------

### Embulk JSON Parser Configuration

Source: https://www.embulk.org/docs/built-in

Illustrates the configuration for the Embulk JSON parser plugin, specifying column names, types, and formats, including how to extract timestamp values.

```yaml
in:
  parser:
    type: json
    columns:
    - {name: time, type: timestamp, format: "%s"}
    - {name: ip, type: string}
    - {name: name, type: string}
```

--------------------------------

### Embulk CSV Parser Timestamp Handling

Source: https://www.embulk.org/docs/built-in

Demonstrates how to configure the Embulk CSV parser to handle UNIX timestamps in seconds using '%s' format. For millisecond timestamps, it shows parsing as 'long' and then using the 'timestamp_format' filter plugin.

```yaml
in:
  type: file
  path_prefix: /my_csv_files
  parser:
    ...
    columns:
    - {name: timestamp_in_seconds, type: timestamp, format: '%s'}
    - {name: timestamp_in_millis, type: long}
filters:
  - type: timestamp_format
    columns:
      - {name: timestamp_in_millis, from_unit: ms}
```

--------------------------------

### Embulk Gzip Encoder Configuration

Source: https://www.embulk.org/docs/built-in

Enables gzip compression for Embulk output files. Allows setting the compression level, ranging from 0 (no compression) to 9 (best compression), with a default of 6.

```yaml
out:
  ...
  encoders:
  - type: gzip
    level: 1
```

--------------------------------

### Embulk BZip2 Decoder Configuration

Source: https://www.embulk.org/docs/built-in

Shows the configuration for the BZip2 decoder plugin in Embulk, used to decompress bzip2 files. Similar to the Gzip decoder, it's specified under the 'decoders' section.

```yaml
in:
  ...
  decoders:
  - {type: bzip2}
```

--------------------------------

### Embulk BZip2 Encoder Configuration

Source: https://www.embulk.org/docs/built-in

Configures the BZip2 encoder plugin for Embulk to compress output files using BZip2. Supports setting the compression level from 1 to 9, with a default of 9.

```yaml
out:
  ...
  encoders:
  - type: bzip2
    level: 6
```

--------------------------------

### Embulk CSV Formatter Configuration

Source: https://www.embulk.org/docs/built-in

Configures the CSV formatter for Embulk, specifying options like delimiter, newline characters, character set, quote policy, escape characters, null string representation, default timezone, and column-specific options for timestamp formatting and timezones.

```yaml
out:
  ...
  formatter:
    type: csv
    delimiter: "\t"
    newline: CRLF
    newline_in_field: LF
    charset: UTF-8
    quote_policy: MINIMAL
    quote: '"'
    escape: "\\"
    null_string: "\\N"
    default_timezone: 'UTC'
    column_options:
      mycol1: {format: '%Y-%m-%d %H:%M:%S'}
      mycol2: {format: '%Y-%m-%d %H:%M:%S', timezone: 'America/Los_Angeles'}
```

--------------------------------

### Embulk Rename: lower_to_upper Rule

Source: https://www.embulk.org/docs/built-in

The lower_to_upper rule converts all lower-case alphabets in a column name to their upper-case equivalents.

```yaml
filters:
  ...
  - type: rename
    rules:
    - rule: lower_to_upper
```

--------------------------------

### Embulk Rename: regex_replace Rule

Source: https://www.embulk.org/docs/built-in

The regex_replace rule replaces parts of column names based on a provided Java-style regular expression. It requires a 'match' pattern and a 'replace' string, which can include backreferences.

```yaml
filters:
  ...
  - type: rename
    rules:
    - rule: regex_replace
      match: "([0-9]+)_dollars"
      replace: "USD$1"
```

--------------------------------

### Embulk Rename: upper_to_lower Rule

Source: https://www.embulk.org/docs/built-in

The upper_to_lower rule converts all upper-case alphabets in a column name to their lower-case equivalents.

```yaml
filters:
  ...
  - type: rename
    rules:
    - rule: upper_to_lower
```

--------------------------------

### Embulk Rename: truncate Rule

Source: https://www.embulk.org/docs/built-in

The truncate rule shortens column names to a specified maximum length. If not specified, it defaults to 128 characters.

```yaml
filters:
  ...
  - type: rename
    rules:
    - rule: truncate
      max_length: 20
```

--------------------------------

### Embulk Rename: unique_number_suffix Rule

Source: https://www.embulk.org/docs/built-in

The unique_number_suffix rule ensures all column names are unique within the schema by appending sequential numbers to duplicate names. It supports custom delimiters, zero-filled digits, and truncation before suffixing.

```yaml
filters:
  ...
  - type: rename
    rules:
    - rule: unique_number_suffix
```
Tokens:
Raw