RedisShake

https://tair-opensource.github.io/RedisShake/en/
Admin
RedisShake is a tool for data migration and processing for Redis-like databases, supporting sync,...
Tokens:10,647
Snippets:110
Trust Score:8.3
Update:9 months ago
Show doc for...
Tokens:
Raw
### Compile RedisShake from Source Code (Shell)

Source: https://tair-opensource.github.io/RedisShake/en/guide/getting-started

This snippet shows how to compile RedisShake from its source code. It requires Git and a Golang environment. The output is the RedisShake executable.

```shell
git clone https://github.com/alibaba/RedisShake
cd RedisShake
sh build.sh
```

--------------------------------

### Run RedisShake Synchronization (Shell)

Source: https://tair-opensource.github.io/RedisShake/en/guide/getting-started

This command initiates the RedisShake process using a specified configuration file. Ensure the RedisShake executable is in the current directory and the configuration file is present.

```shell
./redis-shake shake.toml
```

--------------------------------

### Configure RedisShake Synchronization (TOML)

Source: https://tair-opensource.github.io/RedisShake/en/guide/getting-started

This TOML configuration file defines the source (reader) and destination (writer) Redis instances for RedisShake. It specifies the network addresses for both.

```toml
[sync_reader]
address = "127.0.0.1:6379"

[redis_writer]
address = "127.0.0.1:6380"
```

--------------------------------

### Inline Lua Script for Filtering in RedisShake Configuration

Source: https://tair-opensource.github.io/RedisShake/en/filter/function

This example demonstrates how to embed a Lua script directly within the `[filter]` section of the RedisShake configuration file. The script logs the database, discards data from `db 0`, and forwards data from other databases to the destination. It utilizes `shake.log` for logging and `shake.call` to emit commands.

```toml
[filter]
function = """
shake.log(DB)
if DB == 0
then
    return
end
shake.call(DB, ARGV)
"

[sync_reader]
address = "127.0.0.1:6379"

[redis_writer]
address = "127.0.0.1:6380"

```

--------------------------------

### Get Keyspace Information (Redis CLI)

Source: https://tair-opensource.github.io/RedisShake/en/others/consistent

This snippet shows how to retrieve keyspace information from a Redis instance using the Redis CLI. The output includes the total number of keys and the number of keys with expiration times, which can be used to assess data consistency.

```redis
127.0.0.1:6379> info keyspace
# Keyspace
db0:keys=4463175,expires=2,avg_ttl=333486
```

--------------------------------

### Lua Script to Filter Keys Starting with a Prefix in RedisShake

Source: https://tair-opensource.github.io/RedisShake/en/filter/function

This Lua script, intended for use within RedisShake's `[filter]` section, filters incoming data to only include keys that start with a specified prefix (e.g., 'user:'). It checks the key against the prefix and only proceeds to call `shake.call` if the condition is met. This snippet highlights key-based filtering but does not handle multi-key commands like `MSET`.

```lua
local prefix = "user:"
local prefix_len = #prefix

if not KEYS[1] or string.sub(KEYS[1], 1, prefix_len) ~= prefix then
  return
end

shake.call(DB, ARGV)

```

--------------------------------

### Modify Key Prefixes with Lua

Source: https://tair-opensource.github.io/RedisShake/en/filter/function

This Lua script demonstrates how to modify key prefixes during synchronization. It checks if a key starts with an old prefix and rewrites it with a new prefix before calling `shake.call`. `shake.log` is used for debugging.

```lua
local prefix_old = "prefix_old_"
local prefix_new = "prefix_new_"

shake.log("old=" .. table.concat(ARGV, " "))

for _, index in ipairs(KEY_INDEXES) do
  local key = ARGV[index]
  if key and string.sub(key, 1, #prefix_old) == prefix_old then
    ARGV[index] = prefix_new .. string.sub(key, #prefix_old + 1)
  end
end

shake.log("new=" .. table.concat(ARGV, " "))
shake.call(DB, ARGV)
```

--------------------------------

### Filter Lua Scripts with Lua

Source: https://tair-opensource.github.io/RedisShake/en/filter/function

This Lua script filters out 'SCRIPTING' type data, effectively discarding Lua scripts from the source. This is particularly useful when migrating from a master-slave setup to a cluster that doesn't support all Lua script features.

```lua
if GROUP == "SCRIPTING" then
  return
end
shake.call(DB, ARGV)
```

--------------------------------

### TOML Configuration File Structure for RedisShake

Source: https://tair-opensource.github.io/RedisShake/en/guide/config

This TOML snippet illustrates the basic structure of a RedisShake configuration file, outlining the main sections for reader, writer, filter, and advanced configurations. It serves as a template for setting up data synchronization tasks.

```toml
[xxx_reader]
...

[xxx_writer]
...

[filter]
...

[advanced]
...
```

--------------------------------

### Run redis-benchmark for Incremental Synchronization (Bash)

Source: https://tair-opensource.github.io/RedisShake/en/guide/architecture

This bash script utilizes the 'taskset' command to pin the 'redis-benchmark' process to specific CPU cores (0-15). It's used to generate continuous write traffic for the source Redis instance during the incremental synchronization phase of RedisShake. The command configures benchmark threads, dataset size, operation type, and connection parameters.

```bash
taskset -c 0-15 redis-benchmark \
  --threads 16 -r 10000000 -n 1000000000 -t set \
  -h host -a 'username:password' \
  --cluster -c 256 -d 8 -P 2
```

--------------------------------

### RedisShake Configuration Parameters

Source: https://tair-opensource.github.io/RedisShake/en/reader/scan_reader

Configuration settings for RedisShake, controlling source connection, synchronization behavior, and advanced features. These parameters define how RedisShake connects to and synchronizes data from a Redis source.

```ini
cluster = false
address = "127.0.0.1:6379"
username = ""
password = ""
tls = false
dbs = []
scan = true
ksn = false
count = 1
```

--------------------------------

### Configure RDB Reader - TOML

Source: https://tair-opensource.github.io/RedisShake/en/reader/rdb_reader

Configuration for the RDB reader, specifying the absolute filepath to the RDB backup file. This is essential for the reader to locate and process the data.

```toml
[rdb_reader]
filepath = "/tmp/dump.rdb"

```

--------------------------------

### Sync Reader Configuration (TOML)

Source: https://tair-opensource.github.io/RedisShake/en/reader/sync_reader

Configuration for the sync_reader in RedisShake, used for setting up source database connection details and synchronization preferences. It includes options for cluster mode, source address, authentication, TLS, and whether to synchronize RDB and AOF files.

```toml
[sync_reader]
cluster = false            # set to true if source is a redis cluster
address = "127.0.0.1:6379" # when cluster is true, set address to one of the cluster node
username = ""              # keep empty if not using ACL
password = ""              # keep empty if no authentication is required
tls = false
sync_rdb = true # set to false if you don't want to sync rdb
sync_aof = true # set to false if you don't want to sync aof

```

--------------------------------

### Split MSET Commands into SET Commands with Lua

Source: https://tair-opensource.github.io/RedisShake/en/filter/function

This Lua script optimizes command compatibility by splitting a single `MSET` command into multiple `SET` commands. It iterates through the arguments and calls `shake.call` for each key-value pair, improving compatibility with destinations that prefer single-key operations.

```lua
if CMD == "MSET" then
  for i = 2, #ARGV, 2 do
    shake.call(DB, {"SET", ARGV[i], ARGV[i + 1]})
  end
  return
end

shake.call(DB, ARGV)
```

--------------------------------

### Configure Redis Writer Connection and Authentication (TOML)

Source: https://tair-opensource.github.io/RedisShake/en/writer/redis_writer

This TOML configuration block defines how RedisShake connects to and authenticates with a Redis-like database for writing. It specifies connection address, cluster mode, username, password, and TLS settings.

```toml
[redis_writer]
cluster = false
address = "127.0.0.1:6379"
username = ""
password = ""
tls = false

```

--------------------------------

### Key Filtering Configuration

Source: https://tair-opensource.github.io/RedisShake/en/filter/filter

This TOML configuration defines rules for filtering keys based on exact names, prefixes, suffixes, and regular expressions. It includes both allow and block lists for comprehensive control over key synchronization. Regular expressions follow Go's syntax.

```toml
[filter]
allow_keys = ["user:1001", "product:2001"]          # allow-listed key names
allow_key_prefix = ["user:", "product:"]             # allow-listed key prefixes
allow_key_suffix = [":active", ":valid"]             # allow-listed key suffixes
allow_key_regex = [":\\d{11}:"]                     # allow-listed key regex (11-digit phone numbers)
block_keys = ["temp:1001", "cache:2001"]              # block-listed key names
block_key_prefix = ["temp:", "cache:"]                # block-listed key prefixes
block_key_suffix = [":tmp", ":old"]                  # block-listed key suffixes
block_key_regex = [":test:\\d{11}:"]                # block-listed key regex with "test" prefix

```

--------------------------------

### Configure Redis Sentinel for Sync Reader and Writer in TOML

Source: https://tair-opensource.github.io/RedisShake/en/guide/mode

This TOML configuration shows how to set up RedisShake to use Sentinel for obtaining Redis master addresses for both the sync reader and the redis writer. It specifies Sentinel master names and addresses, along with reader/writer specific credentials and TLS settings.

```toml
[sync_reader]
cluster = false
address = "" # The source Redis address will be obtained from Sentinel
username = ""
password = "redis6380password"
tls = false
[sync_reader.sentinel]
master_name = "mymaster"
address = "127.0.0.1:26380"
username = ""
password = ""
tls = false

[redis_writer]
cluster = false
address = "" # The target Redis address will be obtained from Sentinel
username = ""
password = "redis6381password"
tls = false
[redis_writer.sentinel]
master_name = "mymaster1"
address = "127.0.0.1:26380"
username = ""
password = ""
tls = false
```

--------------------------------

### Swap Databases with Lua

Source: https://tair-opensource.github.io/RedisShake/en/filter/function

This Lua script swaps the source and destination databases for specific database numbers (1 and 2). If the source DB is 1, it's sent to DB 2, and vice-versa. Other databases remain unaffected. This uses `shake.call` to route data to the modified DB.

```lua
local db1 = 1
local db2 = 2

if DB == db1 then
  DB = db2
elseif DB == db2 then
  DB = db1
end
shake.call(DB, ARGV)
```

--------------------------------

### Database Filtering Configuration

Source: https://tair-opensource.github.io/RedisShake/en/filter/filter

This TOML configuration specifies rules for filtering synchronization to specific logical databases. It allows for including a subset of databases or blocking specific ones. If no rules are set, all databases are synchronized.

```toml
[filter]
allow_db = [0, 1, 2]
block_db = [3, 4, 5]

```

--------------------------------

### Command and Command-Group Filtering Configuration

Source: https://tair-opensource.github.io/RedisShake/en/filter/filter

This TOML configuration enables filtering of synchronization based on Redis command names or command groups. This is useful for destinations that may not support certain commands, such as scripting or cluster administration commands.

```toml
[filter]
allow_command = ["GET", "SET"]
block_command = ["DEL", "FLUSHDB"]

allow_command_group = ["STRING", "HASH"]
block_command_group = ["SCRIPTING", "PUBSUB"]

```

--------------------------------

### Lua Script to Filter Data by Database in RedisShake

Source: https://tair-opensource.github.io/RedisShake/en/filter/function

This Lua script demonstrates filtering data based on the source database ID in RedisShake. It logs the current database ID and uses a conditional check (`if DB == 0`) to determine whether to process the command. If the database is not `0`, the command is passed to the destination using `shake.call`. This is a common pattern for segregating data from different Redis databases.

```lua
shake.log(DB)
if DB == 0
then
    return
end
shake.call(DB, ARGV)

```

--------------------------------

### Filter Hash Type Data with Lua

Source: https://tair-opensource.github.io/RedisShake/en/filter/function

This Lua script filters out data of type 'HASH' from the source before writing to the destination. It uses the `shake.call` function to process the remaining data. This is useful for synchronizing specific data types.

```lua
if GROUP == "HASH" then
  return
end
shake.call(DB, ARGV)
```
Tokens:
Raw