### Install blktrace on Ubuntu Source: https://github.com/cockroachdb/pebble/blob/master/docs/io_profiling.md Installs the `blktrace` utility on Ubuntu systems using apt-get. ```bash sudo apt-get install blktrace ``` -------------------------------- ### Range Deletion Example Setup Source: https://github.com/cockroachdb/pebble/blob/master/docs/rocksdb.md Illustrates a multi-level LSM structure with point operations (pX) and range tombstones (rX) for understanding range deletion processing. ```text p0: o r0: m---q p1: n p r1: g---k p2: b d i r2: a---e q----v p3: e r3: ``` -------------------------------- ### Example blktrace output Source: https://github.com/cockroachdb/pebble/blob/master/docs/io_profiling.md Sample output from `blktrace` showing various block I/O events, including queueing, completion, and issue actions. ```text 259,0 4 186 0.016411295 11538 Q WS 129341760 + 296 [pebble] 259,0 4 187 0.016412100 11538 Q WS 129342016 + 40 [pebble] 259,0 4 188 0.016412200 11538 G WS 129341760 + 256 [pebble] 259,0 4 189 0.016412714 11538 G WS 129342016 + 40 [pebble] 259,0 4 190 0.016413148 11538 U N [pebble] 2 259,0 4 191 0.016413255 11538 I WS 129341760 + 256 [pebble] 259,0 4 192 0.016413321 11538 I WS 129342016 + 40 [pebble] 259,0 4 193 0.016414271 11538 D WS 129341760 + 256 [pebble] 259,0 4 194 0.016414860 11538 D WS 129342016 + 40 [pebble] 259,0 12 217 0.016687595 0 C WS 129341760 + 256 [0] 259,0 12 218 0.016700021 0 C WS 129342016 + 40 [0] ``` -------------------------------- ### Marker File Naming Convention Example Source: https://github.com/cockroachdb/pebble/blob/master/vfs/atomicfs/README.md Illustrates the expected file naming pattern for marker files, including marker name, iteration, and value. ```text marker... ``` ```text marker.foo.000001.alpha ``` ```text marker.foo.000002.beta ``` -------------------------------- ### Range Key and Point Key Examples Source: https://github.com/cockroachdb/pebble/blob/master/docs/RFCS/20211018_range_keys.md Illustrates point and range keys with sequence numbers, demonstrating how range keys are split across sstables. ```plaintext Point keys: a#50, b#70, b#49, b#48, b#47, c#46, d#45, e#44 Range key: [a,e)#60 ``` -------------------------------- ### Install Perf Tools on Ubuntu AWS Source: https://github.com/cockroachdb/pebble/blob/master/docs/io_profiling.md Installs common perf tools and specific versions for AWS Linux kernels. ```bash sudo apt-get install linux-tools-common linux-tools-4.4.0-1049-aws linux-cloud-tools-4.4.0-1049-aws ``` -------------------------------- ### RangeKeySet Fragmentation Example Source: https://github.com/cockroachdb/pebble/blob/master/docs/RFCS/20211018_range_keys.md Demonstrates how Pebble may internally fragment a RangeKeySet operation into multiple internal keys to preserve LSM tree performance characteristics. ```go RangeKeySet([a, c), @v3, value) is allowed and may be internally fragmented into: - RangeKeySet([a, a@v1), @v3, value) - RangeKeySet([a@v1, c@v2), @v3, value) - RangeKeySet([c@v2, c), @v3, value) ``` -------------------------------- ### Suffixed Keys and Compaction Split Example Source: https://github.com/cockroachdb/pebble/blob/master/docs/RFCS/20211018_range_keys.md Shows suffixed keys and how compaction splits range keys when a sstable is divided in the middle of a key prefix. ```plaintext first sstable: points: a@100, a@30, b@100, b@40 ranges: [a,c)@50 second sstable: points: b@30, c@40, d@40, e@30, ranges: [c,e)@50 ``` -------------------------------- ### Open, Set, Get, and Close Pebble Database Source: https://github.com/cockroachdb/pebble/blob/master/README.md Demonstrates basic operations: opening a database, setting a key-value pair, retrieving a value, and closing the database. Ensure proper error handling for all operations. ```go package main import ( "fmt" "log" "github.com/cockroachdb/pebble" ) func main() { db, err := pebble.Open("demo", &pebble.Options{}) if err != nil { log.Fatal(err) } key := []byte("hello") if err := db.Set(key, []byte("world"), pebble.Sync); err != nil { log.Fatal(err) } value, closer, err := db.Get(key) if err != nil { log.Fatal(err) } fmt.Printf("%s %s\n", key, value) if err := closer.Close(); err != nil { log.Fatal(err) } if err := db.Close(); err != nil { log.Fatal(err) } } ``` -------------------------------- ### Atomic Marker File Creation and Deletion Source: https://github.com/cockroachdb/pebble/blob/master/vfs/atomicfs/README.md Demonstrates the lifecycle of marker files, starting from an empty directory and progressing through 'Move' operations, showing the resulting file states. ```shell $ ls # (empty) marker, _, err := LocateMarker(fs, "/path", "task") marker.Move("started") $ ls /path marker.task.000001.started marker.Move("halfway") $ ls /path marker.task.000002.halfway marker.RemoveObsolete() $ ls /path marker.task.000002.halfway ``` -------------------------------- ### Analyze Perf Report Output Source: https://github.com/cockroachdb/pebble/blob/master/docs/io_profiling.md Example output from 'perf report --stdio' showing that 96.76% of block device requests originated from the 'pebble' process, with 85.58% attributed to WAL syncing. ```text 96.76% 0.00% pebble pebble [.] runtime.goexit | ---runtime.goexit | |--85.58%-- github.com/cockroachdb/pebble/internal/record.NewLogWriter.func2 | runtime/pprof.Do | github.com/cockroachdb/pebble/internal/record.(*LogWriter).flushLoop-fm | github.com/cockroachdb/pebble/internal/record.(*LogWriter).flushLoop | github.com/cockroachdb/pebble/internal/record.(*LogWriter).flushPending | github.com/cockroachdb/pebble/vfs.(*syncingFile).Sync | github.com/cockroachdb/pebble/vfs.(*syncingFile).syncFdatasync-fm | github.com/cockroachdb/pebble/vfs.(*syncingFile).syncFdatasync | syscall.Syscall | entry_SYSCALL_64_fastpath | sys_fdatasync | do_fsync | vfs_fsync_range | ext4_sync_file | filemap_write_and_wait_range | __filemap_fdatawrite_range | do_writepages | ext4_writepages | blk_finish_plug | blk_flush_plug_list | blk_mq_flush_plug_list | blk_mq_insert_requests ``` -------------------------------- ### Range Key Merging Iterator Initialization Example Source: https://github.com/cockroachdb/pebble/blob/master/docs/RFCS/20211018_range_keys.md Illustrates the initial state of child iterators when initializing a range key merging iterator. This shows the bounds and fragments each child iterator is positioned at. ```text i0: (b, boundKindStart, [ [b,d) ]) i1: (a, boundKindStart, [ [a,c) ]) i2: (a, boundKindStart, [ [a,p) ]) ``` -------------------------------- ### Range Delete Splitting Example Source: https://github.com/cockroachdb/pebble/blob/master/docs/range_deletions.md Demonstrates how a range delete is split across two output files from a compaction. The split point is determined by the first key of the subsequent sstable. ```text Consider the following example of an input range delete `["c", "h")#10` and the following two output files from a compaction: ``` sst1 sst2 last point is "e"#7 | first point is "f"#20 ``` The range delete can be split into `["c", "f")#10` and `["f", "h")#10`, by using the first point key of sst2 as the split point. Then the bounds of sst1 and sst2 will be `[..., "f"#inf]` and `["f"#20, ...]` which are non-overlapping. ``` -------------------------------- ### Range Key Fragmentation Example Source: https://github.com/cockroachdb/pebble/blob/master/docs/RFCS/20211018_range_keys.md This example demonstrates how overlapping range keys are fragmented and surfaced during iteration. The iterator stops at intersection points, surfacing all range keys that cover the current key, potentially fragmented by their intersection points. ```text RangeKeySet([a,z), @1, 'apple') RangeKeySet([c,e), @3, 'banana') RangeKeySet([e,m), @5, 'orange') RangeKeySet([b,k), @7, 'kiwi') @7 → kiwi |-----------------) @5 → orange |---------------) @3 → banana |---) @1 → apple |-------------------------------------------------) a b c d e f g h i j k l m n o p q r s t u v w x y z Key() = a RangeKeyBounds() = [a,b) RangeKeys() = {(@1,apple)} Key() = b RangeKeyBounds() = [b,c) RangeKeys() = {(@7,kiwi), (@1,apple)} Key() = c RangeKeyBounds() = [c,e) RangeKeys() = {(@7,kiwi), (@3,banana), (@1,apple)} Key() = e RangeKeyBounds() = [e,k) RangeKeys() = {(@7,kiwi), (@5,orange), (@1,apple)} Key() = k RangeKeyBounds() = [k,m) RangeKeys() = {(@5,orange), (@1,apple)} Key() = m RangeKeyBounds() = [m,z) RangeKeys() = {(@1,apple)} ``` -------------------------------- ### Interpret perf script block event Source: https://github.com/cockroachdb/pebble/blob/master/docs/io_profiling.md This example shows the output of a `perf script` event for a block I/O operation. It breaks down the fields indicating device, flags, offset, and size. ```text 259,0 WS 0 () 3970952 + 256 | | | + size (sectors) | | | | | + offset (sectors) | | | +- flags: R(ead), W(rite), B(arrier), S(ync), D(iscard), N(one) | +- device: , ``` -------------------------------- ### Iterator API for Range Keys Source: https://github.com/cockroachdb/pebble/blob/master/docs/RFCS/20211018_range_keys.md These methods on the Pebble iterator provide access to range key information. Use `HasPointAndRange` to check for point or range keys, `RangeKeyChanged` to detect changes in range key surfacing, `Key` to get the current key, `RangeBounds` for the start and end of the covering range key, `Value` for the point key's value, and `RangeKeys` to retrieve all range key values at the current position. ```go HasPointAndRange() (hasPoint, hasRange bool) RangeKeyChanged() bool Key() []byte RangeBounds() (start, end []byte) Value() []byte RangeKeys() []RangeKey type RangeKey struct { Suffix []byte Value []byte } ``` -------------------------------- ### Get Max Table Format Version Source: https://github.com/cockroachdb/pebble/blob/master/docs/RFCS/20220112_pebble_sstable_format_versions.md Retrieves the maximum sstable.TableFormat supported by a given FormatMajorVersion. Used to understand feature capabilities. ```go // Existing verisons. FormatDefault.MaxTableFormat() // sstable.TableFormatRocksDBv2 ... FormatSetWithDelete.MaxTableFormat() // sstable.TableFormatRocksDBv2 // Proposed versions with Pebble version scheme. FormatBlockPropertyCollector.MaxTableFormat() // sstable.TableFormatPebbleDBv1 FormatRangeKeys.MaxTableFormat() // sstable.TableFormatPebbleDBv2 ``` -------------------------------- ### FilemetaData Structure Change Source: https://github.com/cockroachdb/pebble/blob/master/docs/RFCS/20221122_virtual_sstable.md Illustrates a change in the FilemetaData structure, adding virtualSizeSumInLatestVersion. ```go type FilemetaData struct { virtualSizeSumInLatestVersion uint64 } ``` -------------------------------- ### Use blktrace and blkparse Source: https://github.com/cockroachdb/pebble/blob/master/docs/io_profiling.md Pipe the output of `blktrace` directly into `blkparse` to analyze block layer I/O events. This is equivalent to using the `btrace` command. ```bash sudo blktrace -d /dev/nvme1n1 -o - | blkparse -i - ``` ```bash sudo btrace /dev/nvme1n1 ``` -------------------------------- ### RocksDB Sliding Window Approach Source: https://github.com/cockroachdb/pebble/blob/master/docs/RFCS/20240701_tombstone_density_heuristic.md This is an adaptation of the sliding window approach used in RocksDB for tombstone density. It schedules compaction if a window of keys contains a high ratio of tombstones. ```go func (c *compactOnDeletionCollector) Collect(key, value []byte) error { if !c.isTombstone(key) { return nil } // If the current key is a tombstone, we increment the tombstone count. c.tombstones++ // If the window size has been reached, we check if the tombstone ratio is high enough. if c.keys >= c.windowSize { // If the tombstone ratio exceeds the threshold, we schedule a compaction. if float64(c.tombstones)/float64(c.keys) >= c.deletionThreshold { c.needsCompaction = true } // Slide the window by removing the oldest key's contribution. firstKey, _ := c.keysQueue.DequeueOrNil() if firstKey != nil { // If the oldest key was a tombstone, decrement the tombstone count. if c.isTombstone(firstKey.([]byte)) { c.tombstones-- } } c.keys-- } // Add the current key to the window. c.keysQueue.Enqueue(key) c.keys++ return nil } ``` -------------------------------- ### Record Block Device Requests with Perf Source: https://github.com/cockroachdb/pebble/blob/master/docs/io_profiling.md Records all block device I/O requests with stack traces. Requires root privileges for access to /sys/kernel/debug/tracing. Use Ctrl-C to stop recording. ```bash # Trace all block device (disk I/O) requests with stack traces, until Ctrl-C. sudo perf record -e block:block_rq_insert -ag ``` ```bash # Trace all block device (disk I/O) issues and completions with stack traces, until Ctrl-C. sudo perf record -e block:block_rq_issue -e block:block_rq_complete -ag ``` -------------------------------- ### SSTable Range Deletions and Bounds Example Source: https://github.com/cockroachdb/pebble/blob/master/docs/range_deletions.md Illustrates how range deletions interact with sstable points and bounds. The delete operation `["c", "g")#8` is applied to sstables with different point data and bounds. ```plaintext sst1 sst2 points: "c"#10 "e"#12 | "e"#10 delete: ["c", "g")#8 | ["c", "g")#8 bounds ["c"#10, "e"#12] | ["e"#10, "g"#inf] ``` ```plaintext sst3 points: "a"#14 "d"#15 bounds ["a"#14, "d"#15] ``` ```plaintext sst4 points: "a"#14 "c"#10 "d"#15 "e"#12 delete: ["c", "g")#8 bounds ["a"#14, "e"#12] ``` ```plaintext sst4 points: "a"#14 "c"#10 "d"#15 delete: ["c", "g")#8 bounds ["a"#14, "d"#15] ``` -------------------------------- ### List all events from perf.data Source: https://github.com/cockroachdb/pebble/blob/master/docs/io_profiling.md Use `perf script` to list all events from a `perf.data` file, including recommended header and fields for detailed analysis. ```bash sudo perf script --header -F comm,pid,tid,cpu,time,event,ip,sym,dso,trace ``` -------------------------------- ### Pebble pickAuto Compaction Logic Sketch Source: https://github.com/cockroachdb/pebble/blob/master/docs/RFCS/20240701_tombstone_density_heuristic.md This sketch outlines the logic within Pebble's pickAuto function for checking SSTables eligible for tombstone density compaction. It involves gathering statistics about tombstones and keys within SSTables and the broader LSM tree. ```go func (p *Pebble) pickAuto(level int) (*compaction, error) { // ... existing pickAuto logic ... // Check for SSTables eligible for tombstone density compaction. for _, sstable := range p.needsTombstoneCheck { // Get tombstone and key counts for the current SSTable. tombstonesInSSTable := p.annotator.TombstoneCount(sstable) internalKeysInSSTable := p.annotator.InternalKeyCount(sstable) // Get tombstone and key counts for the overlapping key range across the whole LSM. a, b := sstable.KeyRange() tombstonesInLSM := p.annotator.TombstoneCountInRange(a, b) internalKeysInLSM := p.annotator.InternalKeyCountInRange(a, b) // Calculate a signal for compaction priority. // This is a placeholder and requires further refinement. var compactionSignal float64 if internalKeysInLSM > 0 { compactionSignal = float64(tombstonesInSSTable) / float64(internalKeysInLSM) } // If the signal indicates a strong need for compaction, schedule it. if compactionSignal > p.tombstoneDensityThreshold { // Ensure the SSTable still exists before scheduling. if p.tableExists(sstable) { return p.createCompaction(sstable, level), nil } } } // ... rest of pickAuto logic ... return nil, nil } ``` -------------------------------- ### Record Block Device Requests for a Duration Source: https://github.com/cockroachdb/pebble/blob/master/docs/io_profiling.md Records block device I/O requests with stack traces for a specified duration. Requires root privileges. ```bash # Trace all block device (disk I/O) requests with stack traces for 10s. sudo perf record -e block:block_rq_insert -ag -- sleep 10 ``` -------------------------------- ### Create Batch for Ingested SSTs Source: https://github.com/cockroachdb/pebble/blob/master/docs/RFCS/20220311_pebble_flushable_ingested_sstable.md Create a new batch and add ingested SSTable paths to it. This is a preparatory step before applying the batch. ```go b := newBatch() for _, path := range paths: b.IngestSSTs([]byte(path), nil) ``` -------------------------------- ### Run All Tests Source: https://github.com/cockroachdb/pebble/blob/master/CLAUDE.md Execute all tests within the current directory and its subdirectories with invariants enabled. ```bash go test -tags invariants . ``` ```bash go test -tags invariants ./... ``` -------------------------------- ### Interpret blktrace output format Source: https://github.com/cockroachdb/pebble/blob/master/docs/io_profiling.md Explains the standard format of `blktrace` output, detailing each field from device to command. ```text + [] ``` -------------------------------- ### Iterator Positions with Range Keys Source: https://github.com/cockroachdb/pebble/blob/master/docs/RFCS/20211018_range_keys.md Illustrates the state of an iterator when configured to surface both point and range keys. It shows the Key, HasPointAndRange status, Value, RangeKeyBounds, and RangeKeys at various iteration positions. ```text Key() HasPointAndRange() Value() RangeKeyBounds() RangeKeys() a (true, true) artichoke [a,b) {(@1,apple)} b (false, true) - [b,c) {(@7,kiwi), (@1,apple)} b@2 (true, true) beet [b,c) {(@7,kiwi), (@1,apple)} c (false, true) - [c,e) {(@7,kiwi), (@3,banana), (@1,apple)} e (false, true) - [e,k) {(@7,kiwi), (@5,orange), (@1,apple)} k (false, true) - [k,m) {(@5,orange), (@1,apple)} m (false, true) - [m,z) {(@1,apple)} t@3 (true, true) turnip [m,z) {(@1,apple)} ``` -------------------------------- ### Upgrade Pebble Database Format Source: https://github.com/cockroachdb/pebble/blob/master/README.md Use this command to upgrade a Pebble database to the latest supported format major version. WARNING: only use if no custom comparer/merger/property collector are necessary. ```bash # WARNING: only use if no custom comparer/merger/property collector are necessary. go run github.com/cockroachdb/pebble/cmd/pebble@v1.1.3 db upgrade ``` -------------------------------- ### RangeKeySet Physical Representation Source: https://github.com/cockroachdb/pebble/blob/master/docs/RFCS/20211018_range_keys.md Illustrates the physical key-value format for RangeKeySet, showing how multiple logical range keys are merged into a single internal key. ```plaintext k1.RangeKeySet#s2 → (k2,[(@t2,v2),(@t1,v1)]) ``` -------------------------------- ### Commit Queue Scenario: Unapplied Batch at Head Source: https://github.com/cockroachdb/pebble/blob/master/docs/rocksdb.md Illustrates the state of the commit queue when an unapplied batch is at the head, preceding the batch whose sequence number is being published. This scenario requires waiting for the unapplied batch to complete. ```text +---------------+-------------+---------------+-----+ | A (unapplied) | B (applied) | C (unapplied) | ... | +---------------+-------------+---------------+-----+ ``` -------------------------------- ### Client-side Iterator Usage for Ingested SSTables Source: https://github.com/cockroachdb/pebble/blob/master/docs/RFCS/20220311_pebble_flushable_ingested_sstable.md Demonstrates how to use the iterator created for ingested SSTables on the client side. It shows how to wrap the base iterator and conditionally initialize a range deletion iterator based on the type of the flushable. ```go var iter internalIteratorWithStats var rangeDelIter keyspan.FragmentIterator iter = base.WrapIterWithStats(mem.newIter(&dbi.opts)) switch mem.flushable.(type) { case *ingestedSSTables: iter.(*levelIter).initRangeDel(&rangeDelIter) default: rangeDelIter = mem.newRangeDelIter(&dbi.opts) } mlevels = append(mlevels, mergingIterLevel{ iter: iter, rangeDelIter: rangeDelIter, }) ``` -------------------------------- ### RocksDB Batch Record Structure Source: https://github.com/cockroachdb/pebble/blob/master/docs/rocksdb.md Illustrates the structure of a single record within a RocksDB batch, including its kind, key, and value. ```text +----------+-----------------+-------------------+ | Kind (1) | Key (varstring) | Value (varstring) | +----------+-----------------+-------------------+ ``` -------------------------------- ### RangeKeyUnset Physical Representation Source: https://github.com/cockroachdb/pebble/blob/master/docs/RFCS/20211018_range_keys.md Shows the physical key-value format for RangeKeyUnset, demonstrating the merging of suffixes for identical bounds within snapshot stripes. ```plaintext k1.RangeKeyUnset#s2 → (k2,[(@t2),(@t1)]) ```