### Running CorfuDB Server with SASL Plain-Text Authentication Source: https://github.com/corfudb/corfudb/blob/master/Security.md Shows an example command to start corfu_server with SASL authentication enabled. It includes options for TLS, keystores, and providing SASL username and password files for server-side authentication. ```bash bin/corfu_server -m -s -e -u test/src/test/resources/security/s1.jks -f test/src/test/resources/security/storepass -r test/src/test/resources/security/s1.jks -w test/src/test/resources/security/storepass -g -o test/src/test/resources/security/username1 -j test/src/test/resources/security/userpass1 9000 ``` -------------------------------- ### Start CorfuDB Single-Server Instance Source: https://github.com/corfudb/corfudb/blob/master/README.md Launches the CorfuDB server in an in-memory, single-server mode on a specified port (e.g., 9000). This setup is ideal for testing purposes, allowing clients to connect to `localhost:9000`. ```Shell ./CorfuDB/bin/corfu_server -ms 9000 ``` -------------------------------- ### Start Multiple CorfuDB Server Instances Source: https://github.com/corfudb/corfudb/blob/master/README.md This snippet demonstrates how to start two non-provisioned CorfuDB server instances on different ports (9000 and 9001) in the background. These instances will later be bootstrapped into a distributed cluster. ```bash $ ./CorfuDB/bin/corfu_server -m 9000 & $ ./CorfuDB/bin/corfu_server -m 9001 & ``` -------------------------------- ### Install Maven on Debian/Ubuntu Source: https://github.com/corfudb/corfudb/blob/master/README.md This command installs Apache Maven on Debian or Ubuntu systems using apt-get. It's used to build Corfu from source. Users should verify the installed version is 3.3 or later using `mvn --version`. ```bash $ sudo apt-get install maven ``` -------------------------------- ### Build CorfuDB Project with Maven Source: https://github.com/corfudb/corfudb/blob/master/README.md Compiles and packages the CorfuDB project using Maven. This command cleans previous builds and installs the necessary binaries, which will be located in the `bin` directory. ```Shell mvn clean install ``` -------------------------------- ### Run CorfuDB Server with TLS (CA Signed Certificates) Source: https://github.com/corfudb/corfudb/blob/master/Security.md Starts the CorfuDB server with TLS enabled, using the server's keystore (`s1.jks`) and the shared trust store (`trust.jks`) for CA-signed certificates. ```bash $ bin/corfu_server -m -e -u s1.jks -f s1pass.txt -r trust.jks -w tpass.txt --single 9000 ``` -------------------------------- ### Verify Maven and Java Prerequisites Source: https://github.com/corfudb/corfudb/blob/master/README.md Confirms the installed Maven version (3.3+) and Java version (11+) using the `mvn --version` command. The output helps identify the active Java Development Kit (JDK) and highlights potential JDK mismatch issues on OS X. ```Shell mvn --version ``` ```Shell Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T08:41:47-08:00) Maven home: /opt/local/share/java/maven3 Java version: 1.8.0_91, vendor: Oracle Corporation Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home/jre Default locale: en_US, platform encoding: UTF-8 OS name: "mac os x", version: "10.11.6", arch: "x86_64", family: "mac" ``` -------------------------------- ### Install Maven on Mac OS X with Homebrew Source: https://github.com/corfudb/corfudb/blob/master/README.md This command installs Apache Maven on Mac OS X using the Homebrew package manager. It's required to build Corfu from source. ```bash $ brew install maven ``` -------------------------------- ### Start New CorfuDB Server for Cluster Expansion Source: https://github.com/corfudb/corfudb/blob/master/README.md This command starts an additional CorfuDB server instance on port 9002. This new server is intended to be added to an existing distributed cluster, expanding its capacity and fault tolerance. ```bash ./CorfuDB/bin/corfu_server -m 9002 ``` -------------------------------- ### Install Maven on Mac OS X with MacPorts Source: https://github.com/corfudb/corfudb/blob/master/README.md This command installs Apache Maven 3 on Mac OS X using the MacPorts package manager. It's required to build Corfu from source. ```bash sudo port install maven3 ``` -------------------------------- ### Import RSA and ECDSA Keystores into Mixed Keystores using Keytool Source: https://github.com/corfudb/corfudb/blob/master/test/src/test/resources/security/README.md These `keytool` commands import both RSA and ECDSA keystores into combined mixed-type keystores. For example, `server_rsa.jks` and `server_ecdsa.jks` are imported into `server_rsa_ecdsa.jks`, creating a single JKS file containing both RSA and ECDSA keys. ```bash keytool -importkeystore -destkeystore server_rsa_ecdsa.jks -srcstoretype jks -deststoretype jks -srcstorepass test123 -deststorepass test123 -srckeystore server_rsa.jks keytool -importkeystore -destkeystore server_rsa_ecdsa.jks -srcstoretype jks -deststoretype jks -srcstorepass test123 -deststorepass test123 -srckeystore server_ecdsa.jks keytool -importkeystore -destkeystore runtime_rsa_ecdsa.jks -srcstoretype jks -deststoretype jks -srcstorepass test123 -deststorepass test123 -srckeystore runtime_rsa.jks keytool -importkeystore -destkeystore runtime_rsa_ecdsa.jks -srcstoretype jks -deststoretype jks -srcstorepass test123 -deststorepass test123 -srckeystore runtime_ecdsa.jks ``` -------------------------------- ### Corfu Node Locator String Format and Example Source: https://github.com/corfudb/corfudb/blob/master/docs/NODE_FORMAT.md Defines the human-readable string format for Corfu node locators, detailing the required and optional components such as protocol, host, port, node identifier, and connection options. An example demonstrates a complete locator string. ```Text (://):(/)(?) ``` ```Text tcp://10.0.0.1:9000/fZPF5eGIScaq9m1DabhaCQ ``` -------------------------------- ### Run CorfuDB Server with SASL Plaintext Authentication Source: https://github.com/corfudb/corfudb/blob/master/Security.md Starts the CorfuDB server with SASL plaintext authentication enabled, specifying keystores, truststores, and user credential files for server-side and client-side authentication. ```bash $ bin/corfu_server -m -e -u s1.jks -f s1pass.txt -r trust.jks -w tpass.txt -g -o username.txt -j userpassword.txt --single 9000 ``` -------------------------------- ### Inject Network Faults in Corfu Tests Source: https://github.com/corfudb/corfudb/blob/master/TestingFramework.md Simulate network failures like dropped packets using the `TestRule` class. Rules can be installed for server-to-client communication using `addServerRule(port, rule)` or client-to-server using `addClientRule(runtime, rule)`. This example demonstrates dropping all packets to server 9002. ```Java import org.corfudb.runtime.view.AbstractViewTest; class TestClass extends AbstractViewTest { @Test public canWriteToAStreamTest() { addServer(9000); addServer(9001); addServer(9002); bootstrapAllServers(new TestLayoutBuilder() .addLayoutServer(9000) .addSequencer(9000) .buildSegment() .setReplicationMode(Layout.ReplicationMode.CHAIN_REPLICATION) .buildStripe() .addLogUnit(9000) .addLogUnit(9001) .addLogUnit(9002) .addToSegment() .addToLayout() .build()); addServerRule(9002, new TestRule() .always() .drop()); CorfuRuntime myRuntime = getRuntime().connect(); StreamView sv = myRuntime.getStreamsView().open("a"); sv.write("hello world"); } } ``` -------------------------------- ### Define and Execute Interleaved Concurrency Test in Java Source: https://github.com/corfudb/corfudb/blob/master/TestingFramework.md This Java example demonstrates defining a multi-step test using `addTestStep()` for controlled thread interleaving. It sets up a `CorfuTable`, starts a transaction, performs a `put` operation, and attempts to commit, handling `TransactionAbortedException`. The test is then executed with `scheduleInterleaved` at a specified concurrency level and iteration count. ```Java @Test public void AbortTest() { Map testMap = getRuntime().getObjectsView().build() .setStreamID(UUID.randomUUID()) .setTypeToken(new TypeToken>() {}) .open(); AtomicInteger aborts = new AtomicInteger(); testMap.clear(); // state 0: start a transaction addTestStep((ignored_task_num) -> { getRuntime().getObjectsView().TXBegin(); }); // state 1: do a put addTestStep( (task_num) -> { testMap.put(Integer.toString(task_num), Integer.toString(task_num)); }); // state 2 (final): ask to commit the transaction addTestStep( (ignored_task_num) -> { try { getRuntime().getObjectsView().TXEnd(); } catch (TransactionAbortedException tae) { aborts.incrementAndGet(); } }); // invoke the interleaving engine scheduleInterleaved(PARAMETERS.CONCURRENCY_SOME, PARAMETERS.CONCURRENCY_SOME*PARAMETERS.NUM_ITERATIONS_LOW); // here, print abort stats, aborts.get(), etc. } ``` -------------------------------- ### Example CorfuDB Cluster Layout with Detected Failures and Statistics Source: https://github.com/corfudb/corfudb/blob/master/docs/failure-detector/filesystem-failures.md This JSON snippet illustrates a CorfuDB cluster layout, detailing its components such as layout servers, sequencers, and log servers within segments. It also provides insights into unresponsive servers and a list of detected failures. Each failure entry specifies the 'node' that failed, the 'failureType' (e.g., ReadOnlyFileSystem, QuotaExceeded), the 'failureDetector' node, a 'timestamp', and the 'clusterState' at the time of detection, including node connectivity. ```json { "layoutServers": [ "192.168.0.1:9000", "192.168.0.2:9000", "192.168.0.3:9000" ], "sequencers": [ "192.168.0.1:9000", "192.168.0.2:9000", "192.168.0.3:9000" ], "segments": [ { "replicationMode": "CHAIN_REPLICATION", "start": 0, "end": -1, "stripes": [ { "logServers": [ "192.168.0.1:9000", "192.168.0.2:9001", "192.168.0.3:9002" ] } ] } ], "unresponsiveServers": [ "192.168.0.3:9000", "192.168.0.2:9000" ], "failures": [ { "node": "192.168.0.3:9000", "failureType": "ReadOnlyFileSystem", "failureDetector": "192.168.0.1:9000", "timestamp": "1633972903", "clusterState": { "nodes": ["192.168.0.1:9000", "192.168.0.2:9000", "192.168.0.3:9000"], "connectivity": [ ["OK", "OK", "FAIL"], ["OK", "OK", "FAIL"], ["FAIL", "FAIL", "OK"] ] } }, { "node": "192.168.0.2:9000", "failureType": "QuotaExceeded", "failureDetector": "192.168.0.1:9000", "timestamp": "1633972903", "clusterState": { "nodes": ["192.168.0.1:9000", "192.168.0.2:9000", "192.168.0.3:9000"], "connectivity": [ ["OK", "FAIL", "OK"], ["FAIL", "OK", "OK"], ["OK", "FAIL", "OK"] ] } } ], "epoch": 0 } ``` -------------------------------- ### Query CorfuDB Cluster Layout Source: https://github.com/corfudb/corfudb/blob/master/README.md Retrieves the current configuration and status of the CorfuDB infrastructure, including layout servers, sequencers, and segments. This helps verify the cluster's setup and ensures the server is running correctly. ```Shell ./CorfuDB/bin/corfu_layouts -c localhost:9000 query ``` ```JSON { "layoutServers": [ "localhost:9000" ], "sequencers": [ "localhost:9000" ], "segments": [ { "replicationMode": "CHAIN_REPLICATION", "start": 0, "end": -1, "stripes": [ { "logServers": [ "localhost:9000" ] } ] } ], "unresponsiveServers": [], "epoch": 0, "clusterId": "fd3802dc-9db4-4a7c-98d6-1aecfbc964ae" } ``` -------------------------------- ### Add New Node to Existing CorfuDB Cluster Source: https://github.com/corfudb/corfudb/blob/master/README.md This command adds a newly started CorfuDB server (on port 9002) to an existing cluster. The command specifies the existing cluster's contact points and the new node's address, integrating it into the distributed system. ```bash ./CorfuDB/bin/corfu_add_node -c localhost:9000,localhost:9001 -n localhost:9002 ``` -------------------------------- ### Importing RSA and ECDSA Keystores for CorfuDB Source: https://github.com/corfudb/corfudb/blob/master/Security.md Demonstrates how to combine existing RSA and ECDSA keystores into a single JKS file for both server and runtime environments using keytool. This consolidates security credentials for easier management. ```bash keytool -importkeystore -destkeystore server_rsa_ecdsa.jks -srcstoretype jks -deststoretype jks -srcstorepass test123 -deststorepass test123 -srckeystore server_rsa.jks keytool -importkeystore -destkeystore server_rsa_ecdsa.jks -srcstoretype jks -deststoretype jks -srcstorepass test123 -deststorepass test123 -srckeystore server_ecdsa.jks keytool -importkeystore -destkeystore runtime_rsa_ecdsa.jks -srcstoretype jks -deststoretype jks -srcstorepass test123 -deststorepass test123 -srckeystore runtime_rsa.jks keytool -importkeystore -destkeystore runtime_rsa_ecdsa.jks -srcstoretype jks -deststoretype jks -srcstorepass test123 -deststorepass test123 -srckeystore runtime_ecdsa.jks ``` -------------------------------- ### Export Certificates from Keystores using Keytool Source: https://github.com/corfudb/corfudb/blob/master/Security.md These `keytool` commands optionally export certificates from the previously generated RSA and EC keystores (`server_ecdsa.jks`, `runtime_ecdsa.jks`, `server_rsa.jks`, `runtime_rsa.jks`) into `.cert` files. This step is noted as optional, as typically only the keystore/truststore files are needed. ```Bash # Saving Certs (OPTIONAL: We only need keystore/truststore, not the certs). keytool -exportcert -alias server_ecdsa -file server_ecdsa.cert -keystore server_ecdsa.jks -storepass test123 -storetype jks -rfc keytool -exportcert -alias runtime_ecdsa -file runtime_ecdsa.cert -keystore runtime_ecdsa.jks -storepass test123 -storetype jks -rfc keytool -exportcert -alias server_rsa -file server_rsa.cert -keystore server_rsa.jks -storepass test123 -storetype jks -rfc keytool -exportcert -alias runtime_rsa -file runtime_rsa.cert -keystore runtime_rsa.jks -storepass test123 -storetype jks -rfc ``` -------------------------------- ### Corfu Server Command-Line Interface (CLI) Reference Source: https://github.com/corfudb/corfudb/blob/master/Security.md Comprehensive reference for `corfu_server` command-line options, covering general server configuration, logging, caching, and detailed TLS and authentication settings. Each option's purpose, parameters, and default values are provided. ```APIDOC Corfu Server, the server for the Corfu Infrastructure. Usage: corfu_server (-l |-m) [-nsQ] [-a
] [-t ] [-c ] [-k seconds] [-d ] [-p ] [-M
:] [-e [-u -f ] [-r -w ] [-b] [-g -o -j ] [-x ] [-z ]] Options: -l , --log-path= Set the path to the storage file for the log unit. -s, --single Deploy a single-node configuration. The server will be bootstrapped with a simple one-unit layout. -a
, --address=
IP address to advertise to external clients [default: localhost]. -m, --memory Run the unit in-memory (non-persistent). Data will be lost when the server exits! -c , --max-cache= The size of the in-memory cache to serve requests from - If there is no log, then this is the max size of the log unit evicted entries will be auto-trimmed. [default: 1000000000]. -t , --initial-token= The first token the sequencer will issue, or -1 to recover from the log. [default: -1]. -p , --compact= The rate the log unit should compact entries (find the, contiguous tail) in seconds [default: 60]. -d , --log-level= Set the logging level, valid levels are: ERROR,WARN,INFO,DEBUG,TRACE [default: INFO]. -Q, --quickcheck-test-mode Run in QuickCheck test mode -M
:, --management-server=
: Layout endpoint to seed Management Server -e, --enable-tls Enable TLS. -u , --keystore= Path to the key store. -f , --keystore-password-file= Path to the file containing the key store password. -b, --enable-tls-mutual-auth Enable TLS mutual authentication. -r , --truststore= Path to the trust store. -w , --truststore-password-file= Path to the file containing the trust store password. -g, --enable-sasl-plain-text-auth Enable SASL Plain Text Authentication. -o , --sasl-plain-text-username-file= Path to the file containing the username for SASL Plain Text Authentication. -j , --sasl-plain-text-password-file= Path to the file containing the password for SASL Plain Text Authentication. -x , --tls-ciphers= Comma separated list of TLS ciphers to use. [default: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256]. -z , --tls-protocols= Comma separated list of TLS protocols to use. [default: TLSv1.1,TLSv1.2]. -h, --help Show this screen ``` -------------------------------- ### Connect CorfuDB Java Runtime with TLS (CA Signed Certificates) Source: https://github.com/corfudb/corfudb/blob/master/Security.md Initializes a `CorfuRuntime` instance in Java, enables TLS using the runtime client's keystore (`r1.jks`) and the shared trust store (`trust.jks`), and connects to the CorfuDB server. ```java CorfuRuntime rt = new CorfuRuntime("localhost:9000"); rt.enableTls("r1.jks", "r1pass.txt", "trust.jks", "tpass.txt"); rt.connect(); ``` -------------------------------- ### CorfuDB Failure Detection Test Suite Overview Source: https://github.com/corfudb/corfudb/blob/master/docs/failure-detector/filesystem-failures.md Lists the key test classes and their functionalities within the CorfuDB failure detection mechanism, demonstrating how various failure scenarios are emulated and validated. ```APIDOC Test Suite Scenarios: - Emulate entire cycle of additional failure detection scenarios using FileSystemStats (boolean values for failure). Test Classes: - BatchProcessorTest#testRestart(): Tests proper status of batch processor in case of exceptions in operations. - FileSystemAdvisorTest: Covers the algorithm of finding failed/healed nodes. - NodeRankTest: Checks proper ordering of the nodes in the list according to their state. - DecisionMakerAgentTest: Checks that the correct decision maker was chosen. - FailureDetectorServiceTest: Tests overall Failure Detector mechanism. - FailureAgentTest: Checks the algorithm of finding failed nodes. ``` -------------------------------- ### Generating Self-Signed Certificates for CorfuDB Source: https://github.com/corfudb/corfudb/blob/master/Security.md Provides a complete sequence of keytool commands to generate self-signed key pairs for CorfuDB infrastructure and runtime. It also shows how to export their certificates and import them into a shared trust store. ```bash # Generate a key pair for the corfu infrastructure: keytool -genkeypair -keystore s1.jks -keyalg RSA -keysize 2048 -alias s1 -storepass password1 # Store the password in a plain text file: cat > s1pass.txt password1 ^C # Export the corfu infrastructure cert to a file keytool -exportcert -keystore s1.jks -storepass password1 -alias s1 -rfc -file s1.cert # Generate a key pair for the corfu runtime: keytool -genkeypair -keystore r1.jks -keyalg RSA -keysize 2048 -alias r1 -storepass password2 # Store the password in a plain text file: cat > r1pass.txt password2 ^C # Export the corfu runtime cert to a file keytool -exportcert -keystore r1.jks -storepass password2 -alias r1 -rfc -file r1.cert # Import the corfu infrastructure and runtime certs into a trust store keytool -importcert -keystore trust.jks -storepass password3 -alias s1 -file s1.cert keytool -importcert -keystore trust.jks -storepass password3 -alias r1 -file r1.cert ``` -------------------------------- ### Import Signed Runtime Certificate into Keystore Source: https://github.com/corfudb/corfudb/blob/master/Security.md Imports the CA-signed runtime client certificate (`r1.cert`) back into the runtime client's keystore (`r1.jks`) after it has been signed by the Certificate Authority. ```bash $ keytool -importcert -keystore r1.jks -storepass password2 -alias r1 -file r1.cert ``` -------------------------------- ### Enable TLS on Corfu Runtime with Java Source: https://github.com/corfudb/corfudb/blob/master/Security.md This Java code snippet demonstrates how to enable TLS on a CorfuRuntime instance by specifying the paths to the keystore, keystore password file, truststore, and truststore password file. It then connects the runtime. ```Java CorfuRuntime rt = new CorfuRuntime("localhost:9000"); rt.enableTls("keystore.jks", "keystore_password.txt", "truststore.jks", "truststore_password.txt"); rt.connect(); ``` -------------------------------- ### Import RSA Keystores into Combined RSA Keystores using Keytool Source: https://github.com/corfudb/corfudb/blob/master/test/src/test/resources/security/README.md These `keytool` commands import individual RSA keystores into combined RSA-only keystores. The `server_rsa.jks` and `server_rsa_2.jks` are imported into `server_rsa_rsa.jks`, and similarly for runtime keystores, consolidating RSA keys into a single JKS file. ```bash keytool -importkeystore -destkeystore server_rsa_rsa.jks -srcstoretype jks -deststoretype jks -srcstorepass test123 -deststorepass test123 -srckeystore server_rsa.jks keytool -importkeystore -destkeystore server_rsa_rsa.jks -srcstoretype jks -deststoretype jks -srcstorepass test123 -deststorepass test123 -srckeystore server_rsa_2.jks keytool -importkeystore -destkeystore runtime_rsa_rsa.jks -srcstoretype jks -deststoretype jks -srcstorepass test123 -deststorepass test123 -srckeystore runtime_rsa.jks keytool -importkeystore -destkeystore runtime_rsa_rsa.jks -srcstoretype jks -deststoretype jks -srcstorepass test123 -deststorepass test123 -srckeystore runtime_rsa_2.jks ``` -------------------------------- ### Generate RSA Keystores for Server and Runtime using Keytool Source: https://github.com/corfudb/corfudb/blob/master/Security.md These `keytool` commands generate RSA key pairs and self-signed certificates for both server and runtime components, storing them in separate JKS keystores (`server_rsa.jks`, `runtime_rsa.jks`). Each keystore is protected by a password and includes subject distinguished names and a long validity period. ```Bash # Creating RSA Keystore. keytool -genkeypair -keystore server_rsa.jks -keyalg RSA -keysize 2048 -alias server_rsa -storepass test123 -keypass test123 -dname "cn=Corfu, ou=NSBU, o=VMware, l=Palo Alto, s=CA c=US" -validity 2000000 keytool -genkeypair -keystore runtime_rsa.jks -keyalg RSA -keysize 2048 -alias runtime_rsa -storepass test123 -keypass test123 -dname "cn=Corfu, ou=NSBU, o=VMware, l=Palo Alto, s=CA c=US" -validity 2000000 ``` -------------------------------- ### Bootstrap CorfuDB Cluster with Layout File Source: https://github.com/corfudb/corfudb/blob/master/README.md This command bootstraps the CorfuDB server instances into a distributed cluster using a predefined layout configuration from `layout.json`. This step provisions the servers according to the specified roles and replication settings. ```bash $ ./CorfuDB/bin/corfu_bootstrap_cluster -l layout.json ``` -------------------------------- ### Connect CorfuDB Java Runtime with TLS and SASL Plaintext Source: https://github.com/corfudb/corfudb/blob/master/Security.md Initializes a `CorfuRuntime` instance in Java, enables both TLS and SASL plaintext authentication, and connects to the CorfuDB server using specified keystores, truststores, and user credential files. ```java CorfuRuntime rt = new CorfuRuntime("localhost:9000"); rt.enableTls("r1.jks", "r1pass.txt", "trust.jks", "tpass.txt"); rt.enableSaslPlainText("r1username.txt", "r1userpassword.txt"); rt.connect(); ``` -------------------------------- ### Store Runtime Keystore Password in Plain Text File Source: https://github.com/corfudb/corfudb/blob/master/Security.md Stores the password for the runtime client's keystore (`r1.jks`) in a plain text file named `r1pass.txt`. ```bash $ cat > r1pass.txt password2 ^C ``` -------------------------------- ### Generate Runtime Key Pair for CA Signed Certificates Source: https://github.com/corfudb/corfudb/blob/master/Security.md Generates an RSA key pair for the CorfuDB runtime client and stores it in a Java Key Store (`r1.jks`) using the `keytool` utility. ```bash # Generate a key pair for the corfu runtime: $ keytool -genkeypair -keystore r1.jks -keyalg RSA -keysize 2048 -alias r1 -storepass password2 ``` -------------------------------- ### Run Longevity App with CLI Arguments Source: https://github.com/corfudb/corfudb/blob/master/generator/README.md Demonstrates how to execute the Longevity application from the command line, specifying the Corfu server endpoint and configuring checkpointing and runtime duration. The 'time_amount' and 'time_unit' arguments are required, while the Corfu endpoint defaults to localhost:9000. ```Shell usage: longevity -c,--corfu_endpoint corfu server to connect to -cp,--checkpoint enable checkpoint -t,--time_amount time amount -u,--time_unit time unit (s, m, h) ``` -------------------------------- ### Import CA-Signed Certificate into Key Store using Keytool Source: https://github.com/corfudb/corfudb/blob/master/Security.md This `keytool` command imports a CA-signed certificate (`s1.crt`) into an existing keystore (`s1.jks`) for the alias `s1`. The `-trustcacerts` option ensures that the certificate's chain of trust is validated against the trust anchors in the default truststore. ```Bash keytool -importcert -keystore s1.jks -storepass test123 -alias s1 -trustcacerts -file s1.crt ``` -------------------------------- ### Corfu Runtime Performance Metrics Reference Source: https://github.com/corfudb/corfudb/blob/master/common/src/main/java/org/corfudb/common/metrics/micrometer/README.md Detailed reference for performance and operational metrics collected by the Corfu Runtime, including timing, counts, and distribution summaries for various operations and components. These metrics help in understanding system behavior and identifying performance bottlenecks. ```APIDOC runtime.fetch_layout.timer: Description: Time in microseconds (mean, max, sum, 0.50p, 0.99p) it takes a client to fetch a layout from Corfu layout servers. chain_replication.write: Description: Time in microseconds (mean, max, sum, 0.50p, 0.99p) it takes a client to write log data (or a hole) into every Corfu logunit server. open_tables.count: Description: Number of currently open tables in the Corfu store. highestSeqNum.numberBatchReads: Description: Number of batches read before finding highest DATA sequence number. highestSeqNum.numberReads: Description: Number of addresses read in batches before finding highest DATA sequence number. highestSequenceNumberDuration: Description: Time in microseconds (mean, max, sum, 0.50p, 0.99p) it takes to complete the latest update. vlo.tx.timer: Description: Time in microseconds (mean, max, sum, 0.50p, 0.99p) it takes to execute a transaction on the corfu object backed by a particular stream id. vlo.no_rollback_exception.count: Description: Number of times we were unable to roll back the particular stream by applying undo records in the reverse order. vlo.sync.read_entries: Description: A distribution summary (mean, max, 0.50p, 0.95p, 0.99p) of the number of updates of a stream to be sync'd. vlo.sync.read_size: Description: A distribution summary (mean, max, 0.50p, 0.95p, 0.99p) of the total size of the updates of a stream to be sync'd. address_space.read_cache.avg_entry_size: Description: The estimated average size of an entry in the address space cache, in bytes. address_space.read_cache.size: Description: The number of entries in the address space cache. address_space.read_cache.hit_ratio: Description: The hit ratio of Corfu client address space cache. address_space.log_data.size.bytes: Description: A size estimate distribution in bytes (mean, max, 0.50p, 0.99p) of the log data payload read or written through the address space API. sequencer.query: Description: Time in microseconds (mean, max, sum, 0.50p, 0.99p) it takes to query the current global tail token in the sequencer or the tails of multiple streams. sequencer.next: Description: Time in microseconds (mean, max, sum, 0.50p, 0.99p) it takes to get the next token in the sequencer for the particular streams. sequencer.tx_resolution: Description: Time in microseconds (mean, max, sum, 0.50p, 0.99p) it takes to acquire a token for a number of streams if there are no transactional conflicts. sequencer.stream_address_range: Description: Time in microseconds (mean, max, sum, 0.50p, 0.99p) it takes to retrieve the address space for the multiple streams. stream.poll.duration: Description: Time in microseconds(mean, max, sum, 0.50p, 0.99p) it takes to poll transaction updates. stream.notify.duration: Description: Time in microseconds(mean, max, sum, 0.50p, 0.99p) it takes to send notification to client with the pre-registered callback. stream_sub.queueDuration.timer: Description: Time in microseconds(mean, max, sum, 0.50p, 0.99p) it takes to wait in the queue. ``` -------------------------------- ### Import ECDSA Keystores into Combined ECDSA Keystores using Keytool Source: https://github.com/corfudb/corfudb/blob/master/test/src/test/resources/security/README.md These `keytool` commands import individual ECDSA keystores into combined ECDSA-only keystores. The `server_ecdsa.jks` and `server_ecdsa_2.jks` are imported into `server_ecdsa_ecdsa.jks`, and similarly for runtime keystores, consolidating ECDSA keys into a single JKS file. ```bash keytool -importkeystore -destkeystore server_ecdsa_ecdsa.jks -srcstoretype jks -deststoretype jks -srcstorepass test123 -deststorepass test123 -srckeystore server_ecdsa.jks keytool -importkeystore -destkeystore server_ecdsa_ecdsa.jks -srcstoretype jks -deststoretype jks -srcstorepass test123 -deststorepass test123 -srckeystore server_ecdsa_2.jks keytool -importkeystore -destkeystore runtime_ecdsa_ecdsa.jks -srcstoretype jks -deststoretype jks -srcstorepass test123 -deststorepass test123 -srckeystore runtime_ecdsa.jks keytool -importkeystore -destkeystore runtime_ecdsa_ecdsa.jks -srcstoretype jks -deststoretype jks -srcstorepass test123 -deststorepass test123 -srckeystore runtime_ecdsa_2.jks ``` -------------------------------- ### Generate EC Keystores for Server and Runtime using Keytool Source: https://github.com/corfudb/corfudb/blob/master/Security.md These `keytool` commands generate Elliptic Curve (EC) key pairs and self-signed certificates for server and runtime components, storing them in separate JKS keystores (`server_ecdsa.jks`, `runtime_ecdsa.jks`). They use the `secp384r1` group and SHA384withECDSA signature algorithm, with specified distinguished names and validity. ```Bash # Creating EC Keystore. keytool -genkeypair -groupname secp384r1 -sigalg SHA384withECDSA -keyalg EC -dname "cn=Corfu, ou=NSBU, o=VMware, l=Palo Alto, s=CA c=US" -alias server_ecdsa -keypass test123 -keystore server_ecdsa.jks -storetype jks -storepass test123 -validity 2000000 keytool -genkeypair -groupname secp384r1 -sigalg SHA384withECDSA -keyalg EC -dname "cn=Corfu, ou=NSBU, o=VMware, l=Palo Alto, s=CA c=US" -alias runtime_ecdsa -keypass test123 -keystore runtime_ecdsa.jks -storetype jks -storepass test123 -validity 2000000 ``` -------------------------------- ### Simulate Single-Node Corfu Deployment Source: https://github.com/corfudb/corfudb/blob/master/TestingFramework.md Open a default `CorfuRuntime` to simulate a single-node system. This is achieved by calling `getDefaultRuntime()` which generates a single Corfu server at port 9000 and allows writing to a stream, simulating communication. ```Java import org.corfudb.runtime.view.AbstractViewTest; class TestClass extends AbstractViewTest { @Test public canWriteToAStreamTest() { CorfuRuntime myRuntime = getDefaultRuntime(); StreamView sv = myRuntime.getStreamsView().open("a"); sv.write("hello world"); } } ``` -------------------------------- ### Simulate Custom Distributed Corfu Deployment Source: https://github.com/corfudb/corfudb/blob/master/TestingFramework.md For distributed testing, simulate a custom layout by adding servers using `addServer(port)` and bootstrapping them with `bootstrapAllServers(layout)`. The `TestLayoutBuilder` is used to construct the layout, defining layout servers, sequencers, and log units across multiple ports. ```Java import org.corfudb.runtime.view.AbstractViewTest; class TestClass extends AbstractViewTest { @Test public canWriteToAStreamTest() { addServer(9000); addServer(9001); addServer(9002); bootstrapAllServers(new TestLayoutBuilder() .addLayoutServer(9000) .addSequencer(9000) .buildSegment() .setReplicationMode(Layout.ReplicationMode.CHAIN_REPLICATION) .buildStripe() .addLogUnit(9000) .addLogUnit(9001) .addLogUnit(9002) .addToSegment() .addToLayout() .build()); CorfuRuntime myRuntime = getRuntime().connect(); StreamView sv = myRuntime.getStreamsView().open("a"); sv.write("hello world"); } } ``` -------------------------------- ### Generate Self-Signed Certificate and Key Store using Keytool Source: https://github.com/corfudb/corfudb/blob/master/Security.md This `keytool` command generates a private/public key pair, creates a self-signed certificate, and stores them in a JKS keystore named `s1.jks` with an alias `s1` and password `test123`. It uses the RSA algorithm with a 2048-bit key size. ```Bash keytool -genkeypair -keystore s1.jks -keyalg RSA -keysize 2048 -alias s1 -storepass test123 ``` -------------------------------- ### Create Certificate Signing Request (CSR) using Keytool Source: https://github.com/corfudb/corfudb/blob/master/Security.md This `keytool` command generates a Certificate Signing Request (CSR) from an existing keystore (`s1.jks`) for the alias `s1`, saving the request to `s1.csr`. This CSR can then be submitted to a Certificate Authority for signing. ```Bash keytool -certreq -keystore s1.jks -storepass test123 -alias s1 -file s1.csr ``` -------------------------------- ### Store Runtime Username and Password for SASL in Separate Files Source: https://github.com/corfudb/corfudb/blob/master/Security.md Creates `r1username.txt` and `r1userpassword.txt` files to store the user username and password, respectively, for client-side runtime SASL authentication. ```bash $ cat > r1username.txt user ^C $ cat > r1userpassword.txt userp@ssw0rd ^C ``` -------------------------------- ### Store Server Keystore Password in Plain Text File Source: https://github.com/corfudb/corfudb/blob/master/Security.md Stores the password for the server's keystore (`s1.jks`) in a plain text file named `s1pass.txt`. ```bash $ cat > s1pass.txt password1 ^C ``` -------------------------------- ### Import CA Certificate into Trust Store Source: https://github.com/corfudb/corfudb/blob/master/Security.md Imports the Certificate Authority's root certificate (`cacert.cert`) into a new trust store (`trust.jks`) for validating server and client certificates. ```bash $ keytool -importcert -keystore trust.jks -storepass password3 -alias ca -file cacert.cert ``` -------------------------------- ### Import Signed Server Certificate into Keystore Source: https://github.com/corfudb/corfudb/blob/master/Security.md Imports the CA-signed server certificate (`s1.cert`) back into the server's keystore (`s1.jks`) after it has been signed by the Certificate Authority. ```bash $ keytool -importcert -keystore s1.jks -storepass password1 -alias s1 -file s1.cert ``` -------------------------------- ### Setting JVM Parameter for JAAS Configuration Source: https://github.com/corfudb/corfudb/blob/master/Security.md Illustrates how to set the 'java.security.auth.login.config' JVM parameter using the SERVER_JVMFLAGS environment variable. This directs the Java Virtual Machine to the specified JAAS configuration file for authentication. ```bash export SERVER_JVMFLAGS="-Djava.security.auth.login.config=test/src/test/resources/security/corfudb_jaas.config" ``` -------------------------------- ### Store Server Password for SASL in Plain Text File Source: https://github.com/corfudb/corfudb/blob/master/Security.md Demonstrates how to store a password for the CorfuDB server's SASL authentication in a plain text file named `tpass.txt` using the `cat` command. ```bash $ cat > tpass.txt password3 ^C ``` -------------------------------- ### CorfuDB Compactor Metrics Source: https://github.com/corfudb/corfudb/blob/master/common/src/main/java/org/corfudb/common/metrics/micrometer/README.md Metrics related to the Corfu Compactor's performance, focusing on checkpointing operations, including timing for single stream checkpoints and distribution summaries for the number and total size of checkpointed entries. ```APIDOC checkpoint.timer: Time in microseconds (mean, max, sum, 0.5p, 0.95p, 0.99p) it takes for a single stream to be checkpointed. checkpoint.write_entries: A distribution summary (mean, max, 0.50p, 0.95p, 0.99p) of the number of entries of a stream that are checkpointed. checkpoint.write_size: A distribution summary (mean, max, 0.50p, 0.95p, 0.99p) of the total size of the entries of a stream that are checkpointed. ``` -------------------------------- ### Store Server Username and Password for SASL in Separate Files Source: https://github.com/corfudb/corfudb/blob/master/Security.md Creates `username.txt` and `userpassword.txt` files to store the admin username and password, respectively, for server-side SASL authentication. ```bash $ cat > username.txt admin ^C $ cat > userpassword.txt @dm1np@ssw0rd ^C ``` -------------------------------- ### CorfuDB Core Component Metrics Source: https://github.com/corfudb/corfudb/blob/master/common/src/main/java/org/corfudb/common/metrics/micrometer/README.md This section details performance metrics related to core CorfuDB components, specifically corfu_table operations (read/write) and logdata processing (compression/decompression). These metrics provide insights into data manipulation and storage efficiency. ```APIDOC corfu_table.read.timer: Time in microseconds(mean, max, sum, 0.50p, 0.99p) it takes to read from the corfu table's map. corfu_table.write.timer: Time in microseconds(mean, max, sum, 0.50p, 0.99p) it takes to write to the corfu table's map. logdata.decompress.timer: Time in microseconds(mean, max, sum, 0.50p, 0.99p) it takes to decompress the payload. logdata.compress.timer: Time in microseconds(mean, max, sum, 0.50p, 0.99p) it takes to compress the payload. logdata.compression.ratio: A distribution summary (mean, max, 0.50p, 0.99p) of the payload's compression ratio. logdata.decompressed.size: A size estimate distribution (mean, max, 0.5p, 0.99p) of the decompressed payload. ``` -------------------------------- ### Generate RSA Key Pairs and Keystores using Keytool Source: https://github.com/corfudb/corfudb/blob/master/test/src/test/resources/security/README.md These `keytool` commands generate RSA key pairs and store them in individual JKS keystores. Each command creates a new keystore with a unique alias, a 2048-bit RSA key, and a validity period of 2,000,000 days, using 'test123' as both store and key password. ```bash keytool -genkeypair -keystore server_rsa.jks -keyalg RSA -keysize 2048 -alias server_rsa -storepass test123 -keypass test123 -dname "cn=Corfu, ou=NSBU, o=VMware, l=Palo Alto, s=CA c=US" -validity 2000000 keytool -genkeypair -keystore server_rsa_2.jks -keyalg RSA -keysize 2048 -alias server_rsa_2 -storepass test123 -keypass test123 -dname "cn=Corfu, ou=NSBU, o=VMware, l=Palo Alto, s=CA c=US" -validity 2000000 keytool -genkeypair -keystore runtime_rsa.jks -keyalg RSA -keysize 2048 -alias runtime_rsa -storepass test123 -keypass test123 -dname "cn=Corfu, ou=NSBU, o=VMware, l=Palo Alto, s=CA c=US" -validity 2000000 keytool -genkeypair -keystore runtime_rsa_2.jks -keyalg RSA -keysize 2048 -alias runtime_rsa_2 -storepass test123 -keypass test123 -dname "cn=Corfu, ou=NSBU, o=VMware, l=Palo Alto, s=CA c=US" -validity 2000000 ``` -------------------------------- ### Generate Server Key Pair for CA Signed Certificates Source: https://github.com/corfudb/corfudb/blob/master/Security.md Generates an RSA key pair for the CorfuDB server and stores it in a Java Key Store (`s1.jks`) using the `keytool` utility. ```bash # Generate a key pair for the corfu infrastructure: $ keytool -genkeypair -keystore s1.jks -keyalg RSA -keysize 2048 -alias s1 -storepass password1 ``` -------------------------------- ### Create a Basic Corfu Test Class Source: https://github.com/corfudb/corfudb/blob/master/TestingFramework.md To write a test in the Corfu testing framework, create a new Java class in `src/test/java/org/corfudb` that extends `AbstractViewTest`. ```Java import org.corfudb.runtime.view.AbstractViewTest; class TestClass extends AbstractViewTest { } ``` -------------------------------- ### Store Trust Store Password in Plain Text File Source: https://github.com/corfudb/corfudb/blob/master/Security.md Stores the password for the trust store (`trust.jks`) in a plain text file named `tpass.txt`. ```bash $ cat > tpass.txt password3 ^C ``` -------------------------------- ### CorfuQueue API Methods Source: https://github.com/corfudb/corfudb/blob/master/docs/corfu-queue/corfu-queue-design.md Detailed API documentation for the primary methods of the `CorfuQueue` class, including how to add, list, and remove elements from the persisted queue. ```APIDOC CorfuQueue: enqueue(E entry) entry: The element to be enqueued. Returns: CorfuRecordId - A unique ID representing the entry's position in the queue. If wrapped in a transaction, this ID captures the transaction's commit order, defining the global cluster-wide order of the entry. ``` ```APIDOC CorfuQueue: entryList() Returns: List> - A list of all entries currently in the queue, along with their comparable CorfuRecordIds. ``` ```APIDOC CorfuQueue: remove(CorfuRecordId id) id: The CorfuRecordId of the entry to be removed, obtained from `enqueue()` or `entryList()`. Returns: E - The removed entry. Note that this operation allows for removal in any order and does not change the original commit order of other entries. ``` -------------------------------- ### Generate and View Aggregate Test Coverage Report Source: https://github.com/corfudb/corfudb/blob/master/coverage/README.md This command runs Unit, Integration, and Universe Tests simultaneously to generate a combined code coverage report. The `-pl coverage -am` arguments build the coverage module and its dependencies. `-Dmaven.test.failure.ignore=true` allows ignoring intermittent test failures, as all three suites are ensured to run successfully in GitHub Actions. `-Dcode-coverage=true` sets the activation property to activate 'it' and 'universe' profiles simultaneously. The full report is accessible via `coverage/target/site/jacoco-aggregate/index.html`, with the aggregate XML at `coverage/target/site/jacoco-aggregate/jacoco.xml`. For activated profiles, use `help:active-profiles`. ```Shell ./mvnw -pl :coverage -am clean verify -Dmaven.test.failure.ignore=true -Dcode-coverage=true ```