### Install Sysbox Package

Source: https://github.com/nestybox/sysbox/blob/master/docs/user-guide/install-package.md

Install the 'jq' tool, which is a prerequisite for the Sysbox installer, and then install the Sysbox package itself using apt-get.

```bash
$ sudo apt-get install jq
$ sudo apt-get install ./sysbox-ce_0.7.0-0.linux_amd64.deb
```

--------------------------------

### Check Capabilities of Root Process in System Container

Source: https://github.com/nestybox/sysbox/blob/master/docs/user-guide/security.md

When a system container's init process is configured with user-ID 0 (root), it starts with all capabilities enabled. This example shows how to verify these capabilities.

```console
$ docker run --runtime=sysbox-runc -it alpine:latest
/ # grep -i cap /proc/self/status
CapInh: 0000003fffffffff
CapPrm: 0000003fffffffff
CapEff: 0000003fffffffff
CapBnd: 0000003fffffffff
CapAmb: 0000003fffffffff
```

--------------------------------

### Run a Sysbox container with Docker

Source: https://github.com/nestybox/sysbox/blob/master/docs/user-guide/dind.md

Starts a Sysbox container that includes a Docker installation. This is the initial step for subsequent Docker operations within the container.

```bash
$ docker run --runtime=sysbox-runc -it --rm nestybox/ubuntu-jammy-systemd-docker
```

--------------------------------

### Install Delve Debugger

Source: https://github.com/nestybox/sysbox/blob/master/docs/developers-guide/debug.md

Use 'go get' to install the Delve debugger. Ensure your Go environment is set up correctly.

```console
rodny@vm-1:~$ go get -u github.com/derekparker/delve/cmd/dlv
```

--------------------------------

### Install Sysbox Binaries

Source: https://github.com/nestybox/sysbox/blob/master/docs/developers-guide/build.md

Install the built Sysbox binaries to your host system's /usr/bin directory. Requires root privileges.

```bash
sudo make install
```

--------------------------------

### Build and Install Shiftfs on Ubuntu-based Cloud VM

Source: https://github.com/nestybox/sysbox/blob/master/docs/user-guide/install-package.md

Clone the shiftfs-dkms repository, update it, build the module using make, and install it. Verify the installation using modinfo.

```bash
sudo apt-get install -y make dkms git wget
git clone -b k5.10 https://github.com/toby63/shiftfs-dkms.git shiftfs-k510
cd shiftfs-k510
./update1
sudo make -f Makefile.dkms
modinfo shiftfs
```

--------------------------------

### Example Stack Trace Output

Source: https://github.com/nestybox/sysbox/blob/master/docs/developers-guide/debug.md

This is an example of the output from the 'bt' command, detailing the function call stack.

```console
(dlv) bt
     0  0x00000000004ead9a in syscall.Syscall6
        at /usr/local/go/src/syscall/asm_linux_amd64.s:53
     1  0x0000000000524f55 in os.(*Process).blockUntilWaitable
        at /usr/local/go/src/os/wait_waitid.go:31
     2  0x00000000005194ae in os.(*Process).wait
        at /usr/local/go/src/os/exec_unix.go:22
     3  0x00000000005180a1 in os.(*Process).Wait
        at /usr/local/go/src/os/exec.go:125
     4  0x00000000007d870f in os/exec.(*Cmd).Wait
        at /usr/local/go/src/os/exec/exec.go:501
     5  0x0000000000d6c2fa in github.com/opencontainers/runc/libcontainer.  (*initProcess).wait
        at /root/nestybox/sysbox/sysbox-runc/libcontainer/process_linux.go:655
     6  0x0000000000d6c43f in github.com/opencontainers/runc/libcontainer.(*initProcess).terminate
        at /root/nestybox/sysbox/sysbox-runc/libcontainer/process_linux.go:668
     7  0x0000000000d89f35 in github.com/opencontainers/runc/libcontainer.(*initProcess).start.func1
        at /root/nestybox/sysbox/sysbox-runc/libcontainer/process_linux.go:353
     8  0x0000000000d6bace in github.com/opencontainers/runc/libcontainer.(*initProcess).start
        at /root/nestybox/sysbox/sysbox-runc/libcontainer/process_linux.go:592
     9  0x0000000000d3f3ae in github.com/opencontainers/runc/libcontainer.(*linuxContainer).start
        at /root/nestybox/sysbox/sysbox-runc/libcontainer/container_linux.go:390
    10  0x0000000000d3e426 in github.com/opencontainers/runc/libcontainer.(*linuxContainer).Start
        at /root/nestybox/sysbox/sysbox-runc/libcontainer/container_linux.go:287
    11  0x0000000000e1da2e in main.(*runner).run
        at /root/nestybox/sysbox/sysbox-runc/utils_linux.go:383
    12  0x0000000000e1f08f in main.startContainer
        at /root/nestybox/sysbox/sysbox-runc/utils_linux.go:553
    13  0x0000000000e1f78c in main.glob..func2
        at /root/nestybox/sysbox/sysbox-runc/create.go:108
    14  0x0000000000bac838 in github.com/urfave/cli.HandleAction
        at /go/pkg/mod/github.com/urfave/cli@v1.22.1/app.go:523

```

--------------------------------

### Start System Container with Docker

Source: https://github.com/nestybox/sysbox/blob/master/docs/quickstart/dind.md

Launches a system container with Docker pre-installed. Use this to set up an environment for Docker-in-Docker.

```bash
$ docker run --runtime=sysbox-runc -it --hostname=syscont nestybox/alpine-docker:latest
```

--------------------------------

### Check Native Docker Installation Path

Source: https://github.com/nestybox/sysbox/blob/master/docs/developers-guide/build.md

Confirm that Docker has been installed natively, indicated by its path in /usr/bin.

```bash
$ which docker
/usr/bin/docker
```

--------------------------------

### Install Sysbox Dependencies

Source: https://github.com/nestybox/sysbox/blob/master/docs/user-guide/troubleshoot.md

If Sysbox installation fails due to missing dependencies, run these commands to update package lists and install missing packages.

```bash
sudo apt-get update
sudo apt-get install -f -y
```

--------------------------------

### Check Capabilities of Non-Root Process in System Container

Source: https://github.com/nestybox/sysbox/blob/master/docs/user-guide/security.md

A system container's init process configured with a non-root user-ID starts with no capabilities. This example demonstrates checking the capabilities for a non-root user.

```console
$ docker run --runtime=sysbox-runc --user 1000 -it alpine:latest
/ $ grep -i cap /proc/self/status
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: 0000003fffffffff
CapAmb: 0000000000000000
```

--------------------------------

### Verify Docker Installation

Source: https://github.com/nestybox/sysbox/blob/master/docs/developers-guide/build.md

Run a simple Docker command to ensure the installation is working correctly and can pull images.

```bash
$ docker run -it alpine
Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
df20fa9351a1: Pull complete
Digest: sha256:185518070891758909c9f839cf4ca393ee977ac378609f700f60a771a2dfe321
Status: Downloaded newer image for alpine:latest
/ #
```

--------------------------------

### Check Docker Installation Path

Source: https://github.com/nestybox/sysbox/blob/master/docs/developers-guide/build.md

Verify if Docker is installed via snap. Sysbox does not support snap installations of Docker.

```bash
$ which docker
/snap/bin/docker
```

--------------------------------

### Verify Docker buildx installation

Source: https://github.com/nestybox/sysbox/blob/master/docs/user-guide/dind.md

Checks the status and supported platforms for Docker's buildx, which is used for multi-arch builds. This command confirms QEMU installation and configuration.

```bash
$ docker buildx inspect --bootstrap
```

--------------------------------

### Deploy Nginx with Multiple Pods

Source: https://github.com/nestybox/sysbox/blob/master/docs/quickstart/kind.md

Example of deploying an Nginx application and scaling it to 10 pods, then verifying the pod status and distribution across nodes.

```console
$ kubectl create deployment nginx --image=nginx
$ kubectl scale --replicas=10 deployment nginx
$ kubectl get pods -o wide
NAME                    READY   STATUS    RESTARTS   AGE   IP            NODE                 NOMINATED NODE   READINESS GATES
nginx-f89759699-6ch9m   1/1     Running   0          21s   10.244.11.4   mycluster-worker-6   <none>           <none>
nginx-f89759699-8jrc8   1/1     Running   0          21s   10.244.10.4   mycluster-worker-5   <none>           <none>
nginx-f89759699-dgxq8   1/1     Running   0          28s   10.244.2.15   mycluster-worker-1   <none>           <none>
nginx-f89759699-hx5tt   1/1     Running   0          21s   10.244.5.15   mycluster-worker-3   <none>           <none>
nginx-f89759699-l9v5p   1/1     Running   0          21s   10.244.1.10   mycluster-worker-0   <none>           <none>
nginx-f89759699-pdnhb   1/1     Running   0          21s   10.244.12.4   mycluster-worker-4   <none>           <none>
nginx-f89759699-qf46b   1/1     Running   0          21s   10.244.2.16   mycluster-worker-1   <none>           <none>
nginx-f89759699-vbnx5   1/1     Running   0          21s   10.244.3.14   mycluster-worker-2   <none>           <none>
nginx-f89759699-whgt7   1/1     Running   0          21s   10.244.13.4   mycluster-worker-8   <none>           <none>
nginx-f89759699-zblsb   1/1     Running   0          21s   10.244.14.4   mycluster-worker-7   <none>           <none>
```

--------------------------------

### Start Systemd System Container

Source: https://github.com/nestybox/sysbox/blob/master/docs/quickstart/systemd.md

Use this command to launch a system container with systemd pre-installed. Ensure you have the 'sysbox-runc' runtime configured.

```bash
$ docker run --runtime=sysbox-runc --rm -it --hostname=syscont nestybox/ubuntu-bionic-systemd

systemd 237 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid)
Detected virtualization container-other.
Detected architecture x86-64.

Welcome to Ubuntu 18.04.3 LTS!

Set hostname to <syscont>.
Failed to read AF_UNIX datagram queue length, ignoring: No such file or directory
Failed to install release agent, ignoring: No such file or directory
File /lib/systemd/system/systemd-journald.service:35 configures an IP firewall (IPAddressDeny=any), but the local system does not support BPF/cgroup based firewalling.
Proceeding WITHOUT firewalling in effect! (This warning is only shown for the first loaded unit using IP firewalling.)
[  OK  ] Reached target Swap.

...

[  OK  ] Reached target Login Prompts.
[  OK  ] Started Login Service.
[  OK  ] Reached target Multi-User System.
[  OK  ] Reached target Graphical Interface.
         Starting Update UTMP about System Runlevel Changes...
[  OK  ] Started Update UTMP about System Runlevel Changes.

Ubuntu 18.04.3 LTS syscont console

syscont login: 

```

--------------------------------

### Start a system container

Source: https://github.com/nestybox/sysbox/blob/master/docs/quickstart/images.md

Launches a system container using the newly created image. The `--runtime=sysbox-runc` flag is essential for this.

```console
$ docker run --runtime=sysbox-runc -it --rm --hostname=syscont nestybox/syscont-with-inner-containers:latest
```

--------------------------------

### Start System Container with Supervisord and Docker

Source: https://github.com/nestybox/sysbox/blob/master/docs/quickstart/dind.md

Launches a system container with supervisord and Docker pre-installed. Use this to create an isolated environment for running Docker commands.

```console
$ docker run --runtime=sysbox-runc -d --rm -P --hostname=syscont nestybox/alpine-supervisord-docker:latest
f3b90976ad0550fc8142568d988c8fa65c54864d04c1637e88323a32f87cf3af
```

--------------------------------

### Install Sysbox on Kubernetes Nodes

Source: https://github.com/nestybox/sysbox/blob/master/docs/user-guide/install-k8s.md

Label nodes for Sysbox installation and apply the Sysbox installation manifest. This process may also install CRI-O on nodes with older containerd versions, potentially restarting Kubelet.

```bash
kubectl label nodes <node-name> sysbox-install=yes
kubectl apply -f https://raw.githubusercontent.com/nestybox/sysbox/master/sysbox-k8s-manifests/sysbox-install.yaml
```

--------------------------------

### Start Sysbox Manually

Source: https://github.com/nestybox/sysbox/blob/master/docs/developers-guide/build.md

Manually start the Sysbox daemons (sysbox-fs and sysbox-mgr) using the provided script. Logs are written to /var/log/sysbox-fs.log and /var/log/sysbox-mgr.log.

```bash
sudo ./scr/sysbox
```

--------------------------------

### Install Sysbox CE Package

Source: https://github.com/nestybox/sysbox/blob/master/docs/user-guide/troubleshoot.md

Use this command to install the Sysbox CE package using apt-get. Ensure you have the correct .deb file path.

```bash
sudo apt-get install ./sysbox-ce_0.5.0-0.linux_amd64.deb
```

--------------------------------

### Example Breakpoint Confirmation

Source: https://github.com/nestybox/sysbox/blob/master/docs/developers-guide/debug.md

This output confirms that a breakpoint has been successfully set at the specified location.

```console
(dlv) b libcontainer.(*initProcess).start
Breakpoint 1 set at 0x55731c80152d for github.com/opencontainers/runc/libcontainer.(*initProcess).start() /home/rodny/go/src/github.com/opencontainers/runc/libcontainer/process_linux.go:263
```

--------------------------------

### Install QEMU for Docker multi-arch emulation

Source: https://github.com/nestybox/sysbox/blob/master/docs/user-guide/dind.md

Installs QEMU user-mode emulation within the Sysbox container to enable Docker to build multi-arch images. This command should be run inside the Sysbox container.

```bash
$ docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
```

--------------------------------

### Start Systemd and Docker System Container

Source: https://github.com/nestybox/sysbox/blob/master/docs/quickstart/dind.md

Run a system container with Systemd and Docker enabled. This command starts the container and maps its ports.

```console
$ docker run --runtime=sysbox-runc -it --rm -P --hostname=syscont nestybox/ubuntu-bionic-systemd-docker:latest
systemd 237 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid)
Detected virtualization container-other.
Detected architecture x86-64.

Welcome to Ubuntu 18.04.3 LTS!

Set hostname to <syscont>.

... 

[  OK  ] Started Docker Application Container Engine.
[  OK  ] Reached target Multi-User System.
[  OK  ] Reached target Graphical Interface.
         Starting Update UTMP about System Runlevel Changes...
[  OK  ] Started Update UTMP about System Runlevel Changes.

Ubuntu 18.04.3 LTS syscont console

syscont login:

```

--------------------------------

### Deploy a System Container with Sysbox

Source: https://github.com/nestybox/sysbox/blob/master/docs/developers-guide/build.md

Run a system container using Sysbox by specifying the `--runtime=sysbox-runc` flag. This example deploys a Debian container.

```bash
$ docker run --runtime=sysbox-runc --rm -it --hostname my_cont debian:latest
```

--------------------------------

### Run a System Container with Docker

Source: https://github.com/nestybox/sysbox/blob/master/README.md

Use this command to launch a system container using Docker. Ensure Sysbox is installed and configured as the runtime.

```bash
$ docker run --runtime=sysbox-runc --rm -it --hostname my_cont registry.nestybox.com/nestybox/ubuntu-bionic-systemd-docker
root@my_cont:/#

```

--------------------------------

### Deploy System Container and Pull Images

Source: https://github.com/nestybox/sysbox/blob/master/docs/quickstart/images.md

Deploy a system container, start dockerd inside it, and pull necessary images. This sets up the environment for committing the container with its inner images.

```console
$ docker run --runtime=sysbox-runc -it --rm nestybox/alpine-docker

/ # dockerd > /var/log/dockerd.log 2>&1 &

/ # docker image ls
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE

/ # docker pull busybox
Using default tag: latest
latest: Pulling from library/busybox
7c9d20b9b6cd: Pull complete
Digest: sha256:fe301db49df08c384001ed752dff6d52b4305a73a7f608f21528048e8a08b51e
Status: Downloaded newer image for busybox:latest

/ # docker pull alpine
Using default tag: latest
latest: Pulling from library/alpine
89d9c30c1d48: Pull complete
Digest: sha256:c19173c5ada610a5989151111163d28a67368362762534d8a8121ce95cf2bd5a
Status: Downloaded newer image for alpine:latest
```

--------------------------------

### Start Sysbox Systemd Service

Source: https://github.com/nestybox/sysbox/blob/master/docs/user-guide/configuration.md

Use this command to start the main Sysbox service after it has been stopped or during system startup.

```bash
sudo systemctl start sysbox
```

--------------------------------

### Start Inner Container

Source: https://github.com/nestybox/sysbox/blob/master/docs/quickstart/dind.md

Deploys a new container (e.g., busybox) using the inner Docker daemon, demonstrating normal Docker functionality within the system container.

```bash
/ # docker run -it busybox
Unable to find image 'busybox:latest' locally
latest: Pulling from library/busybox
7c9d20b9b6cd: Pull complete
Digest: sha256:fe301db49df08c384001ed752dff6d52b4305a73a7f608f21528048e8a08b51e
Status: Downloaded newer image for busybox:latest
/ #
```

--------------------------------

### Start inner Docker and list images inside system container

Source: https://github.com/nestybox/sysbox/blob/master/docs/quickstart/images.md

Starts the Docker daemon within the system container and lists the preloaded images to verify they were included.

```shell
/ # dockerd > /var/log/dockerd.log 2>&1 &

/ # docker image ls
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
alpine              latest              965ea09ff2eb        2 days ago          5.55MB
busybox             latest              19485c79a9bb        7 weeks ago         1.22MB
```

--------------------------------

### Start Sysbox with Debug Logging

Source: https://github.com/nestybox/sysbox/blob/master/docs/developers-guide/build.md

Start Sysbox with debug logging enabled using the --debug flag. This can slow down performance but is useful for diagnosing issues.

```bash
sudo ./scr/sysbox --debug
```

--------------------------------

### Install GDB Package

Source: https://github.com/nestybox/sysbox/blob/master/docs/user-guide/troubleshoot.md

Install the gdb package, which provides the gcore tool for creating core dumps. This is required for generating core-dumps of Sysbox daemons.

```bash
sudo apt-get install gdb
```

```bash
sudo yum install gdb
```

--------------------------------

### Start an Inner Container

Source: https://github.com/nestybox/sysbox/blob/master/docs/quickstart/dind.md

Run a new container (e.g., busybox) from within the system container to verify Docker functionality.

```console
admin@syscont:~$ docker run -it busybox
Unable to find image 'busybox:latest' locally
latest: Pulling from library/busybox
7c9d20b9b6cd: Pull complete
Digest: sha256:fe301db49df08c384001ed752dff6d52b4305a73a7f608f21528048e8a08b51e
Status: Downloaded newer image for busybox:latest
/ # 

```

--------------------------------

### Monitor Sysbox Installation on EKS

Source: https://github.com/nestybox/sysbox/blob/master/docs/user-guide/install-k8s-distros.md

Command to view the logs of the sysbox-deploy-k8s pod for monitoring installation progress and troubleshooting. Use the actual pod ID.

```bash
kubectl -n kube-system logs -f pod/sysbox-deploy-<pod-id>
```

--------------------------------

### Start Docker and Pull Image Inside System Container

Source: https://github.com/nestybox/sysbox/blob/master/docs/quickstart/dind.md

Start the Docker daemon inside the system container and pull an image. The pulled image will be stored in the bind-mounted cache.

```bash
/ # dockerd > /var/log/dockerd.log 2>&1 &

/ # docker pull busybox
Using default tag: latest
latest: Pulling from library/busybox
7c9d20b9b6cd: Pull complete
Digest: sha256:fe301db49df08c384001ed752dff6d52b4305a73a7f608f21528048e8a08b51e
Status: Downloaded newer image for busybox:latest

/ # docker image ls
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
busybox             latest              19485c79a9bb        7 weeks ago         1.22MB
```

--------------------------------

### Deploy System Container with Read-Only Mount

Source: https://github.com/nestybox/sysbox/blob/master/docs/quickstart/security.md

Use this command to launch a system container with a read-only host volume mounted. This demonstrates the setup for immutable mountpoints.

```bash
$ docker run --runtime=sysbox-runc -it --rm --hostname=syscont -v myvol:/mnt/myvol:ro ubuntu
root@syscont:/#

```

--------------------------------

### Launch K8s Cluster with Preloaded Image

Source: https://github.com/nestybox/sysbox/blob/master/docs/quickstart/kind.md

Start a new Kubernetes node container using the custom image that has inner pod images preloaded.

```bash
$ docker run --runtime=sysbox-runc --rm -d --name k8s-node k8s-node-with-inner-nginx:latest
```

--------------------------------

### Run Entire Sysbox Test Suite

Source: https://github.com/nestybox/sysbox/blob/master/docs/developers-guide/test.md

Executes all test targets, including unit and integration tests. Ensure Docker and Make are installed.

```bash
$ make test
```

--------------------------------

### Start Inner Docker Daemon

Source: https://github.com/nestybox/sysbox/blob/master/docs/quickstart/dind.md

Initiates the Docker daemon within the system container. Ensure the log file is checked for successful startup.

```bash
/ # which docker
/usr/bin/docker

/ # dockerd > /var/log/dockerd.log 2>&1 &
```

--------------------------------

### Sysbox Deployment Logs on Kubernetes

Source: https://github.com/nestybox/sysbox/blob/master/docs/user-guide/troubleshoot-k8s.md

Examine these logs from the sysbox-deploy-k8s pod to diagnose installation issues. They show the steps involved in deploying CRI-O, Sysbox, and reconfiguring Kubelet.

```console
Adding K8s label "crio-runtime=installing" to node
node/gke-cluster-3-default-pool-766039d3-68mw labeled
Deploying CRI-O installer agent on the host ...
Running CRI-O installer agent on the host (may take several seconds) ...
Removing CRI-O installer agent from the host ...
Configuring CRI-O ...
Adding K8s label "sysbox-runtime=installing" to node
node/gke-cluster-3-default-pool-766039d3-68mw labeled
Installing Sysbox dependencies on host
Copying shiftfs sources to host
Deploying Sysbox installer helper on the host ...
Running Sysbox installer helper on the host (may take several seconds) ...
Stopping the Sysbox installer helper on the host ...
Removing Sysbox installer helper from the host ...
Installing Sysbox on host
Detected host distro: ubuntu_20.04
Configuring host sysctls
kernel.unprivileged_userns_clone = 1
fs.inotify.max_queued_events = 1048576
fs.inotify.max_user_watches = 1048576
fs.inotify.max_user_instances = 1048576
kernel.keys.maxkeys = 20000
kernel.keys.maxbytes = 400000
Starting Sysbox
Adding Sysbox to CRI-O config
Restarting CRI-O ...
Deploying Kubelet config agent on the host ...
Running Kubelet config agent on the host (will restart Kubelet and temporary bring down all pods on this node for ~1 min) ...
```

```console
Stopping the Kubelet config agent on the host ...
Removing Kubelet config agent from the host ...
Kubelet reconfig completed.
Adding K8s label "crio-runtime=running" to node
node/gke-cluster-3-default-pool-766039d3-68mw labeled
Adding K8s label "sysbox-runtime=running" to node
node/gke-cluster-3-default-pool-766039d3-68mw labeled
The k8s runtime on this node is now CRI-O.
Sysbox installation completed.
Done.
```

--------------------------------

### Verify Image Persistence After Restart

Source: https://github.com/nestybox/sysbox/blob/master/docs/quickstart/dind.md

After exiting and restarting the system container with the same bind mount, start Docker and verify that the previously pulled image is still available in the cache.

```bash
$ docker run --runtime=sysbox-runc -it --rm --hostname syscont --mount type=bind,source=/home/someuser/image-cache,target=/var/lib/docker nestybox/alpine-docker
/ # dockerd > /var/log/dockerd.log 2>&1 &
/ # docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

/ # docker image ls
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
busybox             latest              19485c79a9bb        7 weeks ago         1.22MB
```

--------------------------------

### Run K8s Control Plane Node with Docker

Source: https://github.com/nestybox/sysbox/blob/master/docs/quickstart/kind.md

Starts a Docker container that will serve as the Kubernetes control plane node. Ensure you have sysbox-runc installed and configured.

```bash
$ docker run --runtime=sysbox-runc -d --rm --name=k8s-master --hostname=k8s-master nestybox/k8s-node:v1.18.2
```

--------------------------------

### Demonstrating Startup Time Differences with Inner Docker Image Sharing

Source: https://github.com/nestybox/sysbox/blob/master/docs/user-guide/images.md

Compares the startup time of the first system container instance with subsequent instances when using the `nestybox/k8s-node` image. The first instance is slower due to initial data movement, while subsequent instances benefit from the inner Docker image sharing feature.

```console
cesar@eoan:$ time docker run --runtime=sysbox-runc -d nestybox/k8s-node:v1.18.2
fea256c7dc7dc28e5e4b8bc3a7419888dea99c825b69502545485d76158a678b

real    0m5.858s
user    0m0.027s
sys     0m0.028s
```

```console
cesar@eoan:$ time docker run --runtime=sysbox-runc -d nestybox/k8s-node:v1.18.2
fc62a96cbb372ee5d16b28b5688ab4d331391fe20baa6add9e8758f6962dde47

real    0m0.991s
user    0m0.038s
sys     0m0.018s
```

```console
cesar@eoan:$ time docker run --runtime=sysbox-runc -d nestybox/k8s-node:v1.18.2
26d4c94a3398604ea3f473ab6868ad359ffa1e30c5db20cd98922b1cd1591e5c

real    0m1.061s
user    0m0.030s
sys     0m0.027s
```

```console
cesar@eoan:$ time docker run --runtime=sysbox-runc -d nestybox/k8s-node:v1.18.2
6b886f1c491c6cd593ec46f4f6076126309993add7b6426547283c5c9728da9e

real    0m0.953s
user    0m0.034s
sys     0m0.030s
```

--------------------------------

### Remove Docker Snap Installation

Source: https://github.com/nestybox/sysbox/blob/master/docs/developers-guide/build.md

If Docker is installed via snap, it must be removed before proceeding with native installation.

```bash
$ sudo snap remove docker
docker removed
```

--------------------------------

### Install Sysbox Enterprise Edition on Ubuntu Focal

Source: https://github.com/nestybox/sysbox/blob/master/docs/user-guide/install-package.md

Download and install the Sysbox-EE .deb package for Ubuntu Focal. Verify the installation by checking the status of the sysbox service.

```bash
$ wget https://github.com/nestybox/sysbox-ee/releases/download/v0.4.0/sysbox-ee_0.4.0-0.ubuntu-focal_amd64.deb

$ sudo apt-get install ./sysbox-ee_0.4.0-0.ubuntu-focal_amd64.deb

$ sudo systemctl status sysbox -n20
```

```text
● sysbox.service - Sysbox container runtime
     Loaded: loaded (/lib/systemd/system/sysbox.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2021-07-20 19:35:31 UTC; 18s ago
       Docs: https://github.com/nestybox/sysbox
   Main PID: 7963 (sh)
      Tasks: 2 (limit: 4617)
     Memory: 868.0K
     CGroup: /system.slice/sysbox.service
             ├─7963 /bin/sh -c /usr/bin/sysbox-runc --version && /usr/bin/sysbox-mgr --version && /usr/bin/sysbox-fs --version && /bin/sleep infinity
             └─7986 /bin/sleep infinity

Jul 20 19:35:31 focal systemd[1]: Started Sysbox container runtime.
Jul 20 19:35:32 focal sh[7965]: sysbox-runc
        edition:         Enterprise Edition (EE)
        version:         0.4.0
        commit:         f4daa007da10280095911dde80a8cb95d03c4859
        built at:         Mon Jul 19 18:55:14 UTC 2021
        built by:         Rodny Molina
        oci-specs:         1.0.2-dev
Jul 20 19:35:32 focal sh[7972]: sysbox-mgr
        edition:         Enterprise Edition (EE)
        version:         0.4.0
        commit:         de7cbb47c9a667d4aaa79e4ca8aeadf6d5124bb2
        built at:         Mon Jul 19 18:55:51 UTC 2021
        built by:         Rodny Molina
Jul 20 19:35:32 focal sh[7978]: sysbox-fs
        edition:         Enterprise Edition (EE)
        version:         0.4.0
        commit:         b0cb35cf449f5c929dba24fc940aef151f4432c5
        built at:         Mon Jul 19 18:55:37 UTC 2021
        built by:         Rodny Molina
```

--------------------------------

### Remove Docker Containers During Installation

Source: https://github.com/nestybox/sysbox/blob/master/docs/user-guide/troubleshoot.md

If the Sysbox installer cannot proceed due to active Docker containers, use this command to remove all running and stopped containers before re-launching the installation.

```bash
docker rm $(docker ps -a -q) -f
```

--------------------------------

### Install Sysbox Enterprise Edition on Kubernetes Nodes

Source: https://github.com/nestybox/sysbox/blob/master/docs/user-guide/install-k8s.md

Label nodes for Sysbox Enterprise Edition installation and apply the corresponding manifest. Ensure only one of Sysbox or Sysbox Enterprise is installed per host.

```bash
kubectl label nodes <node-name> sysbox-install=yes
kubectl apply -f https://raw.githubusercontent.com/nestybox/sysbox/master/sysbox-k8s-manifests/sysbox-ee-install.yaml
```

--------------------------------

### Configure Kubectl Context

Source: https://github.com/nestybox/sysbox/blob/master/docs/quickstart/kind.md

Set up your local kubectl to interact with the newly created Kind cluster by exporting the KUBECONFIG environment variable and switching the context.

```console
$ export KUBECONFIG=${KUBECONFIG}:${HOME}/.kube/mycluster-config

$ kubectl config use-context kubernetes-admin@mycluster
Switched to context "kubernetes-admin@mycluster".
```

--------------------------------

### Preconfigure Docker for Sysbox Installation without Restart

Source: https://github.com/nestybox/sysbox/blob/master/docs/user-guide/install-package.md

To install Sysbox without restarting the Docker daemon, preconfigure essential attributes like `bip` and `default-address-pools` in `/etc/docker/daemon.json`. The Sysbox installer will not restart Docker if these are present.

```yaml
{
    "bip": "172.24.0.1/16",
    "default-address-pools": [
        {
            "base": "172.31.0.0/16",
            "size": 24
        }
    ]
}
```

--------------------------------

### Troubleshooting Docker Daemon Start Failure

Source: https://github.com/nestybox/sysbox/blob/master/docs/quickstart/images.md

If the inner Docker daemon fails to start after committing and running a system container, it might be due to stale PID files. This snippet shows how to remove them before starting the daemon.

```console
Error starting daemon: pid file found, ensure docker is not running or delete /var/run/docker.pid
```

```console
Failed to start containerd: timeout waiting for containerd to start
```

--------------------------------

### Inspect Core-dump from Host

Source: https://github.com/nestybox/sysbox/blob/master/docs/developers-guide/debug.md

Load and debug a core-dump file from your host machine. Ensure the correct binary path and core-dump file path are provided. Environment variables should be set as shown.

```console
$ sudo env "PATH=$PATH" env "GOROOT=$GOROOT" env "GOPATH=$GOPATH" env "PWD=$PWD" $(which dlv) core $(which sysbox-fs) ./core.sysbos-fs.1
```

--------------------------------

### Start Docker Daemon Inside System Container

Source: https://github.com/nestybox/sysbox/blob/master/docs/quickstart/dind.md

Start the Docker daemon inside the system container. Its logs will be redirected to `/var/log/dockerd.log`.

```bash
/ # dockerd > /var/log/dockerd.log 2>&1 &
```

--------------------------------

### Unmount Initial Mountpoint (Default Behavior)

Source: https://github.com/nestybox/sysbox/blob/master/docs/quickstart/security.md

Demonstrates unmounting an initial mountpoint, such as /etc/resolv.conf, which is allowed by default. This action exposes the underlying contents of the system container's image.

```bash
root@syscont:/# grep nameserver /etc/resolv.conf
nameserver 75.75.75.75
nameserver 75.75.76.76

root@syscont:/# umount /etc/resolv.conf

root@syscont:/# grep nameserver /etc/resolv.conf
(empty)
```

--------------------------------

### Create Rootfs and OCI Spec for Sysbox Container

Source: https://github.com/nestybox/sysbox/blob/master/docs/user-guide/deploy.md

These console commands demonstrate how to create a root filesystem for a system container and generate its OCI configuration file using `sysbox-runc`.

```bash
# mkdir /root/syscontainer
# cd /root/syscontainer
# mkdir rootfs
# docker export $(docker create debian:latest) | tar -C rootfs -xvf -
```

```bash
# sysbox-runc spec
```

--------------------------------

### Build Sysbox from Source

Source: https://github.com/nestybox/sysbox/blob/master/docs/developers-guide/build.md

Build Sysbox using the 'sysbox-static' target. This process occurs within a temporary container to keep the host system clean.

```bash
make sysbox-static
```

--------------------------------

### Install Sysbox Debugging Symbols

Source: https://github.com/nestybox/sysbox/blob/master/docs/developers-guide/debug.md

Before inspecting a core-dump, install the corresponding Sysbox debugging symbols package. This is crucial as official binaries do not include symbols.

```console
sudo apt-get install ~/sysbox-ce-dbgsym_0.4.0-0.ubuntu-focal_amd64.ddeb
```

--------------------------------

### View Sysbox-fs Help

Source: https://github.com/nestybox/sysbox/blob/master/docs/user-guide/configuration.md

Displays the command-line options available for the sysbox-fs component.

```bash
sysbox-fs --help
```

--------------------------------

### Install Kernel Headers on Amazon Linux 2

Source: https://github.com/nestybox/sysbox/blob/master/docs/developers-guide/test.md

Installs necessary kernel headers on Amazon Linux 2 hosts. Ensure your kernel version is compatible.

```bash
sudo yum install kernel-devel-$(uname -r)
```

--------------------------------

### Build system container image with preloaded images

Source: https://github.com/nestybox/sysbox/blob/master/docs/quickstart/images.md

Builds a Docker image that includes preloaded busybox and alpine images. Ensure DOCKER_BUILDKIT is set to 0 to use the legacy builder.

```console
$ DOCKER_BUILDKIT=0 docker build -t nestybox/syscont-with-inner-containers:latest .
```

--------------------------------

### Create and Scale Nginx Deployment

Source: https://github.com/nestybox/sysbox/blob/master/docs/quickstart/kind.md

Creates a deployment for Nginx pods and then scales it to four replicas. This demonstrates basic Kubernetes workload management after the cluster is set up.

```bash
$ kubectl create deployment nginx --image=nginx

$ kubectl scale --replicas=4 deployment nginx
```

--------------------------------

### Create and Remount New Mountpoint as Read-Write

Source: https://github.com/nestybox/sysbox/blob/master/docs/quickstart/security.md

Shows how to create a new mountpoint (tmpfs) inside the container and successfully remount it as read-write. New mounts are not considered immutable and are not subject to the same restrictions.

```bash
root@syscont:/# mkdir /root/tmp
root@syscont:/# mount -t tmpfs -o ro,size=10M tmpfs /root/tmp
root@syscont:/# mount -o remount,rw,bind /root/tmp
```

--------------------------------

### Build Sysbox Binaries for Debugging

Source: https://github.com/nestybox/sysbox/blob/master/docs/developers-guide/debug.md

Use these Makefile targets to build Sysbox binaries with compiler optimizations disabled, which is necessary for debugging. After building, install the binaries using 'sudo make install'.

```makefile
sysbox-debug
sysbox-runc-debug
sysbox-fs-debug
sysbox-mgr-debug
```

```bash
$ make sysbox-debug && sudo make install
```

--------------------------------

### Deploy a System Container

Source: https://github.com/nestybox/sysbox/blob/master/docs/quickstart/security.md

Launches a Debian system container using Sysbox as the runtime. This is the initial step to explore container isolation features.

```console
$ docker run --runtime=sysbox-runc --rm -it --hostname syscont debian:latest
root@syscont:/#

```

--------------------------------

### Deploy Ubuntu Pod with Systemd and Docker using Sysbox

Source: https://github.com/nestybox/sysbox/blob/master/docs/user-guide/deploy.md

Use this pod spec to deploy an Ubuntu container with systemd as init and Docker installed. Ensure your Kubernetes cluster has Sysbox installed and configured.

```yaml
apiVersion: v1
kind: Pod
metadata:
  name: ubu-focal-systemd-docker
spec:
  runtimeClassName: sysbox-runc
  hostUsers: false
  containers:
  - name: ubu-bio-systemd-docker
    image: registry.nestybox.com/nestybox/ubuntu-focal-systemd-docker
    command: ["/sbin/init"]
  restartPolicy: Never
```

--------------------------------

### Check Ephemeral Systemd Service Logs

Source: https://github.com/nestybox/sysbox/blob/master/docs/user-guide/troubleshoot-k8s.md

Use these journalctl commands to inspect the logs of ephemeral systemd services used during Sysbox installation on K8s worker nodes. These services assist with CRI-O and Sysbox installation, and Kubelet reconfiguration.

```bash
journalctl -eu crio-installer
```

```bash
journalctl -eu sysbox-installer-helper
```

```bash
journalctl -eu kubelet-config-helper
```

--------------------------------

### List Container's Initial Mounts

Source: https://github.com/nestybox/sysbox/blob/master/docs/quickstart/security.md

Execute this command inside the container to view its initial mounts, all of which are considered immutable by Sysbox.

```bash
root@syscont:/# findmnt
TARGET                                                       SOURCE                                                                                                    FSTYPE   OPTIONS
/                                                            .                                                                                                         shiftfs  rw,relatime
|-/sys                                                       sysfs                                                                                                     sysfs    rw,nosuid,nodev,noexec,relatime
| |-/sys/firmware                                            tmpfs                                                                                                     tmpfs    ro,relatime,uid=165536,gid=165536
| |-/sys/fs/cgroup                                           tmpfs                                                                                                     tmpfs    rw,nosuid,nodev,noexec,relatime,mode=755,uid=165536,gid=165536
| | |-/sys/fs/cgroup/systemd                                 systemd                                                                                                   cgroup   rw,nosuid,nodev,noexec,relatime,xattr,name=systemd
| | |-/sys/fs/cgroup/memory                                  cgroup                                                                                                    cgroup   rw,nosuid,nodev,noexec,relatime,memory
| | |-/sys/fs/cgroup/cpuset                                  cgroup                                                                                                    cgroup   rw,nosuid,nodev,noexec,relatime,cpuset
| | |-/sys/fs/cgroup/blkio                                   cgroup                                                                                                    cgroup   rw,nosuid,nodev,noexec,relatime,blkio
| | |-/sys/fs/cgroup/net_cls,net_prio                        cgroup                                                                                                    cgroup   rw,nosuid,nodev,noexec,relatime,net_cls,net_prio
| | |-/sys/fs/cgroup/perf_event                              cgroup                                                                                                    cgroup   rw,nosuid,nodev,noexec,relatime,perf_event
| | |-/sys/fs/cgroup/hugetlb                                 cgroup                                                                                                    cgroup   rw,nosuid,nodev,noexec,relatime,hugetlb
| | |-/sys/fs/cgroup/cpu,cpuacct                             cgroup                                                                                                    cgroup   rw,nosuid,nodev,noexec,relatime,cpu,cpuacct
| | |-/sys/fs/cgroup/freezer                                 cgroup                                                                                                    cgroup   rw,nosuid,nodev,noexec,relatime,freezer
| | |-/sys/fs/cgroup/devices                                 cgroup                                                                                                    cgroup   rw,nosuid,nodev,noexec,relatime,devices
| | |-/sys/fs/cgroup/pids                                    cgroup                                                                                                    cgroup   rw,nosuid,nodev,noexec,relatime,pids
| | `-/sys/fs/cgroup/rdma                                    cgroup                                                                                                    cgroup   rw,nosuid,nodev,noexec,relatime,rdma

```

--------------------------------

### Change Directory to Workspace

Source: https://github.com/nestybox/sysbox/blob/master/docs/developers-guide/debug.md

Navigate to your sysbox workspace directory before starting the debugger.

```console
rodny@vm-1:~$ cd ~/wsp/sysbox
```

--------------------------------

### Inspect /var/lib/sysbox Directory Structure (No Containers)

Source: https://github.com/nestybox/sysbox/blob/master/docs/user-guide/troubleshoot.md

When no system containers are running, the /var/lib/sysbox directory should have a clean structure containing subdirectories for containerd, docker, and kubelet, but without container-specific state.

```bash
# tree /var/lib/sysbox
/var/lib/sysbox
├── containerd
├── docker
│   ├── baseVol
│   ├── cowVol
│   └── imgVol
└── kubelet
```

--------------------------------

### Verify Sysbox Service Status

Source: https://github.com/nestybox/sysbox/blob/master/docs/user-guide/install-package.md

Check the status of the Sysbox systemd service to confirm that it has been installed correctly and is running.

```systemd
$ systemctl status sysbox -n20
● sysbox.service - Sysbox container runtime
     Loaded: loaded (/lib/systemd/system/sysbox.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2026-03-03 14:08:14 PST; 22min ago
       Docs: https://github.com/nestybox/sysbox
   Main PID: 1159354 (sh)
      Tasks: 2 (limit: 38079)
     Memory: 400.0K
        CPU: 13ms
     CGroup: /system.slice/sysbox.service
             ├─1159354 /bin/sh -c "/usr/bin/sysbox-runc --version && /usr/bin/sysbox-mgr --version && /usr/bin/sysbox-fs --version && /bin/sleep infinity"
             └─1159374 /bin/sleep infinity

Mar 03 14:08:14 lenovo systemd[1]: Started Sysbox container runtime.
Mar 03 14:08:14 lenovo sh[1159355]: sysbox-runc
Mar 03 14:08:14 lenovo sh[1159355]:         edition:         Community Edition (CE)
Mar 03 14:08:14 lenovo sh[1159355]:         version:         0.7.0
Mar 03 14:08:14 lenovo sh[1159355]:         commit:          a4dd414f7b9b7455c0fbf0d5e5db7bcfe30645bc
Mar 03 14:08:14 lenovo sh[1159355]:         built at:         Tue Mar  3 21:51:47 UTC 2026
Mar 03 14:08:14 lenovo sh[1159355]:         built by:         Cesar Talledo
Mar 03 14:08:14 lenovo sh[1159355]:         oci-specs:         1.1.0+dev
Mar 03 14:08:14 lenovo sh[1159362]: sysbox-mgr
Mar 03 14:08:14 lenovo sh[1159362]:         edition:         Community Edition (CE)
Mar 03 14:08:14 lenovo sh[1159362]:         version:         0.7.0
Mar 03 14:08:14 lenovo sh[1159362]:         commit:          bff3721f86e737cfa129dfe9fe2e7933692aba26
Mar 03 14:08:14 lenovo sh[1159362]:         built at:         Tue Mar  3 21:52:05 UTC 2026
Mar 03 14:08:14 lenovo sh[1159362]:         built by:         Cesar Talledo
Mar 03 14:08:14 lenovo sh[1159368]: sysbox-fs
Mar 03 14:08:14 lenovo sh[1159368]:         edition:         Community Edition (CE)
Mar 03 14:08:14 lenovo sh[1159368]:         version:         0.7.0
Mar 03 14:08:14 lenovo sh[1159368]:         commit:          b70bd38bbf72bf0e759c5f7d8c38925d717068ae
Mar 03 14:08:14 lenovo sh[1159368]:         built at:         Tue Mar  3 21:52:03 UTC 2026
Mar 03 14:08:14 lenovo sh[1159368]:         built by:         Cesar Talledo
```

--------------------------------

### Enable Core-Dump Creation with Ulimit

Source: https://github.com/nestybox/sysbox/blob/master/docs/user-guide/troubleshoot.md

Before generating a core-dump, enable unlimited core-dump creation for the current process using the ulimit command.

```bash
ulimit -c unlimited
```

--------------------------------

### Build a multi-arch Docker image

Source: https://github.com/nestybox/sysbox/blob/master/docs/user-guide/dind.md

Builds a Docker image for a specified platform (e.g., linux/arm64) using Docker's buildx. This command assumes a Dockerfile is present in the current directory.

```bash
$ docker buildx build --platform linux/arm64 -t alpine-arm-test:arm64 .
```