### Install System Dependencies on Debian/Ubuntu

Source: https://docs.softwareheritage.org/_sources/devel/developer-setup

Installs essential system packages, development libraries, and database systems required for Software Heritage on Debian/Ubuntu systems using apt.

```console
sudo apt install lsb-release wget apt-transport-https

sudo wget https://www.postgresql.org/media/keys/ACCC4CF8.asc -O /etc/apt/trusted.gpg.d/postgresql.asc

echo "deb https://apt.postgresql.org/pub/repos/apt/ $(lsb_release -cs)-pgdg main" | sudo tee -a /etc/apt/sources.list.d/pgdg.list

sudo wget https://downloads.apache.org/cassandra/KEYS -O /etc/apt/trusted.gpg.d/cassandra.asc

echo "deb https://debian.cassandra.apache.org 41x main" | sudo tee -a /etc/apt/sources.list.d/cassandra.list

sudo apt update

sudo apt install \
    build-essential pkg-config lzip rsync \
    python3 python3-pip python3-venv virtualenvwrapper \
    libpython3-dev libsystemd-dev libsvn-dev libffi-dev librdkafka-dev \
    fuse3 libfuse3-dev libcmph-dev libleveldb-dev \
    git myrepos \
    graphviz plantuml inkscape \
    postgresql libpq-dev cassandra redis-server
```

--------------------------------

### Install Cassandra on Fedora

Source: https://docs.softwareheritage.org/_sources/devel/developer-setup

Installs Apache Cassandra version 50x on Fedora systems by configuring the official repository and using dnf.

```console
sudo rpm --import https://downloads.apache.org/cassandra/KEYS

echo "[cassandra]
name=Apache Cassandra
baseurl=https://redhat.cassandra.apache.org/50x/
gpgcheck=1
repo_gpgcheck=0
gpgkey=https://downloads.apache.org/cassandra/KEYS" | sudo tee /etc/yum.repos.d/cassandra.repo

sudo dnf -y update

sudo dnf -y install cassandra
```

--------------------------------

### Install System Dependencies on Fedora

Source: https://docs.softwareheritage.org/_sources/devel/developer-setup

Installs necessary development tools, Python, and system libraries for Software Heritage on Fedora systems using dnf.

```console
sudo dnf -y group install c-development

sudo dnf -y install \
    pkgconf-pkg-config lzip rsync python3.11 python3-virtualenvwrapper \
    python3.11-devel systemd-devel subversion-devel libffi-devel \
    librdkafka fuse3 fuse3-devel leveldb-devel git myrepos graphviz \
    plantuml inkscape postgresql-server postgresql-contrib libpq \
    libpq-devel redis
```

--------------------------------

### Start Local Server with Example Dataset

Source: https://docs.softwareheritage.org/_sources/devel/swh-graph/example-dataset

Launches a local RPC server using the example dataset. This command starts the server in the background and is useful for testing graph queries.

```console
$ swh graph rpc-serve -g swh/graph/example_dataset/compressed/example &
```

--------------------------------

### Common Software Heritage Benchmark Setup

Source: https://docs.softwareheritage.org/devel/swh-shard/benchmarks

These commands cover the common setup steps required before running benchmarks, including disk formatting and mounting, package installation, cloning the repository, setting up a Python virtual environment, installing dependencies, and activating the environment.

```bash
mkfs.ext4 /dev/sdc
```

```bash
mount /dev/sdc /mnt
```

```bash
apt-get install -y python3-venv python3-dev libcmph-dev gcc git emacs-nox
```

```bash
git clone -b wip-benchmark https://git.easter-eggs.org/biceps/swh-perfecthash/
```

```bash
python3 -m venv bench
```

```bash
source bench/bin/activate
```

```bash
cd swh-perfecthash
```

```bash
pip install -r requirements.txt -r requirements-test.txt tox wheel
```

```bash
tox run -e py3
```

--------------------------------

### Install pre-commit and update swh packages

Source: https://docs.softwareheritage.org/_sources/devel/developer-setup

Installs the 'pre-commit' tool and then runs the './bin/update' script to checkout all Software Heritage (swh) packages' source repositories into the current virtual environment. This prepares the development environment for working with swh packages.

```bash
(swh) ~/swh-environment$ pip install pre-commit
(swh) ~/swh-environment$ ./bin/update
```

--------------------------------

### Install postgresql-autodoc on Debian/Ubuntu

Source: https://docs.softwareheritage.org/_sources/devel/developer-setup

Installs the postgresql-autodoc utility on Debian or Ubuntu systems using the apt package manager. This is a straightforward package installation.

```bash
sudo apt install postgresql-autodoc
```

--------------------------------

### Install Node.js on Debian/Ubuntu

Source: https://docs.softwareheritage.org/_sources/devel/developer-setup

Installs Node.js version 20.x on Debian/Ubuntu systems by adding the official NodeSource repository and then using apt.

```console
curl -fsSL https://deb.nodesource.com/setup_20.x | sudo bash -

sudo apt install -y nodejs
```

--------------------------------

### Install Python Requests Module

Source: https://docs.softwareheritage.org/_sources/devel/getting-started/api

Ensures the 'requests' Python module is installed, which is necessary for making HTTP requests to the Software Heritage API. This command-line instruction verifies the Python version and installs the module if it's not already present.

```console
boris@castalia:notebook$ python3 -V
Python 3.7.3
boris@castalia:notebooks$ pip3 install requests
Requirement already satisfied: requests in /usr/lib/python3/dist-packages (2.21.0)
boris@castalia:notebook$
```

--------------------------------

### Start swh.graph API server

Source: https://docs.softwareheritage.org/_sources/devel/swh-graph/quickstart

Starts the swh.graph GRPC API server using a compressed graph dataset. The command requires the path to the compressed graph files.

```console
(venv) $ swh graph rpc-serve -g compressed/graph
Started GRPC using dataset from swh/graph/example_dataset/compressed/example
['/home/dev/.cargo/bin/swh-graph-grpc-serve', '--bind', '[::]:50867', 'compressed/graph']
INFO:swh.graph.grpc_server:Starting gRPC server: /home/dev/.cargo/bin/swh-graph-grpc-serve --bind '[::]:50867' compressed/graph
2024-06-18T09:12:40+02:00 - INFO - Loading graph
2024-06-18T09:12:40+02:00 - INFO - Loading properties
2024-06-18T09:12:40+02:00 - INFO - Loading labels
2024-06-18T09:12:40+02:00 - INFO - Starting server
======== Running on http://0.0.0.0:5009 ========
(Press CTRL+C to quit)
```

--------------------------------

### Example Search Result for Keyword (JSON)

Source: https://docs.softwareheritage.org/_sources/devel/getting-started/api

An example of the JSON response when searching for a keyword in the Software Heritage archive. Each object in the array contains the 'origin_visits_url' and the 'url' of the repository.

```json
[
  {
    "origin_visits_url": "https://archive.softwareheritage.org/api/1/origin/https://github.com/borisbaldassari/alambic/visits/",
    "url": "https://github.com/borisbaldassari/alambic"
  }
]
```

--------------------------------

### Install Dependencies using Apt and Rustup

Source: https://docs.softwareheritage.org/devel/swh-graph/quickstart

Installs necessary system dependencies for swh.graph on Debian-based systems, including build tools, Python, Java, zstd, and protobuf. It also installs Rust using rustup.

```shell
$ sudo apt install build-essential libclang-dev python3 python3-venv \ default-jre zstd protobuf-compiler pv
$ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

```

--------------------------------

### Example Root Directory Listing Output

Source: https://docs.softwareheritage.org/_sources/devel/getting-started/api

This console output displays a list of files and directories found at the root of a Software Heritage repository, as generated by the Python script.

```console
- .dockerignore
- .env
- .gitignore
- CODE_OF_CONDUCT.html
- CODE_OF_CONDUCT.md
- LICENCE.html
- LICENCE.md
- Readme.md
- doc
- docker
- docker-compose.run.yml
- docker-compose.test.yml
- dockercfg.encrypted
- mojo
- resources
```

--------------------------------

### Example Specific Origin JSON

Source: https://docs.softwareheritage.org/_sources/devel/getting-started/api

An example JSON object representing the details of a specific repository fetched from the Software Heritage archive. It includes the 'origin_visits_url' and the 'url' of the repository.

```json
{
    "origin_visits_url": "https://archive.softwareheritage.org/api/1/origin/https://github.com/borisbaldassari/alambic/visits/",
    "url": "https://github.com/borisbaldassari/alambic"
}
```

--------------------------------

### Start and enable Cassandra systemd service

Source: https://docs.softwareheritage.org/_sources/sysadm/data-silos/cassandra/installation

Enables the Cassandra systemd service to start automatically on boot and then starts the service immediately. This command is executed after all configurations, including ZFS, are in place.

```shell
# systemctl enable cassandra@instance1
# systemctl start cassandra@instance1
```

--------------------------------

### Install System Dependencies for swh.graph

Source: https://docs.softwareheritage.org/_sources/devel/swh-graph/quickstart

Installs necessary system dependencies on a Debian-based system for building and running swh.graph, including build tools, Python, Java, zstd, and protobuf.

```console
$ sudo apt install build-essential libclang-dev python3 python3-venv \n       default-jre zstd protobuf-compiler pv
$ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
```

--------------------------------

### Initialize Python Script for Software Heritage API

Source: https://docs.softwareheritage.org/_sources/devel/getting-started/api

Imports necessary Python modules ('json', 'requests') and defines a utility function 'jprint' for pretty-printing JSON data. This setup is required before making any API calls.

```python
import json
import requests

# Utility to pretty-print json.
def jprint(obj):
    # create a formatted string of the Python JSON object
    print(json.dumps(obj, sort_keys=True, indent=4))
```

--------------------------------

### Install swh.graph Python package

Source: https://docs.softwareheritage.org/_sources/devel/swh-graph/quickstart

Sets up a Python virtual environment and installs the swh.graph Python package using pip. This enables the use of the 'swh graph' command-line tool.

```console
python3 -m venv .venv
source .venv/bin/activate
(venv) $ pip install swh.graph
```

--------------------------------

### Install Node.js on Fedora

Source: https://docs.softwareheritage.org/_sources/devel/developer-setup

Installs Node.js on Fedora systems using the dnf package manager.

```console
sudo dnf -y install nodejs
```

--------------------------------

### Start Cassandra Node Service

Source: https://docs.softwareheritage.org/sysadm/data-silos/cassandra/installation

This command initiates the Cassandra service for a specific instance. It's the first step in bringing a new node online. The initial bootstrap process will commence after this command.

```bash
systemctl start cassandra@instance1
```

--------------------------------

### Install Yarn on Fedora

Source: https://docs.softwareheritage.org/_sources/devel/developer-setup

Installs the Yarn package manager on Fedora systems using the dnf package manager.

```console
sudo dnf -y install yarnpkg
```

--------------------------------

### Clone sysadm umbrella repositories using Git

Source: https://docs.softwareheritage.org/sysadm/getting-started/setup-repositories

This snippet demonstrates how to clone the main umbrella repositories for sysadm infrastructure. These repositories act as entry points for managing other related repositories. It uses the standard `git clone` command.

```bash
git clone https://gitlab.softwareheritage.org/swh/infra/sysadm-environment.git
git clone https://gitlab.softwareheritage.org/swh/infra/puppet/puppet-environment.git
git clone https://gitlab.softwareheritage.org/swh/infra/ci-cd/ci-environment.git
```

--------------------------------

### Install Java JDK on Fedora

Source: https://docs.softwareheritage.org/_sources/devel/developer-setup

Installs the Adoptium Temurin JDK 17 on Fedora systems and sets it as the default Java version.

```console
sudo dnf -y install adoptium-temurin-java-repository

sudo dnf config-manager setopt adoptium-temurin-java-repository.enabled=1

sudo dnf -y install temurin-17-jdk

sudo update-alternatives --set java /usr/lib/jvm/temurin-17-jdk/bin/java
```

--------------------------------

### Initialize and Update MyRepos Managed Repositories

Source: https://docs.softwareheritage.org/_sources/sysadm/getting-started/setup-repositories

Iterates through cloned umbrella repositories, adds their .mrconfig to trusted configurations, and then uses 'mr up' to clone all repositories managed by myrepos.

```bash
for repository in sysadm-environment puppet-environment ci-environment; do
  pushd $repository ; 
  readlink -f .mrconfig >> ~/.mrtrust ; 
  mr up ; 
  popd ; 
done
```

--------------------------------

### Install postgresql-autodoc manually on Fedora

Source: https://docs.softwareheritage.org/_sources/devel/developer-setup

Installs postgresql-autodoc on Fedora systems by manually installing its Perl dependencies, cloning the aut-doc repository from GitHub, and then building and installing it. This process is necessary because the utility is not directly available in Fedora repositories.

```bash
sudo dnf -y install perl-DBI perl-HTML-Template perl-lib-relative perl-pgsql_perl5 perl-DBD-Pg

git clone https://github.com/cbbrowne/autodoc.git

cd autodoc

sudo make install
```

--------------------------------

### Clone Umbrella Repositories

Source: https://docs.softwareheritage.org/_sources/sysadm/getting-started/setup-repositories

Clones the main umbrella repositories for sysadm infrastructure. These repositories contain configuration files and READMEs to manage further sub-repositories.

```bash
$ git clone https://gitlab.softwareheritage.org/swh/infra/sysadm-environment.git
$ git clone https://gitlab.softwareheritage.org/swh/infra/puppet/puppet-environment.git
$ git clone https://gitlab.softwareheritage.org/swh/infra/ci-cd/ci-environment.git
```

--------------------------------

### Install Elasticsearch on Fedora

Source: https://docs.softwareheritage.org/_sources/devel/developer-setup

Installs Elasticsearch on Fedora systems by configuring the Elastic YUM repository and then installing the elasticsearch package. This involves creating a .repo file with the repository details, updating the system, and then installing the package using dnf.

```bash
echo "[elasticsearch]
name=Elasticsearch repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
autorefresh=1
type=rpm-md" | sudo tee /etc/yum.repos.d/elasticsearch.repo

sudo dnf -y update

sudo dnf -y install elasticsearch
```

--------------------------------

### Install build dependencies for ipxe

Source: https://docs.softwareheritage.org/_sources/sysadm/server-architecture/howto-install-new-physical-server

Installs necessary packages for building the ipxe bootloader and related utilities. Requires sudo privileges.

```shell
sudo apt build-dep ipxe
sudo apt install whois j2cli isolinux
```

--------------------------------

### Build swh-graph from source

Source: https://docs.softwareheritage.org/_sources/devel/swh-graph/quickstart

Clones the swh-graph repository and builds the rust backend for the swh-graph-grpc-server. This is an alternative to installing directly via cargo.

```console
git clone https://gitlab.softwareheritage.org/swh/devel/swh-graph.git
cd swh-graph
cargo build --release -p swh-graph-grpc-server
```

--------------------------------

### Install Elasticsearch on Debian/Ubuntu

Source: https://docs.softwareheritage.org/_sources/devel/developer-setup

Installs Elasticsearch on Debian-based systems by adding the Elastic APT repository and then installing the elasticsearch package. This involves downloading the GPG key, configuring the repository source, updating the package list, and finally installing the package.

```bash
sudo wget https://artifacts.elastic.co/GPG-KEY-elasticsearch -O /etc/apt/trusted.gpg.d/elasticsearch.asc

echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elasticsearch.list

sudo apt update

sudo apt install elasticsearch
```

--------------------------------

### Install Software Heritage Dependencies on Debian/Ubuntu

Source: https://docs.softwareheritage.org/devel/developer-setup

Installs essential build tools, Python, development libraries, databases (PostgreSQL, Cassandra, Redis), version control, and documentation utilities using apt. Includes setup for PostgreSQL and Cassandra repositories.

```bash
sudo apt install lsb-release wget apt-transport-https

sudo wget https://www.postgresql.org/media/keys/ACCC4CF8.asc -O /etc/apt/trusted.gpg.d/postgresql.asc

echo "deb https://apt.postgresql.org/pub/repos/apt/ $(lsb_release -cs)-pgdg main" | sudo tee -a /etc/apt/sources.list.d/pgdg.list

sudo wget https://downloads.apache.org/cassandra/KEYS -O /etc/apt/trusted.gpg.d/cassandra.asc

echo "deb https://debian.cassandra.apache.org 41x main" | sudo tee -a /etc/apt/sources.list.d/cassandra.list

sudo apt update

sudo apt install \
    build-essential pkg-config lzip rsync \
    python3 python3-pip python3-venv virtualenvwrapper \
    libpython3-dev libsystemd-dev libsvn-dev libffi-dev librdkafka-dev \
    fuse3 libfuse3-dev libcmph-dev libleveldb-dev \
    git myrepos \
    graphviz plantuml inkscape \
    postgresql libpq-dev cassandra redis-server

```

--------------------------------

### Install CMPH Library Manually on Fedora

Source: https://docs.softwareheritage.org/_sources/devel/developer-setup

Manually compiles and installs the CMPH library version 2.0.2 from source on Fedora, as it's not available in standard repositories.

```console
wget https://sourceforge.net/projects/cmph/files/v2.0.2/cmph-2.0.2.tar.gz

tar -xvf cmph-2.0.2.tar.gz

cd cmph-2.0.2

./configure && make && sudo make install
```

--------------------------------

### API server#

Source: https://docs.softwareheritage.org/devel/swh-graph/quickstart

Instructions on how to start the swh.graph API server for a compressed graph dataset using the `rpc-serve` command. It also covers reindexing the graph and handling potential errors related to file formats.

```APIDOC
## API Server Setup and Usage

### Starting the RPC Server
To start an API server for a compressed graph dataset, use the `rpc-serve` command with the basename of the graph. This command loads files with the specified path prefix (e.g., `compressed/graph.{graph,properties,offsets,...}`).

**Command Example:**
```bash
(venv) $ swh graph rpc-serve -g compressed/graph
```

This will start a gRPC server, typically on port 50867, serving the specified graph dataset.

### Reindexing the Graph
If you encounter errors related to missing `.cmph`, `.bin`, `.bits`, or `.ef` files (common for graphs before 2024), you need to reindex the graph. This often requires installing or updating `swh-graph`.

**Installation/Update:**
```bash
RUSTFLAGS="-C target-cpu=native" cargo install --locked swh-graph
```

**Reindexing Command:**
```bash
swh graph reindex compressed/graph
```

### Handling `.ef` File Format Errors
If you receive an error indicating a "Wrong type hash" when deserializing `.ef` files, it means the file format is incompatible with your current `swh-graph` version. You need to regenerate the `.ef` files.

**Regenerating `.ef` files:**
```bash
RUSTFLAGS="-C target-cpu=native" cargo install --locked swh-graph
swh graph reindex --ef compressed/graph
```

### Querying the Compressed Graph (HTTP)
Once the API server is running, you can query the compressed graph using HTTP requests. The following example uses `httpie` to query for leaves associated with a specific directory hash.

**Query Example:**
```bash
~/tmp$ http :50867/graph/leaves/swh:1:dir:432d1b21c1256f7408a07c577b6974bbdbcc1323
```

**Success Response (200):**
```
HTTP/1.1 200 OK
Content-Type: text/plain
Date: Tue, 15 Sep 2020 08:35:19 GMT
Server: Python/3.8 aiohttp/3.6.2
Transfer-Encoding: chunked

swh:1:cnt:33af56e02dd970873d8058154bf016ec73b35dfb
swh:1:cnt:b03b4ffd7189ae5457d8e1c2ee0490b1938fd79f
swh:1:cnt:74d127c2186f7f0e8b14a27249247085c49d548a
swh:1:cnt:c0139aa8e79b338e865a438326629fa22fa8f472
...
swh:1:cnt:a6b60e797063fef707bbaa4f90cfb4a2cbbddd4a
swh:1:cnt:cc0a1deca559c1dd2240c08156d31cde1d8ed406
```

Refer to the specific HTTP graph querying API documentation for more details on available endpoints and usage.
```

--------------------------------

### Start Software Heritage Docker Containers

Source: https://docs.softwareheritage.org/devel/getting-started

This command initiates all the Docker containers defined in the `docker-compose.yml` file. It builds Docker images if necessary and starts the services in detached mode. Ensure you are in the `swh-docker` directory before running this command.

```shell
~/swh-docker$ docker compose up -d

```

--------------------------------

### Common Setup: Install Dependencies and Run Tox

Source: https://docs.softwareheritage.org/_sources/devel/swh-shard/benchmarks

Installs common system and Python dependencies required for benchmarking, clones the swh-perfecthash repository, sets up a Python virtual environment, and then runs the tox tests. This is a prerequisite for executing the benchmark commands.

```shell
apt-get install -y python3-venv python3-dev libcmph-dev gcc git emacs-nox
git clone -b wip-benchmark https://git.easter-eggs.org/biceps/swh-perfecthash/
python3 -m venv bench
source bench/bin/activate
cd swh-perfecthash
pip install -r requirements.txt -r requirements-test.txt tox wheel
tox run -e py3
```

--------------------------------

### Install Dia on Fedora

Source: https://docs.softwareheritage.org/_sources/devel/developer-setup

Installs the Dia diagramming software on Fedora systems using the dnf package manager. This command makes the Dia application available for diagram creation.

```bash
sudo dnf -y install dia
```

--------------------------------

### Install Dia on Debian/Ubuntu

Source: https://docs.softwareheritage.org/_sources/devel/developer-setup

Installs the Dia diagramming software on Debian or Ubuntu systems using the apt package manager. This command ensures the Dia application is available for use.

```bash
sudo apt install dia
```

--------------------------------

### Enable Corepack for Yarn on Debian/Ubuntu

Source: https://docs.softwareheritage.org/_sources/devel/developer-setup

Enables Corepack, the package manager for Node.js, to manage Yarn installations on Debian/Ubuntu systems.

```console
sudo corepack enable
```

--------------------------------

### Create ZFS pools and datasets for Cassandra

Source: https://docs.softwareheritage.org/_sources/sysadm/data-silos/cassandra/installation

Creates ZFS pools and datasets required for Cassandra data and commit logs. It emphasizes using WWNs for device identification and configures specific pool options like ashift, compression, and mount points. Includes commands to load the ZFS module if needed and reboot the server for verification.

```shell
# # get the wwmn name
# ls -al /dev/disk/by-id/nvme-
#
# # Load the zfs module (only if the server was not restarted after initial puppet run)
# modprobe zfs
#
# # Create the zfs pool(s)
# zpool create -o ashift=12 -O atime=off -O relatime=on -O mountpoint=none -O compression=off \
    mixeduse \
    nvme-XX nvme-XY nvme-XZ nvme-YX
# # Only if the server has a write intensive disk for the commit log
# zpool create -o ashift=12 -O atime=off -O relatime=on -O mountpoint=none -O compression=off \
    writeintensive \
    nvme-XX
#
# Create the zfs datasets
# zfs create -o mountpoint=/srv/cassandra/instance1/data mixeduse/cassandra-instance1-data
# # Change the pool to writeintensive if the server has a dedicated disk for the commit logs
# zfs create -o mountpoint=/srv/cassandra/instance1/commitlog mixeduse/cassandra-instance1-commitlog
#
# # Reboot the server to ensure everything is correct
# reboot
#
# # Check the zfs configuration after the reboot
# zpool status
# zfs list
```

--------------------------------

### Example Archive Statistics JSON

Source: https://docs.softwareheritage.org/_sources/devel/getting-started/api

An example JSON structure representing the statistics returned by the Software Heritage archive API. It includes counts for various object types like content, directory, origin, person, release, and revision.

```json
{
    "content": 10049535736,
    "directory": 8390591308,
    "origin": 156388918,
    "person": 42263568,
    "release": 17218891,
    "revision": 2109783249
}
```

--------------------------------

### Software Heritage Database Initialization with Configuration File

Source: https://docs.softwareheritage.org/devel/swh-core/cli

Demonstrates initializing a Software Heritage database using a YAML configuration file. It shows how to specify the database to initialize via different methods, including direct config file path and environment variables.

```yaml
storage:
  cls: pipeline
  steps:
    - cls: masking
      masking_db: postgresql:///?service=swh-masking-proxy
    - cls: buffer
    - cls: postgresql
      db: postgresql://user:passwd@pghost:5433/swh-storage
      objstorage:
        cls: memory
```

```bash
$ swh db -C conf.yml init storage  # or
$ SWH_CONFIG_FILENAME=conf.yml swh db init storage # or
$ swh db init --dbname postgresql://user:passwd@pghost:5433/swh-storage storage
```

```bash
$ # to initialize the "main" storage db (expected to be the last element
$ # of a pipeline config),
$ # or to initialize the masking_db:
$ swh db -C conf.yml init -p storage.steps.0
```

--------------------------------

### Install all swh packages in development mode

Source: https://docs.softwareheritage.org/_sources/devel/developer-setup

Installs all Software Heritage (swh) packages within the active virtual environment in development mode, including testing dependencies. This command is run after checking out the source code and setting up the virtual environment.

```bash
(swh) ~/swh-environment$ bin/install
```

--------------------------------

### Start swh.graph API Server

Source: https://docs.softwareheritage.org/devel/swh-graph/quickstart

Starts the swh.graph API server for a compressed graph dataset using the `rpc-serve` command. It requires the basename of the graph dataset, which is used to locate associated files like `.graph`, `.properties`, and `.offsets`. The server binds to a gRPC port.

```bash
(venv) $ swh graph rpc-serve -g compressed/graph
INFO:swh.graph.grpc_server:Starting gRPC server: /home/dev/.cargo/bin/swh-graph-grpc-serve --bind '[::]:50867' compressed/graph
======== Running on http://0.0.0.0:5009 ========
(Press CTRL+C to quit)

```

--------------------------------

### Example Master Branch Revision JSON Output

Source: https://docs.softwareheritage.org/_sources/devel/getting-started/api

This is an example JSON output representing a Git revision retrieved from the Software Heritage API. It includes details such as author, committer, commit date, directory information, commit message, and parent revisions.

```json
{
    "author": {
        "email": "boris.baldassari@gmail.com",
        "fullname": "Boris Baldassari <boris.baldassari@gmail.com>",
        "name": "Boris Baldassari"
    },
    "committer": {
        "email": "boris.baldassari@gmail.com",
        "fullname": "Boris Baldassari <boris.baldassari@gmail.com>",
        "name": "Boris Baldassari"
    },
    "committer_date": "2020-11-01T12:55:13+01:00",
    "date": "2020-11-01T12:55:13+01:00",
    "directory": "fd9fe3477db3b9b7dea63509832b3fa99bdd7eb8",
    "directory_url": "https://archive.softwareheritage.org/api/1/directory/fd9fe3477db3b9b7dea63509832b3fa99bdd7eb8/",
    "extra_headers": [],
    "history_url": "https://archive.softwareheritage.org/api/1/revision/6dd0504b43b4459d52e9f13f71a91cc0fc445a19/log/",
    "id": "6dd0504b43b4459d52e9f13f71a91cc0fc445a19",
    "merge": false,
    "message": "#163 Fix dygraphs zero padding in forums plugin.\n",
    "metadata": {},
    "parents": [
        {
            "id": "a4a2d8925c1cc43612602ac28e4ca9a31728b151",
            "url": "https://archive.softwareheritage.org/api/1/revision/a4a2d8925c1cc43612602ac28e4ca9a31728b151/"
        }
    ],
    "synthetic": false,
    "type": "git",
    "url": "https://archive.softwareheritage.org/api/1/revision/6dd0504b43b4459d52e9f13f71a91cc0fc445a19/"
}
```

--------------------------------

### Execute Unit Tests, Linting, and Analysis with Tox (Python)

Source: https://docs.softwareheritage.org/devel/developer-setup

This snippet demonstrates running unit tests along with code linting and static analysis using tox. Tox creates isolated environments for each test configuration. This command executes tests against dependencies installed from PyPI for most packages, while the current package is installed from its setup.py. The output includes environment setup, dependency installation, and test execution results, including coverage reports.

```bash
(swh) ~/swh-environment/swh-loader-git$ tox
GLOB sdist-make: ~/swh-environment/swh-loader-git/setup.py
flake8 create: ~/swh-environment/swh-loader-git/.tox/flake8
flake8 installdeps: flake8
flake8 installed: entrypoints==0.3,flake8==3.7.7,mccabe==0.6.1,pycodestyle==2.5.0,pyflakes==2.1.1,swh.loader.git==0.0.48.post3
flake8 run-test-pre: PYTHONHASHSEED='2028963506'
flake8 runtests: commands[0] | ~/swh-environment/swh-loader-git/.tox/flake8/bin/python -m flake8
py3 create: ~/swh-environment/swh-loader-git/.tox/py3
py3 installdeps: .[testing], pytest-cov
py3 inst: ~/swh-environment/swh-loader-git/.tox/.tmp/package/1/swh.loader.git-0.0.48.post3.zip
py3 installed: aiohttp==3.5.4,amqp==2.4.2,arrow==0.13.1,async-timeout==3.0.1,atomicwrites==1.3.0,attrs==19.1.0,billiard==3.5.0.5,celery==4.2.1,certifi==2018.11.29,chardet==3.0.4,Click==7.0,coverage==4.5.2,decorator==4.3.2,dulwich==0.19.11,elasticsearch==6.3.1,Flask==1.0.2,idna==2.8,idna-ssl==1.1.0,itsdangerous==1.1.0,Jinja2==2.10,kombu==4.4.0,MarkupSafe==1.1.1,more-itertools==6.0.0,msgpack-python==0.5.6,multidict==4.5.2,pathlib2==2.3.3,pluggy==0.9.0,psutil==5.6.0,psycopg2==2.7.7,py==1.8.0,pytest==3.10.1,pytest-cov==2.6.1,python-dateutil==2.8.0,pytz==2018.9,PyYAML==3.13,requests==2.21.0,retrying==1.3.3,six==1.12.0,swh.core==0.0.55,swh.loader.core==0.0.39,swh.loader.git==0.0.48.post3,swh.model==0.0.30,swh.objstorage==0.0.30,swh.scheduler==0.0.49,swh.storage==0.0.129,systemd-python==234,typing-extensions==3.7.2,urllib3==1.24.1,vcversioner==2.16.0.0,vine==1.2.0,Werkzeug==0.14.1,yarl==1.3.0
py3 run-test-pre: PYTHONHASHSEED='2028963506'
py3 runtests: commands[0] | pytest --cov=swh --cov-branch
=========================== test session starts ============================
platform linux -- Python 3.5.3, pytest-3.10.1, py-1.8.0, pluggy-0.9.0
rootdir: ~/swh-environment/swh-loader-git, inifile: pytest.ini
plugins: cov-2.6.1, celery-4.2.1
collected 25 items

swh/loader/git/tests/test_converters.py ........                     [ 32%]
swh/loader/git/tests/test_from_disk.py .....                         [ 52%]
swh/loader/git/tests/test_loader.py ......                           [ 76%]
swh/loader/git/tests/test_tasks.py ...                               [88%]
swh/loader/git/tests/test_utils.py ...                               [100%]

----------- coverage: platform linux, python 3.5.3-final-0 -----------
Name                                      Stmts   Miss Branch BrPart  Cover
---------------------------------------------------------------------------
swh/__init__.py                               1      0      0      0   100%
swh/loader/__init__.py                        1      0      0      0   100%
swh/loader/git/__init__.py                    0      0      0      0   100%
swh/loader/git/converters.py                102     10     44      7    86%
swh/loader/git/from_disk.py                 157     44     50      6    67%
swh/loader/git/loader.py                    271     59    114     17    75%
swh/loader/git/tasks.py                      14      0      0      0   100%

```

--------------------------------

### Install Puppet Agent and Run Configuration

Source: https://docs.softwareheritage.org/sysadm/server-architecture/howto-install-new-physical-server

Installs necessary packages (tmux, puppet, gnupg) and then executes the Puppet agent to configure the newly installed server. It points the agent to the 'pergamon' server for its catalog.

```bash
sudo apt install tmux puppet gnupg
sudo puppet agent --vardir /var/lib/puppet \
  --server pergamon.internal.softwareheritage.org -t
```

--------------------------------

### Disable Cassandra and PostgreSQL Services

Source: https://docs.softwareheritage.org/_sources/devel/developer-setup

Disables the Cassandra and PostgreSQL system services from starting automatically on boot and stops them if currently running. This is recommended to save system resources.

```console
sudo systemctl disable --now cassandra postgresql
```

--------------------------------

### Enable Cassandra Service After Configuration (Bash)

Source: https://docs.softwareheritage.org/sysadm/data-silos/cassandra/installation

This command enables the Cassandra service for a specific instance, ensuring it starts automatically on system boot. This should be run after all configurations, including ZFS setup, are complete.

```bash
# systemctl enable cassandra@instance1

```

--------------------------------

### Clone swh-environment repository

Source: https://docs.softwareheritage.org/_sources/devel/developer-setup

Clones the swh-environment Git repository from GitLab into the current directory. This is the first step in setting up the Software Heritage development environment.

```bash
~$ git clone https://gitlab.softwareheritage.org/swh/devel/swh-environment.git
[...]
~$ cd swh-environment
~/swh-environment$
```

--------------------------------

### Build ISO image and installation scripts

Source: https://docs.softwareheritage.org/_sources/sysadm/server-architecture/howto-install-new-physical-server

Builds the ISO image, Debian installer file, and finish_install script using the provided build_iso.sh script. Requires the HOSTNAME environment variable to be set.

```shell
HOSTNAME=cassandra07
./configs/build_iso.sh $HOSTNAME
```

--------------------------------

### Checkout and Sync Puppet Repository

Source: https://docs.softwareheritage.org/_sources/sysadm/puppet/tutorial-my-first-change-in-puppet

This snippet shows how to clone the swh-site repository, checkout the production branch, and merge it with its origin. It also demonstrates syncing the staging branch with its origin, preparing the environment for development.

```bash
you@localhost$ cd swh-site && git pull
you@localhost$ git checkout production && git merge origin/production
# both staging and production should be in sync
you@localhost$ git checkout staging && git merge origin/staging
# you can now start hacking
you@localhost$ # *hack on puppet Git repo*
you@localhost$ rake validate
you@localhost$ git commit
```

--------------------------------

### Example GET Request for Content

Source: https://docs.softwareheritage.org/devel/swh-deposit/endpoints/content

This is an example of an HTTP GET request to the /content/ endpoint to retrieve information about a deposit. It includes necessary headers like Authorization and Accept.

```http
GET /deposit/1/test/1/content/ HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Authorization: Basic xxxxxxxxxx
Connection: keep-alive
Host: deposit.softwareheritage.org


```