### Quick Start Commands Source: https://github.com/siriusscan/sirius/blob/main/documentation/dev/templates/TEMPLATE.guide.md Essential bash commands to quickly get started with the guide's core functionality. These commands are designed for immediate execution to set up or initiate the process. ```bash # Essential commands to get started [Provide working examples] ``` -------------------------------- ### Starting Standard Development Mode (Bash) Source: https://github.com/siriusscan/sirius/blob/main/documentation/dev/README.development.md Initiates the standard development mode, ideal for UI/API development and general testing. This mode utilizes built-in repositories from Docker images and automatically starts all services in development mode with live reloading for UI changes. No local repository setup is required. ```bash ./scripts/dev-setup.sh start ``` -------------------------------- ### Clone Repository and Start Development Environment Source: https://github.com/siriusscan/sirius/blob/main/documentation/dev/README.developer-guide.md This snippet demonstrates the initial steps to clone the Sirius repository and set up the development environment using a provided script. It assumes the user has git installed and navigates into the cloned directory. ```bash # Clone the repository git clone https://github.com/SiriusScan/Sirius.git cd Sirius # Start development environment ./scripts/switch-env.sh dev # Access the application open http://localhost:3000 ``` -------------------------------- ### Troubleshooting Service Startup Issues (Bash) Source: https://github.com/siriusscan/sirius/blob/main/documentation/dev/README.development.md Guides users through diagnosing and resolving problems with services not starting correctly. It includes commands to check container status, view logs for specific services, and perform a clean restart of all services. ```bash # Check container status ./scripts/dev-setup.sh status # View logs for specific service ./scripts/dev-setup.sh logs sirius-engine # Restart clean ./scripts/dev-setup.sh stop ./scripts/dev-setup.sh clean ./scripts/dev-setup.sh start ``` -------------------------------- ### Standard Setup with Docker Compose Source: https://github.com/siriusscan/sirius/blob/main/README.md This bash script demonstrates the standard setup procedure for the Sirius project. It involves cloning the repository, navigating into the project directory, and starting the services using Docker Compose. This setup provides complete scanning capabilities out-of-the-box with pre-configured databases and requires no additional setup. ```bash git clone https://github.com/SiriusScan/Sirius.git cd Sirius docker compose up -d ``` -------------------------------- ### Example GET Endpoint Request (Bash) Source: https://github.com/siriusscan/sirius/blob/main/documentation/dev/templates/TEMPLATE.api.md Illustrates a GET request to a specific API endpoint, including query parameters and authentication headers. This example shows how to retrieve data from the API. ```bash curl -X GET "http://localhost:9001/api/v1/endpoint1?param1=value¶m2=123" \ -H "Authorization: Bearer " ``` -------------------------------- ### UI Pattern Development Startup Script Source: https://github.com/siriusscan/sirius/blob/main/documentation/dev/architecture/README.system-monitor.md This bash script is designed for starting the system monitor in a development environment for the UI pattern. It checks for the /system-monitor directory and the main.go file, then executes 'go run main.go'. It logs the start and the process ID. ```bash # In start-dev.sh if [ -d "/system-monitor" ] && [ -f "/system-monitor/main.go" ]; then echo "📊 Starting System Monitor..." cd /system-monitor CONTAINER_NAME=sirius-ui go run main.go & SYSTEM_MONITOR_PID=$! echo "✅ System Monitor started with PID: $SYSTEM_MONITOR_PID" cd /app fi ``` -------------------------------- ### Setting up Extended Development Mode (Bash) Source: https://github.com/siriusscan/sirius/blob/main/documentation/dev/README.development.md Configures and starts the extended development mode, suitable for working on scanner, terminal, or agent code. This involves cloning external repositories, initializing local overrides, and starting services with local mounts for hot reloading. Changes in these local repositories are reflected without rebuilding containers. ```bash # 1. Set up local repositories (one-time) mkdir -p ../minor-projects && cd ../minor-projects git clone https://github.com/SiriusScan/app-scanner.git git clone https://github.com/SiriusScan/app-terminal.git git clone https://github.com/SiriusScan/app-agent.git cd ../Sirius # 2. Initialize local overrides ./scripts/dev-setup.sh init # 3. Edit docker-compose.local.yaml (uncomment what you need) nano docker-compose.local.yaml # 4. Start extended development ./scripts/dev-setup.sh start-extended ``` -------------------------------- ### Example PostgreSQL Connection String (Go) Source: https://github.com/siriusscan/sirius/blob/main/documentation/dev/architecture/README.architecture.md Illustrates a typical Data Source Name (DSN) string used in Go applications to connect to a PostgreSQL database. This example shows how to specify the host, user, password, database name, and port. ```go // dsn := "host=sirius-postgres user=postgres password=postgres dbname=sirius port=5432 sslmode=disable" ``` -------------------------------- ### Changelog Entry with Migration Guide (Markdown) Source: https://github.com/siriusscan/sirius/blob/main/documentation/dev/operations/README.sdk-releases.md An example of a CHANGELOG.md entry detailing a critical fix that involves a breaking change. It includes a clear 'Upgrading' section with Go code examples showing the 'before' and 'after' states of the affected code. ```markdown ## [0.0.11] - 2025-10-26 ### Fixed #### 🔴 CRITICAL: Port.ID Schema Conflict - **BREAKING CHANGE:** Renamed `Port.ID` field to `Port.Number` - `Port.ID int` → `Port.Number int` - Resolves duplicate key violations ### Upgrading **Before:** ```go port := sirius.Port{ID: 22, Protocol: "tcp"} ``` **After:** ```go port := sirius.Port{Number: 22, Protocol: "tcp"} ``` ``` -------------------------------- ### Quick Start: Clone and Start Sirius with Docker Compose Source: https://github.com/siriusscan/sirius/blob/main/README.md This snippet demonstrates the basic steps to clone the Sirius repository and start the application using Docker Compose. It assumes you have Docker and Docker Compose installed. The command `docker compose up -d` starts the services in detached mode. ```bash git clone https://github.com/SiriusScan/Sirius.git cd Sirius docker compose up -d # Access the web interface open http://localhost:3000 ``` -------------------------------- ### Sirius Engine Dockerfile Setup Source: https://github.com/siriusscan/sirius/blob/main/documentation/dev/architecture/README.architecture.md Configures the Docker environment for the sirius-engine. It starts from a Go base image, installs system dependencies (libpcap-dev, libicu-dev), Nmap, RustScan from source, and PowerShell for different architectures. It also sets up directories for Nmap scripts and clones sub-application repositories. ```dockerfile # sirius-engine Dockerfile FROM golang:latest WORKDIR /engine COPY .air.toml .air.toml # Air configuration for hot-reloading (likely for sub-apps) RUN apt-get update -y RUN apt-get upgrade -y RUN apt-get install -y libpcap-dev libicu-dev # System dependencies # NMAP Installation RUN apt-get install -y nmap # Rust Scan Installation WORKDIR /tmp RUN curl https://sh.rustup.rs -sSf | bash -s -- -y ENV PATH="/root/.cargo/bin:${PATH}" RUN cargo install --git https://github.com/RustScan/RustScan.git --branch master # PowerShell Installation WORKDIR /opt/microsoft/powershell RUN ARCH=$(uname -m) && \ case "$ARCH" in \ "aarch64") \ echo "Installing ARM64 version" && \ wget https://github.com/PowerShell/PowerShell/releases/download/v7.5.0/powershell-7.5.0-linux-arm64.tar.gz && \ tar -xvf powershell-7.5.0-linux-arm64.tar.gz \ ;; \ "x86_64") \ echo "Installing AMD64 version" && \ wget https://github.com/PowerShell/PowerShell/releases/download/v7.5.0/powershell-7.5.0-linux-x64.tar.gz && \ tar -xvf powershell-7.5.0-linux-x64.tar.gz \ ;; \ *) \ echo "Unsupported architecture: $ARCH" && exit 1 \ ;; \ esac RUN chmod +x /opt/microsoft/powershell/pwsh RUN ln -s /opt/microsoft/powershell/pwsh /usr/bin/pwsh # Set up NSE directory structure (for Nmap Scripting Engine used by app-scanner) RUN mkdir -p /opt/sirius/nse && \ chown -R 1000:1000 /opt/sirius && \ chmod -R 755 /opt/sirius # App Repos - Cloning sub-application source code WORKDIR / RUN git clone https://github.com/SiriusScan/app-scanner.git RUN git clone https://github.com/SiriusScan/app-terminal.git RUN git clone https://github.com/SiriusScan/go-api.git # go-api likely contains shared code/models WORKDIR /app-scanner # Setting WORKDIR for app-scanner specific setup #DEV Dependencies for app-scanner (Air) RUN go install github.com/air-verse/air@latest ``` -------------------------------- ### Quick Start Deployment with Docker Compose Source: https://github.com/siriusscan/sirius/blob/main/documentation/dev/deployment/README.docker-container-deployment.md This snippet shows how to clone the Sirius repository and deploy it using Docker Compose with prebuilt images. It also demonstrates how to specify a particular image tag for deployment. ```bash # Clone repository git clone https://github.com/SiriusScan/Sirius.git cd Sirius # Deploy with prebuilt images (default) docker compose up -d # Or specify a version tag IMAGE_TAG=v0.4.1 docker compose up -d ``` -------------------------------- ### UI Pattern Production Startup Script Source: https://github.com/siriusscan/sirius/blob/main/documentation/dev/architecture/README.system-monitor.md This bash script starts the system monitor in a production environment for the UI pattern. It verifies the existence and executability of the compiled binary '/system-monitor/system-monitor' and then runs it. It logs the start and the process ID. ```bash # In start-prod.sh if [ -f "/system-monitor/system-monitor" ] && [ -x "/system-monitor/system-monitor" ]; then echo "📊 Starting system monitor..." cd /system-monitor CONTAINER_NAME=sirius-ui ./system-monitor & SYSTEM_MONITOR_PID=$! echo "✅ System monitor started with PID: $SYSTEM_MONITOR_PID" cd /app fi ``` -------------------------------- ### Installation Option: User-Focused Setup with Docker Compose Source: https://github.com/siriusscan/sirius/blob/main/README.md This installation option uses a simplified Docker Compose configuration (`docker-compose.user.yaml`) for a cleaner experience, excluding development tooling. It's recommended for users who only need the scanning environment without development dependencies. ```bash git clone https://github.com/SiriusScan/Sirius.git cd Sirius docker compose -f docker-compose.user.yaml up -d ``` -------------------------------- ### Troubleshooting Volume Mounts (Bash) Source: https://github.com/siriusscan/sirius/blob/main/documentation/dev/README.development.md Provides commands to diagnose issues with volume mounts not working in the extended development mode. It includes checks for the existence of the local override file and the cloned minor projects, and verifies that the correct start command is being used. ```bash # Check if local file exists ls -la docker-compose.local.yaml # Check if repositories exist ls -la ../minor-projects/ # Verify you're using start-extended ./scripts/dev-setup.sh start-extended ``` -------------------------------- ### Sirius Development File Structure Overview Source: https://github.com/siriusscan/sirius/blob/main/documentation/dev/README.development.md Illustrates the key files and directories within the Sirius project relevant to development setup. It highlights the roles of different docker-compose files, including base configuration, overrides, and local, user-specific configurations, as well as the main development script. ```plaintext Sirius/ ├── docker-compose.yaml # Base configuration ├── docker-compose.override.yaml # 🔒 Committed: Safe development defaults ├── docker-compose.local.example.yaml # 🔒 Committed: Template for local overrides ├── docker-compose.local.yaml # 🚫 Git-ignored: Your personal overrides └── scripts/dev-setup.sh # 🔒 Committed: Development helper ``` -------------------------------- ### YAML Test Workflow Example 1 Source: https://github.com/siriusscan/sirius/blob/main/documentation/dev/ai-rules/README.playwright.md An example YAML workflow for testing a scanner page, demonstrating navigation, login, adding targets, starting scans, and verifying network activity. ```yaml Step 1: Navigate - URL: http://host.docker.internal:3000/scanner - Wait for: Page load Step 2: Login (if needed) - Fill username: admin - Fill password: password - Click: "Join the Pack" - Wait for: Dashboard redirect Step 3: Navigate to Scanner - Click: Scanner link - Wait for: Page load Step 4: Add Target - Type in IP field: 192.168.1.100 - Click: Add button - Verify: Target appears in list Step 5: Start Scan - Select template: "High Risk Scan" - Click: "Start Scan" - Check network: queue.sendMsg called - Check network: store.setValue called - Verify: UI updates with scan details ``` -------------------------------- ### API Endpoint Examples (cURL) Source: https://github.com/siriusscan/sirius/blob/main/documentation/dev/templates/TEMPLATE.reference.md Provides examples of how to interact with API endpoints using cURL. It covers both GET requests with query parameters and POST requests with JSON bodies. ```bash curl -X GET "http://localhost:9001/api/v1/endpoint?param1=value" ``` ```bash curl -X POST "http://localhost:9001/api/v1/endpoint2" \ -H "Content-Type: application/json" \ -d '{"field1": "value", "field2": "value"}' ``` -------------------------------- ### Common Sirius Development Commands (Bash) Source: https://github.com/siriusscan/sirius/blob/main/documentation/dev/README.development.md Lists essential commands for managing the Sirius development environment using the `dev-setup.sh` script. These commands cover initialization, starting/stopping services in different modes, checking status, viewing logs, accessing container shells, and cleaning up resources. ```bash ./scripts/dev-setup.sh init # Create local overrides from template ./scripts/dev-setup.sh start # Standard development mode ./scripts/dev-setup.sh start-extended # Extended development with local repos ./scripts/dev-setup.sh stop # Stop all services ./scripts/dev-setup.sh status # Show container status ./scripts/dev-setup.sh logs [service] # Show logs ./scripts/dev-setup.sh shell # Open shell in container ./scripts/dev-setup.sh clean # Clean containers and volumes ``` -------------------------------- ### Development Mode Startup Logic Source: https://github.com/siriusscan/sirius/blob/main/documentation/dev/architecture/README.system-monitor.md This bash script checks the GO_ENV environment variable and the presence of main.go or the compiled binary to determine how to start the system monitor. It prioritizes 'go run' in development mode and falls back to the binary for production. ```bash if [ "$GO_ENV" = "development" ] && [ -f "/system-monitor/main.go" ]; then CONTAINER_NAME=sirius-engine go run main.go & elif [ -f "/system-monitor/system-monitor" ] && [ -x "/system-monitor/system-monitor" ]; then CONTAINER_NAME=sirius-engine ./system-monitor & fi ``` -------------------------------- ### Start Sirius Full Stack with Docker Compose Source: https://github.com/siriusscan/sirius/blob/main/documentation/dev/architecture/README.architecture.md Builds and starts all services defined in the docker-compose.yaml file. Use -d for detached mode and --build to force image rebuilds. This command is essential for running the entire application locally. ```bash docker compose up docker compose up -d docker compose up --build ``` -------------------------------- ### Consume tRPC Scan Start Mutation in React (TypeScript/JSX) Source: https://github.com/siriusscan/sirius/blob/main/documentation/dev/architecture/README.architecture.md This example demonstrates how a React component in the sirius-ui frontend consumes the `startScan` tRPC mutation defined on the backend. It uses the `useMutation` hook from the tRPC client to trigger the mutation, providing input data and defining callbacks for success and error scenarios. The component renders a button that initiates the scan and shows a loading state. ```tsx import { api } from "~/utils/api"; // Path to the tRPC client function ScanComponent() { const startScanMutation = api.scan.startScan.useMutation({ onSuccess: (data) => { console.log("Scan submitted! ID:", data.scan_id); // navigate to results page or show notification }, onError: (error) => { console.error("Failed to start scan:", error.message); }, }); const handleStartScan = () => { startScanMutation.mutate({ targets: ["127.0.0.1"], scanProfile: "quick_scan", }); }; return ( ); } ``` -------------------------------- ### Production Startup Script for System Monitor Source: https://github.com/siriusscan/sirius/blob/main/documentation/dev/architecture/README.system-monitor.md This bash script is used to start the system monitor in a production environment. It assumes the compiled binary is available in /usr/local/bin and executes it. ```bash # In start-with-monitor.sh echo "📊 Starting system monitor..." cd /usr/local/bin CONTAINER_NAME=sirius-postgres ./system-monitor & ``` -------------------------------- ### Using go-api functions in sirius-engine/app-scanner Source: https://github.com/siriusscan/sirius/blob/main/documentation/dev/architecture/README.architecture.md Conceptual example of how sirius-engine or its sub-modules like app-scanner would call go-api functions to interact with the database after a scan. ```go // Assuming 'go-api' is accessible as a local package import ( "github.com/SiriusScan/go-api/host" "github.com/SiriusScan/go-api/vulnerability" "github.com/SiriusScan/go-api/port" "github.com/SiriusScan/go-api/service" ) func processScanResults(scanData interface{}) { // Update host details host.AddHost(newHostInfo) // Add discovered ports and services port.AddPort(portInfo) service.AddService(serviceInfo) // Store vulnerabilities found vulnerability.AddVulnerability(vulnInfo) } ``` -------------------------------- ### Valkey Key-Value Store Operations in Go Source: https://github.com/siriusscan/sirius/blob/main/documentation/dev/architecture/README.architecture.md Provides a standardized interface for interacting with a Valkey cache. It defines operations for setting, getting, and closing connections to the key-value store. ```Go package store import ( "context" "github.com/redis/go-redis/v9" ) // KVStore interface defines a contract for key-value operations. type KVStore interface { SetValue(ctx context.Context, key, value string) error GetValue(ctx context.Context, key string) (string, error) Close() error } // redisKVStore implements the KVStore interface using go-redis client. type redisKVStore struct { client *redis.Client } // NewRedisKVStore creates a new instance of redisKVStore. func NewRedisKVStore(addr string) KVStore { client := redis.NewClient(&redis.Options{ Addr: addr, // e.g., "sirius-valkey:6379" }) return &redisKVStore{client: client} } // SetValue stores a key-value pair in the Valkey store. func (s *redisKVStore) SetValue(ctx context.Context, key, value string) error { return s.client.Set(ctx, key, value, 0).Err() } // GetValue retrieves a value from the Valkey store by its key. func (s *redisKVStore) GetValue(ctx context.Context, key string) (string, error) { return s.client.Get(ctx, key).Result() } // Close closes the connection to the Valkey store. func (s *redisKVStore) Close() error { return s.client.Close() } ``` -------------------------------- ### RESTful API Endpoint for Hosts in Go (Fiber) Source: https://github.com/siriusscan/sirius/blob/main/documentation/dev/architecture/README.architecture.md This Go code snippet demonstrates how to set up RESTful API endpoints for managing hosts using the Fiber web framework. It includes examples for listing all hosts and retrieving a specific host by its ID. The implementation relies on the `go-api` SDK for database interactions, though actual database calls are commented out for brevity. Error handling and request parsing are also indicated. ```go package routes import ( "github.com/gofiber/fiber/v2" "github.com/SiriusScan/Sirius/go-api/sirius/host" // Assuming go-api is accessible // ... other imports ) // SetupHostRoutes configures routes related to hosts func SetupHostRoutes(app *fiber.App) { api := app.Group("/api/v1") // API versioning hostRoutes := api.Group("/hosts") hostRoutes.Get("/", func(c *fiber.Ctx) error { // Logic to fetch all hosts using go-api's host.GetAllHosts() // Example: hosts, err := host.GetAllHosts( /* db_instance_from_go_api */ ) // if err != nil { return c.Status(fiber.StatusInternalServerError).JSON(...) } // return c.JSON(hosts) return c.SendString("GET /api/v1/hosts - List all hosts") }) hostRoutes.Get("/:hostId", func(c *fiber.Ctx) error { hostId := c.Params("hostId") // Logic to fetch a specific host by ID using go-api's host.GetHost(hostId) // Example: h, err := host.GetHost(hostId, /* db_instance */) // if err != nil { /* handle error, e.g., 404 if not found */ } // return c.JSON(h) return c.SendString("GET /api/v1/hosts/" + hostId) }) hostRoutes.Post("/", func(c *fiber.Ctx) error { // var newHostPayload struct { /* ... fields ... */ } // if err := c.BodyParser(&newHostPayload); err != nil { return c.Status(fiber.StatusBadRequest).JSON(...) } // Logic to add a new host using go-api's host.AddHost(...) return c.SendString("POST /api/v1/hosts - Create a new host") }) } ``` -------------------------------- ### gRPC Service Definition (Proto) Source: https://github.com/siriusscan/sirius/blob/main/documentation/dev/architecture/README.architecture.md This conceptual example illustrates a Protocol Buffers (.proto) file, which defines the service interfaces and message structures for gRPC communication. It is used to ensure strong typing and efficient serialization between the sirius-engine and app-agents. ```Proto // Conceptual .proto definition for sirius-engine and app-agent communication // Syntax for Protocol Buffers version 3 syntax = "proto3"; // Package declaration to prevent naming conflicts package sirius; // Service definition for engine-to-agent communication service AgentService { // Unary RPC for sending a command to an agent rpc ExecuteCommand (CommandRequest) returns (CommandResponse); // Unary RPC for health checks rpc Ping (PingRequest) returns (PingResponse); // Add other service methods as needed, e.g., for registration, data streaming } // Message structure for command execution requests message CommandRequest { string agent_id = 1; // Identifier for the target agent string command = 2; // The command to execute on the agent } // Message structure for command execution responses message CommandResponse { string output = 1; // The output of the executed command int32 exit_code = 2; // The exit code of the command string error = 3; // Any error message encountered } // Message structure for ping requests message PingRequest { string agent_id = 1; // Identifier for the target agent } // Message structure for ping responses message PingResponse { bool is_alive = 1; // Indicates if the agent is responsive } // Add other message types for agent registration, data streaming, etc. ``` -------------------------------- ### Set up sirius-api Development Environment with Air Source: https://github.com/siriusscan/sirius/blob/main/documentation/dev/architecture/README.architecture.md The Dockerfile for sirius-api configures the Go environment, installs the 'air' tool for live reloading, and sets the command to run the application using 'air'. It exposes port 9001 for the API. ```dockerfile # sirius-api Dockerfile FROM golang:latest WORKDIR /api COPY .air.toml .air.toml # Dependencies RUN apt-get update WORKDIR /api #DEV Dependencies RUN go install github.com/air-verse/air@latest # Invoke air to run the server after volume mount CMD ["air", "-c", ".air.toml"] EXPOSE 9001 ``` -------------------------------- ### Dockerfile for sirius-ui Source: https://github.com/siriusscan/sirius/blob/main/documentation/dev/architecture/README.architecture.md This Dockerfile defines the build process for the sirius-ui application. It uses a multi-stage build with a Node.js Alpine base image, installs dependencies using npm, copies application code, generates the Prisma client, and sets up the development environment. ```dockerfile # sirius-ui Dockerfile - Multi-stage build # Base stage with common dependencies FROM node:18-alpine AS base # Install system dependencies RUN apk add --no-cache libc6-compat openssl ca-certificates WORKDIR /app # Copy package files and install dependencies COPY package*.json ./ RUN npm install # Copy source code and build configuration COPY . . # Generate Prisma client RUN npx prisma generate # Development stage FROM base AS development ENV NODE_ENV=development ENV NEXT_TELEMETRY_DISABLED=1 EXPOSE 3000 CMD ["/app/start-dev.sh"] ``` -------------------------------- ### Local Development Setup for sirius-engine Sub-Applications Source: https://github.com/siriusscan/sirius/blob/main/documentation/dev/architecture/README.architecture.md Illustrates the process of setting up local development for sub-applications within sirius-engine, such as app-scanner. This involves cloning the sub-application's repository locally and uncommenting volume mounts in the docker-compose.yaml file to link the local code to the container. This enables direct modification and hot reloading of sub-application code. ```yaml # Example docker-compose.yaml modification for sirius-engine # services: # sirius-engine: # # ... other configurations ... # volumes: # # Uncomment and adjust path for local development # - ../minor-projects/app-scanner:/app-scanner ``` -------------------------------- ### Quick Start Commands Source: https://github.com/siriusscan/sirius/blob/main/documentation/dev/templates/TEMPLATE.documentation-standard.md Essential commands for a quick start with the system. These commands provide a basic entry point for users to begin interacting with the documented functionality. ```bash # Essential commands or steps [Provide working examples] ``` -------------------------------- ### Migrate Docker Compose Overrides to Local Configuration (Bash) Source: https://github.com/siriusscan/sirius/blob/main/documentation/dev/README.development.md This script guides users through migrating from the older `docker-compose.override.yaml` to the new `docker-compose.local.yaml` system. It involves resetting the old file, initializing the new local override setup, manually transferring customizations, and starting the system. ```bash # 1. Reset override file to clean state git checkout docker-compose.override.yaml # 2. Set up new local overrides ./scripts/dev-setup.sh init # 3. Move your customizations to docker-compose.local.yaml nano docker-compose.local.yaml # 4. Start with new system ./scripts/dev-setup.sh start-extended ``` -------------------------------- ### Testing Checklist: Pre-Test Setup Source: https://github.com/siriusscan/sirius/blob/main/documentation/dev/ai-rules/README.playwright.md A checklist for pre-test setup, ensuring that Docker containers are running, services are healthy, and the UI is accessible. ```markdown - [ ] Ensure Docker containers are running - [ ] Verify services are healthy - [ ] Check that UI is accessible at host.docker.internal:3000 ``` -------------------------------- ### Add API Endpoint with Fiber in sirius-api Source: https://github.com/siriusscan/sirius/blob/main/documentation/dev/architecture/README.architecture.md Demonstrates how to define a new GET route and its corresponding handler function within the sirius-api service using the Fiber framework. This involves importing Fiber, defining the route with a path and handler, and implementing the handler logic to process requests and return JSON responses. Assumes an existing Fiber app instance and a handlers package. ```go // Example in sirius-api/main.go or a routes setup function // import "github.com/gofiber/fiber/v2" // import "your-project/sirius-api/handlers" // app is a *fiber.App instance // app.Get("/api/v1/new-endpoint", handlers.HandleNewEndpoint) ``` ```go // Example in sirius-api/handlers/new_handler.go package handlers import "github.com/gofiber/fiber/v2" func HandleNewEndpoint(c *fiber.Ctx) error { // Your logic here: parse request, interact with database (via go-api package), // communicate with sirius-engine via RabbitMQ, etc. return c.JSON(fiber.Map{"status": "success", "message": "New endpoint reached"}) } ``` -------------------------------- ### Cloning go-api in sirius-engine Dockerfile Source: https://github.com/siriusscan/sirius/blob/main/documentation/dev/architecture/README.architecture.md Illustrates the Dockerfile instruction for sirius-engine to clone the go-api repository directly into its build context, ensuring a bundled version. ```dockerfile FROM golang:1.20 AS builder WORKDIR /app # Clone go-api into the engine's context RUN git clone https://github.com/SiriusScan/go-api.git /go-api # Clone other necessary modules like app-scanner RUN git clone https://github.com/SiriusScan/app-scanner.git /app-scanner # Copy application source code COPY . . # Build the sirius-engine application RUN go build -o /sirius-engine ./cmd/sirius-engine FROM alpine:latest WORKDIR /app # Copy the built binary from the builder stage COPY --from=builder /sirius-engine . # Expose port if necessary EXPOSE 8080 CMD ["./sirius-engine"] ``` -------------------------------- ### Troubleshoot Instance Not Starting - Get Console Output Source: https://github.com/siriusscan/sirius/blob/main/documentation/dev/operations/README.terraform-deployment.md Retrieves the system console output for a specific EC2 instance using its ID. This output can contain critical boot messages and error information, aiding in diagnosing why an instance might not be starting correctly. ```bash # View system logs aws ec2 get-console-output --instance-id $(terraform output -raw instance_id) ``` -------------------------------- ### Project Initialization Steps (Bash) Source: https://github.com/siriusscan/sirius/blob/main/documentation/dev/operations/README.new-project.md This script outlines the initial steps for setting up a new project or sprint. It includes creating a feature branch, initializing a task file, creating a project plan document, and making the first commit with these new files. ```bash # 1. Create feature branch git checkout main git pull origin main git checkout -b feature/{sprint-name} # 2. Create task file touch tasks/{sprint-name}.json # [Fill in task structure using template] # 3. Create project plan touch documentation/dev-notes/{sprint-name}-plan.md # [Fill in project plan using template] # 4. Initial commit git add tasks/{sprint-name}.json documentation/dev-notes/{sprint-name}-plan.md git commit -m "feat: initialize {sprint-name} project - Add task breakdown in tasks/{sprint-name}.json - Add project plan in documentation/dev-notes/{sprint-name}-plan.md - Create feature branch for development" git push origin feature/{sprint-name} ``` -------------------------------- ### Define Host Creation API Route (Go) Source: https://github.com/siriusscan/sirius/blob/main/documentation/dev/architecture/README.architecture.md This Go code snippet defines a POST route for creating a new host using the Fiber web framework. It returns a success message indicating the endpoint was hit. ```Go return c.SendString("POST /api/v1/hosts - Create new host") }) // ... other routes for PUT, DELETE etc. } ``` -------------------------------- ### Local go-api Development Mount in sirius-api Docker Compose Source: https://github.com/siriusscan/sirius/blob/main/documentation/dev/architecture/README.architecture.md Shows the commented-out volume mount in the docker-compose.yaml for sirius-api, enabling developers to use a local go-api clone for immediate testing. ```yaml services: sirius-api: build: context: . ports: - "8080:8080" volumes: # - ../minor-projects/go-api:/go-api # Uncomment for local go-api development ``` -------------------------------- ### Configure Valkey Key-Value Store Service (Docker Compose) Source: https://github.com/siriusscan/sirius/blob/main/documentation/dev/architecture/README.architecture.md Sets up the Valkey (Redis fork) service for high-performance in-memory storage. This configuration includes the image, restart policy, container name, hostname, and port mapping for standard Valkey/Redis access. ```yaml sirius-valkey: image: valkey/valkey:latest restart: always container_name: sirius-valkey hostname: sirius-valkey ports: - "6379:6379" ``` -------------------------------- ### RabbitMQ Queue Operations in Go Source: https://github.com/siriusscan/sirius/blob/main/documentation/dev/architecture/README.architecture.md Encapsulates interactions with RabbitMQ for sending and receiving messages. It includes functionality for message processing callbacks and automatic queue declaration. ```Go package queue import ( "github.com/rabbitmq/amqp091-go" "context" "log" ) // MessageProcessor defines a type for callback functions that process incoming messages. type MessageProcessor func(msg string) // Listen subscribes to a specified RabbitMQ queue and invokes the messageProcessor for each received message. // Processors are typically run in separate goroutines for concurrent handling. func Listen(connection *amqp091.Connection, qName string, messageProcessor MessageProcessor) error { channel, err := connection.Channel() if err != nil { return err } defer channel.Close() _, err = channel.QueueDeclare( qName, // name true, // durable false, // delete when unused false, // exclusive false, // no-wait nil, // arguments ) if err != nil { return err } msgs, err := channel.Consume( qName, // queue "", // consumer true, // auto-ack false, // exclusive false, // no-local false, // no-wait nil, // args ) if err != nil { return err } go func() { for d := range msgs { messageProcessor(string(d.Body)) } }() return nil } // Send publishes a message to a specified RabbitMQ queue. func Send(connection *amqp091.Connection, qName string, message string) error { channel, err := connection.Channel() if err != nil { return err } defer channel.Close() _, err = channel.QueueDeclare( qName, // name true, // durable false, // delete when unused false, // exclusive false, // no-wait nil, // arguments ) if err != nil { return err } err = channel.Publish( "", // exchange qName, // routing key false, // mandatory false, // immediate amqp091.Publishing{ ContentType: "text/plain", Body: []byte(message), }, ) if err != nil { log.Printf("Failed to publish message: %v", err) return err } log.Printf(" [x] Sent %s to queue %s", message, qName) return nil } ``` -------------------------------- ### Local go-api Development Mount in sirius-engine Docker Compose Source: https://github.com/siriusscan/sirius/blob/main/documentation/dev/architecture/README.architecture.md Shows the commented-out volume mount in the docker-compose.yaml for sirius-engine, allowing local go-api changes to be tested within the engine's environment. ```yaml services: sirius-engine: build: context: . volumes: # - ../minor-projects/go-api:/go-api # Uncomment for local go-api development ``` -------------------------------- ### RabbitMQ Service Configuration in Docker Compose Source: https://github.com/siriusscan/sirius/blob/main/documentation/dev/architecture/README.architecture.md Defines the RabbitMQ message broker service within the Docker Compose setup. It specifies the image, restart policy, container name, ports for AMQP and management UI, volume mounts for configuration, and a healthcheck to ensure service availability. ```yaml sirius-rabbitmq: image: rabbitmq:3.7.3-management restart: always container_name: sirius-rabbitmq hostname: sirius-rabbitmq ports: - "5672:5672" - "15672:15672" volumes: - ./rabbitmq/rabbitmq.conf:/etc/rabbitmq/rabbitmq.conf:ro healthcheck: test: ["CMD", "rabbitmqctl", "status"] interval: 30s timeout: 15s retries: 5 ``` -------------------------------- ### Go Module Dependency Declaration in sirius-api Source: https://github.com/siriusscan/sirius/blob/main/documentation/dev/architecture/README.architecture.md Demonstrates how the go-api SDK is declared as a module dependency in the go.mod file of the sirius-api service. This ensures versioned and stable integration. ```go module github.com/SiriusScan/sirius-api go 1.20 require ( github.com/SiriusScan/go-api v1.2.0 // Example version ) ``` -------------------------------- ### Example Multi-Stage Dockerfile for Production and Development Source: https://github.com/siriusscan/sirius/blob/main/documentation/dev/architecture/README.docker-architecture.md This Dockerfile demonstrates a multi-stage build process, explicitly defining 'production' and 'development' stages. It ensures that the correct stage can be selected during the build process. ```dockerfile # Ensure production stage is explicitly selected FROM node:18-alpine AS production # ... production-specific setup # Development stage should be clearly separated FROM node:18-alpine AS development # ... development-specific setup ``` -------------------------------- ### Define tRPC Scan Start Procedure (TypeScript) Source: https://github.com/siriusscan/sirius/blob/main/documentation/dev/architecture/README.architecture.md This snippet shows how to define a tRPC mutation procedure on the Next.js backend to initiate a scan. It includes input validation using Zod and outlines the logic for processing scan requests, such as constructing a message for a message queue like RabbitMQ. The procedure expects an array of targets and a scan profile as input and returns a confirmation message with a scan ID. ```typescript import { z } from "zod"; import { createTRPCRouter, publicProcedure } from "~/server/api/trpc"; // import { rabbitMQService } from '~/server/services/rabbitmq'; // Example service export const scanRouter = createTRPCRouter({ startScan: publicProcedure .input( z.object({ targets: z.array(z.string()), scanProfile: z.string(), }) ) .mutation(async ({ input, ctx }) => { // ctx might contain db access, session info, etc. console.log("Received scan request:", input); // 1. Validate input further if necessary // 2. Construct a message for sirius-engine const scanJobMessage = { scan_id: `uuid-scan-${Date.now()}`, user_id: ctx.session?.user?.id, // Example: if auth is used targets: input.targets, scan_type: input.scanProfile, // ... other parameters }; // 3. Publish to RabbitMQ (example call) // await rabbitMQService.publishToQueue('scan_jobs_queue', scanJobMessage); // 4. Return a confirmation or initial status return { message: "Scan job submitted successfully", scan_id: scanJobMessage.scan_id, }; }), // ... other procedures like getScanStatus, getScanResults etc. }); ``` -------------------------------- ### Configure .air.toml for Delve Debugging Source: https://github.com/siriusscan/sirius/blob/main/documentation/dev/architecture/README.architecture.md Modifies the .air.toml configuration file to enable Delve for remote debugging. This involves changing the 'run.cmd' to start the application with Delve listening on a specific port, allowing IDEs to attach for step-through debugging. Ensure the Delve port is exposed in docker-compose.yaml. ```toml # .air.toml example modification for delve [build] cmd = "go build -o ./tmp/main ." # ... other build settings [run] # cmd = "./tmp/main" # Original run command cmd = "dlv --listen=:2345 --headless=true --api-version=2 --accept-multiclient exec ./tmp/main" # Delve command # ... other run settings ``` -------------------------------- ### Create a New UI Page in Sirius (Next.js/TypeScript) Source: https://github.com/siriusscan/sirius/blob/main/documentation/dev/architecture/README.architecture.md Demonstrates creating a new page component within the sirius-ui Next.js application. This involves creating a .tsx file in the pages directory and leveraging tRPC for data fetching. ```typescript // sirius-ui/src/pages/new-feature.tsx import React from 'react'; import { api } from '~/utils/api'; // Assuming tRPC setup const NewFeaturePage: React.FC = () => { // Example of using a tRPC query const { data, error, isLoading } = api.yourRouter.yourProcedure.useQuery(); if (isLoading) return
Loading...
; if (error) return
Error: {error.message}
; return (

New Feature Page

{/* Render data here */}

Data: {JSON.stringify(data)}

); }; export default NewFeaturePage; ``` -------------------------------- ### Go Module Replace Directive for Local go-api Development Source: https://github.com/siriusscan/sirius/blob/main/documentation/dev/architecture/README.architecture.md Illustrates how a 'replace' directive in go.mod can be used to point to a local copy of the go-api repository, useful during development for both sirius-api and sirius-engine. ```go module github.com/SiriusScan/sirius-engine go 1.20 // Use replace directive to point to a local copy of go-api replace github.com/SiriusScan/go-api => ../../go-api ``` -------------------------------- ### View Sirius Service Logs with Docker Compose Source: https://github.com/siriusscan/sirius/blob/main/documentation/dev/architecture/README.architecture.md Displays logs for all running services in real-time. You can also specify a service name to view logs for a particular component, such as the sirius-engine. ```bash docker compose logs -f docker compose logs -f sirius-engine ``` -------------------------------- ### gRPC Service Definition (Protocol Buffers) Source: https://github.com/siriusscan/sirius/blob/main/documentation/dev/architecture/README.architecture.md Defines the AgentService with Ping and ExecuteCommand RPC methods, including request and response message structures. This .proto file is used to generate gRPC stubs for both client and server implementations. ```protobuf service AgentService { rpc Ping (PingRequest) returns (PingResponse); rpc ExecuteCommand (CommandRequest) returns (stream CommandResponse); // Example: streaming for output } message CommandRequest { string agent_id = 1; string command = 2; } message CommandResponse { string output = 1; string error = 2; bool completed = 3; } // ... other message definitions ``` -------------------------------- ### PostgreSQL Data Persistence Source: https://github.com/siriusscan/sirius/blob/main/documentation/dev/architecture/README.architecture.md Details the role of PostgreSQL as the primary relational database for storing structured data within Sirius Scan. ```APIDOC ## PostgreSQL (sirius-postgres) ### Description PostgreSQL serves as the main relational database for Sirius Scan, storing core structured data including asset information, vulnerabilities, and scan configurations. ### Role and Data Stored - **Primary Datastore**: Authoritative source for persistent, relational data. - **Hosts Information**: Details about discovered and scanned hosts (IP addresses, hostnames, operating systems, MAC addresses, etc.). - **Ports & Services**: Information about open ports on hosts and the services running on them (port number, protocol, service name, version, state). - **Vulnerabilities**: Detailed records of identified vulnerabilities (CVE IDs, descriptions, severity, affected hosts/ports, remediation status). - **Scan Configurations & Results**: Parameters for scans performed and the summarized results associated with them. - **User and Access Management**: (If implemented) User accounts, roles, and permissions. - **Audit Trails**: Logs of significant actions performed within the system. The `go-api` SDK, particularly its GORM models, defines and interacts with this schema. ```