### fmt.Sprintf for host and port Source: https://github.com/dragonflyoss/dragonfly/blob/main/docs/security/dragonfly-comprehensive-report-2023.pdf An example use of fmt.Sprintf method to concatenate host and port. Instead, use net.JoinHostPort function. ```go func NewDfstore() *DfstoreConfig { url := url.URL{ Scheme: "http", Host: fmt.Sprintf( "%s:%d", "127.0.0.1", DefaultObjectStorageStartPort), } ``` -------------------------------- ### Deprecated ioutil.ReadFile usage Source: https://github.com/dragonflyoss/dragonfly/blob/main/docs/security/dragonfly-comprehensive-report-2023.pdf An example use of deprecated ioutil.ReadFile method. The ioutil package is deprecated; use os.ReadFile instead. ```go data, err := ioutil.ReadFile(file) ``` -------------------------------- ### Run fuzzer for FuzzTestGrpcToB Source: https://github.com/dragonflyoss/dragonfly/blob/main/docs/security/dragonfly-comprehensive-report-2023.pdf Starts the fuzzer for the FuzzTestGrpcToB harness. The fuzzer runs until the first error is detected. The harness must be copied to client/daemon/rpcserver/rpcserver_test.go first. ```bash go test-fuzz ^\QFuzzTestGrpcToB\E$-run ^$ ``` -------------------------------- ### URL handling with strings methods Source: https://github.com/dragonflyoss/dragonfly/blob/main/docs/security/dragonfly-comprehensive-report-2023.pdf Example code handling a URL with strings methods instead of using dedicated APIs like net/url. The code splits a string and constructs a query string, but the approach is error-prone and not recommended. ```go fileds := strings.Split(polished[|0], ","|) host := strings.Split(fileds[|0], "="|)[1] query := strings.Join(fileds[ return fmt.Sprintf(|1 :], "&", host, query)|) "%s?%s" ``` -------------------------------- ### Use of http.RangePrefix constant Source: https://github.com/dragonflyoss/dragonfly/blob/main/docs/security/dragonfly-comprehensive-report-2023.pdf Example use of the http.RangePrefix constant instead of a hard-coded string. The source recommends defining and using constants whenever possible. ```go urlMeta.Range = strings.TrimLeft(r, http.RangePrefix) ``` -------------------------------- ### Deprecated os.IsExist usage Source: https://github.com/dragonflyoss/dragonfly/blob/main/docs/security/dragonfly-comprehensive-report-2023.pdf An example use of deprecated os.Is* methods family. The code checks if an error is os.IsExist, but the os.Is* functions are deprecated; use errors.Is instead. ```go if err := os.MkdirAll(t.dataDir, dataDirMode); err != nil && !os.IsExist(err) { ``` -------------------------------- ### getLayers method called by CreatePreheat Source: https://github.com/dragonflyoss/dragonfly/blob/main/docs/security/dragonfly-comprehensive-report-2023.pdf Called by CreatePreheat to fetch image layers. It starts a tracing span and calls getManifests with the user-supplied URL. ```go func (p *preheat) getLayers (ctx context.Context, url, tag, filter string, header http.Header, image *preheatImage) ([]internaljob.PreheatRequest, error ) { ctx, span := tracer.Start(ctx, config.SpanGetLayers, trace.WithSpanKind(trace.SpanKindProducer)) defer span.End() resp, err := p.getManifests(ctx, url, header) ``` -------------------------------- ### Difference between strings.TrimLeft and strings.TrimPrefix Source: https://github.com/dragonflyoss/dragonfly/blob/main/docs/security/dragonfly-comprehensive-report-2023.pdf Comparison of strings.TrimLeft and strings.TrimPrefix behavior on the same input. TrimLeft removes any characters in the cutset, while TrimPrefix removes the exact prefix. This example illustrates the difference in behavior. ```go strings.TrimLeft("bytes=bbef02", "bytes=") == "f02" ``` ```go strings.TrimPrefix("bytes=bbef02", "bytes=") == "bbef02" ``` -------------------------------- ### getManifests method performing HTTP request Source: https://github.com/dragonflyoss/dragonfly/blob/main/docs/security/dragonfly-comprehensive-report-2023.pdf Performs the HTTP GET request to the user-supplied URL. It disables TLS verification and does not restrict redirects, contributing to SSRF vulnerabilities. ```go func (p *preheat) getManifests(ctx context.Context, url string, header http.Header) (*http.Response, error ) { req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil ) if err != nil { return nil, err } req.Header = header req.Header.Add(headers.Accept, schema2.MediaTypeManifest) client := &http.Client{ Timeout: defaultHTTPRequesttimeout, Transport: &http.Transport{ TLSClientConfig: &tls.Config{InsecureSkipVerify: true }, }, } resp, err := client.Do(req) if err != nil { return nil, err } return resp, nil } ``` -------------------------------- ### Deprecated os.IsPermission usage Source: https://github.com/dragonflyoss/dragonfly/blob/main/docs/security/dragonfly-comprehensive-report-2023.pdf Another example of using deprecated os.Is* methods family. The code uses os.IsPermission to check for permission errors, which is deprecated. ```go } else if os.IsPermission(err) || dir == "/" { ``` -------------------------------- ### URL parsing with regex Source: https://github.com/dragonflyoss/dragonfly/blob/main/docs/security/dragonfly-comprehensive-report-2023.pdf Example code handling a URL with a regex. The function parseAccessURL uses a regex pattern to extract parts of the URL and returns an error if the pattern does not match exactly 5 groups. ```go func parseAccessURL(url string ) (*preheatImage, error ) { r := accessURLPattern.FindStringSubmatch(url) if len (r) != 5 { return nil, errors.New( "parse access url failed" ) } ``` -------------------------------- ### Backup seeds and link internal fuzzer inputs Source: https://github.com/dragonflyoss/dragonfly/blob/main/docs/security/dragonfly-comprehensive-report-2023.pdf Backs up the seeds directory and replaces it with a symlink to the internal fuzzer's inputs. This is done to get coverage of the fuzzing test. The internal fuzzer's data is a set of 'interesting' files that generate new coverage. ```bash cp-rf ./seeds_backup ln-s "$(go env GOCACHE)/fuzz/d7y.io/dragonfly/v2/client/daemon/rpcserver/FuzzTes tGrpcToB" ``` -------------------------------- ### Hard-coded string literal "bytes=" Source: https://github.com/dragonflyoss/dragonfly/blob/main/docs/security/dragonfly-comprehensive-report-2023.pdf Example use of a hard-coded string literal "bytes=" instead of a constant. The source recommends defining and using constants whenever possible. ```go rg = strings.TrimLeft(r, "bytes=" ) ``` -------------------------------- ### Create and analyze CodeQL database for Go Source: https://github.com/dragonflyoss/dragonfly/blob/main/docs/security/dragonfly-comprehensive-report-2023.pdf Creates a CodeQL database for the Go project and then analyzes it using the go-security-and-quality query suite, outputting SARIF results. The first command creates the database with the -l go flag, and the second analyzes the database with the specified query suite. The source shows the commands without spaces between flags, which may be a formatting artifact. ```bash codeql database create codeql.db-l go ``` ```bash codeql database analyze./codeql.db--format=sarif-latest-o codeql.sarif--go-security-and-quality.qls ``` -------------------------------- ### Run unit tests with make test Source: https://github.com/dragonflyoss/dragonfly/blob/main/test/README.md Use this command to run all unit tests in the project directory. It is the standard command for executing unit tests. ```bash make test ``` -------------------------------- ### Unchecked type assertion in host.go Source: https://github.com/dragonflyoss/dragonfly/blob/main/docs/security/dragonfly-comprehensive-report-2023.pdf Another example of unchecked type assertion. The type assertion may panic if rawPeer is not of type *Peer. ```go return rawPeer.(*Peer), loaded ``` -------------------------------- ### Deploy Dragonfly with run.sh Source: https://github.com/dragonflyoss/dragonfly/blob/main/deploy/docker-compose/README.md Set the host IP address and execute the deployment script. Note that without network=host mode, the host IP must be explicitly provided. ```shell # Without network=host mode,the HOST IP would be the docker network gateway IP address, use the command below to # obtain the ip address, "docker network inspect bridge -f '{{range .IPAM.Config}}{{.Gateway}}{{end}}'" export IP= ./run.sh ``` -------------------------------- ### FuzzTestGrpcToB: fuzz test for dfget daemon Download gRPC Source: https://github.com/dragonflyoss/dragonfly/blob/main/docs/security/dragonfly-comprehensive-report-2023.pdf Fuzz harness for the Download method of the dfget daemon gRPC service. Sets up a local gRPC server using the same code as TestServer_ServeDownload, creates a gRPC client, and calls the server with random data. If the server panics, the client receives an error with codes.Internal due to the recovery handler. The harness is slow (about 3,000 executions per second) because of Unix socket communication and is only a demonstration. ```go func FuzzTestGrpcToB(f *testing.F) { assert := testifyassert.New(f) ctrl := gomock.NewController(f) defer ctrl.Finish() mockPeerTaskManager := peer.NewMockTaskManager(ctrl) srv := &server{ KeepAlive: util.NewKeepAlive( "test" ), peerHost: &schedulerv1.PeerHost{}, peerTaskManager: mockPeerTaskManager, } socketDir, err := ioutil.TempDir(os.TempDir(), "d7y-test-***" ) assert.Nil(err, "make temp dir should be ok" ) socket := path.Join(socketDir, "rpc.sock" ) defer os.RemoveAll(socketDir) if srv.healthServer == nil { srv.healthServer = health.NewServer() } srv.downloadServer = dfdaemonserver.New(srv, srv.healthServer) srv.peerServer = dfdaemonserver.New(srv, srv.healthServer) ln, err := net.Listen( "unix", socket) assert.Nil(err, "listen unix socket should be ok" ) go func () { if err := srv.ServeDownload(ln); err != nil { } }() netAddr := &dfnet.NetAddr{ Type: dfnet.UNIX, Addr: socket, } client, err := dfdaemonclient.GetInsecureV1(context.Background(), netAddr.String()) assert.Nil(err, "grpc dial should be ok" ) f.Fuzz( func (t *testing.T, uu, url, output, tag, filter, rang, digest string, bs, koo, rec bool, uid, gid int64 ) { request := &dfdaemonv1.DownRequest{ Uuid: uu, Url: url, Output: output, DisableBackSource: bs, UrlMeta: &commonv1.UrlMeta{ Tag: tag, Filter: filter, Range: rang, Digest: digest, }, Uid: uid, Gid: gid, KeepOriginalOffset: koo, Recursive: rec, } stream, err := client.Download(context.TODO(), request) if err != nil { // client-side error, skip return } _, err = stream.Recv() if err != nil && status.Code(err) == codes.Internal { } }) } ``` -------------------------------- ### Run unit tests with coverage using make test-coverage Source: https://github.com/dragonflyoss/dragonfly/blob/main/test/README.md Use this command to run unit tests and generate coverage reports. It is useful for measuring test coverage. ```bash make test-coverage ``` -------------------------------- ### CreatePreheat method handling job creation Source: https://github.com/dragonflyoss/dragonfly/blob/main/docs/security/dragonfly-comprehensive-report-2023.pdf Handles Preheat job creation requests. The URL provided by the user is weakly validated, enabling SSRF attacks. The function calls getLayers for image type preheats. ```go func (p *preheat) CreatePreheat(ctx context.Context, schedulers []models.Scheduler, json types.PreheatArgs) (*internaljob.GroupJobState, error ) { [skipped] url := json.URL [skipped] // Generate download files var files []internaljob.PreheatRequest switch PreheatType(json.Type) { case PreheatImageType: // Parse image manifest url skipped, err := parseAccessURL(url) [skipped] files, err = p.getLayers(ctx, url, tag, filter, nethttp.MapToHeader(rawheader), image) [skipped] case PreheatFileType: [skipped] } } ``` -------------------------------- ### Build Base Image with Docker Source: https://github.com/dragonflyoss/dragonfly/blob/main/build/images/base/README.md Builds the base image for debugging with the specified tag. ```shell docker build -t dragonflyoss/base:bpftrace-v0.13.0-go-v1.20.1 -f Dockerfile . ``` -------------------------------- ### Unchecked type assertion in ClientDaemon.Serve Source: https://github.com/dragonflyoss/dragonfly/blob/main/docs/security/dragonfly-comprehensive-report-2023.pdf An example of unchecked type assertion. The type assertion may panic if the cfg variable does not represent a valid config.DaemonOption structure. Use checked assertions instead. ```go daemonConfig := cfg.(*config.DaemonOption) ``` -------------------------------- ### Run E2E tests with coverage using make e2e-test-coverage Source: https://github.com/dragonflyoss/dragonfly/blob/main/test/README.md Use this command to run E2E tests and generate coverage reports. It is useful for measuring E2E test coverage. ```bash make e2e-test-coverage ``` -------------------------------- ### Build Dragonfly Images with Debug Tags Source: https://github.com/dragonflyoss/dragonfly/blob/main/build/images/base/README.md Sets environment variables for debug compilation and builds Dragonfly images. ```shell # tags for condition compiling export GOTAGS="debug" # gcflags for dlv export GOGCFLAGS="all=-N -l" # base image export BASE_IMAGE=dragonflyoss/base:bpftrace-v0.13.0-go-v1.20.1 make docker-build ``` -------------------------------- ### Run E2E tests with make e2e-test Source: https://github.com/dragonflyoss/dragonfly/blob/main/test/README.md Use this command to run end-to-end tests located in the test/e2e directory. It is the standard command for executing E2E tests. ```bash make e2e-test ``` -------------------------------- ### Hardcoded MD5 digest reader in piece downloader Source: https://github.com/dragonflyoss/dragonfly/blob/main/docs/security/dragonfly-comprehensive-report-2023.pdf Creates a digest reader with the MD5 algorithm for a limited response body, using the piece's expected MD5 and a logger. Shows hardcoded use of MD5 in the piece downloader. ```go reader, err = digest.NewReader( digest.AlgorithmMD5, io.LimitReader(resp.Body, int64(req.piece.RangeSize)), digest.WithEncoded(req.piece.PieceMd5), digest.WithLogger(req.log), ) ``` -------------------------------- ### Run Semgrep with Trail of Bits rules Source: https://github.com/dragonflyoss/dragonfly/blob/main/docs/security/dragonfly-comprehensive-report-2023.pdf Runs Semgrep with the Trail of Bits ruleset, producing a SARIF report. The command uses the --sarif flag to output SARIF format, disables metrics with --metrics=off, and writes the report to semgrep.sarif. The source shows the command without spaces between flags, which may be a formatting artifact. ```bash semgrep--config "p/trailofbits"--sarif--metrics=off--output semgrep.sarif ``` -------------------------------- ### Run Debug Container with Docker Source: https://github.com/dragonflyoss/dragonfly/blob/main/build/images/base/README.md Runs a debug container with the base image, mounting kernel and module directories and enabling host network and PID namespace. ```shell BASE_IMAGE=dragonflyoss/base:bpftrace-v0.13.0-go-v1.20.1 # run debug container docker run -ti -v /usr/src:/usr/src:ro \ -v /lib/modules/:/lib/modules:ro \ -v /sys/kernel/debug/:/sys/kernel/debug:rw \ --net=host --pid=host --privileged \ ${BASE_IMAGE} ``` -------------------------------- ### grpcurl command to trigger panic in ImportTask Source: https://github.com/dragonflyoss/dragonfly/blob/main/docs/security/dragonfly-comprehensive-report-2023.pdf A grpcurl command that sends an ImportTask request without the url_meta field, triggering a nil dereference panic in the daemon gRPC server. ```bash grpcurl -plaintext -format json -d \ '{"url":"[http://example.com"](http://example.com"), "path":"x"}' $PEER_IP:65000 dfdaemon.Daemon.ImportTask ``` -------------------------------- ### Prepare Code for Debug with Git Clone Source: https://github.com/dragonflyoss/dragonfly/blob/main/build/images/base/README.md Clones the Dragonfly repository and resets to a specific commit for debugging. ```shell COMMIT_ID=c1c3d652 mkdir -p /go/src/d7y.io/dragonfly git clone https://github.com/dragonflyoss/dragonfly.git /go/src/d7y.io/dragonfly/v2 git reset --hard ${COMMIT_ID} ``` -------------------------------- ### HTTP request to download imported file Source: https://github.com/dragonflyoss/dragonfly/blob/main/docs/security/dragonfly-comprehensive-report-2023.pdf Bob’s response, revealing /etc/passwd contents (figure 12.7). HTTP request to the /download/:task_prefix/:task_id endpoint. ```http GET /download//?peerId=172.17.0.1-1- HTTP / 1.1 Host: $BOB_IP:55002 Range: bytes=0-100 ``` -------------------------------- ### Debug a single fuzzer input Source: https://github.com/dragonflyoss/dragonfly/blob/main/docs/security/dragonfly-comprehensive-report-2023.pdf Debug a single input created by the fuzzer. Replace with a file from the seeds directory at ./client/daemon/rpcserver/testdata/fuzz/FuzzTestGrpcToB. ```bash go test-run=FuzzTestGrpcToB/ ``` -------------------------------- ### gRPC API definition for ImportTask and ExportTask Source: https://github.com/dragonflyoss/dragonfly/blob/main/docs/security/dragonfly-comprehensive-report-2023.pdf Definition of the gRPC API exposed by a peer (api/pkg/apis/dfdaemon/v1/dfdaemon.proto#113–131). Shows the ImportTask and ExportTask endpoints. ```protobuf service Daemon{ [skipped] // Import the given file into P2P cache system rpc ImportTask(ImportTaskRequest) returns (google.protobuf.Empty); // Export or download file from P2P cache system rpc ExportTask(ExportTaskRequest) returns (google.protobuf.Empty); [skipped] } ``` -------------------------------- ### ImportTaskRequest definition with unvalidated url_meta (dfdaemon.proto) Source: https://github.com/dragonflyoss/dragonfly/blob/main/docs/security/dragonfly-comprehensive-report-2023.pdf The ImportTaskRequest protobuf definition. The url_meta field lacks validation rules, allowing it to be nil. The url and path fields have a min_len rule of 1. ```proto message ImportTaskRequest {// Download url. string url = 1 [(validate.rules). string // URL meta info. common.UrlMeta url_meta = 2; // File to be imported. string path = 3 [(validate.rules). string // Task type. common.TaskType type = 4 ;}|.min_len = 1]; .min_len = 1]; ``` -------------------------------- ### DownloadTinyFile with hard-coded slice index Source: https://github.com/dragonflyoss/dragonfly/blob/main/docs/security/dragonfly-comprehensive-report-2023.pdf Slicing operation with hard-coded index in DownloadTinyFile. If p.Task.ID length is less than 3, the code panics. This is from Dragonfly2/scheduler/resource/peer.go#436–446. ```go func (p *Peer) DownloadTinyFile() ([] byte ctx, cancel := context.WithTimeout(context.Background(), downloadTinyFileContextTimeout) defer cancel() // Download url: [http://${host}:${port}/download/${taskIndex}/${taskID}?peerId=${peerID}](http://${host}:${port}/download/${taskIndex}/${taskID}?peerId=${peerID}) targetURL := url.URL{Scheme: "http", Host: fmt.Sprintf("%s:%d" Path: fmt.Sprintf(RawQuery: fmt.Sprintf("peerId=%s"}|, error) {, p.Host.IP, p.Host.DownloadPort), "download/%s/%s", p.Task.ID[: 3], p.Task.ID),, p.ID), ``` -------------------------------- ### GetV1Preheat missing return on parse error Source: https://github.com/dragonflyoss/dragonfly/blob/main/docs/security/dragonfly-comprehensive-report-2023.pdf The GetV1Preheat method in Dragonfly2/manager/service/preheat.go (lines 66-70) parses the rawID argument as an unsigned integer. If parsing fails, it logs an error but does not return, leaving id with zero or max_uint value. This missing return statement is the second instance of TOB-DF2-16. ```go func (s *service) GetV1Preheat(ctx context.Context, rawID string) (*types.GetV1PreheatResponse, error) { id, err := strconv.ParseUint(rawID, 10, 32) if err != nil { logger.Errorf("preheat convert error") } } ``` -------------------------------- ### calculateParentHostUploadSuccessScore with incorrect comment Source: https://github.com/dragonflyoss/dragonfly/blob/main/docs/security/dragonfly-comprehensive-report-2023.pdf Function with an incorrect comment: the comment says the output ranges from 0 to unlimited, but the output actually ranges only from 0 to 1. ```go // calculateParentHostUploadSuccessScore 0.0~unlimited larger and better. func calculateParentHostUploadSuccessScore(peer *resource.Peer) float64 { uploadCount := peer.Host.UploadCount.Load() uploadFailedCount := peer.Host.UploadFailedCount.Load() if uploadCount < uploadFailedCount { return minScore } // Host has not been scheduled, then it is scheduled first. if uploadCount == 0 && uploadFailedCount == 0 { return maxScore } return float64 (uploadCount-uploadFailedCount) / float64 (uploadCount) } ```