### Example: Create Tar Archive Source: https://github.com/mholt/archives/blob/main/_autodocs/06-archive-formats.md Demonstrates how to create a new tar archive from files on disk. Ensure the output file is properly closed. ```go import ( "context" "os" "github.com/mholt/archives" ) // Create tar archive files, _ := archives.FilesFromDisk(context.Background(), nil, map[string]string{ "/path/to/files": "dest", }) out, _ := os.Create("archive.tar") defer out.Close() archives.Tar{}.Archive(context.Background(), out, files) ``` -------------------------------- ### Get mholt/archives Go Package Source: https://github.com/mholt/archives/blob/main/README.md Install the mholt/archives Go package using the go get command. ```bash go get github.com/mholt/archives ``` -------------------------------- ### ArchiveFS Usage Examples Source: https://github.com/mholt/archives/blob/main/_autodocs/05-filesystems.md Provides practical examples of how to use the ArchiveFS type, including creating an ArchiveFS from a file, walking through the archive, opening and reading files, and using a subdirectory prefix or a stream. ```APIDOC ## Examples ```go import ( "context" "io/fs" "github.com/mholt/archives" ) // Create ArchiveFS from file fsys := &archives.ArchiveFS{ Path: "archive.zip", Format: archives.Zip{}, } // Walk archive fs.WalkDir(fsys, ".", func(path string, d fs.DirEntry, err error) error { fmt.Println(path) return nil }) // Open and read file f, _ := fsys.Open("folder/file.txt") def f.Close() data, _ := io.ReadAll(f) // Open with subdirectory prefix fsys2 := &archives.ArchiveFS{ Path: "archive.tar", Format: archives.Tar{}, Prefix: "subfolder", } // Now "." returns contents of subfolder // Use with stream stream := io.NewSectionReader(file, 0, size) fsys3 := &archives.ArchiveFS{ Stream: stream, Format: archives.Zip{}, } ``` ``` -------------------------------- ### Example: Create and Extract Compressed Archive Source: https://github.com/mholt/archives/blob/main/_autodocs/04-identify.md Demonstrates creating a .tar.gz archive using CompressedArchive and then extracting its contents. Ensure necessary imports are included. ```go import ( "context" "os" "github.com/mholt/archives" ) // Create a .tar.gz archive ca := archives.CompressedArchive{ Archival: archives.Tar{}, Extraction: archives.Tar{}, Compression: archives.Gz{}, } out, _ := os.Create("archive.tar.gz") defer out.Close() files, _ := archives.FilesFromDisk(context.Background(), nil, map[string]string{ "/path": "dest", }) ca.Archive(context.Background(), out, files) // Extract from compressed archive in, _ := os.Open("archive.tar.gz") defer in.Close() ca.Extract(context.Background(), in, func(ctx context.Context, file archives.FileInfo) error { f, _ := file.Open() defer f.Close() // Process f return nil }) ``` -------------------------------- ### Gzip Compression and Decompression Examples Source: https://github.com/mholt/archives/blob/main/_autodocs/07-compression-formats.md Demonstrates how to compress data using Gzip with multithreading and how to decompress data. Ensure to close the compressor/decompressor after use. ```go import ( "github.com/mholt/archives" "os" ) // Compress with multithreading compressor, _ := archives.Gz{Multithreaded: true}.OpenWriter(outputFile) deferr compressor.Close() io.Copy(compressor, inputFile) // Decompress decompressor, _ := archives.Gz{}.OpenReader(inputFile) deferr decompressor.Close() io.Copy(outputFile, decompressor) ``` -------------------------------- ### Example: Creating a Tar Archive Source: https://github.com/mholt/archives/blob/main/_autodocs/03-interfaces.md Demonstrates how to create a tar archive using the Archiver interface. Ensure files are correctly prepared and the output stream is handled properly. ```go import ( "context" "os" "github.com/mholt/archives" ) files, _ := archives.FilesFromDisk(context.Background(), nil, map[string]string{ "/path/to/file": "dest.txt", }) out, _ := os.Create("archive.tar") defer out.Close() archives.Tar{}.Archive(context.Background(), out, files) ``` -------------------------------- ### Example: Custom File Extraction Handler Source: https://github.com/mholt/archives/blob/main/_autodocs/03-interfaces.md Provides an example of a FileHandler implementation that skips .git directories and stops processing if a file ends with .tmp. It demonstrates conditional logic for file processing during extraction. ```go handler := func(ctx context.Context, file archives.FileInfo) error { if file.Name() == ".git" { return fs.SkipDir } if strings.HasSuffix(file.NameInArchive, ".tmp") { return fs.SkipAll } f, err := file.Open() if err != nil { return err } defer f.Close() // Process file return nil } ``` -------------------------------- ### Create Zip with Selective Compression Source: https://github.com/mholt/archives/blob/main/_autodocs/06-archive-formats.md Example of creating a zip archive with selective compression enabled, using Deflate compression. ```go import ( "context" "github.com/mholt/archives" ) // Create zip with selective compression files, _ := archives.FilesFromDisk(context.Background(), nil, map[string]string{ "/my/files": "", }) out, _ := os.Create("archive.zip") defer out.Close() archives.Zip{ SelectiveCompression: true, Compression: 8, // Deflate }.Archive(context.Background(), out, files) ``` -------------------------------- ### DeepFS Usage Example: Walking and Opening Files Source: https://github.com/mholt/archives/blob/main/_autodocs/05-filesystems.md Demonstrates creating a DeepFS instance and using fs.WalkDir to traverse directories and archives. Also shows how to open a specific file within a compressed archive. ```go import ( "context" "io/fs" "github.com/mholt/archives" ) // Create DeepFS rooted at directory fsys := &archives.DeepFS{Root: "/home/user/files"} // Walk, transparently entering archives fs.WalkDir(fsys, ".", func(path string, d fs.DirEntry, err error) error { if err != nil { return err } // path might be: "document.txt", "archives/old.zip", "archives/old.zip/config.json" fmt.Println(path) return nil }) // Open file inside archive f, err := fsys.Open("backups/2023.tar.gz/data/file.txt") if err != nil { return err } deffer f.Close() data, _ := io.ReadAll(f) ``` -------------------------------- ### Progressive Compression with OpenWriter Source: https://github.com/mholt/archives/blob/main/_autodocs/11-compatibility.md Use OpenWriter for progressive compression, which is stream-friendly. This example shows how to obtain a compressor. ```go // Progressive compression - use OpenWriter compressor, _ := format.OpenWriter(w) // Stream-friendly ``` -------------------------------- ### Implementing a Custom Compression Format Source: https://github.com/mholt/archives/blob/main/_autodocs/12-quick-reference.md Example of implementing the necessary methods for a custom compression format, including Extension, MediaType, Match, OpenWriter, and OpenReader. Remember to register the format. ```go type MyFormat struct{} func (MyFormat) Extension() string { return ".myext" } func (MyFormat) MediaType() string { return "application/x-myext" } func (MyFormat) Match(ctx context.Context, filename string, stream io.Reader) (archives.MatchResult, error) { ... } func (MyFormat) OpenWriter(w io.Writer) (io.WriteCloser, error) { ... } func (MyFormat) OpenReader(r io.Reader) (io.ReadCloser, error) { ... } func init() { archives.RegisterFormat(MyFormat{}) } ``` -------------------------------- ### Example: Extract from Tar Archive Source: https://github.com/mholt/archives/blob/main/_autodocs/06-archive-formats.md Shows how to extract files from an existing tar archive. The provided handler function processes each extracted file. ```go import ( "context" "os" "github.com/mholt/archives" ) // Extract from tar in, _ := os.Open("archive.tar") defer in.Close() archives.Tar{}.Extract(context.Background(), in, func(ctx context.Context, f archives.FileInfo) error { file, _ := f.Open() defer file.Close() // Process file return nil }) ``` -------------------------------- ### Implementing a Custom Archive Format Source: https://github.com/mholt/archives/blob/main/_autodocs/12-quick-reference.md Example of implementing the necessary methods for a custom archive format, including Extension, MediaType, Match, Archive, and Extract. Remember to register the format. ```go type MyArchive struct{} func (MyArchive) Extension() string { return ".myarch" } func (MyArchive) MediaType() string { return "application/x-myarch" } func (MyArchive) Match(ctx context.Context, filename string, stream io.Reader) (archives.MatchResult, error) { ... } func (MyArchive) Archive(ctx context.Context, output io.Writer, files []archives.FileInfo) error { ... } func (MyArchive) Extract(ctx context.Context, archive io.Reader, handleFile archives.FileHandler) error { ... } func init() { archives.RegisterFormat(MyArchive{}) } ``` -------------------------------- ### Creating Archives Source: https://github.com/mholt/archives/blob/main/_autodocs/12-quick-reference.md Examples of how to create different types of archives, including standard Zip, Tar, and compressed archives with various compression algorithms. ```APIDOC ## Zip Archive ### Description Creates a Zip archive with DEFLATE compression. ### Code Example ```go archives.Zip{Compression: 8} ``` ## Tar Archive ### Description Creates a standard Tar archive, preserving Unix permissions and symlinks. ### Code Example ```go archives.Tar{} ``` ## Compressed Archive (Max Compression) ### Description Creates a compressed archive using Tar for archival and Zstd for compression with a high compression level. ### Code Example ```go archives.CompressedArchive{ Archival: archives.Tar{}, Extraction: archives.Tar{}, Compression: archives.Zstd{CompressionLevel: 22}, } ``` ## Compressed Archive (Fast, Large Files) ### Description Creates a compressed archive using Tar for archival and multi-threaded Gzip for fast compression, suitable for large files. ### Code Example ```go archives.CompressedArchive{ Archival: archives.Tar{}, Extraction: archives.Tar{}, Compression: archives.Gz{Multithreaded: true}, } ``` ``` -------------------------------- ### Create Zip Archive from Embedded Filesystem Source: https://github.com/mholt/archives/blob/main/_autodocs/09-patterns.md This example demonstrates creating a zip archive from an embedded filesystem using `FilesFromFS`. Ensure the `//go:embed` directive is correctly set up to include the desired assets. ```Go import ( "embed" "context" "github.com/mholt/archives" ) //go:embed assets/* var assets embed.FS func createZip(outputPath string) error { ctx := context.Background() // Gather from embedded FS files, err := archives.FilesFromFS(ctx, assets, nil, nil) if err != nil { return err } out, _ := os.Create(outputPath) defer out.Close() return archives.Zip{Compression: 8}.Archive(ctx, out, files) } ``` -------------------------------- ### Example: Append to Tar Archive Source: https://github.com/mholt/archives/blob/main/_autodocs/06-archive-formats.md Illustrates how to append new files to an existing tar archive using the Insert method. The archive must be opened in read-write mode. ```go import ( "context" "os" "github.com/mholt/archives" ) // Append to tar tarball, _ := os.OpenFile("archive.tar", os.O_RDWR, 0644) defer tarball.Close() newFiles, _ := archives.FilesFromDisk(context.Background(), nil, map[string]string{ "/new": "new-content", }) archives.Tar{}.Insert(context.Background(), tarball, newFiles) ``` -------------------------------- ### Example: Appending Files to a Tar Archive Source: https://github.com/mholt/archives/blob/main/_autodocs/03-interfaces.md Demonstrates how to use the Inserter interface to append new files to an existing tar archive. Ensure the archive file is opened in read-write mode. ```go import ( "context" "os" "github.com/mholt/archives" ) tarball, _ := os.OpenFile("archive.tar", os.O_RDWR, 0644) deftarball.Close() files, _ := archives.FilesFromDisk(context.Background(), nil, map[string]string{ "/new/file": "newfile.txt", }) archives.Tar{}.Insert(context.Background(), tarball, files) ``` -------------------------------- ### Zip Archive Example Source: https://github.com/mholt/archives/blob/main/_autodocs/06-archive-formats.md Example of creating a Zip archive with selective compression and extracting from a Zip archive. ```APIDOC ## Zip Archive Example ### Description Example of creating a Zip archive with selective compression and extracting from a Zip archive. ### Code Example ```go import ( "context" "github.com/mholt/archives" ) // Create zip with selective compression files, _ := archives.FilesFromDisk(context.Background(), nil, map[string]string{ "/my/files": "", }) out, _ := os.Create("archive.zip") defer out.Close() archives.Zip{ SelectiveCompression: true, Compression: 8, // Deflate }.Archive(context.Background(), out, files) // Extract from zip in, _ := os.Open("archive.zip") defer in.Close() archives.Zip{}.Extract(context.Background(), in, func(ctx context.Context, f archives.FileInfo) error { file, _ := f.Open() defer file.Close() return nil }) ``` ``` -------------------------------- ### Find and Process JSON Files in an Archive Source: https://github.com/mholt/archives/blob/main/_autodocs/05-filesystems.md Use `fs.WalkDir` to iterate over all entries in an archive filesystem. This example specifically finds files with a `.json` extension and reads their content for further processing. ```go fsys, _ := archives.FileSystem(ctx, "data.zip", nil) fs.WalkDir(fsys, ".", func(path string, d fs.DirEntry, err error) error { if filepath.Ext(path) == ".json" { data, _ := fs.ReadFile(fsys, path) // Process data } return nil }) ``` -------------------------------- ### Compress Data Source: https://github.com/mholt/archives/blob/main/_autodocs/INDEX.md Use `OpenWriter` from a specific format to get a compressor, then write data to it. Ensure the compressor is closed afterwards. ```go // Compress compressor, err := format.OpenWriter(writer) defer compressor.Close() compressor.Write(data) ``` -------------------------------- ### Example Usage of Compressor Source: https://github.com/mholt/archives/blob/main/_autodocs/03-interfaces.md Demonstrates how to use a `Compressor` to write compressed data. The `OpenWriter` method is called on a compressor implementation (e.g., `archives.Zstd{}`), and the returned `WriteCloser` is used for writing. Remember to `defer` closing the `WriteCloser`. ```go import "github.com/mholt/archives" comp, err := archives.Zstd{}.OpenWriter(outputFile) if err != nil { return err } def comp.Close() // Writes to comp are compressed comp.Write([]byte("data")) ``` -------------------------------- ### Example: Extracting Files from a Zip Archive Source: https://github.com/mholt/archives/blob/main/_autodocs/03-interfaces.md Shows how to extract files from a zip archive using the Extractor interface. The FileHandler callback is invoked for each file, and files must be closed within the handler. ```go import ( "context" "fmt" "github.com/mholt/archives" ) var format archives.Zip archives.Zip{}.Extract(context.Background(), archiveReader, func(ctx context.Context, file archives.FileInfo) error { fmt.Println("Found:", file.NameInArchive) f, err := file.Open() if err != nil { return err } defer f.Close() // Read and process f return nil }) ``` -------------------------------- ### Example Usage of Decompressor Source: https://github.com/mholt/archives/blob/main/_autodocs/03-interfaces.md Demonstrates how to use a `Decompressor` to read decompressed data. The `OpenReader` method is called on a decompressor implementation (e.g., `archives.Brotli{}`), and the returned `ReadCloser` is used for reading. Remember to `defer` closing the `ReadCloser`. ```go import "github.com/mholt/archives" decomp, err := archives.Brotli{}.OpenReader(inputFile) if err != nil { return err } def decomp.Close() // Reads from decomp are decompressed io.Copy(outputFile, decomp) ``` -------------------------------- ### Browse Archive Contents Source: https://github.com/mholt/archives/blob/main/_autodocs/09-patterns.md Use `archives.FileSystem` to get an `fs.FS` interface for an archive. Then, use `fs.WalkDir` to iterate through its contents, distinguishing between directories and files. This is useful for inspecting archive structure without extracting. ```go import ( "io/fs" "github.com/mholt/archives" ) func listArchiveContents(archivePath string) error { ctx := context.Background() fsys, err := archives.FileSystem(ctx, archivePath, nil) if err != nil { return err } // Walk archive return fs.WalkDir(fsys, ".", func(path string, d fs.DirEntry, err error) error { if err != nil { return err } if d.IsDir() { fmt.Printf("DIR: %s\n", path) } else { info, _ := d.Info() fmt.Printf("FILE: %s (%d bytes)\n", path, info.Size()) } return nil }) } ``` -------------------------------- ### DeepFS Struct Definition Source: https://github.com/mholt/archives/blob/main/_autodocs/05-filesystems.md Defines the DeepFS struct with a Root field for the starting directory path. ```go type DeepFS struct { Root string // Root directory path on disk } ``` -------------------------------- ### FilesFromDisk with Options Source: https://github.com/mholt/archives/blob/main/_autodocs/02-filesgather.md Demonstrates using FilesFromDisk with custom options, such as enabling symlink following and clearing file attributes. This allows for more control over how files are gathered. ```go // With options files, err := archives.FilesFromDisk(context.Background(), &archives.FromDiskOptions{ FollowSymlinks: true, ClearAttributes: true, }, map[string]string{ "/path": "target", }) ``` -------------------------------- ### Extract from Zip Archive Source: https://github.com/mholt/archives/blob/main/_autodocs/06-archive-formats.md Example of extracting files from a zip archive. The handler function is called for each extracted file. ```go import ( "context" "github.com/mholt/archives" ) // Extract from zip in, _ := os.Open("archive.zip") defer in.Close() archives.Zip{}.Extract(context.Background(), in, func(ctx context.Context, f archives.FileInfo) error { file, _ := f.Open() defer file.Close() return nil }) ``` -------------------------------- ### Create and Walk ArchiveFS from File Source: https://github.com/mholt/archives/blob/main/_autodocs/05-filesystems.md Demonstrates creating an ArchiveFS instance from a zip file path and then walking its directory structure using fs.WalkDir. ```go import ( "context" "io/fs" "github.com/mholt/archives" ) // Create ArchiveFS from file fsys := &archives.ArchiveFS{ Path: "archive.zip", Format: archives.Zip{}, } // Walk archive fs.WalkDir(fsys, ".", func(path string, d fs.DirEntry, err error) error { fmt.Println(path) return nil }) ``` -------------------------------- ### Basic File Mapping with FilesFromDisk Source: https://github.com/mholt/archives/blob/main/_autodocs/02-filesgather.md Demonstrates basic file mapping using FilesFromDisk, where specific files are mapped to paths within the archive. The second entry shows how to place a file directly in the archive's root using an empty string for the archive path. ```go import ( "context" "github.com/mholt/archives" ) // Basic file mapping files, err := archives.FilesFromDisk(context.Background(), nil, map[string]string{ "/path/to/file.txt": "documents/file.txt", "/path/to/file2.txt": "", // basename in root: file2.txt }) ``` -------------------------------- ### Create Tar.Gz Archive from Disk Files Source: https://github.com/mholt/archives/blob/main/README.md Demonstrates creating a .tar.gz archive by mapping files from disk to archive paths. Supports recursive directory inclusion and custom folder names within the archive. Requires context, output file, and a map of source files to archive paths. ```go ctx := context.TODO() // map files on disk to their paths in the archive using default settings (second arg) files, err := archives.FilesFromDisk(ctx, nil, map[string]string{ "/path/on/disk/file1.txt": "file1.txt", "/path/on/disk/file2.txt": "subfolder/file2.txt", "/path/on/disk/file3.txt": "", // put in root of archive as file3.txt "/path/on/disk/file4.txt": "subfolder/", // put in subfolder as file4.txt "/path/on/disk/folder": "Custom Folder", // contents added recursively }) if err != nil { return err } // create the output file we'll write to out, err := os.Create("example.tar.gz") if err != nil { return err } defer out.Close() // we can use the CompressedArchive type to gzip a tarball // (since we're writing, we only set Archival, but if you're // going to read, set Extraction) format := archives.CompressedArchive{ Compression: archives.Gz{}, Archival: archives.Tar{}, } // create the archive er r = format.Archive(ctx, out, files) if err != nil { return err } ``` -------------------------------- ### Cancellable Archive Extraction Source: https://github.com/mholt/archives/blob/main/_autodocs/10-advanced.md Perform archive extraction while honoring context cancellation. This allows operations to be interrupted gracefully, for example, by a timeout. ```go func extractWithTimeout(archivePath string, timeoutSeconds int) error { ctx, cancel := context.WithTimeout(context.Background(), time.Duration(timeoutSeconds)*time.Second) defer cancel() file, _ := os.Open(archivePath) defer file.Close() format, stream, _ := archives.Identify(ctx, archivePath, file) // If timeout expires during extraction, ctx.Err() will be non-nil return format.(archives.Extractor).Extract(ctx, stream, func(ctx context.Context, file archives.FileInfo) error { if err := ctx.Err(); err != nil { return err // Propagate cancellation } // Process file... return nil }) } ``` -------------------------------- ### Create and Read from Compressed FileFS Source: https://github.com/mholt/archives/blob/main/_autodocs/05-filesystems.md Demonstrates creating a FileFS for a compressed file (e.g., .gz) and then opening and reading its decompressed content. Ensure to close the file after reading. ```go // Create FileFS for a compressed file fsys := archives.FileFS{ Path: "file.txt.gz", Compression: archives.Gz{}, } // Open and read decompressed content f, _ := fsys.Open(".") defer f.Close() data, _ := io.ReadAll(f) ``` -------------------------------- ### Decompress Data Source: https://github.com/mholt/archives/blob/main/_autodocs/INDEX.md Use `OpenReader` from a specific format to get a decompressor, then copy data from it to a writer. Ensure the decompressor is closed afterwards. ```go // Decompress decompressor, err := format.OpenReader(reader) defer decompressor.Close() io.Copy(writer, decompressor) ``` -------------------------------- ### Create FileInfo Manually for Archiving Source: https://github.com/mholt/archives/blob/main/_autodocs/01-fileinfo.md Demonstrates how to manually create a FileInfo instance for archiving. Requires embedding an existing fs.FileInfo and providing a NameInArchive and an Open callback. ```go import ( "os" "github.com/mholt/archives" ) // Create a FileInfo manually for archiving file := archives.FileInfo{ FileInfo: fileInfo, // from os.Stat() or similar NameInArchive: "path/in/archive.txt", Open: func() (fs.File, error) { return os.Open("/path/on/disk/file.txt") }, } ``` -------------------------------- ### Specify Zip Text Encoding Source: https://github.com/mholt/archives/blob/main/_autodocs/11-compatibility.md Set Zip.TextEncoding to specify a fallback encoding for filenames in older zip archives. This example uses Windows-1252. ```go archives.Zip{ TextEncoding: charmap.Windows1252, // For Windows-1252 encoded filenames } ``` -------------------------------- ### Dockerfile for Building Go Application Source: https://github.com/mholt/archives/blob/main/_autodocs/11-compatibility.md This Dockerfile demonstrates how to build a Go application using a multi-stage build. It first compiles the application in a Go environment and then copies the resulting binary to a minimal Alpine Linux image. ```dockerfile FROM golang:1.25-alpine AS builder RUN apk add --no-cache git WORKDIR /build COPY . . RUN go build -o app . FROM alpine:latest COPY --from=builder /build/app /app ENTRYPOINT ["/app"] ``` -------------------------------- ### 7z Archive Format Source: https://github.com/mholt/archives/blob/main/_autodocs/06-archive-formats.md Handles 7z archives, supporting password-protected files. Includes methods for getting the extension, media type, matching archives, and extracting their contents. ```APIDOC ## 7z Archive Format ### Description Handles 7z archives, supporting password-protected files. Includes methods for getting the extension, media type, matching archives, and extracting their contents. ### Configuration - **Password** (string): Password for decrypting encrypted archives. ### Methods - **Extension()**: Returns ".7z". - **MediaType()**: Returns "application/x-7z-compressed". - **Match(ctx context.Context, filename string, stream io.Reader)**: Matches by ".7z" extension or by reading 7z header. - **Extract(ctx context.Context, sourceArchive io.Reader, handleFile FileHandler)**: Extracts files from 7z archive. ### Example ```go import ( "context" "github.com/mholt/archives" ) // Extract from 7z archives.SevenZ{ Password: "mypassword", }.Extract(context.Background(), sevenZFile, func(ctx context.Context, f archives.FileInfo) error { file, _ := f.Open() defer file.Close() // Process file return nil }) ``` ``` -------------------------------- ### Format Interface Definition Source: https://github.com/mholt/archives/blob/main/_autodocs/03-interfaces.md The base interface for all archive and compression formats. It defines methods to get the file extension, MIME type, and to match the format against input. ```go type Format interface { Extension() string MediaType() string Match(ctx context.Context, filename string, stream io.Reader) (MatchResult, error) } ``` -------------------------------- ### Configure FileSystem Gathering Options Source: https://github.com/mholt/archives/blob/main/_autodocs/12-quick-reference.md Customize file system gathering with `archives.FromDiskOptions`. Options include `FollowSymlinks` to dereference symbolic links and `ClearAttributes` to remove modification time and owner information. ```go archives.FilesFromDisk(ctx, &archives.FromDiskOptions{ FollowSymlinks: true, // Dereference symlinks ClearAttributes: true, // Remove modtime, owner info }, files) ``` -------------------------------- ### Compress File with Gzip Source: https://github.com/mholt/archives/blob/main/_autodocs/09-patterns.md This pattern shows how to compress a file using Gzip with a specified compression level. Ensure input and output files are properly opened and closed. ```go func compressFile(inputPath, outputPath string, level int) error { inFile, _ := os.Open(inputPath) defer inFile.Close() outFile, _ := os.Create(outputPath) defer outFile.Close() compressor, _ := archives.Gz{CompressionLevel: level}.OpenWriter(outFile) defer compressor.Close() _, err := io.Copy(compressor, inFile) return err } ``` -------------------------------- ### RAR Archive Format Source: https://github.com/mholt/archives/blob/main/_autodocs/06-archive-formats.md Handles RAR archives, supporting password-protected files. Includes methods for getting the extension, media type, matching archives, and extracting their contents. ```APIDOC ## RAR Archive Format ### Description Handles RAR archives, supporting password-protected files. Includes methods for getting the extension, media type, matching archives, and extracting their contents. ### Configuration - **Password** (string): Password for decrypting encrypted archives. ### Methods - **Extension()**: Returns ".rar". - **MediaType()**: Returns "application/x-rar-compressed". - **Match(ctx context.Context, filename string, stream io.Reader)**: Matches by ".rar" extension or by reading rar header. - **Extract(ctx context.Context, sourceArchive io.Reader, handleFile FileHandler)**: Extracts files from rar archive, calling handleFile for each. ### Example ```go import ( "context" "github.com/mholt/archives" ) // Extract from RAR (including password-protected) archives.Rar{ Password: "secret", }.Extract(context.Background(), rarFile, func(ctx context.Context, f archives.FileInfo) error { file, _ := f.Open() defer file.Close() // Process file return nil }) ``` ``` -------------------------------- ### Use Standard fs.FS Operations on Virtual Filesystem Source: https://github.com/mholt/archives/blob/main/_autodocs/INDEX.md After creating a virtual filesystem, you can use standard `fs.FS` operations like `ReadDir`, `ReadFile`, and `WalkDir` to interact with the archive's contents. ```go // Use standard fs.FS operations entries, err := fs.ReadDir(fsys, "path") data, err := fs.ReadFile(fsys, "path/file") err := fs.WalkDir(fsys, ".", walkFn) ``` -------------------------------- ### Format Interface Source: https://github.com/mholt/archives/blob/main/_autodocs/03-interfaces.md The base interface for all archive and compression formats. It provides methods to get the file extension, MIME type, and to match the format against a given file and stream. ```APIDOC ## Format Interface ### Description The base interface for all archive and compression formats. It provides methods to get the file extension, MIME type, and to match the format against a given file and stream. ### Methods #### Extension() Returns the conventional file extension including dot. Examples: ".tar", ".gz", ".zip". #### MediaType() Returns the MIME type (RFC 2046). Examples: "application/x-tar", "application/gzip", "application/zip". #### Match(ctx context.Context, filename string, stream io.Reader) (MatchResult, error) Tests if the input matches this format. ##### Parameters - **ctx** (context.Context) - Context for cancellation - **filename** (string) - Base filename (no path) for matching by extension; may be empty - **stream** (io.Reader) - Stream to read from for header matching; may be nil Returns MatchResult indicating whether match occurred by name and/or stream, or error. ``` -------------------------------- ### Directory Recursion with FilesFromDisk Source: https://github.com/mholt/archives/blob/main/_autodocs/02-filesgather.md Shows how to use FilesFromDisk to recursively gather all contents of a directory and place them into a specified folder within the archive. ```go // Directory recursion with subfolder files, err := archives.FilesFromDisk(context.Background(), nil, map[string]string{ "/path/to/folder": "archive-folder", // adds folder and contents }) ``` -------------------------------- ### Root ArchiveFS at a Subdirectory Source: https://github.com/mholt/archives/blob/main/_autodocs/10-advanced.md Set the 'Prefix' field in ArchiveFS to treat a subdirectory within an archive as the root of the filesystem. This allows relative path operations to start from that subdirectory. ```go func readSubdirectory(archivePath, subdir string) error { ctx := context.Background() fsys := &archives.ArchiveFS{ Path: archivePath, Format: archives.Zip{}, Prefix: subdir, // Root filesystem here } // Now "." refers to subdir contents entries, _ := fsys.ReadDir(".") for _, e := range entries { fmt.Println(e.Name()) } return nil } ``` -------------------------------- ### Configure Tar Archive Options Source: https://github.com/mholt/archives/blob/main/_autodocs/12-quick-reference.md Configure Tar archive creation using `archives.Tar` struct. Options include format (USTAR, PAX, GNU), error handling (`ContinueOnError`), and overriding owner information (UID, Uname). ```go archives.Tar{ Format: tar.FormatPAX, // USTAR, PAX, or GNU ContinueOnError: true, // Don't fail on bad files Uid: 1000, // Override owner UID Uname: "user", // Override owner name } ``` -------------------------------- ### Identify Function Source: https://github.com/mholt/archives/blob/main/_autodocs/04-identify.md Automatically detects archive and compression formats by inspecting the filename and/or stream. It returns the identified `Format` type and a stream repositioned to its original starting point. ```APIDOC ## Identify Function ### Description Automatically detects archive and compression formats by inspecting the filename and/or stream. It returns the identified `Format` type and a stream repositioned to its original starting point. ### Function Signature ```go func Identify( ctx context.Context, filename string, stream io.Reader, ) (Format, io.Reader, error) ``` ### Parameters #### Function Parameters - **ctx** (context.Context) - Required - Context for cancellation. - **filename** (string) - Optional - Filename (base name only, no paths) for extension matching. Can be empty. - **stream** (io.Reader) - Optional - Stream to read for header matching. Can be nil. *Note: At least one of filename or stream must be provided.* ### Return Values #### Function Return Values - **format** (Format) - The identified format, or nil if no match. - **reader** (io.Reader) - Stream repositioned to original position; same as input if seekable, buffered otherwise. - **error** (error) - `NoMatch` if no format matched; other errors for I/O or context issues. ### Returned Format Types Based on input, the function returns one of: - **Compression** (e.g., Gzip): Standalone compressed file - **Archival** (e.g., Tar): Archive that can be read and written - **Extraction** (e.g., RAR): Read-only archive - **CompressedArchive** (e.g., Tar+Gzip): Archive wrapped in compression ### Examples #### Identify from filename only ```go import ( "context" "github.com/mholt/archives" ) format, _, err := archives.Identify(context.Background(), "archive.tar.gz", nil) if err != nil { // Could be NoMatch error } // Type-assert to see what we have if ex, ok := format.(archives.Extractor); ok { // Can extract from this format // ex.Extract(context.Background(), reader, handler) } ``` #### Identify from stream (more reliable) ```go import ( "context" "os" "github.com/mholt/archives" ) file, _ := os.Open("unknown-file") defer file.Close() format, rewindedStream, err := archives.Identify(context.Background(), "unknown", file) if err == archives.NoMatch { fmt.Println("Unknown format") return } // Returned stream can be read from beginning if ca, ok := format.(archives.CompressedArchive); ok { // ca.Extract(context.Background(), rewindedStream, handler) } ``` ### Behavior Details **Matching Priority:** 1. Tries all Compression formats first (outer layer) 2. Then tries all Archival/Extraction formats 3. Returns most specific match: Compression > Archive > Archival > other **Stream Handling:** - If stream is an `io.Seeker`, no buffering occurs; `Seek()` returns stream to original position. - If stream is not seekable, bytes are buffered and returned via a `MultiReader`. - Identifies only by reading necessary bytes (peeking); minimal stream consumption. **Filename Handling:** - Only the base name is used; any path separators are stripped. - Matching is case-insensitive. - Filename is a hint; stream inspection is more reliable. ### Common Errors ```go import ( "context" "errors" "fmt" "github.com/mholt/archives" ) // Assume ctx, filename, stream are defined // format, _, err := archives.Identify(ctx, filename, stream) if errors.Is(err, archives.NoMatch) { // No format matched the input fmt.Println("Unknown archive format") } if errors.Is(err, context.DeadlineExceeded) { // Context expired during identification } ``` ``` -------------------------------- ### Create ArchiveFS from Stream Source: https://github.com/mholt/archives/blob/main/_autodocs/05-filesystems.md Demonstrates initializing an ArchiveFS using an io.SectionReader stream instead of a file path, suitable for archives read from network or other sources. ```go // Use with stream stream := io.NewSectionReader(file, 0, size) fsys3 := &archives.ArchiveFS{ Stream: stream, Format: archives.Zip{}, } ``` -------------------------------- ### Tar Methods Source: https://github.com/mholt/archives/blob/main/_autodocs/06-archive-formats.md Methods available for interacting with Tar archives, including getting file extensions, media types, matching archives, creating archives, extracting contents, and inserting new files. ```APIDOC ## Tar Methods ### Extension - **Description**: Returns the default file extension for Tar archives. - **Signature**: `func (Tar) Extension() string` ### MediaType - **Description**: Returns the media type for Tar archives. - **Signature**: `func (Tar) MediaType() string` ### Match - **Description**: Determines if a given reader's content matches the Tar format, either by file extension or by reading the Tar header. - **Signature**: `func (t Tar) Match(ctx context.Context, filename string, stream io.Reader) (MatchResult, error)` ### Archive - **Description**: Creates a Tar archive from a slice of file information. - **Signature**: `func (t Tar) Archive(ctx context.Context, output io.Writer, files []FileInfo) error` ### ArchiveAsync - **Description**: Creates a Tar archive asynchronously from a channel of jobs. - **Signature**: `func (t Tar) ArchiveAsync(ctx context.Context, output io.Writer, jobs <-chan ArchiveAsyncJob) error` ### Extract - **Description**: Extracts files from a Tar archive. For each extracted file, it calls the provided `handleFile` function. - **Signature**: `func (t Tar) Extract(ctx context.Context, sourceArchive io.Reader, handleFile FileHandler) error` ### Insert - **Description**: Appends new files to an existing Tar archive without rewriting the entire archive. Requires a `ReadWriteSeeker` for the archive. - **Signature**: `func (t Tar) Insert(ctx context.Context, archive io.ReadWriteSeeker, files []FileInfo) error` ``` -------------------------------- ### ArchiveFS Open Method Source: https://github.com/mholt/archives/blob/main/_autodocs/05-filesystems.md Opens a file from the archive. Use '.' to open the archive root as a directory. Returns fs.File for reading or fs.ReadDirFile for directories. ```go func (f ArchiveFS) Open(name string) (fs.File, error) ``` -------------------------------- ### Configure Zip Archive Options Source: https://github.com/mholt/archives/blob/main/_autodocs/12-quick-reference.md Configure Zip archive creation with `archives.Zip`. Options include selective compression, compression method (Store, Deflate, Bzip2, Zstd, Xz), and skipping bad files (`ContinueOnError`). ```go archives.Zip{ SelectiveCompression: true, // Don't recompress compressed files Compression: 8, // Method: Store(0), Deflate(8), Bzip2(12), Zstd(93), Xz(95) ContinueOnError: true, // Skip bad files } ``` -------------------------------- ### Create Tar Archive Source: https://github.com/mholt/archives/blob/main/_autodocs/12-quick-reference.md Use this to create standard TAR archives, preserving Unix permissions and symlinks. ```go archives.Tar{} ``` -------------------------------- ### Find Files by Extension in Archive Source: https://github.com/mholt/archives/blob/main/_autodocs/09-patterns.md Search for files within an archive that match a specific extension. This involves getting the archive's filesystem interface and then using `fs.WalkDir` combined with `filepath.Ext` to filter the results. ```go func findFiles(archivePath, extension string) ([]string, error) { ctx := context.Background() var matches []string fsys, _ := archives.FileSystem(ctx, archivePath, nil) fs.WalkDir(fsys, ".", func(path string, d fs.DirEntry, err error) error { if err == nil && !d.IsDir() && filepath.Ext(path) == extension { matches = append(matches, path) } return nil }) return matches, nil } ``` -------------------------------- ### Create Archive from Files Source: https://github.com/mholt/archives/blob/main/_autodocs/12-quick-reference.md Creates a compressed tar.gz archive from specified files on disk. Ensure the output file and input files are correctly specified. ```go files, _ := archives.FilesFromDisk(ctx, nil, map[string]string{ "/path/to/files": "archive-folder", }) out, _ := os.Create("archive.tar.gz") defer out.Close() archives.CompressedArchive{ Archival: archives.Tar{}, Extraction: archives.Tar{}, Compression: archives.Gz{}, }.Archive(ctx, out, files) ``` -------------------------------- ### CompressedArchive Methods Source: https://github.com/mholt/archives/blob/main/_autodocs/04-identify.md Provides methods for CompressedArchive to get concatenated extensions, MIME type, match compression and archival formats, create compressed archives, extract from compressed archives, and open compressed readers/writers. ```go func (ca CompressedArchive) Extension() string ``` ```go func (ca CompressedArchive) MediaType() string ``` ```go func (ca CompressedArchive) Match(ctx context.Context, filename string, stream io.Reader) (MatchResult, error) ``` ```go func (ca CompressedArchive) Archive(ctx context.Context, output io.Writer, files []FileInfo) error ``` ```go func (ca CompressedArchive) Extract(ctx context.Context, sourceArchive io.Reader, handleFile FileHandler) error ``` ```go func (ca CompressedArchive) OpenWriter(w io.Writer) (io.WriteCloser, error) ``` ```go func (ca CompressedArchive) OpenReader(r io.Reader) (io.ReadCloser, error) ``` -------------------------------- ### FileFS Open Method Source: https://github.com/mholt/archives/blob/main/_autodocs/05-filesystems.md Opens the file represented by FileFS. The 'name' parameter must be '.', the full path, or the basename of the file. ```go func (f FileFS) Open(name string) (fs.File, error) ``` -------------------------------- ### Decompress File with Gzip Source: https://github.com/mholt/archives/blob/main/_autodocs/09-patterns.md This pattern demonstrates how to decompress a Gzip-compressed file. It opens the input compressed file and writes the decompressed content to the output file. ```go func decompressFile(inputPath, outputPath string) error { inFile, _ := os.Open(inputPath) defer inFile.Close() outFile, _ := os.Create(outputPath) defer outFile.Close() decompressor, _ := archives.Gz{}.OpenReader(inFile) defer decompressor.Close() _, err := io.Copy(outFile, decompressor) return err } ``` -------------------------------- ### Identify Archive Format from Filename and Stream Source: https://github.com/mholt/archives/blob/main/_autodocs/04-identify.md Use this snippet to identify an archive format by providing a filename and an io.Reader stream. The returned stream is repositioned to its original starting point. Handle potential errors, including `archives.NoMatch` if no format is recognized. ```go import ( "context" "os" "github.com/mholt/archives" ) // Identify from filename only format, _, err := archives.Identify(context.Background(), "archive.tar.gz", nil) if err != nil { // Could be NoMatch error } // Type-assert to see what we have if ex, ok := format.(archives.Extractor); ok { // Can extract from this format ex.Extract(context.Background(), reader, handler) } // Identify from stream (more reliable) file, _ := os.Open("unknown-file") deferr file.Close() format, rewindedStream, err := archives.Identify(context.Background(), "unknown", file) if err == archives.NoMatch { fmt.Println("Unknown format") return } // Returned stream can be read from beginning if ca, ok := format.(archives.CompressedArchive); ok { ca.Extract(context.Background(), rewindedStream, handler) } ``` -------------------------------- ### Create Tar.gz Archive from Disk Source: https://github.com/mholt/archives/blob/main/_autodocs/09-patterns.md This snippet shows how to create a compressed tar.gz archive from files located on disk. It uses `FilesFromDisk` to gather files and `CompressedArchive` with `Tar` and `Gz` to create the archive. ```Go import ( "context" "os" "github.com/mholt/archives" ) func createTarGz(inputPath, outputPath string) error { ctx := context.Background() // Gather files from disk files, err := archives.FilesFromDisk(ctx, nil, map[string]string{ inputPath: "contents", // Add inputPath contents under "contents" folder }) if err != nil { return err } // Create output file out, err := os.Create(outputPath) if err != nil { return err } defer out.Close() // Create compressed archive ca := archives.CompressedArchive{ Archival: archives.Tar{}, Extraction: archives.Tar{}, Compression: archives.Gz{Multithreaded: true}, } return ca.Archive(ctx, out, files) } ``` -------------------------------- ### Create Tar Archive with Manually Constructed Files Source: https://github.com/mholt/archives/blob/main/_autodocs/09-patterns.md This snippet illustrates how to create a tar archive by manually constructing `FileInfo` objects. This is useful when file metadata needs to be specified explicitly, including custom `Open` functions for file content. ```Go func createArchiveManually() error { ctx := context.Background() // Create FileInfo manually files := []archives.FileInfo{ { FileInfo: someFileInfo, // From os.Stat() NameInArchive: "file.txt", Open: func() (fs.File, error) { return os.Open("/path/to/file.txt") }, }, } out, _ := os.Create("archive.tar") defer out.Close() return archives.Tar{}.Archive(ctx, out, files) } ``` -------------------------------- ### Create Virtual Filesystem from Archive Source: https://github.com/mholt/archives/blob/main/_autodocs/INDEX.md Use the `FileSystem` function to create a virtual filesystem representation of an archive's contents. ```go // Create filesystem fsys, err := archives.FileSystem(ctx, filename, stream) ``` -------------------------------- ### FromFSOptions Source: https://github.com/mholt/archives/blob/main/_autodocs/02-filesgather.md Options for FilesFromFS. ```APIDOC ## FromFSOptions Options for FilesFromFS. ```go type FromFSOptions struct { FollowSymlinks bool // Dereference symlinks instead of preserving them ClearAttributes bool // Zero out file attributes (modtime, owner, etc.) } ``` | Field | Type | Default | Description | |-------|------|---------|-------------| | FollowSymlinks | bool | false | If true, symlinks are dereferenced; if symlink points to directory, contents are added recursively | | ClearAttributes | bool | false | If true, file attributes like modtime are cleared; name, size, type, permissions preserved | ``` -------------------------------- ### Gz OpenReader Method Source: https://github.com/mholt/archives/blob/main/_autodocs/07-compression-formats.md Wraps an io.Reader to enable gzip decompression. ```go func (gz Gz) OpenReader(r io.Reader) (io.ReadCloser, error) ``` -------------------------------- ### Gz OpenWriter Method Source: https://github.com/mholt/archives/blob/main/_autodocs/07-compression-formats.md Wraps an io.Writer to enable gzip compression. ```go func (gz Gz) OpenWriter(w io.Writer) (io.WriteCloser, error) ``` -------------------------------- ### Create ArchiveFS with Subdirectory Prefix Source: https://github.com/mholt/archives/blob/main/_autodocs/05-filesystems.md Illustrates creating an ArchiveFS instance with a Prefix field set, allowing access to a specific subdirectory within the archive as the root. ```go // Open with subdirectory prefix fsys2 := &archives.ArchiveFS{ Path: "archive.tar", Format: archives.Tar{}, Prefix: "subfolder", } // Now "." returns contents of subfolder ``` -------------------------------- ### Gz Methods for Extension and MediaType Source: https://github.com/mholt/archives/blob/main/_autodocs/07-compression-formats.md Returns the file extension and media type for the Gzip format. ```go func (Gz) Extension() string func (Gz) MediaType() string ``` -------------------------------- ### Open Method Source: https://github.com/mholt/archives/blob/main/_autodocs/05-filesystems.md Opens a file from the archive. A name of '.' opens the archive root as a directory. Returns an fs.File for reading or an fs.ReadDirFile for directories. ```APIDOC ## Open Method ```go func (f ArchiveFS) Open(name string) (fs.File, error) ``` Opens file from archive. Name '.' opens archive root as directory. Returns fs.File for reading or fs.ReadDirFile for directories. ```