### Install Goclone via Brew Source: https://github.com/goclone-dev/goclone/blob/master/README.md Tap the repository and install the goclone tool using Homebrew. ```bash # tap brew tap goclone-dev/goclone # install tool brew install goclone ``` -------------------------------- ### Install Goclone Manually with Go Source: https://github.com/goclone-dev/goclone/blob/master/README.md Install the goclone tool using 'go install'. Requires Go version 1.20 or higher. ```bash # Go version >= 1.20 go install github.com/goclone-dev/goclone/cmd/goclone@latest ``` -------------------------------- ### Install Goclone with Homebrew Source: https://context7.com/goclone-dev/goclone/llms.txt Installs Goclone using the Homebrew package manager. ```bash brew tap goclone-dev/goclone brew install goclone ``` -------------------------------- ### Install Goclone with Go Install Source: https://context7.com/goclone-dev/goclone/llms.txt Installs the latest version of Goclone using 'go install'. Requires Go version 1.20 or higher. ```bash # Requires Go >= 1.20 go install github.com/goclone-dev/goclone/cmd/goclone@latest ``` -------------------------------- ### Start Local Static File Server with Go Source: https://context7.com/goclone-dev/goclone/llms.txt Use `server.Serve` to start an Echo HTTP server that serves static files from a directory. It supports graceful shutdown and CORS. This function blocks until an OS interrupt signal is received. ```go package main import ( "log" "github.com/goclone-dev/goclone/pkg/server" ) func main() { // Serve the cloned site at http://localhost:8080 // Blocks until Ctrl+C is received, then shuts down gracefully if err := server.Serve("./example.com", 8080); err != nil { log.Fatalf("server error: %v", err) } // Access the cloned site at: http://localhost:8080/index.html } ``` -------------------------------- ### CloneSite - Programmatic Website Cloning Source: https://context7.com/goclone-dev/goclone/llms.txt The top-level Go function that orchestrates the full clone pipeline: cookie setup, crawling, asset extraction, and HTML link restructuring. It accepts a context for cancellation support. ```APIDOC ## CloneSite (Go API) ### Description `CloneSite` is the top-level function that orchestrates the full clone pipeline: cookie setup, crawling, asset extraction, and HTML link restructuring. It accepts a context for cancellation support. ### Signature ```go func CloneSite(ctx context.Context, urls []string, opts CloneOptions) error ``` ### Parameters - `ctx` (context.Context): Context for cancellation support. - `urls` ([]string): A slice of URLs or domains to clone. - `opts` (CloneOptions): Options for the cloning process. - `Serve` (bool): Whether to serve the site locally after cloning. - `Open` (bool): Whether to auto-open the site in the default browser. - `ServePort` (int): The port to use for serving the site (default: 5000). - `Cookies` ([]string): A slice of cookie strings (e.g., `"session=abc123"`). - `Proxy` (string): The proxy string to use (e.g., `"socks5://127.0.0.1:9050"`). - `UserAgent` (string): The custom User-Agent string. ### Example ```go package main import ( "context" "fmt" "log" "os/signal" "os" "github.com/goclone-dev/goclone/cmd" ) func main() { ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt) defer stop() opts := cmd.CloneOptions{ Serve: false, Open: false, ServePort: 5000, Cookies: []string{"session=abc123"}, Proxy: "", UserAgent: "Mozilla/5.0 (compatible; Goclone/1.0)", } err := cmd.CloneSite(ctx, []string{"https://example.com"}, opts) if err != nil { log.Fatalf("clone failed: %v", err) } fmt.Println("Clone complete. Files saved to ./example.com/") } ``` ``` -------------------------------- ### Initialize Project Directory with CreateProject Source: https://context7.com/goclone-dev/goclone/llms.txt CreateProject initializes a new site's directory structure, including subdirectories for CSS, JS, and images, along with an empty index.html file. ```go package main import ( "fmt" "github.com/goclone-dev/goclone/pkg/file" ) func main() { projectPath := file.CreateProject("example.com") fmt.Println(projectPath) // Output: /home/user/projects/example.com // Created structure: // /home/user/projects/example.com/ // /home/user/projects/example.com/index.html // /home/user/projects/example.com/css/ // /home/user/projects/example.com/js/ // /home/user/projects/example.com/imgs/ } ``` -------------------------------- ### Build Goclone from Source Source: https://github.com/goclone-dev/goclone/blob/master/README.md Clone the repository, build the goclone binary, and optionally move it to your PATH. Requires Go version 1.20 or higher. ```bash # Clone the repository git clone https://github.com/goclone-dev/goclone.git cd goclone # Build and run go build -o goclone cmd/goclone/main.go # Move binary to a directory in your PATH (optional) mv goclone /usr/local/bin/ ``` -------------------------------- ### Goclone CLI with All Flags Source: https://context7.com/goclone-dev/goclone/llms.txt Demonstrates cloning a website using Goclone with various optional flags for serving, proxy, user agent, and cookies. ```bash goclone https://example.com \ --serve \ # Serve locally after cloning --open \ # Auto-open in default browser --servePort 8080 \ # Custom serve port (default: 5000) --proxy_string "socks5://127.0.0.1:9050" \ # Route through proxy --user_agent "Mozilla/5.0 ..." \ # Custom User-Agent header --cookie "session=abc123; token=xyz" # Pre-set cookies ``` -------------------------------- ### Basic Goclone Usage Source: https://github.com/goclone-dev/goclone/blob/master/README.md Download a website to your local directory by providing the URL. Goclone mirrors the site's structure and files. ```bash # goclone goclone https://configtree.co ``` -------------------------------- ### Serve Cloned Site with Goclone CLI Source: https://context7.com/goclone-dev/goclone/llms.txt Clones a website and serves it locally. Options to specify the port and automatically open in the browser are available. Press Ctrl+C to stop the server. ```bash # Clone and immediately serve at http://localhost:5000 goclone https://example.com --serve # Clone, serve on port 9000, and open in browser automatically goclone https://example.com --serve --open --servePort 9000 # Press Ctrl+C to stop the server ``` -------------------------------- ### Build Goclone from Source Source: https://context7.com/goclone-dev/goclone/llms.txt Builds Goclone from source code by cloning the repository and running the Go build command. ```bash git clone https://github.com/goclone-dev/goclone.git cd goclone go build -o goclone cmd/goclone/main.go mv goclone /usr/local/bin/ ``` -------------------------------- ### Basic Website Clone with Goclone CLI Source: https://context7.com/goclone-dev/goclone/llms.txt Clones a website to a local directory named after the domain. Supports cloning by full URL or just the domain name. ```bash # Clone by full URL goclone https://example.com # Output: creates ./example.com/index.html, ./example.com/css/, ./example.com/js/, ./example.com/imgs/ # Clone by domain (https:// is prepended automatically) goclone example.com # Output: creates ./example.com/ with the same structure ``` -------------------------------- ### Clone with All Flags (CLI) Source: https://context7.com/goclone-dev/goclone/llms.txt Demonstrates cloning a website with various optional flags for serving, proxying, user agent, and cookies. ```APIDOC ## Clone Website with Flags (CLI) ### Description Clones a website with various optional flags for serving, proxying, user agent, and cookies. ### Usage ```bash goclone [flags] ``` ### Flags - `--serve`: Serve locally after cloning. - `--open`: Auto-open in default browser after serving. - `--servePort `: Custom port for serving (default: 5000). - `--proxy_string `: Route through a proxy (e.g., `socks5://127.0.0.1:9050`). - `--user_agent `: Custom User-Agent header. - `--cookie ""`: Pre-set cookies (e.g., `session=abc123; token=xyz`). ### Example ```bash goclone https://example.com \ --serve \ --open \ --servePort 8080 \ --proxy_string "socks5://127.0.0.1:9050" \ --user_agent "Mozilla/5.0 ..." \ --cookie "session=abc123; token=xyz" ``` ``` -------------------------------- ### Version Subcommand (CLI) Source: https://context7.com/goclone-dev/goclone/llms.txt Prints the current build version information of the Goclone utility. ```APIDOC ## Version (CLI) ### Description Prints the current build version information of the Goclone utility. ### Usage ```bash goclone version ``` ``` -------------------------------- ### Serve Cloned Site (CLI) Source: https://context7.com/goclone-dev/goclone/llms.txt Clones a website and optionally serves it locally, with options to specify the port and auto-open in the browser. ```APIDOC ## Serve Cloned Site (CLI) ### Description Clones a website and optionally serves it locally, with options to specify the port and auto-open in the browser. Press Ctrl+C to stop the server. ### Usage ```bash goclone --serve [flags] ``` ### Flags - `--serve`: Required to enable serving. - `--open`: Auto-open in default browser. - `--servePort `: Custom port for serving (default: 5000). ### Example ```bash # Clone and immediately serve at http://localhost:5000 goclone https://example.com --serve # Clone, serve on port 9000, and open in browser automatically goclone https://example.com --serve --open --servePort 9000 ``` ``` -------------------------------- ### Basic Website Clone (CLI) Source: https://context7.com/goclone-dev/goclone/llms.txt Clones a website to a local subdirectory named after the domain. HTTPS is automatically prepended if only the domain is provided. ```APIDOC ## Clone Website (CLI) ### Description Clones a website to a local subdirectory named after the domain. HTTPS is automatically prepended if only the domain is provided. ### Usage ```bash goclone ``` ### Example ```bash # Clone by full URL goclone https://example.com # Output: creates ./example.com/index.html, ./example.com/css/, ./example.com/js/, ./example.com/imgs/ # Clone by domain (https:// is prepended automatically) goclone example.com # Output: creates ./example.com/ with the same structure ``` ``` -------------------------------- ### Goclone Version Subcommand Source: https://context7.com/goclone-dev/goclone/llms.txt Retrieves the current build version information for the Goclone utility. ```bash goclone version # Prints current build version information ``` -------------------------------- ### Programmatic Website Cloning with Go API Source: https://context7.com/goclone-dev/goclone/llms.txt Uses the `CloneSite` function from the Goclone Go API to clone a website. Supports context for cancellation and various options like cookies and user agent. ```go package main import ( "context" "fmt" "log" "os/signal" "os" "github.com/goclone-dev/goclone/cmd" ) func main() { ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt) defer stop() opts := cmd.CloneOptions{ Serve: false, Open: false, ServePort: 5000, Cookies: []string{"session=abc123"}, Proxy: "", UserAgent: "Mozilla/5.0 (compatible; Goclone/1.0)", } err := cmd.CloneSite(ctx, []string{"https://example.com"}, opts) if err != nil { log.Fatalf("clone failed: %v", err) } fmt.Println("Clone complete. Files saved to ./example.com/") } // Output directory structure: // ./example.com/ // index.html // css/ // js/ // imgs/ ``` -------------------------------- ### Format HTML with FormatHTML Source: https://context7.com/goclone-dev/goclone/llms.txt Use FormatHTML to read an HTML file and pretty-print its content in place, overwriting the original file with formatted output. ```go package main import "github.com/goclone-dev/goclone/pkg/html" func main() { // Before: ./example.com/index.html is minified or unindented html.FormatHTML("./example.com/index.html") // After: ./example.com/index.html is pretty-printed with proper indentation } ``` -------------------------------- ### Asset Crawling Engine with Go API Source: https://context7.com/goclone-dev/goclone/llms.txt Utilizes the `crawler.Crawl` function to download website assets. It accepts a project path, cookie jar, proxy string, and user agent. The project path is created using `file.CreateProject`. ```go package main import ( "context" "log" "net/http/cookiejar" "github.com/goclone-dev/goclone/pkg/crawler" "github.com/goclone-dev/goclone/pkg/file" ) func main() { jar, _ := cookiejar.New(nil) projectPath := file.CreateProject("example.com") // Creates: ./example.com/, ./example.com/css/, ./example.com/js/, ./example.com/imgs/ err := crawler.Crawl( context.Background(), "https://example.com", projectPath, jar, "", // proxy: empty = direct connection "", // userAgent: empty = Colly default ) if err != nil { log.Fatalf("crawl error: %v", err) } // After: ./example.com/index.html contains raw HTML // ./example.com/css/*.css downloaded // ./example.com/js/*.js downloaded // ./example.com/imgs/*.{png,jpg,gif,svg} downloaded } ``` -------------------------------- ### Check File Existence and Read Content Source: https://context7.com/goclone-dev/goclone/llms.txt Utilize Exists to verify if a file or directory is present and GetFileContent to read its content. GetFileContent returns an empty string for non-existent files. ```go package main import ( "fmt" "github.com/goclone-dev/goclone/pkg/file" ) func main() { path := "./example.com/index.html" if file.Exists(path) { content := file.GetFileContent(path) fmt.Printf("File content (%d bytes):\n%s\n", len(content), content[:200]) } else { fmt.Println("File does not exist yet") } } ``` -------------------------------- ### Download HTML Page with HTMLExtractor Source: https://context7.com/goclone-dev/goclone/llms.txt Use HTMLExtractor to download the raw HTML of a given URL and save it to index.html within a project directory. TLS verification is disabled. ```go package main import ( "log" "github.com/goclone-dev/goclone/pkg/crawler" "github.com/goclone-dev/goclone/pkg/file" ) func main() { projectPath := file.CreateProject("mysite") err := crawler.HTMLExtractor("https://example.com", projectPath) if err != nil { log.Fatalf("HTML extraction failed: %v", err) } // Output: ./mysite/index.html now contains the raw HTML of https://example.com } ``` -------------------------------- ### Parse URL Filename and Extension with Go Source: https://context7.com/goclone-dev/goclone/llms.txt Use `parser.URLFilename` and `parser.URLExtension` to extract filename and extension from a URL. `parser.PathFilename` can be used for path-based extraction. ```go package main import ( "fmt" "github.com/goclone-dev/goclone/pkg/parser" ) func main() { url := "https://cdn.example.com/assets/styles/main.css?v=3" filename := parser.URLFilename(url) fmt.Println(filename) // main.css ext := parser.URLExtension(url) fmt.Println(ext) // .css // Path-based variant pathFile := parser.PathFilename("/js/vendor/analytics.js") fmt.Println(pathFile) // analytics.js } ``` -------------------------------- ### Validate URLs and Domains with Parser Functions Source: https://context7.com/goclone-dev/goclone/llms.txt Use ValidateURL and ValidateDomain for input validation. Helper functions CreateURL and GetDomain are also provided for URL manipulation. ```go package main import ( "fmt" "github.com/goclone-dev/goclone/pkg/parser" ) func main() { fmt.Println(parser.ValidateURL("https://example.com")) // true fmt.Println(parser.ValidateURL("example.com")) // false fmt.Println(parser.ValidateDomain("example.com")) // true fmt.Println(parser.ValidateDomain("example")) // false fmt.Println(parser.ValidateDomain("https://example.com")) // false // Convert domain to full URL url := parser.CreateURL("example.com") fmt.Println(url) // https://example.com // Extract domain from full URL domain := parser.GetDomain("https://sub.example.com/path?q=1") fmt.Println(domain) // sub.example.com } ``` -------------------------------- ### Pulling a New Branch Source: https://github.com/goclone-dev/goclone/blob/master/docs/CONTRIBUTING.md After creating a new branch on GitHub, use this command to pull it down to your local development environment. ```bash git pull ``` -------------------------------- ### Rewrite HTML Links with LinkRestructure Source: https://context7.com/goclone-dev/goclone/llms.txt Employ LinkRestructure to parse an index.html file and rewrite script, link, and image source attributes to use local relative paths for offline browsing. ```go package main import ( "log" "github.com/goclone-dev/goclone/pkg/html" ) func main() { // Before: index.html contains // // // Logo err := html.LinkRestructure("./example.com") if err != nil { log.Fatalf("link restructure failed: %v", err) } // After: index.html is rewritten to // // // Logo } ``` -------------------------------- ### crawler.Crawl - Asset Crawling Engine Source: https://context7.com/goclone-dev/goclone/llms.txt Drives the async Colly-based scraper that downloads all CSS, JS, and image assets linked from the target page. It accepts an optional cookie jar, proxy string, and user agent. ```APIDOC ## crawler.Crawl (Go API) ### Description `Crawl` drives the async Colly-based scraper that downloads all CSS, JS, and image assets linked from the target page. It accepts an optional cookie jar, proxy string, and user agent. ### Signature ```go func Crawl(ctx context.Context, url string, projectPath string, jar http.CookieJar, proxy string, userAgent string) error ``` ### Parameters - `ctx` (context.Context): Context for cancellation support. - `url` (string): The starting URL to crawl. - `projectPath` (string): The local directory path to save assets. - `jar` (http.CookieJar): An optional cookie jar. - `proxy` (string): The proxy string to use (empty string means direct connection). - `userAgent` (string): The custom User-Agent string (empty string uses Colly default). ### Example ```go package main import ( "context" "log" "net/http/cookiejar" "github.com/goclone-dev/goclone/pkg/crawler" "github.com/goclone-dev/goclone/pkg/file" ) func main() { jar, _ := cookiejar.New(nil) projectPath := file.CreateProject("example.com") // Creates: ./example.com/, ./example.com/css/, ./example.com/js/, ./example.com/imgs/ err := crawler.Crawl( context.Background(), "https://example.com", projectPath, jar, "", // proxy: empty = direct connection "", // userAgent: empty = Colly default ) if err != nil { log.Fatalf("crawl error: %v", err) } // After: ./example.com/index.html contains raw HTML // ./example.com/css/*.css downloaded // ./example.com/js/*.js downloaded // ./example.com/imgs/*.{png,jpg,gif,svg} downloaded } ``` ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.