### Build and Run Site Generator - Bash Source: https://context7.com/avelino/awesome-go/llms.txt This section provides bash commands for building and deploying the static site generator locally. It outlines the necessary steps to get the site up and running on a local development environment. ```bash # Commands for building and running the site generator locally would go here. ``` -------------------------------- ### Fetch Repository Metadata with Caching Source: https://context7.com/avelino/awesome-go/llms.txt Retrieves repository metadata from GitHub or GitLab APIs with a 7-day file-based cache. It includes rate limiting and environment variable checks to skip fetching. ```go func fetchProjectMeta(projects []*Project) error { if os.Getenv("AWESOME_SKIP_FETCH") != "" { return nil } token := os.Getenv("GITHUB_TOKEN") client := &http.Client{Timeout: 10 * time.Second} for _, p := range projects { cached, err := readCachedMeta(p) if err == nil && cached != nil { p.Meta = cached continue } var meta *RepoMeta switch p.Host { case "github": meta = fetchGitHubMeta(client, p.Owner, p.Repo, token) case "gitlab": meta = fetchGitLabMeta(client, p.URL) } if meta != nil { meta.FetchedAt = time.Now().Format("2006-01-02") p.Meta = meta writeCachedMeta(p, meta) } time.Sleep(50 * time.Millisecond) } return nil } ``` -------------------------------- ### Validate Entry Format - Go Source: https://context7.com/avelino/awesome-go/llms.txt The TestSeparator function in Go validates that all README entries adhere to the required format, specifically checking for proper description separators and punctuation. It reads the README.md file, splits it into lines, and uses regular expressions to ensure links are followed by a description ending with a period. ```go var ( reContainsLink = regexp.MustCompile(`\* \[.*\](.*)`) reOnlyLink = regexp.MustCompile(`\* \[.*\]\([^()]*\)$`) reLinkWithDescription = regexp.MustCompile(`\* \[.*\](.*\) - \S.*[\.\!]`) ) func TestSeparator(t *testing.T) { input, _ := os.ReadFile("README.md") lines := strings.Split(string(input), "\n") for _, line := range lines { line = strings.Trim(line, " ") containsLink := reContainsLink.MatchString(line) if containsLink { noDescription := reOnlyLink.MatchString(line) if noDescription { continue } matched := reLinkWithDescription.MatchString(line) if !matched { t.Errorf("expected `* [link] - description.`, got '%s'", line) } } } } ``` -------------------------------- ### Fetch GitHub Repository Metadata using Go Source: https://context7.com/avelino/awesome-go/llms.txt The `fetchGitHubMeta` function retrieves repository metadata from the GitHub REST API. It requires an HTTP client, owner, repository name, and an optional GitHub token for authentication. It returns a RepoMeta struct containing stars, forks, license, language, topics, last push date, open issues, and archived status, or nil if an error occurs. ```go func fetchGitHubMeta(client *http.Client, owner, repo, token string) *RepoMeta { apiURL := fmt.Sprintf("https://api.github.com/repos/%s/%s", owner, repo) req, _ := http.NewRequest("GET", apiURL, nil) req.Header.Set("Accept", "application/vnd.github.v3+json") if token != "" { req.Header.Set("Authorization", "Bearer "+token) } resp, err := client.Do(req) if err != nil || resp.StatusCode != http.StatusOK { return nil } defer resp.Body.Close() var ghRepo struct { StargazersCount int `json:"stargazers_count"` ForksCount int `json:"forks_count"` License *struct { SpdxID string `json:"spdx_id"` } `json:"license"` Language string `json:"language"` Topics []string `json:"topics"` PushedAt string `json:"pushed_at"` OpenIssuesCount int `json:"open_issues_count"` Archived bool `json:"archived"` } json.NewDecoder(resp.Body).Decode(&ghRepo) return &RepoMeta{ Stars: ghRepo.StargazersCount, Forks: ghRepo.ForksCount, License: ghRepo.License.SpdxID, Language: ghRepo.Language, Topics: ghRepo.Topics, LastPush: ghRepo.PushedAt[:10], // Format: "2024-01-15" OpenIssues: ghRepo.OpenIssuesCount, Archived: ghRepo.Archived, } } // Set GITHUB_TOKEN environment variable for higher rate limits: // export GITHUB_TOKEN=ghp_xxxxxxxxxxxxxxxxxxxx // go run main.go ``` -------------------------------- ### Parse GitHub and GitLab Repository URLs using Go Source: https://context7.com/avelino/awesome-go/llms.txt The `parseRepoURL` function parses a raw URL string to extract the host, owner, and repository name for GitHub and GitLab. It returns these values along with a boolean indicating success. Errors during URL parsing or if the host is not recognized will result in `false` being returned. ```go func parseRepoURL(rawURL string) (host, owner, repo string, ok bool) { u, err := url.Parse(rawURL) if err != nil { return "", "", "", false } parts := strings.Split(strings.Trim(u.Path, "/"), "/") switch u.Hostname() { case "github.com": if len(parts) < 2 { return "", "", "", false } return "github", parts[0], parts[1], true case "gitlab.com": if len(parts) < 2 { return "", "", "", false } return "gitlab", parts[0], parts[len(parts)-1], true } return "", "", "", false } // Examples: // parseRepoURL("https://github.com/gin-gonic/gin") // Returns: "github", "gin-gonic", "gin", true // parseRepoURL("https://gitlab.com/group/subgroup/project") // Returns: "gitlab", "group", "project", true // parseRepoURL("https://example.com/other") // Returns: "", "", "", false ``` -------------------------------- ### Orchestrate Static Site Generation in Go Source: https://context7.com/avelino/awesome-go/llms.txt The buildStaticSite function manages the end-to-end generation process. It handles directory creation, markdown-to-HTML conversion, category parsing, metadata fetching from external APIs, and asset deployment. ```go func buildStaticSite() error { if err := dropCreateDir("out/"); err != nil { return fmt.Errorf("drop-create out dir: %w", err) } if err := renderIndex("README.md", "out/index.html"); err != nil { return fmt.Errorf("convert markdown to html: %w", err) } input, _ := os.ReadFile("out/index.html") doc, _ := goquery.NewDocumentFromReader(bytes.NewReader(input)) categories, _ := extractCategories(doc) projects := buildProjects(categories) fetchProjectMeta(projects) renderCategories(categories) renderProjects(projects) rewriteLinksInIndex(doc, categories) renderSitemap(categories, projects) cp.Copy("tmpl/assets", "out/assets") cp.Copy("tmpl/robots.txt", "out/robots.txt") return nil } ``` -------------------------------- ### Convert Markdown to HTML in Go Source: https://context7.com/avelino/awesome-go/llms.txt The ToHTML function utilizes the goldmark library to parse markdown into HTML. It supports GitHub Flavored Markdown (GFM) and custom heading ID generation for consistent anchor links. ```go func ToHTML(markdown []byte) ([]byte, error) { md := goldmark.New( goldmark.WithExtensions(extension.GFM), goldmark.WithParserOptions( parser.WithAutoHeadingID(), ), goldmark.WithRendererOptions( html.WithXHTML(), html.WithUnsafe(), ), ) ctx := parser.NewContext( parser.WithIDs(&IDGenerator{}), ) var buf bytes.Buffer if err := md.Convert(markdown, &buf, parser.WithContext(ctx)); err != nil { return nil, err } return buf.Bytes(), nil } ``` -------------------------------- ### Extract Categories from HTML using Goquery Source: https://context7.com/avelino/awesome-go/llms.txt Parses an HTML document to identify and map categories, their links, and descriptions. It relies on the goquery library to traverse the DOM and extract specific category structures. ```go func extractCategories(doc *goquery.Document) (map[string]Category, error) { categories := make(map[string]Category) contentsHeading := doc.Find("body #contents") contentsList := contentsHeading.NextFiltered("ul") if contentsList.Length() == 0 { contentsList = contentsHeading.NextFiltered("details").Find("ul").First() } contentsList.Find("ul").Each(func(_ int, selUl *goquery.Selection) { selUl.Find("li a").Each(func(_ int, s *goquery.Selection) { selector, exists := s.Attr("href") if !exists { return } category, err := extractCategory(doc, selector) if err != nil { return } categories[selector] = *category }) }) return categories, nil } ``` -------------------------------- ### Detect Duplicate Links in README Test Suite using Go Source: https://context7.com/avelino/awesome-go/llms.txt The `TestDuplicatedLinks` function is a Go test that verifies the uniqueness of package URLs listed in a README.md file. It iterates through all the links, storing them in a map to detect any duplicates. If a duplicate link is found, the test fails, ensuring that each entry in the project list is unique. ```go func TestDuplicatedLinks(t *testing.T) { doc := goqueryFromReadme(t) links := make(map[string]bool) doc.Find("body li > a:first-child").Each(func(_ int, s *goquery.Selection) { t.Run(s.Text(), func(t *testing.T) { href, ok := s.Attr("href") if !ok { t.Error("expected to have href") } if links[href] { t.Fatalf("duplicated link '%s'", href) } links[href] = true }) }) } // Run: go test -run TestDuplicatedLinks -v // Ensures unique URLs across all ~2000 entries ``` -------------------------------- ### Validate Alphabetical Order in README Test Suite using Go Source: https://context7.com/avelino/awesome-go/llms.txt The `TestAlpha` function is part of a Go test suite that validates the alphabetical ordering of entries within categories in a README.md file. It uses the `goquery` library to parse the HTML and checks each list (`