### Install gotreesitter Source: https://github.com/odvcencio/gotreesitter/blob/main/README.md Command to install the gotreesitter library using go get. ```sh go get github.com/odvcencio/gotreesitter ``` -------------------------------- ### GoParser - Create and Use Parsers in Go Source: https://context7.com/odvcencio/gotreesitter/llms.txt Demonstrates how to create a parser for a specific language (Go in this example) using GoTreeSitter. It shows parsing source code, accessing the root node, and checking for errors. Requires the 'gotreesitter' and 'gotreesitter/grammars' packages. ```Go package main import ( "fmt" "github.com/odvcencio/gotreesitter" "github.com/odvcencio/gotreesitter/grammars" ) func main() { // Get a language from the registry lang := grammars.GoLanguage() // Create a parser for that language parser := gotreesitter.NewParser(lang) // Parse source code src := []byte(`package main func main() { fmt.Println("Hello, World!") } `) tree, err := parser.Parse(src) if err != nil { panic(err) } defer tree.Release() // Access the root node root := tree.RootNode() fmt.Printf("Root type: %s\n", root.Type(lang)) fmt.Printf("Children: %d\n", root.ChildCount()) fmt.Printf("Has errors: %v\n", root.HasError()) // Output: // Root type: source_file // Children: 2 // Has errors: false } ``` -------------------------------- ### Code Highlighting with GoTreeSitter Source: https://github.com/odvcencio/gotreesitter/blob/main/README.md Provides an example of using `gotreesitter.NewHighlighter` to generate highlighting information for source code. It initializes a highlighter with a language and a query, then calls `Highlight` to get a list of ranges with associated capture names. The output can be used to apply syntax highlighting to the source text. ```go hl, _ := gotreesitter.NewHighlighter(lang, highlightQuery) ranges := hl.Highlight(src) for _, r := range ranges { fmt.Printf("%s: %q\n", r.Capture, src[r.StartByte:r.EndByte]) } ``` -------------------------------- ### Generate DOT Output and Copy Trees in Go Source: https://context7.com/odvcencio/gotreesitter/llms.txt Demonstrates generating DOT graph output for tree visualization, writing DOT to a string builder, and creating an independent copy of a tree for concurrent modification. It also shows how to apply edits to the original tree without affecting the copy and how to get a root node with an offset. ```go package main import ( "fmt" "strings" "github.com/odvcencio/gotreesitter" "github.com/odvcencio/gotreesitter/grammars" ) func main() { lang := grammars.GoLanguage() parser := gotreesitter.NewParser(lang) src := []byte(`package main func main() {} `) tree, _ := parser.Parse(src) defer tree.Release() // Generate DOT graph for visualization dot := tree.DOT(lang) fmt.Printf("DOT output (first 200 chars):\n%s...\n", dot[:min(200, len(dot))]) // Write DOT to a writer var sb strings.Builder tree.WriteDOT(&sb, lang) // Copy tree for independent modification treeCopy := tree.Copy() defer treeCopy.Release() // Edit original without affecting copy tree.Edit(gotreesitter.InputEdit{ StartByte: 0, OldEndByte: 7, NewEndByte: 7, }) fmt.Printf("Original has changes: %v\n", tree.RootNode().HasChanges()) fmt.Printf("Copy has changes: %v\n", treeCopy.RootNode().HasChanges()) // Get root with offset (for embedding in larger documents) offsetRoot := tree.RootNodeWithOffset(100, gotreesitter.Point{Row: 5, Column: 0}) fmt.Printf("Offset root start: byte=%d, row=%d\n", offsetRoot.StartByte(), offsetRoot.StartPoint().Row) } ``` -------------------------------- ### Typed Query Example with gotreesitter Source: https://github.com/odvcencio/gotreesitter/blob/main/README.md Illustrates the usage of type-safe Go wrappers generated by tsquery. It shows how to instantiate a generated query and iterate through its typed matches. ```go type FunctionDeclarationMatch struct { Name *gotreesitter.Node Body *gotreesitter.Node } q, _ := queries.NewGoFunctionsQuery(lang) cursor := q.Exec(tree.RootNode(), lang, src) for { match, ok := cursor.Next() if !ok { break } fmt.Println(match.Name.Text(src)) } ``` -------------------------------- ### Go Syntax Highlighting with Highlighter Source: https://context7.com/odvcencio/gotreesitter/llms.txt Demonstrates how to use the Highlighter to perform syntax highlighting on source code. It covers initialization, obtaining highlight ranges, and performing incremental highlighting after code edits. Dependencies include the gotreesitter library and grammars. ```go package main import ( "fmt" "github.com/odvcencio/gotreesitter" "github.com/odvcencio/gotreesitter/grammars" ) func main() { entry := grammars.DetectLanguage("example.go") lang := entry.Language() // Create highlighter with the language's highlight query highlighter, err := gotreesitter.NewHighlighter(lang, entry.HighlightQuery) if err != nil { panic(err) } src := []byte(`package main import "fmt" func main() { x := 42 fmt.Println(x) } `) // Get highlight ranges ranges := highlighter.Highlight(src) fmt.Printf("Highlight ranges: %d\n", len(ranges)) for _, r := range ranges[:10] { // Show first 10 text := string(src[r.StartByte:r.EndByte]) fmt.Printf(" [%d:%d] %s = %q\n", r.StartByte, r.EndByte, r.Capture, text) } // Incremental highlighting after edits tree, _ := gotreesitter.NewParser(lang).Parse(src) // Edit source newSrc := append(src[:len(src)-2], []byte("\n y := 100\n}")...) tree.Edit(gotreesitter.InputEdit{ StartByte: uint32(len(src) - 2), OldEndByte: uint32(len(src)), NewEndByte: uint32(len(newSrc)), StartPoint: gotreesitter.Point{Row: 6, Column: 0}, OldEndPoint: gotreesitter.Point{Row: 7, Column: 1}, NewEndPoint: gotreesitter.Point{Row: 8, Column: 1}, }) // Re-highlight incrementally newRanges, newTree := highlighter.HighlightIncremental(newSrc, tree) defer newTree.Release() fmt.Printf("New highlight ranges: %d\n", len(newRanges)) } ``` -------------------------------- ### Running GoTreeSitter Benchmarks Source: https://github.com/odvcencio/gotreesitter/blob/main/README.md Provides the command-line instructions to run the GoTreeSitter benchmarks locally. It includes commands for running pure Go benchmarks, CGo binding benchmarks, and native C benchmarks, along with instructions for generating a benchmark matrix for multi-workload tracking. ```sh # Pure Go (this repo): GOMAXPROCS=1 go test . -run '^$' \ -bench 'BenchmarkGoParseFullDFA|BenchmarkGoParseIncrementalSingleByteEditDFA|BenchmarkGoParseIncrementalNoEditDFA' \ -benchmem -count=10 -benchtime=1s # CGo binding benchmarks: cd cgo_harness GOMAXPROCS=1 go test . -run '^$' -tags treesitter_c_bench \ -bench 'BenchmarkCTreeSitterGoParseFull|BenchmarkCTreeSitterGoParseIncrementalSingleByteEdit|BenchmarkCTreeSitterGoParseIncrementalNoEdit' \ -benchmem -count=10 -benchtime=750ms # Native C benchmarks (no Go, direct C binary): ./pure_c/run_go_benchmark.sh 500 2000 20000 ``` ```sh go run ./cmd/benchmatrix --count 10 ``` -------------------------------- ### Run Head-to-Head Comparison and Claim Suite Source: https://github.com/odvcencio/gotreesitter/blob/main/cgo_harness/README.md Runs comparative benchmarks between Go and C implementations, including multi-language matrices and full 3-way claim suites. ```bash ./pure_c/run_go_head_to_head.sh ./pure_c/run_matrix_head_to_head.sh ./pure_c/run_claim_suite.sh RUNS=7 SIZES="500 2000 10000" CFLAGS_EXTRA="-march=native -flto" ./pure_c/run_claim_suite.sh ``` -------------------------------- ### Basic Parsing with gotreesitter Source: https://github.com/odvcencio/gotreesitter/blob/main/README.md Demonstrates basic parsing of Go source code using the gotreesitter library. It initializes a parser with the Go language grammar and parses a given source byte slice. ```go import ( "fmt" "github.com/odvcencio/gotreesitter" "github.com/odvcencio/gotreesitter/grammars" ) func main() { src := []byte(`package main func main() {} `) lang := grammars.GoLanguage() parser := gotreesitter.NewParser(lang) tree, _ := parser.Parse(src) fmt.Println(tree.RootNode()) } ``` -------------------------------- ### Perform Incremental Parsing with GoTreeSitter Source: https://context7.com/odvcencio/gotreesitter/llms.txt Demonstrates how to perform an initial parse, record a byte-level edit on the syntax tree, and execute an incremental reparse to optimize performance. It also shows how to identify changed ranges between the original and updated trees. ```go package main import ( "fmt" "github.com/odvcencio/gotreesitter" "github.com/odvcencio/gotreesitter/grammars" ) func main() { lang := grammars.GoLanguage() parser := gotreesitter.NewParser(lang) // Initial parse src := []byte(`package main func main() {} `) tree, _ := parser.Parse(src) // Simulate user typing "x" at byte offset 28 (inside {}) newSrc := make([]byte, 0, len(src)+1) newSrc = append(newSrc, src[:28]...) newSrc = append(newSrc, 'x') newSrc = append(newSrc, src[28:]...) // Record the edit on the tree tree.Edit(gotreesitter.InputEdit{ StartByte: 28, OldEndByte: 28, NewEndByte: 29, StartPoint: gotreesitter.Point{Row: 2, Column: 14}, OldEndPoint: gotreesitter.Point{Row: 2, Column: 14}, NewEndPoint: gotreesitter.Point{Row: 2, Column: 15}, }) // Incrementally reparse - reuses unchanged subtrees newTree, _ := parser.ParseIncremental(newSrc, tree) fmt.Printf("New root: %s\n", newTree.RootNode().Type(lang)) fmt.Printf("Stop reason: %s\n", newTree.ParseStopReason()) // Check parse runtime diagnostics rt := newTree.ParseRuntime() fmt.Printf("Tokens consumed: %d\n", rt.TokensConsumed) fmt.Printf("Truncated: %v\n", rt.Truncated) // Get changed ranges between old and new tree changedRanges := gotreesitter.DiffChangedRanges(tree, newTree) for _, r := range changedRanges { fmt.Printf("Changed: [%d, %d)\n", r.StartByte, r.EndByte) } tree.Release() newTree.Release() } ``` -------------------------------- ### Run All Harness Modes with Go Source: https://github.com/odvcencio/gotreesitter/blob/main/cgo_harness/HARNESS_FRAMEWORK.md Executes the full local harness, including root tests, cgo parity checks, and performance benchmarks, using the `harnessgate` command. ```shell go run ./cmd/harnessgate -mode all ``` -------------------------------- ### Querying Syntax Trees with gotreesitter Source: https://github.com/odvcencio/gotreesitter/blob/main/README.md Shows how to create and execute queries on a parsed syntax tree using gotreesitter. It defines a query for function declarations and iterates through the matches. ```go q, _ := gotreesitter.NewQuery(`(function_declaration name: (identifier) @fn)`, lang) cursor := q.Exec(tree.RootNode(), lang, src) for { match, ok := cursor.NextMatch() if !ok { break } for _, cap := range match.Captures { fmt.Println(cap.Node.Text(src)) } } ``` -------------------------------- ### Run C Baseline Benchmarks Source: https://github.com/odvcencio/gotreesitter/blob/main/cgo_harness/README.md Executes C-based tree-sitter benchmarks with specific Go test tags and benchmarking flags. ```bash GOMAXPROCS=1 go test . -run '^$' -tags treesitter_c_bench -bench 'BenchmarkCTreeSitterGoParseFull|BenchmarkCTreeSitterGoParseIncrementalSingleByteEdit|BenchmarkCTreeSitterGoParseIncrementalNoEdit' -benchmem -count=10 -benchtime=750ms ``` -------------------------------- ### Run Performance Harness with Baseline Comparison in Go Source: https://github.com/odvcencio/gotreesitter/blob/main/cgo_harness/HARNESS_FRAMEWORK.md Executes the performance tests and compares the results against an existing baseline output file using the `harnessgate` command. ```shell go run ./cmd/harnessgate -mode perf -bench-base harness_out/base.txt ``` -------------------------------- ### Pattern Matching with S-Expressions and Queries in Go Source: https://context7.com/odvcencio/gotreesitter/llms.txt Illustrates how to use the Query type in GoTreeSitter to compile tree-sitter query patterns and execute them against syntax trees. It covers finding nodes, capturing specific parts of matches, using a streaming cursor API, and implementing queries with predicates. ```Go package main import ( "fmt" "github.com/odvcencio/gotreesitter" "github.com/odvcencio/gotreesitter/grammars" ) func main() { lang := grammars.GoLanguage() parser := gotreesitter.NewParser(lang) src := []byte(`package main func add(a, b int) int { return a + b } func multiply(x, y int) int { return x * y } `) tree, _ := parser.Parse(src) defer tree.Release() // Compile a query to find function declarations query, err := gotreesitter.NewQuery(` (function_declaration name: (identifier) @fn_name parameters: (parameter_list) @params result: (_) @return_type ) @function `, lang) if err != nil { panic(err) } // Execute query - returns all matches matches := query.Execute(tree) fmt.Printf("Found %d functions\n", len(matches)) for _, m := range matches { fmt.Printf("Pattern %d matched:\n", m.PatternIndex) for _, cap := range m.Captures { fmt.Printf(" @%s: %q\n", cap.Name, cap.Node.Text(src)) } } // Streaming cursor API for large trees cursor := query.Exec(tree.RootNode(), lang, src) cursor.SetMatchLimit(10) // Limit matches for { match, ok := cursor.NextMatch() if !ok { break } for _, cap := range match.Captures { if cap.Name == "fn_name" { fmt.Printf("Function: %s\n", cap.Node.Text(src)) } } } // Query with predicates callQuery, _ := gotreesitter.NewQuery(` (call_expression function: (identifier) @fn (#match? @fn "^Print")) `, lang) callMatches := callQuery.Execute(tree) fmt.Printf("Print calls: %d\n", len(callMatches)) } ``` -------------------------------- ### Run Pure-C Runtime Benchmarks Source: https://github.com/odvcencio/gotreesitter/blob/main/cgo_harness/README.md Executes performance benchmarks against the native C runtime without using Go cgo bindings. ```bash ./pure_c/run_matrix.sh ./pure_c/run_go_benchmark.sh ./pure_c/run_go_benchmark.sh 500 2000 20000 CFLAGS_EXTRA="-march=native -flto" ./pure_c/run_go_benchmark.sh ``` -------------------------------- ### Benchmarking GoTreeSitter Performance Source: https://github.com/odvcencio/gotreesitter/blob/main/README.md Presents benchmark results comparing GoTreeSitter's performance against native C and CGo bindings for full parsing, incremental parsing with a single-byte edit, and incremental parsing with no edit. The results highlight GoTreeSitter's efficiency, especially for incremental operations. ```text goos: linux goarch: amd64 cpu: Intel(R) Core(TM) Ultra 9 285 | Runtime | Full parse | Incremental (1-byte edit) | Incremental (no edit) | |---|---:|---:|---:| | Native C (pure C runtime) | 1.76 ms | 102.3 μs | 101.7 μs | | CGo binding (C runtime via cgo) | ~2.0 ms | ~130 μs | — | | gotreesitter (pure Go) | 4.20 ms | 1.49 μs | 2.18 ns | ``` -------------------------------- ### Go Symbol Extraction with Tagger Source: https://context7.com/odvcencio/gotreesitter/llms.txt Illustrates how to use the Tagger to extract symbols (definitions and references) from source code using tree-sitter tags queries. It shows how to initialize the Tagger, extract tags from source code, and mentions incremental tagging. Dependencies include the gotreesitter library and grammars. ```go package main import ( "fmt" "github.com/odvcencio/gotreesitter" "github.com/odvcencio/gotreesitter/grammars" ) func main() { entry := grammars.DetectLanguage("example.go") lang := entry.Language() // Get the tags query (may need to resolve) tagsQuery := grammars.ResolveTagsQuery(*entry) // Create tagger tagger, err := gotreesitter.NewTagger(lang, tagsQuery) if err != nil { panic(err) } src := []byte(`package main type User struct { Name string Age int } func NewUser(name string) *User { return &User{Name: name} } func (u *User) Greet() string { return "Hello, " + u.Name } `) // Extract tags tags := tagger.Tag(src) fmt.Printf("Found %d tags:\n", len(tags)) for _, tag := range tags { fmt.Printf(" %s: %s at %d:%d\n", tag.Kind, tag.Name, tag.NameRange.StartPoint.Row+1, tag.NameRange.StartPoint.Column+1) } // Incremental tagging tree, _ := gotreesitter.NewParser(lang).Parse(src) // ... edit tree ... // newTags, newTree := tagger.TagIncremental(newSrc, tree) tree.Release() } ``` -------------------------------- ### Source Rewriting with gotreesitter Source: https://github.com/odvcencio/gotreesitter/blob/main/README.md Shows how to perform source code modifications using gotreesitter's Rewriter. It demonstrates replacing, inserting, and deleting nodes, and applying these changes to generate new source and edits for incremental reparsing. ```go rw := gotreesitter.NewRewriter(src) rw.Replace(funcNameNode, []byte("newName")) rw.InsertBefore(bodyNode, []byte("// added\n")) rw.Delete(unusedNode) newSrc, _ := rw.ApplyToTree(tree) newTree, _ := parser.ParseIncremental(newSrc, tree) ``` -------------------------------- ### Configure External Grammar Blobs - Go Build Source: https://github.com/odvcencio/gotreesitter/blob/main/README.md Builds the Go project with external grammar blobs, requiring a specified directory for grammar files and optionally disabling memory mapping on Unix-like systems. ```sh go build -tags grammar_blobs_external GOTREESITTER_GRAMMAR_BLOB_DIR=/path/to/blobs # required GOTREESITTER_GRAMMAR_BLOB_MMAP=false # disable mmap (Unix only) ``` -------------------------------- ### Run Corpus Parity (dump.v1) Source: https://github.com/odvcencio/gotreesitter/blob/main/cgo_harness/README.md Compares GoTreeSitter against the native C oracle, generates 'dump.v1' artifacts, writes JSONL results, and updates the PARITY.md scoreboard. Supports specifying languages, corpus directories, output paths, and artifact modes. ```go go run -tags treesitter_c_parity ./cmd/corpus_parity \ --lang top10 \ --corpus ./corpus \ --out ./parity_out/results.jsonl \ --artifact-dir ./parity_out/dump_v1 \ --artifact-mode failures \ --scoreboard ./PARITY.md ``` -------------------------------- ### GoGrammars - Automatic Language Detection Source: https://context7.com/odvcencio/gotreesitter/llms.txt Illustrates how to use the 'grammars' package in GoTreeSitter for automatic language detection based on filename, extension, shebang, or display name. It supports lazy loading of grammars and lists available languages. Requires the 'gotreesitter/grammars' package. ```Go package main import ( "fmt" "github.com/odvcencio/gotreesitter/grammars" ) func main() { // Detect by filename/extension entry := grammars.DetectLanguage("main.go") if entry != nil { fmt.Printf("Detected: %s\n", entry.Name) lang := entry.Language() // Lazy loads grammar fmt.Printf("Symbol count: %d\n", lang.SymbolCount) } // Detect by shebang entry = grammars.DetectLanguageByShebang("#!/usr/bin/env python3") if entry != nil { fmt.Printf("Shebang detected: %s\n", entry.Name) } // Detect by display name or alias entry = grammars.DetectLanguageByName("JavaScript") if entry != nil { fmt.Printf("By name: %s\n", entry.Name) } // List all available languages (metadata only, no loading) all := grammars.AllLanguages() fmt.Printf("Available languages: %d\n", len(all)) // Get display name entry = grammars.DetectLanguage("example.rs") displayName := grammars.DisplayName(entry) fmt.Printf("Display name: %s\n", displayName) // Output: // Detected: go // Symbol count: 213 // Shebang detected: python // By name: javascript // Available languages: 206 // Display name: Rust } ``` -------------------------------- ### Initialize Java Main Class Source: https://github.com/odvcencio/gotreesitter/blob/main/cgo_harness/pure_c/samples/java.txt Defines a standard Java Main class with a main method that prints a greeting to the console. This serves as the entry point for Java applications. ```java public class Main { public static void main(String[] args) { System.out.println("hello"); } } ``` -------------------------------- ### Build Real Corpus using Go Source: https://github.com/odvcencio/gotreesitter/blob/main/cgo_harness/HARNESS_FRAMEWORK.md Command to build the real corpus for testing using the `build_real_corpus` tool. It specifies the manifest file and the output directory. ```shell go run ./cgo_harness/cmd/build_real_corpus \ -profile cgo_harness/cmd/build_real_corpus/top50_manifest.json \ -out cgo_harness/corpus_real ``` -------------------------------- ### Incremental Reparsing with GoTreeSitter Source: https://github.com/odvcencio/gotreesitter/blob/main/README.md Demonstrates how to perform incremental reparsing of source code after an edit. It shows the process of updating the source, applying an edit to the existing tree, and then using `ParseIncremental` to efficiently re-parse only the modified section. This method reuses unchanged subtrees, significantly improving performance for frequent edits. ```go tree, _ := parser.Parse(src) // User types "x" at byte offset 42 src = append(src[:42], append([]byte("x"), src[42:]...)...) tree.Edit(gotreesitter.InputEdit{ StartByte: 42, OldEndByte: 42, NewEndByte: 43, StartPoint: gotreesitter.Point{Row: 3, Column: 10}, OldEndPoint: gotreesitter.Point{Row: 3, Column: 10}, NewEndPoint: gotreesitter.Point{Row: 3, Column: 11}, }) tree2, _ := parser.ParseIncremental(src, tree) ``` -------------------------------- ### Navigate Syntax Trees with TreeCursor Source: https://context7.com/odvcencio/gotreesitter/llms.txt Explains how to use the TreeCursor type for efficient O(1) navigation through a syntax tree. It covers moving between parents, children, and siblings, as well as navigating by field names, byte positions, and copying cursors for parallel traversal. ```go package main import ( "fmt" "github.com/odvcencio/gotreesitter" "github.com/odvcencio/gotreesitter/grammars" ) func main() { lang := grammars.GoLanguage() parser := gotreesitter.NewParser(lang) src := []byte(`package main import "fmt" func main() { fmt.Println("Hello") } `) tree, _ := parser.Parse(src) defer tree.Release() // Create cursor from tree cursor := gotreesitter.NewTreeCursorFromTree(tree) // Start at root fmt.Printf("Root: %s\n", cursor.CurrentNodeType()) fmt.Printf("Depth: %d\n", cursor.Depth()) // Navigate to first child if cursor.GotoFirstChild() { fmt.Printf("First child: %s\n", cursor.CurrentNodeType()) } // Navigate to next sibling if cursor.GotoNextSibling() { fmt.Printf("Next sibling: %s\n", cursor.CurrentNodeType()) } // Navigate by field name cursor.Reset(tree.RootNode()) cursor.GotoFirstChild() // package_clause cursor.GotoNextSibling() // import_declaration cursor.GotoNextSibling() // function_declaration if cursor.GotoChildByFieldName("body") { fmt.Printf("Function body: %s\n", cursor.CurrentNodeType()) } // Navigate to first named child (skip anonymous tokens) cursor.Reset(tree.RootNode()) if cursor.GotoFirstNamedChild() { fmt.Printf("First named child: %s\n", cursor.CurrentNodeType()) } // Navigate by byte position cursor.Reset(tree.RootNode()) idx := cursor.GotoFirstChildForByte(50) // Find child containing byte 50 if idx >= 0 { fmt.Printf("Child at byte 50: %s\n", cursor.CurrentNodeType()) } // Get field information cursor.Reset(tree.RootNode()) cursor.GotoFirstNamedChild() cursor.GotoNextNamedSibling() cursor.GotoNextNamedSibling() // function_declaration cursor.GotoFirstChild() fieldName := cursor.CurrentFieldName() fmt.Printf("Field name: %s\n", fieldName) // Copy cursor for parallel traversal cursor2 := cursor.Copy() cursor2.GotoParent() fmt.Printf("Copy at: %s\n", cursor2.CurrentNodeType()) } ``` -------------------------------- ### Configure Parser Options: Timeout, Cancellation, and Ranges Source: https://context7.com/odvcencio/gotreesitter/llms.txt Illustrates how to configure the GoTreeSitter parser with advanced options including setting a parsing timeout, enabling cancellation via context, and specifying included ranges for partial parsing. This is useful for performance optimization and handling large or dynamic codebases. ```go package main import ( "context" "fmt" "time" "github.com/odvcencio/gotreesitter" "github.com/odvcencio/gotreesitter/grammars" ) func main() { lang := grammars.GoLanguage() parser := gotreesitter.NewParser(lang) src := []byte(`package main func main() {} `) // Parse with timeout tree, err := parser.ParseWith(src, gotreesitter.WithTimeout(5*time.Second), ) if err != nil { panic(err) } rt := tree.ParseRuntime() fmt.Printf("Stop reason: %s\n", rt.StopReason) tree.Release() // Parse with cancellation ctx, cancel := context.WithCancel(context.Background()) go func() { time.Sleep(100 * time.Millisecond) cancel() }() tree2, _ := parser.ParseWith(src, gotreesitter.WithCancellation(ctx), ) if tree2 != nil { fmt.Printf("Parse completed: %s\n", tree2.ParseStopReason()) tree2.Release() } // Parse specific ranges (for embedded languages) ranges := []gotreesitter.Range{ {StartByte: 0, EndByte: 12}, // "package main" } parser.SetIncludedRanges(ranges) tree3, _ := parser.Parse(src) if tree3 != nil { fmt.Printf("Partial parse root: %s\n", tree3.RootNode().Type(lang)) tree3.Release() } // Output: // Stop reason: accepted // Parse completed: accepted // Partial parse root: source_file } ``` -------------------------------- ### Verify Gotreesitter Implementation Source: https://github.com/odvcencio/gotreesitter/blob/main/README.md Verifies the gotreesitter implementation by running a parity report and unit tests for grammars. This command ensures that the implemented parsers are correct and consistent with expected behavior. ```Go go run ./cmd/parity_report && go test ./grammars/... ``` -------------------------------- ### Accessing Syntax Tree Structure with Node in Go Source: https://context7.com/odvcencio/gotreesitter/llms.txt Demonstrates how to use the Node type in GoTreeSitter to traverse syntax trees, access node properties like byte ranges and types, and extract source text. It covers navigating children, named children, field-based navigation, sibling traversal, and descendant lookup. ```Go package main import ( "fmt" "github.com/odvcencio/gotreesitter" "github.com/odvcencio/gotreesitter/grammars" ) func main() { lang := grammars.GoLanguage() parser := gotreesitter.NewParser(lang) src := []byte(`package main func add(a, b int) int { return a + b } `) tree, _ := parser.Parse(src) defer tree.Release() root := tree.RootNode() // Navigate children for i := 0; i < root.ChildCount(); i++ { child := root.Child(i) fmt.Printf("Child %d: %s [%d:%d]\n", i, child.Type(lang), child.StartByte(), child.EndByte()) } // Access named children only fmt.Printf("Named children: %d\n", root.NamedChildCount()) funcDecl := root.NamedChild(1) // Skip package clause // Field-based navigation funcName := funcDecl.ChildByFieldName("name", lang) if funcName != nil { fmt.Printf("Function name: %s\n", funcName.Text(src)) } // Get field name for a child for i := 0; i < funcDecl.ChildCount(); i++ { fieldName := funcDecl.FieldNameForChild(i, lang) if fieldName != "" { fmt.Printf("Child %d field: %s\n", i, fieldName) } } // Sibling navigation if sibling := funcDecl.PrevSibling(); sibling != nil { fmt.Printf("Previous sibling: %s\n", sibling.Type(lang)) } // Descendant lookup by byte range node := root.DescendantForByteRange(35, 38) // "add" fmt.Printf("Descendant at 35-38: %s = %q\n", node.Type(lang), node.Text(src)) // S-expression output fmt.Printf("S-expr: %s\n", funcDecl.SExpr(lang)) } ``` -------------------------------- ### Execute Performance Benchmarks Source: https://github.com/odvcencio/gotreesitter/blob/main/AGENTS.md Runs Go performance benchmarks with specific stable settings, including GOMAXPROCS, count, benchtime, and memory profiling. It targets specific benchmark functions for parsing and incremental edits. ```bash GOMAXPROCS=1 go test . -run '^$' -bench 'BenchmarkGoParseFullDFA|BenchmarkGoParseIncrementalSingleByteEditDFA|BenchmarkGoParseIncrementalNoEditDFA' -benchmem -count=10 -benchtime=750ms ``` -------------------------------- ### Configure Curated Language Set - Go Build Source: https://github.com/odvcencio/gotreesitter/blob/main/README.md Builds the Go project with a curated, embedded grammar set for a smaller binary size. It allows runtime restriction to specific languages. ```sh go build -tags grammar_set_core # curated Core100 embedded grammar set GOTREESITTER_GRAMMAR_SET=go,json,python # runtime restriction ``` -------------------------------- ### Navigating Syntax Trees with TreeCursor Source: https://github.com/odvcencio/gotreesitter/blob/main/README.md Illustrates the usage of `TreeCursor` for efficient traversal of a syntax tree. It shows how to move between nodes (first child, named child, next sibling, parent) and access node information like type and byte offset. The cursor allows O(1) movement with zero allocations. Note that cursors need to be recreated after tree modifications. ```go c := gotreesitter.NewTreeCursorFromTree(tree) c.GotoFirstChild() c.GotoChildByFieldName("body") for ok := c.GotoFirstNamedChild(); ok; ok = c.GotoNextNamedSibling() { fmt.Printf("%s at %d\n", c.CurrentNodeType(), c.CurrentNode().StartByte()) } idx := c.GotoFirstChildForByte(128) ``` -------------------------------- ### Run Correctness Harness Only with Go Source: https://github.com/odvcencio/gotreesitter/blob/main/cgo_harness/HARNESS_FRAMEWORK.md Executes only the correctness tests within the harness framework using the `harnessgate` command. ```shell go run ./cmd/harnessgate -mode correctness ``` -------------------------------- ### Run Focused Grammargen Targets Source: https://github.com/odvcencio/gotreesitter/blob/main/cgo_harness/README.md Uses Docker to run grammargen targets for specific languages, isolating OOM risks by processing one language at a time. ```bash bash cgo_harness/docker/run_grammargen_focus_targets.sh --mode real-corpus --langs typescript bash cgo_harness/docker/run_grammargen_focus_targets.sh --mode cgo --langs typescript ``` -------------------------------- ### Parse Multi-Language Documents with InjectionParser Source: https://context7.com/odvcencio/gotreesitter/llms.txt Demonstrates how to use InjectionParser to handle documents containing multiple languages, such as HTML with embedded JavaScript and CSS. It covers registering languages, defining injection queries, and parsing the source code. ```go package main import ( "fmt" "github.com/odvcencio/gotreesitter" "github.com/odvcencio/gotreesitter/grammars" ) func main() { // Create injection parser ip := gotreesitter.NewInjectionParser() // Register languages htmlEntry := grammars.DetectLanguage("index.html") jsEntry := grammars.DetectLanguage("script.js") cssEntry := grammars.DetectLanguage("style.css") ip.RegisterLanguage("html", htmlEntry.Language()) ip.RegisterLanguage("javascript", jsEntry.Language()) ip.RegisterLanguage("css", cssEntry.Language()) // Register injection query for HTML injectionQuery := ` (script_element (raw_text) @injection.content (#set! injection.language "javascript")) (style_element (raw_text) @injection.content (#set! injection.language "css")) ` err := ip.RegisterInjectionQuery("html", injectionQuery) if err != nil { panic(err) } // Parse multi-language document src := []byte(` `) result, err := ip.Parse(src, "html") if err != nil { panic(err) } defer result.Tree.Release() fmt.Printf("Parent tree root: %s\n", result.Tree.RootNode().Type(htmlEntry.Language())) fmt.Printf("Injections found: %d\n", len(result.Injections)) for _, inj := range result.Injections { fmt.Printf(" Language: %s\n", inj.Language) if inj.Tree != nil { lang := grammars.DetectLanguageByName(inj.Language).Language() fmt.Printf(" Root: %s\n", inj.Tree.RootNode().Type(lang)) } for _, r := range inj.Ranges { fmt.Printf(" Range: [%d, %d)\n", r.StartByte, r.EndByte) } } // Incremental reparsing with child tree reuse // result2, _ := ip.ParseIncremental(newSrc, "html", result) // Output: // Parent tree root: document // Injections found: 2 // Language: css // Root: stylesheet // Range: [45, 75) // Language: javascript // Root: program // Range: [109, 142) } ``` -------------------------------- ### Run Docker Parity Test (Bash) Source: https://github.com/odvcencio/gotreesitter/blob/main/docs/superpowers/plans/2026-03-13-grammargen-functional-parity.md This bash script executes a Docker container to run grammar parity tests for a specific target grammar. It sets various environment variables to configure the test run, including corpus paths, profiling options, and limits, before executing the Go test command. ```bash SEED_DIR="/home/draco/work/gotreesitter/.claude/worktrees/grammargen-pr9-resume/cgo_harness/grammar_seed" GRAMMAR="scala" # repeat for each target grammar docker run --rm \ --memory 16g --cpus 4 \ -v /tmp/gts-grammargen-rebase:/workspace:ro \ -v "$SEED_DIR":/tmp/grammar_parity:ro \ -w /workspace \ -e GTS_GRAMMARGEN_REAL_CORPUS_ENABLE=1 \ -e GTS_GRAMMARGEN_REAL_CORPUS_ROOT=/tmp/grammar_parity \ -e GTS_GRAMMARGEN_REAL_CORPUS_PROFILE=aggressive \ -e GTS_GRAMMARGEN_REAL_CORPUS_MAX_CASES=25 \ -e GTS_GRAMMARGEN_REAL_CORPUS_ALLOW_PARTIAL=1 \ -e GTS_GRAMMARGEN_REAL_CORPUS_ONLY=$GRAMMAR \ -e GTS_GRAMMARGEN_REAL_CORPUS_FLOORS_PATH=/workspace/grammargen/testdata/real_corpus_parity_floors.json \ golang:1.24-bookworm \ go test ./grammargen -run '^TestMultiGrammarImportRealCorpusParity$' -count=1 -v -timeout 90m \ 2>&1 | tee /tmp/rr-fix-$GRAMMAR.log ``` -------------------------------- ### Register Custom Languages in Go Source: https://context7.com/odvcencio/gotreesitter/llms.txt Shows how to register custom language grammars with GoTreeSitter, either by providing an extension entry for grammargen-generated languages or a full LangEntry for more control. This enables the parser to recognize and process files with custom extensions. ```go package main import ( "fmt" "github.com/odvcencio/gotreesitter" "github.com/odvcencio/gotreesitter/grammars" ) func main() { // Register a custom language extension grammars.RegisterExtension(grammars.ExtensionEntry{ Name: "mylang", Extensions: []string{'.mylang', '.ml'}, Aliases: []string{"ml", "mylang"}, GenerateLanguage: func() (*gotreesitter.Language, error) { // In practice, use grammargen.GenerateLanguage(grammar) // For demo, return nil return nil, fmt.Errorf("not implemented") }, HighlightQuery: "(identifier) @variable", InheritHighlights: "go", // Inherit Go highlights }) // Now DetectLanguage will recognize .mylang files entry := grammars.DetectLanguage("test.mylang") if entry != nil { fmt.Printf("Detected custom language: %s\n", entry.Name) } // Register with full LangEntry for more control grammars.Register(grammars.LangEntry{ Name: "custom", Extensions: []string{'.custom'}, Language: func() *gotreesitter.Language { // Return your Language instance return nil }, GrammarSource: grammars.GrammarSourceGrammargen, HighlightQuery: "(comment) @comment", }) } ``` -------------------------------- ### Test Execution Commands Source: https://github.com/odvcencio/gotreesitter/blob/main/docs/superpowers/plans/2026-03-13-grammargen-functional-parity.md Standard commands for verifying compilation and committing changes to the grammargen repository. ```bash cd /tmp/gts-grammargen-rebase && go test -run '^$' -count=1 ./grammargen cd /tmp/gts-grammargen-rebase && buckley commit --yes --minimal-output ``` -------------------------------- ### Run Parallel Worktree Experiments Source: https://github.com/odvcencio/gotreesitter/blob/main/cgo_harness/README.md Executes multiple bounded container experiments across different worktrees in parallel. It preserves per-experiment artifacts and metadata. Dependencies include the 'run_parity_experiments.sh' script. Inputs are experiment paths, max parallel jobs, memory, and CPU limits. Custom commands can also be provided. ```shell chmod +x cgo_harness/docker/run_parity_experiments.sh cgo_harness/docker/run_parity_experiments.sh \ --experiment main=/home/me/work/gotreesitter \ --experiment glr-a=/home/me/work/gts-glr-a \ --experiment glr-b=/home/me/work/gts-glr-b \ --max-parallel 2 \ --memory 6g \ --cpus 2 ``` ```shell cgo_harness/docker/run_parity_experiments.sh \ --experiment scala=/home/me/work/gts-scala \ --max-parallel 1 \ -- "cd /workspace/cgo_harness && GTS_PARITY_SCALA_REALWORLD_STRICT=1 go test . -tags treesitter_c_parity -run '^TestParityScalaRealWorldCorpus$' -count=1 -v" ``` -------------------------------- ### Multi-language Document Parsing with gotreesitter Source: https://github.com/odvcencio/gotreesitter/blob/main/README.md Demonstrates parsing documents containing embedded languages like HTML with JavaScript and CSS. It involves registering languages and injection queries with an InjectionParser. ```go ip := gotreesitter.NewInjectionParser() ip.RegisterLanguage("html", htmlLang) ip.RegisterLanguage("javascript", jsLang) ip.RegisterLanguage("css", cssLang) ip.RegisterInjectionQuery("html", injectionQuery) result, _ := ip.Parse(source, "html") for _, inj := range result.Injections { fmt.Printf("%s: %d ranges\n", inj.Language, len(inj.Ranges)) // inj.Tree is the child language's parse tree } ``` -------------------------------- ### Code Tagging with GoTreeSitter Source: https://github.com/odvcencio/gotreesitter/blob/main/README.md Demonstrates how to use `gotreesitter.NewTagger` for code tagging, which can be used for features like code navigation or auto-completion. It first detects the language of a file, then creates a tagger using the language's tags query. The `Tag` method returns a list of tags, each containing information about the code element's kind, name, and location. ```go entry := grammars.DetectLanguage("main.go") lang := entry.Language() tagge, _ := gotreesitter.NewTagger(lang, entry.TagsQuery) tags := tagger.Tag(src) for _, tag := range tags { fmt.Printf("%s %s at %d:%d\n", tag.Kind, tag.Name, tag.NameRange.StartPoint.Row, tag.NameRange.StartPoint.Column) } ``` -------------------------------- ### Verify Compilation After LR Splitting Change Source: https://github.com/odvcencio/gotreesitter/blob/main/docs/superpowers/plans/2026-03-13-grammargen-functional-parity.md Runs a Go test command to verify that the code compiles correctly after the changes made to enable LR splitting by default. ```bash cd /tmp/gts-grammargen-rebase && go test -run '^$' -count=1 ./grammargen ``` -------------------------------- ### Run Single Grammar Parity Test - Docker Source: https://github.com/odvcencio/gotreesitter/blob/main/README.md Executes a Docker command to run a single grammar parity test for a specified language, useful for local correctness and parity work. ```sh bash cgo_harness/docker/run_single_grammar_parity.sh typescript ``` -------------------------------- ### Add Weighted Confidence Gate to Correctness Harness in Go Source: https://github.com/odvcencio/gotreesitter/blob/main/cgo_harness/HARNESS_FRAMEWORK.md Runs the correctness harness with real corpus data and applies a weighted confidence gate using a built-in profile and minimum confidence threshold. ```shell go run ./cmd/harnessgate -mode correctness \ -real-corpus-dir cgo_harness/corpus_real \ -real-corpus-langs top10 \ -confidence-profile core90 \ -confidence-min 0.90 ``` -------------------------------- ### Quality Check for Real Corpus Manifest in Go Source: https://github.com/odvcencio/gotreesitter/blob/main/cgo_harness/HARNESS_FRAMEWORK.md Shell command to perform a quality check on the real corpus manifest using Go tests. It requires navigating to the `cgo_harness` directory and setting an environment variable. ```shell cd cgo_harness GTS_REAL_CORPUS_MANIFEST=corpus_real/manifest.json \ go test . -run TestRealCorpusManifestQuality -count=1 ``` -------------------------------- ### Run Single Grammar Parity Tests Source: https://github.com/odvcencio/gotreesitter/blob/main/AGENTS.md Initiates parity tests for a single specified grammar within a Dockerized environment. This helps isolate issues related to a particular grammar. ```bash bash cgo_harness/docker/run_single_grammar_parity.sh ``` -------------------------------- ### Run Grammagen Focus Targets - Docker Source: https://github.com/odvcencio/gotreesitter/blob/main/README.md Executes Docker commands to focus grammagen on specific targets, either for real-corpus parity or CGO parity, for a given language. These runs are optimized for safety and resource efficiency. ```sh # Real-corpus parity for one grammar bash cgo_harness/docker/run_single_grammar_parity.sh typescript # Focused grammargen real-corpus lane for one language bash cgo_harness/docker/run_grammargen_focus_targets.sh --mode real-corpus --langs typescript # Focused grammargen-vs-C lane for one language bash cgo_harness/docker/run_grammargen_focus_targets.sh --mode cgo --langs typescript ``` -------------------------------- ### Run Grammargen Focus Targets Source: https://github.com/odvcencio/gotreesitter/blob/main/AGENTS.md Executes grammargen focus targets in Docker, either with real-corpus or cgo modes for specified languages. This is used for broader correctness and parity checks. ```bash bash cgo_harness/docker/run_grammargen_focus_targets.sh --mode real-corpus --langs ``` ```bash bash cgo_harness/docker/run_grammargen_focus_targets.sh --mode cgo --langs ```