### Query File Example (highlights.scm) Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/06-configuration-guide.md An example of a Tree-sitter query file for syntax highlighting. ```scm ; Identifier naming conventions (identifier) @variable ((identifier) @constructor (#match? @constructor "^[A-Z]")) ; Function definitions (function_definition name: (identifier) @function) ; Literals [ (integer) (float) ] @number ``` -------------------------------- ### Query File Example (tags.scm) Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/06-configuration-guide.md An example of a Tree-sitter query file for code navigation (tags). ```scm (class_definition name: (identifier) @name) @definition.class (function_definition name: (identifier) @name) @definition.function (call function: [ (identifier) @name (attribute attribute: (identifier) @name) ]) @reference.call ``` -------------------------------- ### Incremental Parsing Example Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/06-configuration-guide.md Example demonstrating incremental parsing in Python by providing the old tree. ```python tree = parser.parse(new_code, old_tree) ``` -------------------------------- ### Python Usage with tree-sitter Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/06-configuration-guide.md Example of how to use the tree-sitter-python package when the `tree-sitter` library is installed. ```python # WITH tree-sitter installed: from tree_sitter import Language, Parser import tree_sitter_python language = tree_sitter_python.language() parser = Parser(language) tree = parser.parse(code) ``` -------------------------------- ### Rust Build Script Example Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/06-configuration-guide.md A simplified example of the Rust build script (`bindings/rust/build.rs`) used to compile the C parser files. ```rust // Simplified example fn main() { let parser_dir = "src"; cc::Build::new() .file(format!("{}/parser.c", parser_dir)) .file(format!("{}/scanner.c", parser_dir)) .include(parser_dir) .compile("tree_sitter_python"); } ``` -------------------------------- ### Python Integration Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/06-configuration-guide.md Steps to install tree-sitter-python for Python projects and basic parser setup. ```bash pip install tree-sitter-python ``` ```python from tree_sitter import Language, Parser import tree_sitter_python language = tree_sitter_python.language() parser = Parser(language) tree = parser.parse(code.encode()) ``` -------------------------------- ### Custom Query Example Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/06-configuration-guide.md Example of defining a custom query in SCM format to find class definitions and their methods. ```scm ; Find all class definitions with their methods (class_definition name: (identifier) @class_name body: (block (function_definition name: (identifier) @method_name))) ``` -------------------------------- ### Query Optimization Example (Bad vs. Better) Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/06-configuration-guide.md Illustrates a less efficient way of filtering identifiers compared to a more optimized approach within the query itself. ```scm ; Bad: (identifier) @name (#match? @name "^test_") ; Better: ((identifier) @name (#match? @name "^test_")) ``` -------------------------------- ### Loading Custom Queries in Python Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/06-configuration-guide.md Example of loading a custom SCM query string in Python. ```python with open("custom.scm") as f: custom_query = f.read() query = Query(language, custom_query) ``` -------------------------------- ### Python Build Commands Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/06-configuration-guide.md Commands for building and installing the Python package. ```bash # Build in-place pip install -e . # Build wheel python -m build # Build for specific Python versions (with cibuildwheel) cibuildwheel --platform linux # Install from PyPI pip install tree-sitter-python ``` -------------------------------- ### Python Usage without tree-sitter Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/06-configuration-guide.md Example of how to use the tree-sitter-python package when the `tree-sitter` library is NOT installed, accessing the native binding directly. ```python # WITHOUT tree-sitter (native access only): import tree_sitter_python lang = tree_sitter_python.language() # Can pass to other bindings or use via cffi ``` -------------------------------- ### Loading Custom Queries in Rust Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/06-configuration-guide.md Example of loading a custom SCM query string in Rust. ```rust let custom_query = std::fs::read_to_string("custom.scm")?; let query = Query::new(&language, &custom_query)?; ``` -------------------------------- ### String Start Example Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/04-grammar-types.md Illustrates various opening quote types for string literals. ```python " or ' or """ or """ or r" or f" or b", etc. ``` -------------------------------- ### Installation Commands Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/CMakeLists.txt Installs the library, header files, pkgconfig file, and queries. ```cmake configure_file(bindings/c/tree-sitter-python.pc.in "${CMAKE_CURRENT_BINARY_DIR}/tree-sitter-python.pc" @ONLY) install(DIRECTORY "${CMAKE_CURRENT_SOURCE_DIR}/bindings/c/tree_sitter" DESTINATION "${CMAKE_INSTALL_INCLUDEDIR}" FILES_MATCHING PATTERN "*.h") install(FILES "${CMAKE_CURRENT_BINARY_DIR}/tree-sitter-python.pc" DESTINATION "${CMAKE_INSTALL_LIBDIR}/pkgconfig") install(TARGETS tree-sitter-python LIBRARY DESTINATION "${CMAKE_INSTALL_LIBDIR}") file(GLOB QUERIES queries/*.scm) install(FILES ${QUERIES} DESTINATION "${CMAKE_INSTALL_DATADIR}/tree-sitter/queries/python") ``` -------------------------------- ### Rust Integration Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/06-configuration-guide.md Steps to add tree-sitter and tree-sitter-python to a Rust project and basic parser setup. ```bash cargo add tree-sitter tree-sitter-python ``` ```rust use tree_sitter::Parser; use tree_sitter_python::LANGUAGE; let mut parser = Parser::new(); parser.set_language(&LANGUAGE.into())?; let tree = parser.parse(code, None)?; ``` -------------------------------- ### JavaScript/Node.js Integration Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/06-configuration-guide.md Steps to add tree-sitter and tree-sitter-python to a JavaScript/Node.js project and basic parser setup. ```bash npm install tree-sitter tree-sitter-python ``` ```javascript const Parser = require("tree-sitter"); const Python = require("tree-sitter-python"); const parser = new Parser(); parser.setLanguage(Python); const tree = parser.parse(code); ``` -------------------------------- ### Return Statement Examples Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/04-grammar-types.md Examples of return statements. ```python return return value return a, b ``` -------------------------------- ### Print AST - Rust Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/07-usage-examples.md Example of printing the AST in Rust Debug format. ```Rust println!("{}", tree.root_node()); # Rust Debug format ``` -------------------------------- ### Installation Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/01-python-api.md Installs the tree-sitter-python package along with the core tree-sitter library. ```bash pip install tree-sitter-python tree-sitter ``` -------------------------------- ### Python Highlighting Query Example Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/05-query-types.md Example of how to use the HIGHLIGHTS_QUERY in Python. ```python import tree_sitter_python from tree_sitter import Language, Parser, Query language = tree_sitter_python.language() query_str = tree_sitter_python.HIGHLIGHTS_QUERY # Create a Query object parser = Parser(language) query = Query(language, query_str) # Parse code code = b"x = 42\nprint(x)" tree = parser.parse(code) ``` -------------------------------- ### Tuple Expression Examples Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/04-grammar-types.md Examples of implicit tuple creation in Python. ```python 1, 2, 3 x, y, z a, ``` -------------------------------- ### Global Statement Example Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/04-grammar-types.md Example of a global statement. ```python global x, y ``` -------------------------------- ### Usage example for Language Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/02-javascript-api.md This example demonstrates how to create a parser, set the Python language, and parse Python code. ```javascript const Parser = require("tree-sitter"); const PythonLanguage = require("tree-sitter-python"); // Create parser and set language const parser = new Parser(); parser.setLanguage(PythonLanguage); // Parse Python code const sourceCode = ` def greet(name): print(f"Hello, {name}!") `; const tree = parser.parse(sourceCode); console.log(tree.rootNode.type); // "module" ``` -------------------------------- ### Pattern 1: Extract All Definitions Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/07-usage-examples.md A Python example demonstrating how to extract all symbols (names and types) from a code string using Tree-sitter queries. ```python # Python def extract_all_symbols(code): query = Query(tree_sitter_python.language(), tree_sitter_python.TAGS_QUERY) tree = parser.parse(code.encode()) symbols = [] for match in query.matches(tree.root_node(), code.encode()): symbol = {'position': None, 'type': None} for capture in match.captures: name = query.capture_names()[capture.index] if name == '@name': symbol['name'] = capture.node.text.decode() elif name.startswith('@definition'): symbol['type'] = name.split('.')[-1] symbol['position'] = capture.node.start_point if 'name' in symbol: symbols.append(symbol) return symbols ``` -------------------------------- ### Raise Statement Examples Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/04-grammar-types.md Examples of raise statements. ```python raise raise ValueError("msg") raise exc from cause ``` -------------------------------- ### Assert Statement Examples Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/04-grammar-types.md Examples of assert statements. ```python assert condition assert x > 0, "x must be positive" ``` -------------------------------- ### Continue Statement Example Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/04-grammar-types.md Example of a continue statement. ```python continue ``` -------------------------------- ### Return statements Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/test/corpus/statements.txt Examples of return statements. ```python return return a + b, c return not b ``` -------------------------------- ### Print AST - JavaScript Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/07-usage-examples.md Example of printing the AST in Tree format for JavaScript code. ```JavaScript console.log(tree.rootNode.toString()); # Tree format ``` -------------------------------- ### language() Usage Example Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/01-python-api.md Demonstrates how to get the tree-sitter Language object for Python and use it with a Parser to parse Python code and inspect the root node. ```python from tree_sitter import Language, Parser import tree_sitter_python # Create a parser with Python language language = tree_sitter_python.language() parser = Parser(language) # Parse Python code code = """ def hello(name): print(f"Hello, {name}!") """ tree = parser.parse(code.encode('utf-8')) root_node = tree.root_node() # Query the tree def_node = root_node.child(0) print(f"Found {def_node.type} node") ``` -------------------------------- ### JavaScript Installation Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/00-README.md Install the tree-sitter and tree-sitter-python packages using npm. ```bash npm install tree-sitter tree-sitter-python ``` -------------------------------- ### Delete Statement Examples Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/04-grammar-types.md Examples of delete statements. ```python del x del a, b del obj.attr del list[0] ``` -------------------------------- ### Print AST - Python Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/07-usage-examples.md Example of printing the Abstract Syntax Tree (AST) in S-expression format for Python code. ```Python def print_ast(code): tree = parser.parse(code.encode()) print(tree.root_node().sexp()) # S-expression format ``` -------------------------------- ### Query Example Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/00-README.md An example of a tree-sitter query to match function definitions. ```scm (function_definition name: (identifier) @function_name) ``` -------------------------------- ### Sets Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/test/corpus/literals.txt Examples of set literals. ```python {a, b, c,} {*{}} ``` ```python (module (set (identifier) (identifier) (identifier)) (set (list_splat (dictionary)))) ``` -------------------------------- ### Usage example for NodeInfo Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/02-javascript-api.md This example shows how to access and log information about the 'function_definition' node type from the NodeInfo array. ```javascript const PythonLanguage = require("tree-sitter-python"); // Access node type information const functionDefInfo = PythonLanguage.nodeTypeInfo.find( info => info.type === "function_definition" ); if (functionDefInfo) { console.log("function_definition fields:"); Object.entries(functionDefInfo.fields).forEach(([name, field]) => { console.log(` ${name}: ${field.types.map(t => t.type).join("|")} (required: ${field.required})`); }); } ``` -------------------------------- ### Rust Installation Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/00-README.md Install the tree-sitter and tree-sitter-python packages using Cargo. ```bash cargo add tree-sitter tree-sitter-python ``` -------------------------------- ### Builtin Classes Example Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/test/corpus/pattern_matching.txt Example demonstrating pattern matching with built-in classes and actions. ```python for action in actions: match action: case {"text": str(message), "color": str(c)}: ui.set_text_color(c) ui.display(message) case {"sleep": float(duration)}: ui.wait(duration) case {"sound": str(url), "format": "ogg"}: ui.play(url) case {"sound": _, "format": _}: warning("Unsupported audio format") ``` -------------------------------- ### Simple Tuples Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/test/corpus/literals.txt Examples of simple tuple literals. ```python () (a, b) (a, b, c,) (print, exec) ``` ```python (module (tuple) (tuple (identifier) (identifier)) (tuple (identifier) (identifier) (identifier)) (tuple (identifier) (identifier))) ``` -------------------------------- ### f-string Interpolation Examples Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/04-grammar-types.md Examples demonstrating expression interpolation within f-strings. ```python f"Result: {x + 1}" f"Value={value!r:.2f}" ``` -------------------------------- ### For statements Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/test/corpus/statements.txt Examples of for statements. ```python for line, i in lines: print line for character, j in line: print character else: print x for x, in [(1,), (2,), (3,)]: x ``` -------------------------------- ### Type Parameter List Example Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/04-grammar-types.md Example of a generic type parameter list. ```python [T, U] [T: Bound, U] ``` -------------------------------- ### Function definitions Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/test/corpus/statements.txt Examples of various function definitions in Python. ```python def e((a,b)): return (a,b) def e(*list: str): pass def e(**list: str): pass def f(): nonlocal a def g(h, i, /, j, *, k=100, **kwarg): return h,i,j,k,kwarg def h(*a): i((*a)) j(((*a))) def foo(): pass \ \ \ ``` -------------------------------- ### If statements Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/test/corpus/statements.txt Example of if statements. ```python if a: b c ``` -------------------------------- ### Assert statements Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/test/corpus/statements.txt Examples of assert statements. ```python assert a assert b, c ``` -------------------------------- ### Visitor Pattern Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/07-usage-examples.md An example of using a visitor pattern to traverse the syntax tree and extract information like function and class definitions, as well as syntax errors. ```rust struct Analyzer { functions: Vec, classes: Vec, errors: Vec, } impl Analyzer { fn new() -> Self { Analyzer { functions: Vec::new(), classes: Vec::new(), errors: Vec::new(), } } fn analyze(&mut self, node: tree_sitter::Node, code: &str) { match node.kind() { "function_definition" => { for child in node.named_children(&mut node.walk()) { if child.kind() == "identifier" { let name = &code[child.byte_range()]; self.functions.push(name.to_string()); break; } } } "class_definition" => { for child in node.named_children(&mut node.walk()) { if child.kind() == "identifier" { let name = &code[child.byte_range()]; self.classes.push(name.to_string()); break; } } } "ERROR" => { let err = format!( "Syntax error at {}:வைக்{}", node.start_position().row, node.start_position().column ); self.errors.push(err); } _ => {} } let mut cursor = node.walk(); for child in node.children(&mut cursor) { self.analyze(child, code); } } } fn main() -> Result<(), Box> { let code = r#"# class Calculator: def add(self, a, b): return a + b def main(): calc = Calculator() print(calc.add(1, 2)) "#; let mut parser = Parser::new(); parser.set_language(&LANGUAGE.into())?; let tree = parser.parse(code, None)?; let mut analyzer = Analyzer::new(); analyzer.analyze(tree.root_node(), code); println!("Functions: {:?}", analyzer.functions); println!("Classes: {:?}", analyzer.classes); println!("Errors: {:?}", analyzer.errors); Ok(()) } ``` -------------------------------- ### Entry Point (bindings/node/index.js) Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/06-configuration-guide.md The main JavaScript entry point for the Node.js binding, handling loading of prebuilt binaries or compilation from source. ```javascript const root = require("path").join(__dirname, "..", ".."); module.exports = typeof process.versions.bun === "string" ? require(`../../prebuilds/${process.platform}-${process.arch}/tree-sitter-python.node`) : require("node-gyp-build")(root); try { module.exports.nodeTypeInfo = require("../../src/node-types.json"); } catch (_) {} ``` -------------------------------- ### Tree-sitter Query Language Example Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/00-README.md Provides an example of the Tree-sitter query language syntax. ```scm ; Match nodes by type (identifier) ; Capture nodes (identifier) @name ; Match with fields (function_definition name: (identifier) @func_name parameters: (parameters) @params) ; Alternation [ (integer) (float) ] ; Predicates ((identifier) @const (#match? @const "^[A-Z][A-Z_]*$")) ``` -------------------------------- ### Control-flow statements Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/test/corpus/statements.txt Example of while, break, and continue statements. ```python while true: pass break continue ``` -------------------------------- ### With statements Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/test/corpus/statements.txt Examples of with statements, including multiple context managers and aliasing. ```python with a as b: c ``` ```python with (open('d') as d, open('e') as e): f ``` ```python with e as f, g as h,: i ``` -------------------------------- ### Pattern 2: Breadth-First Search Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/07-usage-examples.md A JavaScript example implementing a Breadth-First Search (BFS) algorithm to traverse the syntax tree and find nodes matching a given predicate. ```javascript // JavaScript function bfs(rootNode, predicate) { const queue = [rootNode]; const results = []; while (queue.length > 0) { const node = queue.shift(); if (predicate(node)) { results.push(node); } queue.push(...node.children); } return results; } // Find all function_definition nodes const functions = bfs(tree.rootNode, n => n.type === "function_definition"); ``` -------------------------------- ### Import-from statements Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/test/corpus/statements.txt Examples of import-from statement syntaxes, including relative imports. ```python from a import b from a import * from a import (b, c) from a.b import c from . import b from .. import b from .a import b from ..a import b ``` -------------------------------- ### Using Highlights Query Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/07-usage-examples.md Demonstrates how to use the HIGHLIGHTS_QUERY to identify and categorize different parts of the code for syntax highlighting. ```rust fn highlight_code(code: &str) -> Result<(), Box> { use tree_sitter_python::HIGHLIGHTS_QUERY; let mut parser = Parser::new(); let language = LANGUAGE.into(); parser.set_language(&language)?; let tree = parser.parse(code, None)?; let query = Query::new(&language, HIGHLIGHTS_QUERY)?; let mut cursor = tree_sitter::QueryCursor::new(); for m in cursor.matches(&query, tree.root_node(), code.as_bytes()) { for capture in m.captures { let capture_name = &query.capture_names()[capture.index as usize]; let text = &code[capture.node.byte_range()]; println!(ירת: {:?}", capture_name, text); } } Ok(()) } ``` -------------------------------- ### Import statements Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/test/corpus/statements.txt Examples of various import statement syntaxes in Python. ```python import a, b import b.c as d import a.b.c ``` -------------------------------- ### Querying with Highlighting Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/07-usage-examples.md Demonstrates how to use Tree-sitter's querying capabilities to find specific syntax elements, using the HIGHLIGHTS_QUERY for syntax highlighting information. ```python from tree_sitter import Query # Get highlights query query_text = tree_sitter_python.HIGHLIGHTS_QUERY # Create and run query query = Query(tree_sitter_python.language(), query_text) code = b"DEBUG = True" tree = parser.parse(code) captures = query.captures(tree.root_node(), code) for node, capture_name in captures: print(f"{capture_name}: {node.text.decode()}") ``` -------------------------------- ### If else statements Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/test/corpus/statements.txt Examples of if, elif, and else statements. ```python if a: b elif c: d else: f if a: b else: f if a: b if a: b; c ``` -------------------------------- ### Field-Based Extraction Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/07-usage-examples.md Example of extracting variable names and their assigned values using a Tree-sitter query. ```Rust // Rust fn extract_assignments(code: &str) -> Result> { let mut parser = Parser::new(); parser.set_language(&LANGUAGE.into())?; let tree = parser.parse(code, None)?; let mut results = Vec::new(); let query = Query::new( &LANGUAGE.into(), r#"(assignment left: (identifier) @var right: (_) @value)""# )?; let mut cursor = tree_sitter::QueryCursor::new(); for m in cursor.matches(&query, tree.root_node(), code.as_bytes()) { let mut var_name = ""; let mut value_text = ""; for capture in &m.captures { let name = &query.capture_names()[capture.index as usize]; let text = &code[capture.node.byte_range()]; if name == "@var" { var_name = text; } else if name == "@value" { value_text = text; } } if !var_name.is_empty() && !value_text.is_empty() { results.push((var_name.to_string(), value_text.to_string())); } } Ok(results) } ``` -------------------------------- ### Python Installation Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/00-README.md Install the tree-sitter-python package using pip. The tree-sitter package is also required. ```bash pip install tree-sitter-python # Also requires: pip install tree-sitter ``` -------------------------------- ### Set Comprehensions Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/test/corpus/literals.txt Examples of set comprehensions. ```python {a[b][c] for a, b, c in items} {r for s in qs for n in ms} ``` ```python (module (set_comprehension (subscript (subscript (identifier) (identifier)) (identifier)) (for_in_clause (pattern_list (identifier) (identifier) (identifier)) (identifier))) (set_comprehension (identifier) (for_in_clause (identifier) (identifier)) (for_in_clause (identifier) (identifier)))) ``` -------------------------------- ### Basic Parser Setup Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/01-python-api.md Demonstrates how to initialize the parser, parse Python code, and access the root node of the parse tree. ```python from tree_sitter import Language, Parser import tree_sitter_python # Configure the parser language = tree_sitter_python.language() parser = Parser(language) # Parse code code = b"x = 1 + 2\nprint(x)" tree = parser.parse(code) # Access root node root = tree.root_node() print(f"Root type: {root.type}") print(f"Has errors: {root.has_error()}") ``` -------------------------------- ### Rust Build Commands Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/06-configuration-guide.md Common commands for building, testing, and documenting the Rust crate. ```bash # Build library car go build # Run tests car go test # Build docs car go doc --open # Publish (maintainers only) car go publish ``` -------------------------------- ### Querying AST Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/07-usage-examples.md Shows how to query the Abstract Syntax Tree (AST) using Tree-sitter queries to find specific code constructs like function definitions. ```rust use tree_sitter::Query; fn find_functions(code: &str) -> Result, Box> { let mut parser = Parser::new(); let language = LANGUAGE.into(); parser.set_language(&language)?; let tree = parser.parse(code, None)?; let query = Query::new( &language, r#"(function_definition name: (identifier) @name)"# )?; let mut cursor = tree_sitter::QueryCursor::new(); let mut functions = Vec::new(); for m in cursor.matches(&query, tree.root_node(), code.as_bytes()) { for capture in m.captures { let name_text = &code[capture.node.byte_range()]; functions.push(name_text.to_string()); } } Ok(functions) } fn main() -> Result<(), Box> { let code = r#" def add(a, b): return a + b def multiply(x, y): return x * y "#; let functions = find_functions(code)?; println!("Functions: {:?}", functions); Ok(()) } ``` -------------------------------- ### While statements Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/test/corpus/statements.txt Examples of while statements with and without else clauses. ```python while a: b while c: d else: e f ``` -------------------------------- ### Parameters Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/04-grammar-types.md Example of a function parameter list in Python. ```python (a, b, c=1, *args, **kwargs) ``` -------------------------------- ### Global statements Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/test/corpus/statements.txt Examples of global statements declaring one or more variables. ```python global a global a, b ``` -------------------------------- ### Delete statements Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/test/corpus/statements.txt Example of delete statements. ```python del a[1], b[2] ``` -------------------------------- ### Nonlocal Statement Example Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/04-grammar-types.md Example of a nonlocal statement. ```python nonlocal x, y ``` -------------------------------- ### Basic Parsing Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/07-usage-examples.md Demonstrates the basic process of initializing the parser, parsing Python code, and accessing the root node of the syntax tree. ```python from tree_sitter import Language, Parser import tree_sitter_python # Initialize parser language = tree_sitter_python.language() parser = Parser(language) # Parse Python code code = b""" def hello(name): print(f"Hello, {name}!") """ tree = parser.parse(code) root = tree.root_node() print(f"Root type: {root.type}") print(f"Root text: {root.text.decode()}") print(f"Has errors: {root.has_error()}") ``` -------------------------------- ### binding.gyp Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/06-configuration-guide.md Configuration for the native C++ module build using node-gyp. ```python { "targets": [{ "target_name": "tree_sitter_python", "sources": [ "src/parser.c", "src/scanner.c" ], "include_dirs": ["src"] }] } ``` -------------------------------- ### Dictionary Comprehensions Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/test/corpus/literals.txt Examples of dictionary comprehensions. ```python {a: b for a, b in items} {a: b for c in d for e in items} ``` ```python (module (dictionary_comprehension (pair (identifier) (identifier)) (for_in_clause (pattern_list (identifier) (identifier)) (identifier))) (dictionary_comprehension (pair (identifier) (identifier)) (for_in_clause (identifier) (identifier)) (for_in_clause (identifier) (identifier)))) ``` -------------------------------- ### Basic Parsing Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/07-usage-examples.md Demonstrates basic parsing of Python code using tree-sitter-python. ```rust use tree_sitter::Parser; use tree_sitter_python::LANGUAGE; fn main() -> Result<(), Box> { let mut parser = Parser::new(); parser.set_language(&LANGUAGE.into())?; let code = r#" def hello(name): print(f"Hello, {name}!") "#; let tree = parser.parse(code, None)?; let root = tree.root_node(); println!("Root type: {}", root.kind()); println!("Root text length: {}", root.utf8_byte_count()); Ok(()) } ``` -------------------------------- ### Print statements with redirection Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/test/corpus/statements.txt Examples of print statements with output redirection. ```python print >> a print >> a, "b", "c" ``` -------------------------------- ### Break Statement Example Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/04-grammar-types.md Example of a break statement. ```python break ``` -------------------------------- ### Future import statements Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/test/corpus/statements.txt Examples of future import statements. ```python from __future__ import print_statement from __future__ import python4 from __future__ import (absolute_import, division, print_function, unicode_literals) ``` -------------------------------- ### Pass Statement Example Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/04-grammar-types.md Example of a pass statement. ```python pass ``` -------------------------------- ### JavaScript: Basic Parsing Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/07-usage-examples.md Shows how to initialize the tree-sitter parser and parse Python code using the tree-sitter-python grammar in a Node.js environment. ```javascript const Parser = require("tree-sitter"); const Python = require("tree-sitter-python"); // Initialize parser const parser = new Parser(); parser.setLanguage(Python); // Parse Python code const code = ` def hello(name): print(f"Hello, {name}!") `; const tree = parser.parse(code); const root = tree.rootNode; console.log(`Root type: ${root.type}`); console.log(`Root text length: ${root.text.length}`); console.log(`Has error: ${root.hasError()}`); ``` -------------------------------- ### Python pyproject.toml Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/06-configuration-guide.md Configuration for the Python package, including build system requirements, project metadata, optional dependencies, and cibuildwheel settings. ```toml [build-system] requires = ["setuptools>=62.4.0", "wheel"] build-backend = "setuptools.build_meta" [project] name = "tree-sitter-python" version = "0.25.0" description = "Python grammar for tree-sitter" requires-python = ">=3.10" [project.optional-dependencies] core = ["tree-sitter~=0.24"] [tool.cibuildwheel] build = "cp310-*" build-frontend = "build" ``` -------------------------------- ### Tree Traversal with Cursor Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/07-usage-examples.md Illustrates how to traverse the syntax tree using a cursor to inspect nodes. ```rust fn print_tree(node: tree_sitter::Node, depth: usize) { let indent = " ".repeat(depth); println!( "{}{} : {}", indent, node.kind(), node.utf8_text(code.as_bytes()).unwrap(), ); let mut cursor = node.walk(); for child in node.children(&mut cursor) { print_tree(child, depth + 1); } } fn main() { let mut parser = Parser::new(); parser.set_language(&LANGUAGE.into()).unwrap(); let code = "x = 1 + 2"; let tree = parser.parse(code, None).unwrap(); print_tree(tree.root_node(), 0); } ``` -------------------------------- ### Yield Expression Examples Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/04-grammar-types.md Examples of yield expressions. ```python yield value yield from generator ``` -------------------------------- ### Finding Functions and Classes Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/07-usage-examples.md Illustrates how to use the TAGS_QUERY to extract definitions of functions and classes from Python code. ```python def find_definitions(code_text): """Extract all function and class definitions.""" query_text = tree_sitter_python.TAGS_QUERY query = Query(tree_sitter_python.language(), query_text) code = code_text.encode() tree = parser.parse(code) matches = query.matches(tree.root_node(), code) definitions = {} for match in matches: for capture in match.captures: name = query.capture_names()[capture.index] if name == "@definition.class": # Find the @name child for cap2 in match.captures: if query.capture_names()[cap2.index] == "@name": class_name = cap2.node.text.decode() definitions[class_name] = "class" break elif name == "@definition.function": for cap2 in match.captures: if query.capture_names()[cap2.index] == "@name": func_name = cap2.node.text.decode() definitions[func_name] = "function" break return definitions code = """ class Calculator: def add(self, a, b): return a + b def main(): calc = Calculator() print(calc.add(1, 2)) """ defs = find_definitions(code) print(defs) ``` -------------------------------- ### Parenthesized Expression Examples Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/04-grammar-types.md Examples of expressions enclosed in parentheses. ```python (x + y) (func()) (a,) ``` -------------------------------- ### Expression Statement Example Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/04-grammar-types.md Examples of expressions used as statements. ```python 42 func() x + y ``` -------------------------------- ### Constrained Type Example Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/04-grammar-types.md Example of a constrained type variable. ```python T: int | str ``` -------------------------------- ### Generic Type Example Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/04-grammar-types.md Example of a generic type with parameters. ```python list[int] dict[str, Any] Generic[T, U] ``` -------------------------------- ### HIGHLIGHTS_QUERY Usage Example Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/01-python-api.md Shows how to access the HIGHLIGHTS_QUERY string and print its length and a prefix, illustrating its use in syntax highlighting. ```python import tree_sitter_python from tree_sitter import Query, Language, Parser language = tree_sitter_python.language() # Get the highlighting query query_str = tree_sitter_python.HIGHLIGHTS_QUERY # Create a query object from the query string # (Assumes tree-sitter library supports Query creation from string) # This would typically be used by a syntax highlighter to color code Python # Example: accessing the query content print(f"Query length: {len(query_str)} characters") print("First 100 chars:", query_str[:100]) ``` -------------------------------- ### package.json Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/06-configuration-guide.md The package.json file for tree-sitter-python, detailing dependencies, devDependencies, and scripts. ```json { "name": "tree-sitter-python", "version": "0.25.0", "main": "bindings/node", "types": "bindings/node", "dependencies": { "node-addon-api": "^8.5.0", "node-gyp-build": "^4.8.4" }, "devDependencies": { "tree-sitter-cli": "^0.25.9", "prebuildify": "^6.0.1" }, "peerDependencies": { "tree-sitter": "^0.25.0" }, "scripts": { "install": "node-gyp-build", "test": "node --test bindings/node/*_test.js", "start": "tree-sitter playground" } } ``` -------------------------------- ### Incremental Updates Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/07-usage-examples.md Demonstrates how to perform incremental parsing by reusing the previous syntax tree when updating the code. ```rust fn main() -> Result<(), Box> { let mut parser = Parser::new(); parser.set_language(&LANGUAGE.into())?; // Parse initial code let code_v1 = "x = 1"; let tree = parser.parse(code_v1, None)?; println!("v1 root: {}", tree.root_node().kind()); // Update with new code (reuses tree structure) let code_v2 = "x = 1\ny = 2"; let tree = parser.parse(code_v2, Some(&tree))?; println!("v2 root: {}", tree.root_node().kind()); println!("v2 child count: {}", tree.root_node().child_count()); Ok(()) } ``` -------------------------------- ### Rust Cargo.toml Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/06-configuration-guide.md Configuration for the Rust crate, including package metadata, library path, and dependencies. ```toml [package] name = "tree-sitter-python" version = "0.25.0" edition = "2021" [lib] path = "bindings/rust/lib.rs" [dependencies] tree-sitter-language = "0.1" [build-dependencies] cc = "1.2" [dev-dependencies] tree-sitter = "0.25.8" ``` -------------------------------- ### Async Function definitions Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/test/corpus/statements.txt Examples of async function definitions with various parameter types, type hints, and return types. ```python async def a(): b ``` ```python async def c(d): e ``` ```python async def g(g, h,): i ``` ```python async def c(a: str): a ``` ```python async def c(a: b.c): a ``` ```python async def d(a: Sequence[T]) -> T: a ``` ```python async def i(a, b=c, *c, **d): a ``` ```python async def d(a: str) -> None: return None ``` ```python async def d(a:str="default", b=c) -> None: return None ``` -------------------------------- ### Escape Interpolation Example Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/04-grammar-types.md Example of literal braces within an f-string. ```python f"{{ literal brace }}" ``` -------------------------------- ### Example of capturing matches in JavaScript (conceptual) Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/05-query-types.md Illustrates how to set up the parser and parse code in JavaScript. Note that direct query exposure might differ in older versions. ```javascript const Parser = require("tree-sitter"); const PythonLanguage = require("tree-sitter-python"); const parser = new Parser(); parser.setLanguage(PythonLanguage); const code = "x = 42"; const tree = parser.parse(code); // Note: JavaScript tree-sitter doesn't directly expose queries in older versions // You would typically use a library wrapper for highlighting ``` -------------------------------- ### Splat Type Example Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/04-grammar-types.md Example of splat usage in type contexts. ```python *args: *Args **kwargs: **Kwargs ``` -------------------------------- ### Basic Parser Example Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/03-rust-api.md Demonstrates how to create a parser, set the language to Python, and parse a simple Python code string. ```rust use tree_sitter::Parser; use tree_sitter_python::LANGUAGE; fn main() -> Result<(), Box> { // Create a parser let mut parser = Parser::new(); // Set the language to Python parser.set_language(&LANGUAGE.into())?; // Parse some Python code let code = r#"# def fibonacci(n): if n <= 1: return n return fibonacci(n - 1) + fibonacci(n - 2) "#; let tree = parser.parse(code, None)?; // Inspect the root node let root = tree.root_node(); println!("Root: {:?}", root); println!("Root kind: {}", root.kind()); Ok(()) } ``` -------------------------------- ### Decorator Expression Example Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/04-grammar-types.md Example of a single decorator expression in Python. ```python @app.route("/") ``` -------------------------------- ### Creating Entirely Custom Queries Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/05-query-types.md Example of creating a completely custom query string for specific use cases. ```python custom_query_str = """ (function_definition name: (identifier) @public_fn (#match? @public_fn "^[a-z]")) (function_definition name: (identifier) @private_fn (#match? @private_fn "^_")) """ query = Query(language, custom_query_str) ``` -------------------------------- ### Decorated Definition Example Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/04-grammar-types.md Examples of decorated functions and classes in Python. ```python @decorator1 @decorator2 def func(): ... @dataclass class MyClass: ... ``` -------------------------------- ### Check for Errors Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/07-usage-examples.md Example of checking for errors in the parsed tree and finding error nodes. ```Python if tree.root_node().has_error(): # Find error nodes def find_errors(node): if "error" in node.type: yield node for child in node.children: yield from find_errors(child) for error_node in find_errors(tree.root_node()): print(f"Error: {error_node.text}") ``` -------------------------------- ### Expression List Examples Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/04-grammar-types.md Examples of comma-separated expressions, often forming tuples. ```python a, b, c x, y ``` -------------------------------- ### Block Statement Example Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/04-grammar-types.md An example illustrating an indented block of statements in Python. ```python if x: a b c ``` -------------------------------- ### Grammar DSL Structure (grammar.js) Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/06-configuration-guide.md The basic structure of the grammar definition in tree-sitter's DSL. ```javascript module.exports = grammar({ name: 'python', // External tokens (handled by lexer): externals: $ => [ $._newline, $._indent, $._dedent, $.string_start, $.comment, ], // Inline rules (don't create nodes): inline: $ => [ $._simple_statement, $._compound_statement, ], // Rules (create nodes): rules: { module: $ => repeat($._statement), _statement: $ => choice( $._simple_statements, $._compound_statement, ), // ... 150+ more rules } }); ``` -------------------------------- ### Member Type Example Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/04-grammar-types.md Example of a member type within a type annotation. ```python typing.List collections.abc.Iterable ``` -------------------------------- ### Union Type Example Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/04-grammar-types.md Example of a union type using pipe separation. ```python int | str list[int] | None ``` -------------------------------- ### Type Annotation Examples Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/04-grammar-types.md Examples of type expressions used in annotations and subscripts. ```python int list[str] dict[str, int] A | B T: Bound ``` -------------------------------- ### HIGHLIGHTS_QUERY Usage Example Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/03-rust-api.md Demonstrates how to use the HIGHLIGHTS_QUERY to parse Python code and extract syntax highlighting information. ```rust use tree_sitter::{Parser, Query}; use tree_sitter_python::{LANGUAGE, HIGHLIGHTS_QUERY}; fn main() { let mut parser = Parser::new(); parser.set_language(&LANGUAGE.into()).unwrap(); let code = "x = 42 # A constant"; let tree = parser.parse(code, None).unwrap(); // Create query from the highlights string let query = Query::new(&LANGUAGE.into(), HIGHLIGHTS_QUERY) .expect("Failed to compile highlights query"); let mut cursor = tree_sitter::QueryCursor::new(); let matches = cursor.matches(&query, tree.root_node(), code.as_bytes()); for m in matches { for capture in m.captures { let capture_name = &query.capture_names()[capture.index as usize]; println!("Capture: {} at {}..{}", capture_name, capture.node.start_byte(), capture.node.end_byte() ); } } } ``` -------------------------------- ### Error Handling Example Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/00-README.md Illustrates error tolerance in the parser with an example of an incomplete expression. ```python tree = parser.parse(b"x = 1 +") # Incomplete expression root = tree.root_node() root.has_error() # True # tree contains ERROR node for the incomplete expression ``` -------------------------------- ### Incremental Parsing Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/07-usage-examples.md Demonstrates incremental parsing, where a previously generated syntax tree is reused to parse updated code, leading to faster parsing times. ```python # Initial parse code_v1 = b"x = 1" tree = parser.parse(code_v1) # Update: reuse previous tree for faster parsing code_v2 = b"x = 1\ny = 2" tree = parser.parse(code_v2, tree) # Pass old tree # Parser now uses incremental strategy print(tree.root_node().child_count) # 2 statements ``` -------------------------------- ### Example of capturing matches in Python Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/05-query-types.md Demonstrates how to use a query to find captures within a parsed tree and print the capture name and node text. ```python matches = query.captures(tree.root_node(), code) for node, capture_name in matches: print(f"{capture_name}: {node.text.decode()}") ``` -------------------------------- ### Conditional Matching Example Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/test/corpus/pattern_matching.txt Example of a case statement with positional and keyword arguments, and a trailing comma. ```python case Point3D(34, x=0, y=0, z=0,): foo8() ``` -------------------------------- ### Python code example using 'match' as a keyword argument. Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/test/corpus/pattern_matching.txt Shows how 'match' can be used as a keyword argument in a function call. ```python field = call(match=r".*\.txt$") ``` -------------------------------- ### f-string Format Specifier Examples Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/04-grammar-types.md Examples showing format specifiers used within f-strings. ```python f"{value:.2f}" f"{value:>10}" ``` -------------------------------- ### Usage example for LANGUAGE constant Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/03-rust-api.md Demonstrates how to initialize the tree-sitter parser with the Python language and parse a simple Python code snippet. ```rust use tree_sitter::Parser; use tree_sitter_python::LANGUAGE; fn main() { let mut parser = Parser::new(); // Load Python language parser.set_language(&LANGUAGE.into()) .expect("Error loading Python parser"); // Parse Python code let code = r#"# def greet(name): print(f"Hello, {name}!") #"#; let tree = parser.parse(code, None).unwrap(); let root = tree.root_node(); println!("Root node type: {}", root.kind()); println!("Has error: {}", root.has_error()); } ``` -------------------------------- ### Example of using TAGS_QUERY in Python Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/05-query-types.md Demonstrates how to load the TAGS_QUERY and use it to find definitions and references within Python code. ```python import tree_sitter_python from tree_sitter import Language, Parser, Query code = b""" class Calculator: def add(self, a, b): return a + b def test(): calc = Calculator() result = calc.add(1, 2) """ language = tree_sitter_python.language() query_str = tree_sitter_python.TAGS_QUERY parser = Parser(language) query = Query(language, query_str) tree = parser.parse(code) # Find all definitions and references matches = query.captures(tree.root_node(), code) definitions = {} references = {} for node, capture_name in matches: if capture_name == "@name": # Check the parent context pass # Would need to track which definition this name belongs to elif capture_name.startswith("@definition"): def_type = capture_name.split(".")[-1] name_text = node.text.decode() if node.type == "identifier" else "?" definitions[name_text] = def_type elif capture_name == "@reference.call": # Find the @name child pass print("Definitions found:") for name, def_type in definitions.items(): print(f" {name}: {def_type}") ``` -------------------------------- ### Basic print statements Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/test/corpus/statements.txt Demonstrates simple print statements and comments. ```python hi print b # bye print c ``` ```tree-sitter-grammar (module (print_statement (identifier)) (comment) (print_statement (identifier)) (comment) (print_statement (identifier))) ``` -------------------------------- ### Chevron (Output Redirection) Example Source: https://github.com/tree-sitter/tree-sitter-python/blob/master/_autodocs/04-grammar-types.md Example of output redirection using 'chevron' in Python 2 print statements. ```python print >> sys.stderr ```