### TinyPreprocessor Usage Example Source: https://github.com/dsisco11/tinypreprocessor/blob/master/docs/06-preprocessor-orchestrator.md A comprehensive example demonstrating the creation of preprocessor components, configuration, processing of a root resource, and handling of results and diagnostics. It also shows how to use the source map for error mapping. ```csharp // 1. Create components parser = new IncludeParser() directiveModel= new IncludeDirectiveModel() resolver = new FileSystemResolver("/project/src") merger = new ConcatenatingMergeStrategy() contentModel = new ReadOnlyMemoryCharContentModel() // 2. Create configuration config = new PreprocessorConfiguration, IncludeDirective, object>( parser, directiveModel, resolver, merger, contentModel) // 3. Create preprocessor preprocessor = new Preprocessor, IncludeDirective, object>(config) // 4. Load root resource root = new Resource("main.c", ReadFile("/project/src/main.c")) // 5. Process result = await preprocessor.ProcessAsync(root, context: null) // 6. Check results if result.Success: WriteFile("/project/out/bundle.c", result.Content) else: for each error in result.Diagnostics.GetBySeverity(Error): print "[{error.Code}] {error.Resource}: {error.Message}" // 6. Use source map for error mapping runtimeError = SourcePosition(line: 150, column: 10) originalLocation = result.SourceMap.Query(runtimeError) if originalLocation exists: print "Error originated in: {originalLocation}" ``` -------------------------------- ### Offset Mapping Usage Example Source: https://github.com/dsisco11/tinypreprocessor/blob/master/docs/04-source-mapping.md Example demonstrating how to use the SourceMapBuilder during a merge operation and query the resulting SourceMap. ```APIDOC ## Usage Example ### Description Illustrates the process of using `SourceMapBuilder` during content merging and querying the final `SourceMap`. ### Merge Process ```csharp // Assume 'preprocessor' is an instance of your processing class // Assume 'root' and 'context' are defined var result = await preprocessor.ProcessAsync(root, context); // Querying the SourceMap // Map generated offset 0 back to its origin var location = result.SourceMap.Query(generatedOffset: 0); if (location is not null) { Console.WriteLine($"Originated from {location.Resource.Path} at original offset {location.OriginalOffset}"); } // Query a generated range and map it back to original ranges var ranges = result.SourceMap.QueryRangeByLength(generatedStartOffset: 0, length: 20); ``` ### Builder Usage During Merge ```csharp // When emitting content during a merge strategy: // let generatedStart = current output offset // emit N content units from resource X starting at original offset O // builder.AddOffsetSegment(X, generatedStart, O, length: N) ``` ``` -------------------------------- ### Example: Querying source map for a generated range Source: https://github.com/dsisco11/tinypreprocessor/blob/master/docs/04-source-mapping.md Shows how to query a range of generated offsets and map it back to its original source ranges. This is useful for understanding the origin of larger code blocks. ```csharp // Query a generated range and map it back to original ranges var ranges = result.SourceMap.QueryRangeByLength(generatedStartOffset: 0, length: 20); ``` -------------------------------- ### Topological Sort Processing Order Example Source: https://github.com/dsisco11/tinypreprocessor/blob/master/docs/03-dependency-graph.md Demonstrates how to obtain a processing order for resources using the GetProcessingOrder method, which performs a topological sort. The example shows a graph with specific dependencies and the resulting valid processing orders, highlighting that interchangeable dependencies can lead to multiple valid outputs. ```csharp // Given dependency structure: // main includes header, utils // utils includes types // header includes types graph.AddDependency("main", "header") graph.AddDependency("main", "utils") graph.AddDependency("utils", "types") graph.AddDependency("header", "types") order = graph.GetProcessingOrder() // Result: [types, header, utils, main] // or: [types, utils, header, main] // (header and utils are interchangeable since neither depends on the other) ``` -------------------------------- ### Implement Custom Merge Strategy (C#) Source: https://github.com/dsisco11/tinypreprocessor/blob/master/README.md Provides an example of how to implement a custom merge strategy for TinyPreprocessor. This allows for custom output formatting and handling of directives like `@import` or `#include`. It highlights the use of `MergeContext.SourceMapBuilder` for mapping generation and `MergeContext.ResolvedReferences` for handling inlined directives. ```csharp public sealed record JsonMergeOptions; public sealed class JsonMergeStrategy : IMergeStrategy, IncludeDirective, JsonMergeOptions> { public ReadOnlyMemory Merge( IReadOnlyList, IncludeDirective>> orderedResources, JsonMergeOptions userContext, MergeContext, IncludeDirective> mergeContext) { // Custom merge logic here // Use mergeContext.SourceMapBuilder to record mappings. // Use offset-based segments for precise mappings: // mergeContext.SourceMapBuilder.AddOffsetSegment(resourceId, generatedStartOffset, originalStartOffset, length) // Use mergeContext.Diagnostics to report issues // If you need to inline/expand directives: // - Use mergeContext.ResolvedReferences to map (requestingResourceId, directiveIndex) -> resolved dependency ResourceId // - Then fetch the already-processed dependency content from mergeContext.ResolvedCache // Do NOT re-derive ResourceIds from raw reference strings (resolvers may implement custom mapping). return ReadOnlyMemory.Empty; } } ``` -------------------------------- ### Example: Querying source map for generated offset 0 Source: https://github.com/dsisco11/tinypreprocessor/blob/master/docs/04-source-mapping.md Demonstrates how to use the SourceMap to query the origin of a specific generated offset. If a mapping exists, it prints the originating resource path and offset. ```csharp var result = await preprocessor.ProcessAsync(root, context); // Map generated offset 0 back to its origin var location = result.SourceMap.Query(generatedOffset: 0); if (location is not null) { Console.WriteLine($"Originated from {location.Resource.Path} at original offset {location.OriginalOffset}"); } ``` -------------------------------- ### Install TinyPreprocessor using .NET CLI Source: https://github.com/dsisco11/tinypreprocessor/blob/master/README.md This snippet shows how to add the TinyPreprocessor package to your .NET project using the 'dotnet add package' command. It's a standard way to manage NuGet dependencies. ```bash dotnet add package TinyPreprocessor ``` -------------------------------- ### Configure Preprocessor Options in C# Source: https://github.com/dsisco11/tinypreprocessor/blob/master/README.md This code snippet shows how to configure the preprocessor with various options. These options control behaviors like duplicate include handling, maximum include depth, and error handling. The example demonstrates setting these options and then running the preprocessor with the specified configuration. ```csharp var options = new PreprocessorOptions( DeduplicateIncludes: true, // Currently informational (resources are processed once per call) MaxIncludeDepth: 100, // Safety limit for recursion ContinueOnError: true // Collect all diagnostics instead of stopping early ); // Note: The current implementation processes each resource at most once per call, // so `DeduplicateIncludes` does not currently change output. var result = await preprocessor.ProcessAsync(root, context, options); ``` -------------------------------- ### Directive Stripping Example Transformation Source: https://github.com/dsisco11/tinypreprocessor/blob/master/docs/05-merge-system.md Illustrates how directives are removed from content, shifting line numbers in the process. This is a key part of the default merge strategy. ```text - Original Line 1: #include "utils.h" → _Removed (directive)_ - Original Line 2: void main() { → **New Line 1** - Original Line 3: doSomething(); → **New Line 2** ``` -------------------------------- ### IMergeStrategy Interface and ModuleBundleMergeStrategy Implementation Source: https://context7.com/dsisco11/tinypreprocessor/llms.txt The IMergeStrategy interface defines how resolved resources are combined into the final output. ModuleBundleMergeStrategy is an example implementation that bundles resources into a JavaScript module format. ```APIDOC ## IMergeStrategy Interface ### Description The `IMergeStrategy` interface controls how resolved resources are combined into final output. Resources arrive in topological order (dependencies first). ### Method `Merge(IReadOnlyList, IncludeDirective>> orderedResources, BundleOptions userContext, MergeContext, IncludeDirective> context)` ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Request Example ```csharp // Usage within the preprocessor pipeline ``` ### Response #### Success Response (200) - **ReadOnlyMemory**: The merged content of the resources. #### Response Example ```json // JavaScript module code representing the merged resources "(function(modules) { ... })();" ``` ## ModuleBundleMergeStrategy Class ### Description A concrete implementation of `IMergeStrategy` that merges resources into a JavaScript module bundle. It strips directives and generates code for module loading. ### Method `Merge` (implements `IMergeStrategy.Merge`) ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Request Example ```csharp // This class is typically used internally by the preprocessor. var mergeStrategy = new ModuleBundleMergeStrategy(); var mergedContent = mergeStrategy.Merge(orderedResources, userContext, mergeContext); ``` ### Response #### Success Response (200) - **ReadOnlyMemory**: The generated JavaScript module bundle code. #### Response Example ```javascript (function(modules) { var cache = {}; function require(id) { return cache[id] || (cache[id] = modules[id]()); } modules['file1.js'] = function() { /* content of file1 */ }; modules['file2.js'] = function() { /* content of file2 */ }; })({}); ``` ``` -------------------------------- ### Build Graph, Detect Cycles, and Get Processing Order Source: https://github.com/dsisco11/tinypreprocessor/blob/master/docs/03-dependency-graph.md This pseudocode demonstrates the usage of ResourceDependencyGraph. It covers building the dependency graph during resource resolution, detecting circular dependencies, and obtaining a valid processing order. Dependencies include a parser and a resolver. ```pseudocode graph = new ResourceDependencyGraph() function ResolveRecursive(resource, depth) graph.AddResource(resource.Id) directives = parser.Parse(resource.Content, resource.Id) for each directive in directives: result = await resolver.ResolveAsync(directive.Reference, resource, ct) if result.IsSuccess: graph.AddDependency(resource.Id, result.Resource.Id) await ResolveRecursive(result.Resource, depth + 1) // Detect cycles before processing for each cycle in graph.DetectCycles(): diagnostics.Add(CircularDependencyDiagnostic(cycle + [cycle[0]])) // Get processing order (skip if cycles detected) if not diagnostics.HasErrors: order = graph.GetProcessingOrder() // Process resources in this order ``` -------------------------------- ### Model Directives with Location and Reference Extraction (C#) Source: https://github.com/dsisco11/tinypreprocessor/blob/master/docs/01-core-abstractions.md Models directives using an unconstrained `TDirective` type, providing semantics via `IDirectiveModel` for extracting location and dependency references. Example directive types `IncludeDirective` and `ImportDirective` are shown. ```csharp public interface IDirectiveModel { Range GetLocation(TDirective directive); bool TryGetReference(TDirective directive, out string reference); } public sealed record IncludeDirective(string Reference, Range Location); public sealed record ImportDirective(string Module, bool IsRelative, Range Location); ``` -------------------------------- ### Query Source Map for Original Locations (C#) Source: https://github.com/dsisco11/tinypreprocessor/blob/master/README.md Demonstrates how to query the source map generated by TinyPreprocessor to find the original file and offset for a given generated offset or range. It also shows how to resolve boundary-based locations like line numbers using a custom boundary resolver. ```csharp var result = await preprocessor.ProcessAsync(root, context); // Find where generated offset 0 in output came from var location = result.SourceMap.Query(generatedOffset: 0); if (location is not null) { Console.WriteLine($"Originated from {location.Resource.Path} at original offset {location.OriginalOffset}"); } // For precise diagnostic spans, query a range. // The range may map to multiple original resources (e.g., it crosses file boundaries). var ranges = result.SourceMap.QueryRangeByLength(generatedStartOffset: 0, length: 20); foreach (var range in ranges) { Console.WriteLine( $"Generated [{range.GeneratedStartOffset} - {range.GeneratedEndOffset}) -> {range.Resource.Path} [{range.OriginalStartOffset} - {range.OriginalEndOffset})"); } // Boundary-based "line number" resolution (content-agnostic) // // If you want a line number for an offset, provide an IContentBoundaryResolver for a boundary kind // (e.g., TinyPreprocessor.Text.LineBoundary), and compose it with the source map. // // Example (LF-only): boundary offsets are start offsets of lines after the first line. public sealed class LfLineBoundaryResolver : IContentBoundaryResolver, LineBoundary> { public IEnumerable ResolveOffsets(ReadOnlyMemory content, ResourceId resourceId, int startOffset, int endOffset) { var s = content.Span.Slice(startOffset, Math.Min(endOffset, content.Length) - startOffset); for (var o = startOffset, i = s.IndexOf('\n'); i >= 0 && i + 1 < s.Length; s = s.Slice(i + 1), o += i + 1, i = s.IndexOf('\n')) yield return o; } } var boundaryLocation = result.SourceMap.ResolveOriginalBoundaryLocation( generatedOffset: 0, contentProvider: id => files[id].AsMemory(), boundaryResolver: new LfLineBoundaryResolver()); if (boundaryLocation is not null) { Console.WriteLine($"Line index: {boundaryLocation.BoundaryIndex}"); } ``` -------------------------------- ### Query SourceMap Positions and Ranges in C# Source: https://context7.com/dsisco11/tinypreprocessor/llms.txt Demonstrates how to use the SourceMap class to query specific generated code offsets or ranges and map them back to their original source file locations. It also shows how to use a custom boundary resolver to find line numbers. ```csharp using TinyPreprocessor.SourceMaps; var result = await preprocessor.ProcessAsync(root, context); // Query a single offset var location = result.SourceMap.Query(generatedOffset: 150); if (location is not null) { Console.WriteLine($"Offset 150 originated from {location.Resource.Path} at offset {location.OriginalOffset}"); } // Query a range (may span multiple original files) var ranges = result.SourceMap.QueryRangeByLength(generatedStartOffset: 100, length: 50); foreach (var range in ranges) { Console.WriteLine( $"Generated [{range.GeneratedStartOffset}-{range.GeneratedEndOffset}) -> " + $"{range.Resource.Path} [{range.OriginalStartOffset}-{range.OriginalEndOffset})"); } // Get line numbers using boundary resolver public sealed class LfLineBoundaryResolver : IContentBoundaryResolver, LineBoundary> { public IEnumerable ResolveOffsets(ReadOnlyMemory content, ResourceId resourceId, int startOffset, int endOffset) { var span = content.Span.Slice(startOffset, Math.Min(endOffset, content.Length) - startOffset); for (var o = startOffset; ; ) { var i = span.IndexOf('\n'); if (i < 0 || o + i + 1 >= endOffset) break; yield return o + i + 1; span = span.Slice(i + 1); o += i + 1; } } } var boundaryLocation = result.SourceMap.ResolveOriginalBoundaryLocation( generatedOffset: 150, contentProvider: id => files[id].AsMemory(), boundaryResolver: new LfLineBoundaryResolver()); if (boundaryLocation is not null) { Console.WriteLine($"Line {boundaryLocation.BoundaryIndex + 1} in {boundaryLocation.Resource.Path}"); } ``` -------------------------------- ### Define OffsetSpan struct for offset ranges Source: https://github.com/dsisco11/tinypreprocessor/blob/master/docs/04-source-mapping.md Represents a half-open range of offsets [Start, End). It includes a computed Length property. This is a fundamental type for defining spans within source code. ```csharp public readonly record struct OffsetSpan(int Start, int End) { public int Length => End - Start; } ``` -------------------------------- ### Manage Resource Dependencies and Cycles in C# Source: https://context7.com/dsisco11/tinypreprocessor/llms.txt Shows how to use the ResourceDependencyGraph class to manage dependencies between resources. It covers querying dependencies and dependents, detecting cycles, and obtaining a valid processing order. ```csharp using TinyPreprocessor.Graph; // Access the graph from preprocessing result var result = await preprocessor.ProcessAsync(root, context); var graph = result.DependencyGraph; // Query dependencies var deps = graph.GetDependencies(new ResourceId("main.txt")); Console.WriteLine($"main.txt depends on: {string.Join(", ", deps.Select(d => d.Path))}"); // Query dependents (reverse lookup) var dependents = graph.GetDependents(new ResourceId("utils.txt")); Console.WriteLine($"Files that include utils.txt: {string.Join(", ", dependents.Select(d => d.Path))}"); // Check for cycles if (graph.HasCycles()) { var cycles = graph.DetectCycles(); foreach (var cycle in cycles) { Console.WriteLine($"Cycle: {string.Join(" -> ", cycle.Select(c => c.Path))}"); } } // Get processing order (dependencies first) var order = graph.GetProcessingOrder(); Console.WriteLine($"Processing order: {string.Join(", ", order.Select(o => o.Path))}"); ``` -------------------------------- ### Define Content Model Interface (C#) Source: https://github.com/dsisco11/tinypreprocessor/blob/master/docs/01-core-abstractions.md The IContentModel interface defines how to measure and slice content. Implementations are crucial for abstracting 'offset' concepts for different content types, enabling the preprocessor pipeline to correctly handle various data structures. ```csharp public interface IContentModel { int GetLength(TContent content); TContent Slice(TContent content, int start, int length); } ``` -------------------------------- ### Core API: Preprocessor Class Usage Source: https://context7.com/dsisco11/tinypreprocessor/llms.txt Demonstrates the core usage of the Preprocessor class in TinyPreprocessor. This involves defining directive types, implementing directive semantics, configuring the preprocessor with various strategies, and processing a root resource. ```csharp using TinyPreprocessor; using TinyPreprocessor.Core; using TinyPreprocessor.Text; // Define your directive type public sealed record IncludeDirective(string Reference, Range Location); // Implement directive semantics public sealed class IncludeDirectiveModel : IDirectiveModel { public Range GetLocation(IncludeDirective directive) => directive.Location; public bool TryGetReference(IncludeDirective directive, out string reference) { reference = directive.Reference; return true; } } // Create and configure the preprocessor var config = new PreprocessorConfiguration, IncludeDirective, object>( directiveParser: new IncludeParser(), directiveModel: new IncludeDirectiveModel(), resourceResolver: new InMemoryResolver(files), mergeStrategy: new ConcatenatingMergeStrategy(), contentModel: new ReadOnlyMemoryCharContentModel()); var preprocessor = new Preprocessor, IncludeDirective, object>(config); // Process a root resource var root = new Resource>("main.txt", content.AsMemory()); var result = await preprocessor.ProcessAsync(root, context: null); // Check results if (!result.Diagnostics.HasErrors) { Console.WriteLine(result.Content.ToString()); // Access: result.SourceMap, result.ProcessedResources, result.DependencyGraph } ``` -------------------------------- ### Core API: Preprocessor Source: https://context7.com/dsisco11/tinypreprocessor/llms.txt The main entry point for orchestrating the preprocessing pipeline, including recursive resolution, cycle detection, topological ordering, and merging. ```APIDOC ## POST /process ### Description Processes a root resource through the configured preprocessing pipeline, performing dependency resolution, cycle detection, and merging. ### Method POST ### Endpoint /process ### Parameters #### Request Body - **rootResource** (Resource) - Required - The root resource to start processing from. - **context** (object) - Optional - An optional context object to pass through the pipeline. ### Request Example ```json { "rootResource": { "id": "main.txt", "content": "// Content of main.txt" }, "context": null } ``` ### Response #### Success Response (200) - **content** (TContent) - The processed content after all preprocessing steps. - **sourceMap** (SourceMap) - The generated source map detailing the origin of the processed content. - **processedResources** (IEnumerable>) - A collection of all resources processed during the pipeline. - **dependencyGraph** (DependencyGraph) - The constructed dependency graph of all processed resources. - **diagnostics** (Diagnostics) - Any diagnostic messages (errors, warnings) generated during processing. #### Response Example ```json { "content": "// Processed content...", "sourceMap": { /* ... source map details ... */ }, "processedResources": [ /* ... processed resource details ... */ ], "dependencyGraph": { /* ... dependency graph details ... */ }, "diagnostics": { "errors": [], "warnings": [] } } ``` ``` -------------------------------- ### Manage and Filter Diagnostics in C# Source: https://context7.com/dsisco11/tinypreprocessor/llms.txt Illustrates how to use the DiagnosticCollection class to collect, store, and filter preprocessor diagnostics. It covers checking for errors, filtering by severity, and filtering by the resource where the diagnostic occurred. ```csharp using TinyPreprocessor.Diagnostics; var result = await preprocessor.ProcessAsync(root, context); // Quick error check if (result.Diagnostics.HasErrors) { Console.WriteLine("Preprocessing failed!"); } // Filter by severity foreach (var error in result.Diagnostics.GetBySeverity(DiagnosticSeverity.Error)) { Console.WriteLine($"ERROR [{error.Code}]: {error.Message}"); if (error.Resource.HasValue) Console.WriteLine($" in {error.Resource.Value.Path}"); if (error.Location.HasValue) Console.WriteLine($" at offset {error.Location.Value.Start}..{error.Location.Value.End}"); } // Filter by resource var mainErrors = result.Diagnostics.GetByResource(new ResourceId("main.txt")); // Built-in diagnostic types: // - CircularDependencyDiagnostic (TPP0001): Cycle detected in includes // - MaxDepthExceededDiagnostic (TPP0002): Include depth limit reached // - ResolutionFailedDiagnostic (TPP0100): Failed to resolve reference // - ParseErrorDiagnostic (TPP0200): Directive parsing failed // - NonWholeLineDirectiveDiagnostic (TPP0300): Directive not on its own line ``` -------------------------------- ### PreprocessorOptions Configuration Source: https://context7.com/dsisco11/tinypreprocessor/llms.txt The PreprocessorOptions record allows customization of the preprocessing behavior, including include deduplication, maximum include depth, and error handling. ```APIDOC ## PreprocessorOptions ### Description The `PreprocessorOptions` record configures preprocessing behavior including depth limits, deduplication, and error handling. ### Method `ProcessAsync(string root, PreprocessorContext context, PreprocessorOptions? options = null)` (Part of the main Preprocessor class) ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Request Example **Default Options:** ```csharp var result = await preprocessor.ProcessAsync(root, context); ``` **Custom Options:** ```csharp var options = new PreprocessorOptions( DeduplicateIncludes: true, // Include each resource only once (like #pragma once) MaxIncludeDepth: 50, // Safety limit for recursion (default: 100) ContinueOnError: true // Collect all diagnostics instead of stopping at first error ); var result = await preprocessor.ProcessAsync(root, context, options); ``` ### Response #### Success Response (200) - **ProcessResult**: Contains the processed output and any diagnostics. #### Response Example ```json { "Output": "// Processed content...", "Diagnostics": { "Messages": [ { "Severity": "Error", "Code": "ERR001", "Resource": "path/to/file.txt", "Message": "An example error occurred." } ] } } ``` ## Error Handling Example ### Description Demonstrates how to check the `Diagnostics` property of the `ProcessResult` for errors. ### Method `ProcessAsync` (when `ContinueOnError` is true) ### Endpoint N/A (Local Object Method) ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Request Example ```csharp if (result.Diagnostics.HasErrors) { foreach (var error in result.Diagnostics.GetBySeverity(DiagnosticSeverity.Error)) { Console.WriteLine($"[{error.Code}] {error.Resource}: {error.Message}"); } } ``` ### Response #### Success Response (200) - **void**: Prints error messages to the console. #### Response Example ``` [ERR001] path/to/file.txt: An example error occurred. ``` ``` -------------------------------- ### IResourceResolver Interface and FileSystemResolver Implementation Source: https://context7.com/dsisco11/tinypreprocessor/llms.txt The IResourceResolver interface defines how the preprocessor resolves string references to actual resources. The FileSystemResolver is a concrete implementation that resolves resources from the local file system. ```APIDOC ## IResourceResolver Interface ### Description The `IResourceResolver` interface resolves string references to actual resources. It supports async I/O for file systems, databases, or network sources. ### Method `ResolveAsync(string reference, IResource>? relativeTo, CancellationToken ct)` ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Request Example ```csharp var resolver = new FileSystemResolver("/project/src"); var result = await resolver.ResolveAsync("utils/helpers.h", currentResource, CancellationToken.None); ``` ### Response #### Success Response (200) - **ResourceResolutionResult>**: Contains the resolved resource or a diagnostic if resolution failed. #### Response Example ```json { "Resource": { "Id": { "Path": "utils/helpers.h" }, "Content": "// File content here..." }, "Diagnostic": null } ``` ## FileSystemResolver Class ### Description A concrete implementation of `IResourceResolver` that resolves resources from the local file system. It takes a base path during initialization. ### Method `FileSystemResolver(string basePath)` ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Request Example ```csharp var resolver = new FileSystemResolver("/project/src"); ``` ### Response #### Success Response (200) - **FileSystemResolver**: An instance of the FileSystemResolver. #### Response Example (Constructor, no direct response example) ## Resource Resolution Example ### Description Demonstrates how to use the `FileSystemResolver` to resolve a file. ### Method `ResolveAsync` ### Endpoint N/A (Local Object Method) ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Request Example ```csharp var resolver = new FileSystemResolver("/project/src"); var result = await resolver.ResolveAsync("utils/helpers.h", currentResource, CancellationToken.None); if (result.IsSuccess) { Console.WriteLine($"Loaded: {result.Resource!.Id.Path}"); } ``` ### Response #### Success Response (200) - **ResourceResolutionResult>**: Indicates success or failure of the resource resolution. #### Response Example ```json { "IsSuccess": true, "Resource": { "Id": { "Path": "utils/helpers.h" }, "Content": "// Content of helpers.h" }, "Diagnostic": null } ``` ``` -------------------------------- ### Cycle Detection and Reporting Source: https://github.com/dsisco11/tinypreprocessor/blob/master/docs/03-dependency-graph.md Illustrates the process of detecting cycles within the dependency graph and reporting them as CircularDependencyDiagnostic objects. It shows how detected cycles are formatted for user-friendly output, including a direct cycle, an indirect cycle, and a self-reference. ```csharp function DetectCyclesAsDiagnostics() → IEnumerable for each cycle in DetectCycles(): displayCycle = cycle + [cycle[0]] // append first to show closure yield CircularDependencyDiagnostic(displayCycle) ``` -------------------------------- ### Process Includes in Memory with C# Source: https://github.com/dsisco11/tinypreprocessor/blob/master/README.md This code defines and uses an in-memory preprocessor to handle include directives. It sets up the necessary components: directive definition, directive model, parser, resolver, and merge strategy. The preprocessor then processes the includes, resolving the file content and merging them to produce a final output. ```csharp using System; using System.Collections.Generic; using System.Threading; using System.Threading.Tasks; using TinyPreprocessor; using TinyPreprocessor.Core; using TinyPreprocessor.Diagnostics; using TinyPreprocessor.Text; // 1) Define your directive type. public sealed record IncludeDirective(string Reference, Range Location); // 2) Provide directive semantics to the pipeline. public sealed class IncludeDirectiveModel : IDirectiveModel { public Range GetLocation(IncludeDirective directive) => directive.Location; public bool TryGetReference(IncludeDirective directive, out string reference) { reference = directive.Reference; return true; } } // 3) Implement a tiny directive parser for lines like: #include other.txt public sealed class IncludeParser : IDirectiveParser, IncludeDirective> { public IEnumerable Parse(ReadOnlyMemory content, ResourceId resourceId) { var text = content.ToString(); var lines = text.Split('\n'); var offset = 0; foreach (var line in lines) { if (line.StartsWith("#include ")) { var path = line[9..].Trim().Trim('"'); yield return new IncludeDirective(path, offset..(offset + line.Length)); } offset += line.Length + 1; } } } // 4) Implement an in-memory resolver. public sealed class InMemoryResolver : IResourceResolver> { private readonly IReadOnlyDictionary _files; public InMemoryResolver(IReadOnlyDictionary files) => _files = files; public ValueTask>> ResolveAsync( string reference, IResource>? context, CancellationToken ct) { if (!_files.TryGetValue(new ResourceId(reference), out var content)) { return ValueTask.FromResult(new ResourceResolutionResult>( null, new ResolutionFailedDiagnostic(reference, $"Not found: {reference}"))); } var resource = new Resource>(reference, content.AsMemory()); return ValueTask.FromResult(new ResourceResolutionResult>(resource, null)); } } // 5) Wire everything together. var files = new Dictionary { ["main.txt"] = "#include a.txt\nMAIN\n", ["a.txt"] = "A\n#include b.txt\n", ["b.txt"] = "B\n" }; var parser = new IncludeParser(); var directiveModel = new IncludeDirectiveModel(); var resolver = new InMemoryResolver(files); var merger = new ConcatenatingMergeStrategy(); var contentModel = new ReadOnlyMemoryCharContentModel(); var context = new object(); var config = new PreprocessorConfiguration, IncludeDirective, object>( parser, directiveModel, resolver, merger, contentModel); var preprocessor = new Preprocessor, IncludeDirective, object>(config); var root = new Resource>("main.txt", files["main.txt"].AsMemory()); var result = await preprocessor.ProcessAsync(root, context); if (!result.Diagnostics.HasErrors) { Console.WriteLine(result.Content.ToString()); } else { foreach (var diagnostic in result.Diagnostics) { Console.WriteLine($"[{diagnostic.Code}] {diagnostic.Message}"); } } ``` -------------------------------- ### IDirectiveParser Interface Implementation and Usage Source: https://context7.com/dsisco11/tinypreprocessor/llms.txt Shows how to implement and use the IDirectiveParser interface from TinyPreprocessor. This interface is responsible for extracting directive information from resource content, allowing for custom directive syntax parsing. ```csharp using TinyPreprocessor.Core; public sealed class IncludeParser : IDirectiveParser, IncludeDirective> { public IEnumerable Parse(ReadOnlyMemory content, ResourceId resourceId) { var text = content.ToString(); var lines = text.Split('\n'); var offset = 0; foreach (var line in lines) { // Parse lines like: #include "path/to/file.txt" if (line.StartsWith("#include ")) { var path = line[9..].Trim().Trim('"'); yield return new IncludeDirective(path, offset..(offset + line.Length)); } offset += line.Length + 1; // +1 for newline } } } // Usage var parser = new IncludeParser(); var directives = parser.Parse("code\n#include \"utils.h\"\nmore code".AsMemory(), "main.c"); foreach (var directive in directives) { Console.WriteLine($"Include: {directive.Reference} at {directive.Location}"); } ``` -------------------------------- ### Adding Diagnostics During Processing (Pseudocode) Source: https://github.com/dsisco11/tinypreprocessor/blob/master/docs/02-diagnostics-system.md Illustrates patterns for adding diagnostics to a DiagnosticCollection during various processing steps, such as resolving references or detecting cycles. ```pseudocode diagnostics = new DiagnosticCollection() // During resolution result = await resolver.ResolveAsync(reference, currentResource, ct) if not result.IsSuccess: diagnostics.Add(result.Error) // During cycle detection for each cycle in graph.DetectCycles(): diagnostics.Add(new CircularDependencyDiagnostic(cycle)) ``` -------------------------------- ### IDirectiveParser Interface Source: https://context7.com/dsisco11/tinypreprocessor/llms.txt Interface for parsing directives from resource content. Implement this to define how your specific directive syntax is extracted. ```APIDOC ## POST /directives/parse ### Description Parses a given content to extract directives based on the implemented `IDirectiveParser` logic. ### Method POST ### Endpoint /directives/parse ### Parameters #### Request Body - **content** (TContent) - Required - The content to parse for directives. - **resourceId** (ResourceId) - Required - The identifier of the resource being parsed. ### Request Example ```json { "content": "#include \"utils.h\"", "resourceId": "main.c" } ``` ### Response #### Success Response (200) - **directives** (IEnumerable) - A collection of directives found in the content. #### Response Example ```json { "directives": [ { "reference": "utils.h", "location": { "start": 5, "end": 24 } } ] } ``` ``` -------------------------------- ### Represent Single Resource with Content and Metadata (C#) Source: https://github.com/dsisco11/tinypreprocessor/blob/master/docs/01-core-abstractions.md Defines the `IResource` interface for representing a single resource in the preprocessing pipeline, including its ID, content, and optional metadata. A default generic implementation `Resource` is also provided. ```csharp public interface IResource { ResourceId Id { get; } ReadOnlyMemory Content { get; } IReadOnlyDictionary? Metadata { get; } } public sealed record Resource( ResourceId Id, TContent Content, IReadOnlyDictionary? Metadata = null ) : IResource; ``` -------------------------------- ### Resolve String References to Resources Asynchronously (C#) Source: https://github.com/dsisco11/tinypreprocessor/blob/master/docs/01-core-abstractions.md Defines the `IResourceResolver` interface for resolving string references (from directives) into actual resources asynchronously. It supports relative path resolution and returns a `ResourceResolutionResult` to handle success or failure. ```csharp public interface IResourceResolver { ValueTask> ResolveAsync( string reference, IResource? relativeTo, CancellationToken ct); } public sealed record ResourceResolutionResult( IResource? Resource, IPreprocessorDiagnostic? Error) { public bool IsSuccess => Resource is not null; public static ResourceResolutionResult Success(IResource resource) => new(resource, null); public static ResourceResolutionResult Failure(IPreprocessorDiagnostic error) => new(null, error); } ``` -------------------------------- ### IContentModel Interface Source: https://github.com/dsisco11/tinypreprocessor/blob/master/docs/01-core-abstractions.md The IContentModel interface defines how the pipeline measures and slices content, specifying the meaning of an 'offset' for a given TContent type. ```APIDOC ## Interface IContentModel ### Description Defines how the pipeline measures and slices content. This interface is crucial for establishing the meaning of an 'offset' for a specific `TContent` type used within the preprocessing pipeline. ### Method Interface Definition ### Endpoint N/A ### Parameters #### Type Parameters - **TContent**: The generic type representing the content being processed. #### Methods - **GetLength**(TContent content) (int) - Returns the total length of the provided content. - **Slice**(TContent content, int start, int length) (TContent) - Returns a slice of the content starting from the specified `start` index with the given `length`. ### Request Example ```csharp public interface IContentModel { int GetLength(TContent content); TContent Slice(TContent content, int start, int length); } ``` ### Response #### Success Response (N/A) This is an interface definition, not an API endpoint. #### Response Example N/A ``` -------------------------------- ### Core Abstractions Overview Source: https://github.com/dsisco11/tinypreprocessor/blob/master/docs/01-core-abstractions.md This section outlines the fundamental types that are essential for the TinyPreprocessor's operation, focusing on resource identification, representation, parsing, and resolution. ```APIDOC ## Overview The core abstractions define how resources are identified, represented, parsed, and resolved. These types are intentionally minimal and abstract to allow downstream users maximum flexibility. ## Types ### ResourceId A lightweight, immutable identifier for resources in the preprocessing system. **Design Decisions:** - **Readonly struct**: Zero-allocation equality checks, value semantics - **String-based path**: Flexible enough for file paths, URIs, or abstract identifiers - **IEquatable**: Enables efficient dictionary lookups and hash set operations - **Implicit conversion from string**: Convenient API for simple cases **Usage:** ```csharp ResourceId id = "path/to/resource.txt"; ResourceId id2 = new ResourceId("path/to/resource.txt"); ``` --- ### IResource Represents a single resource (file, module, or abstract content unit) in the preprocessing pipeline. **Design Decisions:** - **Interface**: Allows custom resource implementations (lazy-loaded, cached, virtual, etc.) - **TContent-generic**: The pipeline does not assume content is text or a sequence of symbols. - **Nullable Metadata**: Optional extensibility point for custom data (timestamps, checksums, etc.) **Default Implementation:** ```csharp public sealed record Resource( ResourceId Id, TContent Content, IReadOnlyDictionary? Metadata = null ) : IResource; ``` --- ### Directives (TDirective) + IDirectiveModel Directives are modeled as an unconstrained `TDirective` type. Directive semantics needed by the pipeline (location + dependency reference extraction) are provided via `IDirectiveModel`. **Example Directive Types:** ```csharp public sealed record IncludeDirective(string Reference, Range Location); public sealed record ImportDirective(string Module, bool IsRelative, Range Location); ``` --- ### IResourceResolver Resolves string references (from directives) into actual resources. **Design Decisions:** - **Async-first (ValueTask)**: Resolution may involve I/O (file system, network, database) - **relativeTo parameter**: Enables relative path resolution from the including resource - **Returns result object**: Separates success/failure without exceptions for expected failures **ResourceResolutionResult:** ```csharp public sealed record ResourceResolutionResult( IResource? Resource, IPreprocessorDiagnostic? Error) { public bool IsSuccess => Resource is not null; public static ResourceResolutionResult Success(IResource resource) => new(resource, null); public static ResourceResolutionResult Failure(IPreprocessorDiagnostic error) => new(null, error); } ``` --- ### IDirectiveParser Extracts directives from resource content. **Design Decisions:** - **Generic TDirective**: Type-safe parsing for specific directive types - **IEnumerable return**: Lazy evaluation, allows streaming large files - **ResourceId parameter**: Enables context-aware parsing and better error messages - **Sync API**: Parsing is CPU-bound; async would add overhead without benefit **Example Implementation:** ```csharp public sealed class CStyleIncludeParser : IDirectiveParser, IncludeDirective> { // Parses: #include "path" or #include public IEnumerable Parse(ReadOnlyMemory content, ResourceId resourceId) { // Implementation using Regex or manual parsing } } ``` --- ## Relationships ```mermaid flowchart LR subgraph Resolution IResourceResolver -->|resolves to| IResource IResource -->|has| ResourceId end subgraph Parsing IDirectiveParserT["IDirectiveParser<T>"] -->|extracts| TDirective end ``` ``` -------------------------------- ### SourceMapBuilder API Source: https://github.com/dsisco11/tinypreprocessor/blob/master/docs/04-source-mapping.md APIs for building a SourceMap by adding offset mapping segments. ```APIDOC ## SourceMapBuilder API ### Description Used to incrementally build a `SourceMap` by adding mapping segments during content generation. ### Class `SourceMapBuilder` ### Methods #### `AddOffsetSegment` - **Description**: Adds a mapping segment between a generated offset span and an original resource offset span. - **Method**: `AddOffsetSegment` - **Parameters**: - `resource` (ResourceId) - The original resource identifier. - `generatedStartOffset` (int) - The starting offset in the generated content. - `originalStartOffset` (int) - The starting offset in the original resource. - `length` (int) - The length of the content segment being mapped. #### `Build` - **Description**: Finalizes the mapping process and returns an immutable `SourceMap`. - **Method**: `Build` - **Returns**: `SourceMap` - The constructed source map. #### `Clear` - **Description**: Resets the builder, clearing all accumulated mapping segments. - **Method**: `Clear` ### Thread Safety Not thread-safe. Use one builder instance per merge operation. ``` -------------------------------- ### Diagnostic Collection Class (Pseudocode) Source: https://github.com/dsisco11/tinypreprocessor/blob/master/docs/02-diagnostics-system.md Describes a thread-safe collection for accumulating diagnostics during preprocessing. It provides properties for checking errors/warnings and methods for adding and filtering diagnostics. ```pseudocode class DiagnosticCollection : IReadOnlyCollection // Thread-safe storage with lock-based synchronization Properties: Count → number of diagnostics HasErrors → true if any Error severity exists HasWarnings → true if any Warning severity exists Methods: Add(diagnostic) → append single diagnostic AddRange(diagnostics) → append multiple diagnostics GetByResource(resourceId) → filter by source resource GetBySeverity(severity) → filter by severity level GetEnumerator() → returns snapshot for thread-safe iteration ``` -------------------------------- ### Check TinyPreprocessor Processing Results Source: https://github.com/dsisco11/tinypreprocessor/blob/master/docs/02-diagnostics-system.md This C# code snippet demonstrates how to process the result of `preprocessor.ProcessAsync`. It checks for errors in the diagnostics, prints any errors found, and then proceeds to use the generated content if no errors are present. This is a common pattern for handling the output of a preprocessing step. ```csharp result = await preprocessor.ProcessAsync(root, context, ct) if result.Diagnostics.HasErrors: for each error in result.Diagnostics.GetBySeverity(Error): print "[{error.Code}] {error.Resource}: {error.Message}" return // Use result.Content ``` -------------------------------- ### MergeContext Structure Source: https://github.com/dsisco11/tinypreprocessor/blob/master/docs/05-merge-system.md Provides shared context to merge strategies for source map building and diagnostics. It includes a SourceMapBuilder, DiagnosticCollection, and read-only dictionaries for cache and resolved references. Also includes models for content and directives. ```csharp class MergeContext Properties: SourceMapBuilder : SourceMapBuilder // for recording mappings Diagnostics : DiagnosticCollection // for reporting issues ResolvedCache : IReadOnlyDictionary> // for cross-referencing ResolvedReferences : IReadOnlyDictionary // directive occurrence -> resolved dependency id DirectiveModel : IDirectiveModel // for interpreting directive locations ContentModel : IContentModel // for interpreting offsets + slicing content ``` -------------------------------- ### Define Preprocessor Options (Record) Source: https://github.com/dsisco11/tinypreprocessor/blob/master/docs/06-preprocessor-orchestrator.md Defines configuration options for preprocessing behavior, including deduplication, include depth limits, and error handling. It provides sensible defaults and supports immutable updates. ```csharp record PreprocessorOptions DeduplicateIncludes : bool = true // include each resource only once MaxIncludeDepth : int = 100 // safety limit against infinite recursion ContinueOnError : bool = true // collect all diagnostics vs. stop at first static Default : PreprocessorOptions // default instance ``` -------------------------------- ### SourceMap API Source: https://github.com/dsisco11/tinypreprocessor/blob/master/docs/04-source-mapping.md APIs for querying source location information from a generated SourceMap. ```APIDOC ## SourceMap API ### Description Provides methods to query the source location of generated content based on offsets. ### Class `SourceMap` ### Methods #### `Query` - **Description**: Maps a single generated offset back to its original resource and offset. - **Method**: `Query` - **Parameters**: - `generatedOffset` (int) - The offset in the generated content. - **Returns**: `SourceLocation?` - The original source location or null if not found. #### `QueryRangeByLength` - **Description**: Queries a range of generated content by its start offset and length, returning all overlapping original source ranges. - **Method**: `QueryRangeByLength` - **Parameters**: - `generatedStartOffset` (int) - The starting offset of the generated content range. - `length` (int) - The length of the generated content range. - **Returns**: `IReadOnlyList` - A list of source range locations. #### `QueryRangeByEnd` - **Description**: Queries a range of generated content by its start and end offsets, returning all overlapping original source ranges. - **Method**: `QueryRangeByEnd` - **Parameters**: - `generatedStartOffset` (int) - The starting offset of the generated content range. - `generatedEndOffset` (int) - The ending offset of the generated content range. - **Returns**: `IReadOnlyList` - A list of source range locations. ### Thread Safety Immutable after construction, thread-safe for queries. ```