### Elixir Module and Struct Documentation Example Source: https://github.com/nshkrdotcom/json_remedy/blob/main/CODE_QUALITY.md This snippet demonstrates how to document an Elixir module and its struct using `@moduledoc` and `@doc`. It defines a `StartOpts` struct with enforced keys and a type specification, along with a `new` function for instantiation, including an `iex` example. ```elixir defmodule Quantum.NodeSelectorBroadcaster.StartOpts do @moduledoc """ A struct for configuring a `Quantum.NodeSelectorBroadcaster`. Defines required fields for initializing the broadcaster process. See `@type t` for the struct's type specification. """ @enforce_keys [:name, :execution_broadcaster_reference, :task_supervisor_reference] defstruct @enforce_keys @type t :: %__MODULE___{ name: GenServer.server(), execution_broadcaster_reference: GenServer.server(), task_supervisor_reference: GenServer.server() } @doc """ Creates a new `StartOpts` struct. ## Examples iex> %Quantum.NodeSelectorBroadcaster.StartOpts{ ...> name: :broadcaster, ...> execution_broadcaster_reference: {:global, :exec_broadcaster}, ...> task_supervisor_reference: {:global, :task_sup} ...> } %Quantum.NodeSelectorBroadcaster.StartOpts{...} """ def new(name, exec_ref, task_ref) do %__MODULE___{ name: name, execution_broadcaster_reference: exec_ref, task_supervisor_reference: task_ref } end end ``` -------------------------------- ### Run Elixir CharUtils Unit Tests Source: https://github.com/nshkrdotcom/json_remedy/blob/main/docs/desktop/RED_CONTINUE.md Command to execute unit tests specifically for the `char_utils.ex` module. This helps verify that tests are failing as expected before starting the implementation work. ```Shell mix test test/unit/utils/char_utils_test.exs ``` -------------------------------- ### Define Quantum NodeSelectorBroadcaster StartOpts Module in Elixir Source: https://github.com/nshkrdotcom/json_remedy/blob/main/CODE_QUALITY.md This snippet defines the `Quantum.NodeSelectorBroadcaster.StartOpts` module, a struct for configuring a broadcaster process in the Quantum job scheduler. It enforces required fields (`name`, `execution_broadcaster_reference`, `task_supervisor_reference`) and includes type specifications (`@type t`). It also defines `new/3` for creating new structs and `validate/1` for validating them, complete with examples. ```Elixir defmodule Quantum.NodeSelectorBroadcaster.StartOpts do @moduledoc """ A struct for configuring a `Quantum.NodeSelectorBroadcaster` process. This struct defines the required configuration for initializing a broadcaster process in the Quantum job scheduler. All fields are enforced to ensure proper configuration. ## Fields - `name`: The name of the broadcaster process (`GenServer.server()`). - `execution_broadcaster_reference`: Reference to the execution broadcaster. - `task_supervisor_reference`: Reference to the task supervisor. See `@type t` for the type specification. """ @enforce_keys [:name, :execution_broadcaster_reference, :task_supervisor_reference] defstruct @enforce_keys @type t :: %__MODULE___{ name: GenServer.server(), execution_broadcaster_reference: GenServer.server(), task_supervisor_reference: GenServer.server() } @doc """ Creates a new `StartOpts` struct with the given parameters. ## Parameters - `name`: The name of the broadcaster process. - `exec_ref`: The execution broadcaster reference. - `task_ref`: The task supervisor reference. ## Returns - A `%StartOpts{}` struct if all parameters are valid. - Raises a `KeyError` if any enforced key is missing or `nil`. ## Examples iex> Quantum.NodeSelectorBroadcaster.StartOpts.new( ...> :broadcaster, ...> {:global, :exec_broadcaster}, ...> {:global, :task_sup} ...> ) %Quantum.NodeSelectorBroadcaster.StartOpts{ name: :broadcaster, execution_broadcaster_reference: {:global, :exec_broadcaster}, task_supervisor_reference: {:global, :task_sup} } """ @spec new(GenServer.server(), GenServer.server(), GenServer.server()) :: t() def new(name, exec_ref, task_ref) do %__MODULE___{ name: name, execution_broadcaster_reference: exec_ref, task_supervisor_reference: task_ref } end @doc """ Validates a `StartOpts` struct. ## Parameters - `opts`: The `%StartOpts{}` struct to validate. ## Returns - `{:ok, opts}` if valid. - `{:error, reason}` if invalid. ## Examples iex> opts = Quantum.NodeSelectorBroadcaster.StartOpts.new(:broadcaster, {:global, :exec}, {:global, :task}) iex> Quantum.NodeSelectorBroadcaster.StartOpts.validate(opts) {:ok, opts} """ @spec validate(t()) :: {:ok, t()} | {:error, term()} def validate(%__MODULE__{} = opts) do {:ok, opts} end end ``` -------------------------------- ### Start Elixir Interactive Shell Source: https://github.com/nshkrdotcom/json_remedy/blob/main/CLAUDE.md Starts an interactive Elixir shell (IEx) with the project's environment loaded, useful for debugging and experimentation. ```Elixir iex -S mix ``` -------------------------------- ### Define Elixir Module with Struct, Type, and Enforced Keys Source: https://github.com/nshkrdotcom/json_remedy/blob/main/CODE_QUALITY.md This example demonstrates how to define an Elixir module, `Quantum.NodeSelectorBroadcaster.StartOpts`, using `@moduledoc` for module documentation, `@enforce_keys` to specify required fields for the struct, and `@type t` to define a type specification for the struct. It ensures clarity, type safety, and proper initialization of configuration options. ```Elixir defmodule Quantum.NodeSelectorBroadcaster.StartOpts do @moduledoc """ Configuration struct for starting a Quantum NodeSelectorBroadcaster. Defines required options for initializing the broadcaster process. """ # Module attributes for struct and type specs @enforce_keys [:name, :execution_broadcaster_reference, :task_supervisor_reference] defstruct @enforce_keys @type t :: %__MODULE___{ name: GenServer.server(), execution_broadcaster_reference: GenServer.server(), task_supervisor_reference: GenServer.server() } end ``` -------------------------------- ### Running Performance Examples for JsonRemedy Source: https://github.com/nshkrdotcom/json_remedy/blob/main/README.md Command to execute performance-related examples, covering fast path optimization, layer-specific performance, throughput measurements, and memory usage patterns. ```bash mix run examples/quick_performance.exs ``` -------------------------------- ### Running Basic Usage Examples for JsonRemedy Source: https://github.com/nshkrdotcom/json_remedy/blob/main/README.md Command to execute the basic usage examples script, demonstrating fixes for unquoted keys, quote normalization, boolean/null variants, structural issues, and LLM outputs. ```bash mix run examples/basic_usage.exs ``` -------------------------------- ### Setup Python Analysis Environment Source: https://github.com/nshkrdotcom/json_remedy/blob/main/docs/design/15.md This Bash script prepares an isolated Python virtual environment for JSON analysis, installing necessary libraries like `json-repair`, `pytest`, `coverage`, and instrumentation tools such as `ast`, `line_profiler`, and `memory_profiler`. ```bash python -m venv json_analysis_env pip install json-repair pytest coverage # Setup instrumentation tools pip install ast line_profiler memory_profiler ``` -------------------------------- ### Install Elixir Project Dependencies Source: https://github.com/nshkrdotcom/json_remedy/blob/main/CLAUDE.md Fetches and installs all necessary dependencies for the Elixir project, required before compilation or testing. ```Elixir mix deps.get ``` -------------------------------- ### Elixir Pattern Matching for Struct Validation Source: https://github.com/nshkrdotcom/json_remedy/blob/main/CODE_QUALITY.md This example demonstrates using pattern matching in Elixir function clauses to validate input structs. It ensures that the `start_link` function only accepts arguments that match the `StartOpts` struct. ```elixir def start_link(%__MODULE__.StartOpts{} = opts), do: GenServer.start_link(__MODULE__, opts) ``` -------------------------------- ### Running JsonRemedy Examples via Mix Source: https://github.com/nshkrdotcom/json_remedy/blob/main/README.md These Bash commands illustrate how to execute various JsonRemedy examples and benchmarks using `mix run`. It includes commands for general performance testing, stress testing, and notes a known issue with the `performance_benchmarks.exs` script on large datasets, suggesting alternative commands. ```Bash # These work fine: mix run examples/performance_benchmarks.exs # May hang on large datasets # Alternatives that complete successfully: mix run examples/quick_performance.exs # Lightweight performance testing mix run examples/simple_stress_test.exs # Stress testing without hanging ``` -------------------------------- ### Elixir Behaviour Definition Example Source: https://github.com/nshkrdotcom/json_remedy/blob/main/CODE_QUALITY.md This snippet shows how to define a behaviour for an Elixir module using `@behaviour`. It indicates that `Quantum.NodeSelectorBroadcaster` adheres to the `GenServer` behaviour, ensuring a common interface. ```elixir defmodule Quantum.NodeSelectorBroadcaster do @behaviour GenServer # ... end ``` -------------------------------- ### Elixir Example Configuration Options Source: https://github.com/nshkrdotcom/json_remedy/blob/main/docs/desktop/LAYER_3.md An example Elixir map demonstrating how to define and pass configuration options to Layer 3, enabling or disabling specific normalization and formatting behaviors. ```Elixir options = [ strict_mode: false, preserve_formatting: true, normalize_quotes: true, normalize_booleans: true, fix_commas: true ] ``` -------------------------------- ### Elixir Examples of Unpredictable JSON Malformations Source: https://github.com/nshkrdotcom/json_remedy/blob/main/docs/desktop/CRITIQUE.md Presents various real-world JSON malformation examples originating from different sources like LLMs, legacy systems, and copy-paste errors. These examples highlight the diverse and unpredictable nature of errors that a robust JSON repair solution must handle. ```Elixir # From GPT models: "{\"name\": \"Alice\", age: 30, active: true" # From legacy Python systems: "{'name': 'Alice', 'active': True, 'scores': [1, 2, 3,]}" # From copy-paste errors: "```json\n{\"name\": \"Alice\"}\n```" # From incomplete responses: "{\"users\": [{\"name\": \"Alice\"" ``` -------------------------------- ### Elixir Struct Field Alignment for Readability Source: https://github.com/nshkrdotcom/json_remedy/blob/main/CODE_QUALITY.md This example illustrates the recommended formatting for Elixir struct definitions, emphasizing vertical alignment of fields for improved readability, as enforced by `mix format`. ```elixir @type t :: %__MODULE___{ name: GenServer.server(), execution_broadcaster_reference: GenServer.server(), task_supervisor_reference: GenServer.server() } ``` -------------------------------- ### Running Real-World Scenario Examples for JsonRemedy Source: https://github.com/nshkrdotcom/json_remedy/blob/main/README.md Command to execute examples showcasing JsonRemedy's ability to handle complex, real-world problematic JSON from sources like LLMs, legacy systems, user input, config files, API responses, database dumps, JavaScript objects, and log outputs. ```bash mix run examples/real_world_scenarios.exs ``` -------------------------------- ### Bash Commands for Performance Monitoring Setup Source: https://github.com/nshkrdotcom/json_remedy/blob/main/docs/desktop/QUADRATIC_SOLVE.md These Bash commands provide a simple interface for monitoring the performance of the JSON processing library. They allow setting a baseline, tracking progress after optimizations, and checking current performance metrics. ```Bash # 1. Set baseline (before optimizations) mix run scripts/perf_monitor.exs baseline # 2. Track progress (after each phase) mix run scripts/perf_monitor.exs compare # 3. Check current performance mix run scripts/perf_monitor.exs current ``` -------------------------------- ### Elixir Struct Definition with Enforced Keys Source: https://github.com/nshkrdotcom/json_remedy/blob/main/CODE_QUALITY.md This Elixir code demonstrates how to define a struct using `defmodule` and `defstruct`, enforcing required fields with `@enforce_keys`. It includes an example of valid struct instantiation and illustrates `KeyError` scenarios when enforced keys are missing or assigned `nil` values. ```Elixir defmodule Quantum.NodeSelectorBroadcaster.StartOpts do @enforce_keys [:name, :execution_broadcaster_reference, :task_supervisor_reference] defstruct @enforce_keys # Valid creation def example do %__MODULE___{ name: :broadcaster, execution_broadcaster_reference: {:global, :exec_broadcaster}, task_supervisor_reference: {:global, :task_sup} } end end # Missing key %Quantum.NodeSelectorBroadcaster.StartOpts{name: :broadcaster} # => ** (KeyError) key :execution_broadcaster_reference not found # Nil value for enforced key %Quantum.NodeSelectorBroadcaster.StartOpts{ name: :broadcaster, execution_broadcaster_reference: nil, task_supervisor_reference: {:global, :task_sup} } # => ** (KeyError) key :execution_broadcaster_reference cannot be nil ``` -------------------------------- ### Elixir: Collect Real-World Malformed JSON Data Source: https://github.com/nshkrdotcom/json_remedy/blob/main/docs/design/102.md This module outlines strategies for collecting diverse and representative real-world malformed JSON examples. It suggests mining GitHub issues, corrupting API documentation examples, collecting community examples, and converting Python `json_repair` test cases. The goal is to gather examples that cover the error space without needing millions of entries. ```elixir defmodule JsonRemedy.DataCollection do @doc """ The key insight: You don't need millions of examples. You need *diverse, representative* examples that cover the error space. """ def collect_real_world_data do # Strategy 1: GitHub mining for malformed JSON in issues/PRs github_examples = mine_github_for_json_errors() # Strategy 2: Generate from API documentation examples api_examples = corrupt_api_documentation_examples() # Strategy 3: Elixir community - ask for malformed JSON examples community_examples = collect_community_examples() # Strategy 4: Convert Python json_repair test cases python_test_cases = convert_python_test_cases() combine_and_deduplicate([github_examples, api_examples, community_examples, python_test_cases]) end defp mine_github_for_json_errors do # Search GitHub for issues containing malformed JSON # Look for: "invalid json", "json parse error", "malformed json" # Extract the malformed JSON from issue descriptions end defp convert_python_test_cases do # This is your goldmine! Python json_repair probably has test cases # Convert their test cases to your format # Each test case is: input -> expected_output end end ``` -------------------------------- ### Elixir Repair Context Flow Example Source: https://github.com/nshkrdotcom/json_remedy/blob/main/docs/desktop/LAYER_3.md Illustrates the structure of the repair context map, showing how repair histories from previous layers are accumulated and options are passed through the pipeline. ```Elixir # Input context from Layer 2 %{ repairs: [layer1_repairs, layer2_repairs], options: [...] } ``` -------------------------------- ### Elixir: Generate Bootstrap Dataset for JSON Remediation Source: https://github.com/nshkrdotcom/json_remedy/blob/main/docs/design/102.md This module defines functions to generate a minimal 'seed knowledge' dataset for bootstrapping a learning system. It starts with valid JSON examples, systematically applies various corruptions to create malformed inputs, and then pairs them with their correct outputs and corruption types. The generated dataset serves as the initial training set. ```elixir defmodule JsonRemedy.DataGeneration do @moduledoc """ Generate the minimal dataset needed to bootstrap the learning system. This is your 'seed knowledge' that everything else builds from. """ def generate_bootstrap_dataset do # Start with valid JSON, then systematically break it valid_examples = generate_valid_json_corpus() # Apply systematic corruptions to create malformed examples malformed_examples = apply_systematic_corruptions(valid_examples) # Each example is: {malformed_input, correct_output, corruption_type} labeled_examples = create_labeled_pairs(malformed_examples, valid_examples) # Save as your initial training set save_bootstrap_dataset(labeled_examples) end defp generate_valid_json_corpus do [ # Simple objects ~s|"{"name": "Alice", "age": 30}"|, ~s|"{"active": true, "tags": ["dev", "senior"]}"|, # Nested structures ~s|"{"user": {"name": "Bob", "settings": {"theme": "dark"}}}"|, # Arrays ~s|"[1, 2, 3, "hello", true, null]"|, # Complex mixed ~s|"{"users": [{"id": 1, "active": true}, {"id": 2, "active": false}]}"|, # Edge cases ~s|"{"": "empty key", "unicode": "café", "numbers": [1.5, -2, 0]}"| ] end defp apply_systematic_corruptions(valid_examples) do corruption_types = [ &remove_random_comma/1, &change_quotes_to_single/1, &add_python_literals/1, &remove_closing_quote/1, &add_trailing_comma/1, &remove_quote_from_key/1, &mismatch_brackets/1 ] # Apply each corruption to each example for example <- valid_examples, corruption <- corruption_types do {corruption.(example), example, get_corruption_name(corruption)} end end end ``` -------------------------------- ### JsonRemedy Test Data: LLM Output Examples Source: https://github.com/nshkrdotcom/json_remedy/blob/main/docs/desktop/01_COMPREHENSIVE_TEST_PLAN_response.md Illustrative examples of malformed JSON outputs generated by Large Language Models (LLMs) like ChatGPT and Claude, including code fences, comments, truncated responses, and mixed syntax issues. These examples serve as test fixtures for JsonRemedy's repair capabilities. ```json // ChatGPT with code fences and comments // Claude with reasoning text // Truncated responses // Mixed syntax issues ``` -------------------------------- ### Elixir Type Specifications for Modules and Functions Source: https://github.com/nshkrdotcom/json_remedy/blob/main/CODE_QUALITY.md This Elixir example illustrates how to use `@type`, `@type t`, and `@spec` to define clear type specifications. It shows defining a public type `t` for a module's struct, a private internal state type (`@typep`), and a function specification (`@spec`) for a `GenServer.start_link` function, enhancing static analysis and documentation. ```Elixir defmodule Quantum.NodeSelectorBroadcaster do @moduledoc """ A GenServer for broadcasting node selection events in the Quantum scheduler. """ @type t :: %__MODULE__.StartOpts{ name: GenServer.server(), execution_broadcaster_reference: GenServer.server(), task_supervisor_reference: GenServer.server() } @typep internal_state :: %{ broadcaster: GenServer.server(), tasks: map() } @spec start_link(t()) :: GenServer.on_start() def start_link(%__MODULE__.StartOpts{} = opts) do GenServer.start_link(__MODULE__, opts, name: opts.name) end end ``` -------------------------------- ### JSON Comprehensive Syntax Normalization Example Source: https://github.com/nshkrdotcom/json_remedy/blob/main/docs/desktop/LAYER_3.md This snippet provides a comprehensive example demonstrating Layer 3's ability to perform multiple syntax normalizations in a single pass, addressing various issues like unquoted keys, single quotes, non-standard literals, and punctuation errors. ```JSON // Input (multiple syntax issues) {name: 'Alice', 'age': 30 active: True, items: [1 2 3,], data: None,} // Output (Layer 3 fully normalized) {"name": "Alice", "age": 30, "active": true, "items": [1, 2, 3], "data": null} ``` -------------------------------- ### Create Initial Dataset for JSON Remediation in Elixir Source: https://github.com/nshkrdotcom/json_remedy/blob/main/docs/design/102.md This module defines functions to generate a diverse initial dataset for JSON error correction. It includes examples of common JSON errors like unquoted keys, single quotes, missing commas, and trailing commas, along with real-world and edge cases. The `create_initial_dataset` function orchestrates the generation of these examples. ```elixir defmodule JsonRemedy.InitialDataset do def create_initial_dataset do %{ # 20 examples of each major error type unquoted_keys: generate_unquoted_key_examples(20), single_quotes: generate_single_quote_examples(20), python_literals: generate_python_literal_examples(20), missing_commas: generate_missing_comma_examples(20), trailing_commas: generate_trailing_comma_examples(20), # Real-world examples from Python library python_cases: load_python_test_cases(), # Edge cases that break most parsers edge_cases: generate_edge_cases(10) } end def generate_unquoted_key_examples(count) do base_templates = [ ~s|{name: "value"}|, ~s|{user_id: 123, active: true}|, ~s|{nested: {inner_key: "value"}}| ] # Generate variations Enum.flat_map(base_templates, fn template -> generate_variations(template, count) end) |> Enum.take(count) |> Enum.map(fn malformed -> corrected = fix_unquoted_keys(malformed) {malformed, corrected, :unquoted_keys} end) end end ``` -------------------------------- ### Future Extensibility Examples for JsonRemedy Configuration Source: https://github.com/nshkrdotcom/json_remedy/blob/main/FEATURE_FLAGS.md This snippet provides examples of how JsonRemedy's configuration system is designed for future extensibility. It shows potential new feature flags for advanced optimizations like SIMD, platform-specific code, and different performance profiles (e.g., memory-optimized) to evolve the library. ```Elixir # Future: Phase 3 could add SIMD optimizations config :json_remedy, :layer3_optimization_phase, 3 # Future: Platform-specific optimizations config :json_remedy, :layer3_platform_optimization, :native_code # Future: Memory vs speed profiles config :json_remedy, :layer3_performance_profile, :memory_optimized ``` -------------------------------- ### Elixir Layer 1 Configuration Options Source: https://github.com/nshkrdotcom/json_remedy/blob/main/docs/desktop/LAYER_1.md Provides an example of how to configure Layer 1 with boolean options to enable or disable its various content cleaning functionalities. ```elixir options = [ remove_comments: true, remove_code_fences: true, extract_from_html: false, normalize_encoding: true ] ``` -------------------------------- ### Elixir Error Handling with `with` and Pattern Matching Source: https://github.com/nshkrdotcom/json_remedy/blob/main/CODE_QUALITY.md This snippet illustrates a robust error handling approach in Elixir using the `with` special form and pattern matching. It validates multiple fields of a `StartOpts` struct, returning `{:ok, opts}` on success or `{:error, error}` on the first validation failure. ```elixir def validate_opts(%__MODULE__.StartOpts{} = opts) do with :ok <- validate_name(opts.name), :ok <- validate_reference(opts.execution_broadcaster_reference), :ok <- validate_reference(opts.task_supervisor_reference) do {:ok, opts} else error -> {:error, error} end end ``` -------------------------------- ### Elixir `configure` Function Specification Source: https://github.com/nshkrdotcom/json_remedy/blob/main/docs/desktop/04_API_CONTRACTS.md Specifies the `configure` function in Elixir, detailing its input `config` as a keyword list and its return type as either `:ok` or an error tuple, crucial for initial system setup. ```Elixir @spec configure(config :: keyword()) :: :ok | {:error, String.t()} def configure(config) end ``` -------------------------------- ### Elixir: Example of Original Optimization Infrastructure Source: https://github.com/nshkrdotcom/json_remedy/blob/main/CLAUDECODE_FAILED_REFACTOR.md This Elixir snippet from `key_quoter.ex` illustrates the original sophisticated optimization infrastructure. It uses an application environment variable to dynamically select between an O(1) IO list-based operation and an O(n²) character-by-character string concatenation, showcasing the feature flag-controlled performance options. ```Elixir if Application.get_env(:json_remedy, :layer3_iolist_optimization, true) do quote_unquoted_keys_iolist(input) # O(1) IO list operations else quote_unquoted_keys_char_by_char(input, "", 0, false, false, nil, []) # O(n²) string concat end ``` -------------------------------- ### Running Stress Testing Examples for JsonRemedy Source: https://github.com/nshkrdotcom/json_remedy/blob/main/README.md Command to execute stress tests to verify JsonRemedy's reliability under load, including repeated repair operations, nested structure handling, large array processing, and memory usage stability. ```bash mix run examples/simple_stress_test.exs ``` -------------------------------- ### Inspecting Repair Actions and Context in Elixir Source: https://github.com/nshkrdotcom/json_remedy/blob/main/docs/desktop/LAYER_2.md This example shows how to process an input with `StructuralRepair` and then inspect the `context.repairs` list to understand the specific actions taken by the structural repair layer, such as adding missing delimiters. ```Elixir {:ok, result, context} = StructuralRepair.process(input, %{repairs: [], options: []}) IO.inspect(context.repairs) # => [%{layer: :structural_repair, action: "added missing closing brace", ...}] ``` -------------------------------- ### Inefficient String Concatenation (O(n²)) Example in Elixir Source: https://github.com/nshkrdotcom/json_remedy/blob/main/docs/desktop/QUADRATIC_SOLVE.md This Elixir function demonstrates the O(n²) string concatenation pattern where `result <> char` copies the entire string in each iteration. This pattern is a major performance bottleneck. ```Elixir defp process_char(input, result, pos) do char = String.at(input, pos) process_char(input, result <> char, pos + 1) # ❌ O(n²) end ``` -------------------------------- ### Testing Individual Syntax Normalization Functions in Elixir Source: https://github.com/nshkrdotcom/json_remedy/blob/main/docs/desktop/LAYER_3.md This Elixir example shows how to directly call specific normalization functions like `normalize_quotes` and `normalize_booleans`. It also illustrates how to capture the `repairs` list to inspect the actions performed by each function. ```Elixir {result, repairs} = SyntaxNormalization.normalize_quotes("{'key': 'value'}") # => {"{\"key\": \"value\"}", [%{action: "normalized quotes", ...}]} {result, repairs} = SyntaxNormalization.normalize_booleans("{\"active\": True}") # => {"{\"active\": true}", [%{action: "normalized boolean True -> true", ...}]} ``` -------------------------------- ### Performance Monitor Usage Commands Source: https://github.com/nshkrdotcom/json_remedy/blob/main/PERF_MONITOR.md This section provides the command-line instructions for interacting with the performance monitor. Users can set an initial performance baseline, check the current performance metrics, and compare recent runs against the established baseline to evaluate optimization efforts. ```bash # 1. Set baseline (run once before optimizations) mix run scripts/perf_monitor.exs baseline # 2. Check current performance (run anytime) mix run scripts/perf_monitor.exs current # 3. Compare with baseline (run after changes) mix run scripts/perf_monitor.exs compare ``` -------------------------------- ### Setting Up JsonRemedy Development Environment Source: https://github.com/nshkrdotcom/json_remedy/blob/main/README.md This Bash script outlines the steps to set up the JsonRemedy development environment. It includes cloning the repository, navigating into the directory, and fetching project dependencies using `mix deps.get`. ```bash git clone https://github.com/nshkrdotcom/json_remedy.git cd json_remedy mix deps.get ``` -------------------------------- ### Get JsonRemedy Performance Metrics Source: https://github.com/nshkrdotcom/json_remedy/blob/main/docs/desktop/04_API_CONTRACTS.md Get performance metrics for recent operations. ```APIDOC JsonRemedy.performance_stats() :: JsonRemedy.Performance.performance_metrics() ``` -------------------------------- ### README.md Implementation Status Section Source: https://github.com/nshkrdotcom/json_remedy/blob/main/docs/desktop/DOCUMENTATION_UPDATE_SUMMARY.md This snippet shows the new 'Implementation Status' table added to README.md, detailing the current operational status of each layer (1-4 complete, 5 planned) for the JsonRemedy project. It provides a clear overview of the project's current capabilities. ```Markdown ## Implementation Status JsonRemedy is currently in **Phase 1** implementation with **Layers 1-4 fully operational**: | Layer | Status | Description | |-------|--------|-------------| | **Layer 1** | ✅ **Complete** | Content cleaning | | **Layer 2** | ✅ **Complete** | Structural repair | | **Layer 3** | ✅ **Complete** | Syntax normalization | | **Layer 4** | ✅ **Complete** | Fast validation | | **Layer 5** | ⏳ **Planned** | Tolerant parsing | ``` -------------------------------- ### Get JsonRemedy Library Version Information Source: https://github.com/nshkrdotcom/json_remedy/blob/main/docs/desktop/04_API_CONTRACTS.md Get current library version and build information. ```APIDOC JsonRemedy.version_info() :: %{version: String.t(), build_date: String.t(), git_commit: String.t(), elixir_version: String.t()} ``` -------------------------------- ### Run Elixir Project Tests with Mix Source: https://github.com/nshkrdotcom/json_remedy/blob/main/docs/desktop/RED_CONTINUE.md Provides a set of `mix test` commands for running different levels of tests within the Elixir project. These commands allow developers to execute unit tests for specific utilities (like CharUtils), run all context-related tests, verify critical baseline functionality, and perform a full test suite including integration tests. ```bash # Run CharUtils tests (currently failing) mix test test/unit/utils/char_utils_test.exs # Run all context tests (should pass) mix test test/unit/context/ # Run critical baseline (must always pass) mix test test/critical # Run full test suite with integration mix test --include integration ``` -------------------------------- ### CHANGELOG.md Version 0.1.1 Entry Source: https://github.com/nshkrdotcom/json_remedy/blob/main/docs/desktop/DOCUMENTATION_UPDATE_SUMMARY.md This snippet presents the CHANGELOG.md entry for version 0.1.1, highlighting a complete architectural rewrite, the new 5-layer pipeline design, improved performance, and the future plan for Layer 5. It emphasizes the significant changes introduced in this version. ```Markdown ## [0.1.1] - 2025-01-07 ### Changed - **BREAKING: Complete architectural rewrite** - Brand new 5-layer pipeline design - **New layered approach**: Regex → State Machine → Character Parsing → Validation → Tolerant Parsing - **Improved performance**: Significantly faster with intelligent fast-path optimization [... detailed changes] ### Future - **Layer 5 - Tolerant Parsing**: Planned for next major release ### Note This is a **100% rewrite** - all previous code has been replaced with the new layered architecture. ``` -------------------------------- ### Elixir Code Quality and Documentation Tools Overview Source: https://github.com/nshkrdotcom/json_remedy/blob/main/CODE_QUALITY.md This section outlines essential tools for maintaining Elixir code quality and generating documentation. It covers `mix format` for consistent formatting, Dialyzer for static type checking, Credo for linting and best practices, and ExDoc for generating API documentation from source code. Each tool includes instructions on how to run and configure it. ```APIDOC Tools for Code Quality: mix format: Description: Enforce consistent code formatting. Run Command: mix format Configuration: Update .formatter.exs for project-specific rules. Dialyzer: Description: Static analysis for type checking. Run Command: mix dialyzer Guideline: Ensure all public functions and structs have @spec and @type. Credo: Description: Linting for code style and best practices. Run Command: mix credo Configuration: Customize .credo.exs for project-specific checks. ExDoc: Description: Generate documentation from @moduledoc and @doc. Run Command: mix docs Guideline: Ensure all public APIs are documented. ``` -------------------------------- ### Run Elixir Integration Tests Source: https://github.com/nshkrdotcom/json_remedy/blob/main/CLAUDE.md Executes the full suite of integration tests, validating the interaction between different layers and components. ```Elixir mix test test/integration/ ``` -------------------------------- ### Layer 2 Configuration Options and Example Source: https://github.com/nshkrdotcom/json_remedy/blob/main/docs/desktop/LAYER_2.md Describes the configurable options for Layer 2, such as maximum nesting depth, processing timeout, and strict mode, along with their validation rules. Includes an Elixir example demonstrating how to set these options. ```APIDOC Configuration Options: - :max_nesting_depth - Maximum allowed nesting depth (positive integer) - :timeout_ms - Processing timeout in milliseconds (positive integer) - :strict_mode - Enable strict structural validation (boolean) Option Validation: - max_nesting_depth: Must be positive integer - timeout_ms: Must be positive integer - strict_mode: Must be boolean - Unknown options result in validation errors ``` ```Elixir options = [ max_nesting_depth: 50, timeout_ms: 5000, strict_mode: false ] ``` -------------------------------- ### JSON5 Example with Comments and Hexadecimals Source: https://github.com/nshkrdotcom/json_remedy/blob/main/docs/desktop/02_COMPREHENSIVE_TEST_PLAN.md This snippet demonstrates various JSON5 features, including unquoted keys, hexadecimal numbers, block and trailing comments, and trailing commas in arrays. It serves as an example of malformed JSON that a robust parser or repair library should handle. ```JSON { // JSON5 features name: 'Alice', age: 0x1E, // Hexadecimal pi: 3.14159, active: true, metadata: { /* Block comment */ created: '2024-01-15', tags: [ 'user', 'active', ], // Trailing comma }, } ``` -------------------------------- ### Run Critical Elixir Test Suite Source: https://github.com/nshkrdotcom/json_remedy/blob/main/CLAUDE.md Executes the critical test suite, focusing on 82 key tests that validate core functionalities and real-world scenarios. ```Elixir mix test test/critical ``` -------------------------------- ### Extract Patterns from Dataset in Elixir Source: https://github.com/nshkrdotcom/json_remedy/blob/main/docs/design/102.md This module focuses on extracting generalizable patterns from a curated dataset of malformed and corrected JSON examples. It groups examples by error type, identifies transformations between malformed and corrected versions, and converts these transformations into executable rules for JSON correction. ```elixir defmodule JsonRemedy.PatternExtraction do @doc """ This is where the magic happens: extracting generalizable patterns from your carefully curated examples. """ def extract_patterns_from_dataset(dataset) do # Group examples by error type grouped_examples = Enum.group_by(dataset, &elem(&1, 2)) # Extract patterns for each error type patterns = Enum.map(grouped_examples, fn {error_type, examples} -> {error_type, extract_patterns_for_error_type(examples)} end) # Validate patterns work across examples validated_patterns = validate_pattern_generalization(patterns, dataset) validated_patterns end defp extract_patterns_for_error_type(examples) do # For each pair of {malformed, corrected}, find the transformation transformations = Enum.map(examples, fn {malformed, corrected, _type} -> extract_transformation(malformed, corrected) end) # Find common patterns in the transformations common_patterns = find_common_transformation_patterns(transformations) # Convert to executable rules Enum.map(common_patterns, &convert_to_rule/1) end defp extract_transformation(malformed, corrected) do # This is your "edit distance with memory" algorithm edit_sequence = compute_detailed_edit_sequence(malformed, corrected) # Abstract the specific edits to pattern rules abstract_pattern = abstract_edit_sequence(edit_sequence) %{ original_example: malformed, corrected_example: corrected, edit_sequence: edit_sequence, abstract_pattern: abstract_pattern } end end ``` -------------------------------- ### Examples of Malformed JSON Inputs for JsonRemedy Source: https://github.com/nshkrdotcom/json_remedy/blob/main/README.md Illustrates various real-world scenarios of malformed JSON strings that JsonRemedy is designed to repair. These examples highlight common syntax errors and structural inconsistencies found in outputs from LLMs, legacy systems, JavaScript consoles, streaming APIs, and human input. ```text // LLM output with mixed issues ```json { users: [ {name: 'Alice Johnson', active: True, scores: [95, 87, 92,]}, {name: "Bob Smith", active: False /* incomplete ], metadata: None }``` ``` ```python # Legacy Python system output {'users': [{'name': 'Alice', 'verified': True, 'data': None}]} ``` ```javascript // Copy-paste from JavaScript console {name: "Alice", getValue: function() { return "test"; }, data: [1,2,3]} ``` ```text // Streaming API with connection drop {"status": "processing", "results": [{"id": 1, "name": "Alice" ``` ```text // Human input with common mistakes {name: Alice, "age": 30, "scores": [95 87 92], active: true,} ``` -------------------------------- ### Run All Elixir Tests Source: https://github.com/nshkrdotcom/json_remedy/blob/main/CLAUDE.md Executes the complete test suite for the JSON Remedy project, ensuring all 449 tests pass without failures. ```Elixir mix test ``` -------------------------------- ### Profiling JsonRemedy Performance with Elixir Tools Source: https://github.com/nshkrdotcom/json_remedy/blob/main/docs/desktop/QUADRATIC.md Commands to set up a minimal reproduction case and profile JsonRemedy using Elixir's built-in `fprof` and `observer` tools for performance and memory analysis. ```bash # 1. Create minimal reproduction case mix run scripts/perf_repro.exs # 2. Profile with :fprof mix run scripts/profile_layers.exs # 3. Memory analysis with :observer mix run scripts/memory_analysis.exs ``` -------------------------------- ### Elixir Binary Pattern Matching for Character Processing Source: https://github.com/nshkrdotcom/json_remedy/blob/main/docs/desktop/QUADRATIC_SOLVE.md This Elixir code demonstrates the use of binary pattern matching to efficiently process characters in a binary string. This approach replaces position-based access (like `String.at/2`) with O(1) character access, significantly improving performance for large inputs. ```Elixir defp process_chars(<>, state) do new_state = handle_char(char, state) process_chars(rest, new_state) end defp process_chars(<<>>, state), do: state ``` -------------------------------- ### Repairing Environment Configuration with Mixed Syntax Source: https://github.com/nshkrdotcom/json_remedy/blob/main/docs/desktop/05_DETAILED_TEST_SPEC_AND_CASES.md Illustrates how `JsonRemedy` handles environment configuration files that combine standard JSON elements with non-standard syntax such as unquoted keys and comments, making them parseable. ```JSON { // Environment: Production NODE_ENV: "production", PORT: 3000, // Database ``` ```Elixir {:ok, result} = JsonRemedy.repair(env_config) # Assertions would follow here, e.g.: # assert result["NODE_ENV"] == "production" # assert result["PORT"] == 3000 ``` -------------------------------- ### JsonRemedy's Context-Aware JSON Repair Examples in Elixir Source: https://github.com/nshkrdotcom/json_remedy/blob/main/README.md Demonstrates how JsonRemedy intelligently processes and repairs JSON while preserving valid content and normalizing syntax. Examples include handling commas inside string content, removing trailing commas, quoting unquoted keys, normalizing boolean values, and correctly parsing escape sequences. ```elixir # ✅ PRESERVE: Comma inside string content {"message": "Hello, world", "status": "ok"} # ✅ REMOVE: Trailing comma {"items": [1, 2, 3,]} # ✅ PRESERVE: Numbers stay numbers {"count": 42} # ✅ QUOTE: Unquoted keys get quoted {name: "Alice"} # ✅ PRESERVE: Boolean content in strings {"note": "Set active to True"} # ✅ NORMALIZE: Boolean values {"active": True} # ✅ PRESERVE: Escape sequences in strings {"path": "C:\\Users\\Alice"} # ✅ PARSE: Unicode escapes {"unicode": "\\u0048\\u0065\\u006c\\u006c\\u006f"} ``` -------------------------------- ### Elixir: Setting Up Real-World Data Collection for JsonRemedy Source: https://github.com/nshkrdotcom/json_remedy/blob/main/docs/design/102.md This Elixir module outlines strategies for continuous data collection to improve the JsonRemedy system. It covers setting up anonymous usage telemetry, a community contribution system for user-provided examples, comparative testing against other JSON libraries, and synthetic data generation based on learned patterns. ```Elixir defmodule JsonRemedy.DataPipeline do @doc """ Once your system is working, collect real-world usage data to continuously improve it. """ def setup_data_collection do # Strategy 1: Anonymous usage metrics (with user consent) setup_telemetry_collection() # Strategy 2: Community contributions setup_community_contribution_system() # Strategy 3: Automated testing against other JSON libraries setup_comparative_testing() # Strategy 4: Synthetic data generation based on learned patterns setup_synthetic_data_generation() end defp setup_telemetry_collection do # Collect (with permission): # - Input patterns that fail # - Successful repair patterns # - Performance metrics # - Error types encountered end defp setup_community_contribution_system do # Create a simple way for users to contribute examples: # mix json_remedy.contribute "malformed json" "corrected json" end end ``` -------------------------------- ### Elixir Binary Pattern Matching for Known Structures Source: https://github.com/nshkrdotcom/json_remedy/blob/main/docs/desktop/CRITIQUE.md Illustrates how binary pattern matching is effective for parsing well-defined, fixed binary structures like HTTP headers, where the format is predictable and known. ```Elixir <<"HTTP/1.1 ", status::binary-size(3), " ", rest::binary>> ``` -------------------------------- ### Optimized Character Access with Binary Pattern Matching (O(n)) in Elixir Source: https://github.com/nshkrdotcom/json_remedy/blob/main/docs/desktop/QUADRATIC_SOLVE.md This Elixir function demonstrates efficient character processing using binary pattern matching. This approach provides O(1) access to the head of the binary and O(1) termination, eliminating the O(n²) bottleneck of `String.at/2`. ```Elixir defp process_binary(<>) do # ✅ O(1) access process_binary(rest) end defp process_binary(<<>>) do # ✅ O(1) termination :done end ``` -------------------------------- ### Markdown Code Fence Example Source: https://github.com/nshkrdotcom/json_remedy/blob/main/docs/desktop/LAYER_1.md Demonstrates a markdown-style code block with a JSON payload, which Layer 1's code fence removal functionality processes. ```markdown ```json {"key": "value"} ``` ``` -------------------------------- ### Identifying O(n²) Character Access Bottleneck with String.at/2 in Elixir Source: https://github.com/nshkrdotcom/json_remedy/blob/main/docs/desktop/QUADRATIC_SOLVE.md This snippet shows the use of `String.at(input, pos)` within O(n) loops, which causes an O(n²) performance bottleneck. `String.at/2` scans from the beginning of the string, making each access proportional to the position. ```Elixir # Critical character access points (Lines 346, 664, 766, 916, 1231) char = String.at(input, pos) # O(pos) operation in O(n) loop ``` -------------------------------- ### JSON Boolean and Null Literal Normalization Example Source: https://github.com/nshkrdotcom/json_remedy/blob/main/docs/desktop/LAYER_3.md This snippet illustrates how Layer 3 standardizes non-JSON compliant boolean and null literals (e.g., Python/JavaScript style) to their JSON-compliant lowercase forms. ```JSON // Input (Python/JavaScript style literals) {"active": True, "verified": False, "data": None, "status": NULL} // Output (Layer 3 normalized) {"active": true, "verified": false, "data": null, "status": null} ``` -------------------------------- ### JSONParser Context Values: ContextValues Enum Source: https://github.com/nshkrdotcom/json_remedy/blob/main/docs/desktop/1.md The `ContextValues` Enum defines the different parsing contexts used by the JSON parser: `OBJECT_KEY`, `OBJECT_VALUE`, and `ARRAY`, guiding context-aware parsing decisions. ```APIDOC ContextValues Enum: - Defines the different parsing contexts: - OBJECT_KEY - OBJECT_VALUE - ARRAY ``` -------------------------------- ### Repairing Truncated LLM JSON Responses Source: https://github.com/nshkrdotcom/json_remedy/blob/main/docs/desktop/05_DETAILED_TEST_SPEC_AND_CASES.md This example highlights JsonRemedy's capability to repair and complete malformed or truncated JSON outputs, a common issue with streaming or interrupted LLM responses. ```json { "users": [ { "id": 1, "name": "Alice", "email": "alice@example.com", "active": true, "profile": { "city": "New York", "age": 30, "interests": ["coding", "design" ``` ```elixir {:ok, result} = JsonRemedy.repair(truncated_response) assert result["users"] |> hd() |> Map.get("name") == "Alice" assert result["users"] |> hd() |> Map.get("email") == "alice@example.com" assert result["users"] |> hd() |> get_in(["profile", "city"]) == "New York" ``` -------------------------------- ### Optimize Elixir Character Access with Binary Pattern Matching Source: https://github.com/nshkrdotcom/json_remedy/blob/main/docs/desktop/QUADRATIC_SOLVE.md This snippet illustrates how to replace inefficient position-based character access (O(n) for length, O(pos) for String.at/2) in Elixir with highly efficient binary pattern matching (O(1)). This technique processes binaries segment by segment, avoiding costly string length calculations and random access, leading to significant performance gains. ```Elixir # Template for converting position-based character access defp original_function(input, pos, state) do if pos >= String.length(input) do # ❌ O(n) state else char = String.at(input, pos) # ❌ O(pos) new_state = process_char(char, state) original_function(input, pos + 1, new_state) end end ``` ```Elixir # ↓ CONVERT TO ↓ defp optimized_function(<>, state) do # ✅ O(1) new_state = process_char(char, state) optimized_function(rest, new_state) end defp optimized_function(<<>>, state), do: state # ✅ O(1) ``` -------------------------------- ### JsonRemedy Project File Structure Source: https://github.com/nshkrdotcom/json_remedy/blob/main/docs/desktop/LAYER3_QUAD.md Details the new and modified files within the `lib/json_remedy/layer3/` directory, outlining the organization of optimized modules and performance utilities for the JsonRemedy project. ```Filesystem lib/json_remedy/layer3/ ├── syntax_normalization.ex # Updated main module ├── optimized/ │ ├── iolist_builder.ex # IO list utilities │ ├── binary_parser.ex # Binary pattern matching │ ├── single_pass_processor.ex # Unified processor │ └── context_tracker.ex # State management └── performance/ ├── benchmarks.ex # Performance test suite └── profiler.ex # Instrumentation utilities Modified Files: lib/json_remedy/layer3/syntax_normalization.ex # Main API preserved, implementation optimized test/unit/layer3_syntax_normalization_test.exs # Add performance tests test/critical/performance_stress_layer_3_test.exs # Update performance targets ``` -------------------------------- ### Validating Syntax Normalization Options in Elixir Source: https://github.com/nshkrdotcom/json_remedy/blob/main/docs/desktop/LAYER_3.md This Elixir example illustrates how to use `SyntaxNormalization.validate_options` to ensure that configuration options provided to the normalization layer are correctly typed and valid. It helps prevent runtime errors due to misconfigured settings. ```Elixir SyntaxNormalization.validate_options([normalize_quotes: "yes"]) # => {:error, "Option normalize_quotes must be a boolean"} ```