### OpenCode Installation Instructions Source: https://github.com/bitflight-devops/hallucination-detector/blob/main/README.md Instruct OpenCode to fetch and follow installation instructions from a specific URL. This method is used for platforms that support fetching external installation scripts. ```text Fetch and follow instructions from https://raw.githubusercontent.com/bitflight-devops/hallucination-detector/refs/heads/main/.opencode/INSTALL.md ``` -------------------------------- ### Install Cursor Plugin Source: https://github.com/bitflight-devops/hallucination-detector/blob/main/README.md Install the hallucination detector plugin directly from the marketplace within Cursor Agent chat. This is a single-command installation. ```text /plugin-add hallucination-detector ``` -------------------------------- ### Codex Installation Instructions Source: https://github.com/bitflight-devops/hallucination-detector/blob/main/README.md Instruct Codex to fetch and follow installation instructions from a specific URL. This method is used for platforms that support fetching external installation scripts. ```text Fetch and follow instructions from https://raw.githubusercontent.com/bitflight-devops/hallucination-detector/refs/heads/main/.codex/INSTALL.md ``` -------------------------------- ### Update manual installation Source: https://github.com/bitflight-devops/hallucination-detector/blob/main/README.md Navigate to the installation directory and pull the latest changes from the repository. ```bash cd && git pull ``` -------------------------------- ### Quick Install Hallucination Detector Source: https://github.com/bitflight-devops/hallucination-detector/blob/main/README.md Use the Vercel Skills CLI for a quick installation of the hallucination detector plugin. This command installs the plugin for any supported agent. ```bash npx skills add bitflight-devops/hallucination-detector ``` -------------------------------- ### Install Claude Code Plugin Source: https://github.com/bitflight-devops/hallucination-detector/blob/main/README.md Register the marketplace and install the hallucination detector plugin within Claude Code. This requires two separate commands. ```bash /plugin marketplace add bitflight-devops/hallucination-detector ``` ```bash /plugin install hallucination-detector@hallucination-detector ``` -------------------------------- ### SessionStart Hook Source: https://github.com/bitflight-devops/hallucination-detector/blob/main/docs/architecture.md Triggered at the start of a session to provide context to Claude. ```APIDOC ## SessionStart Hook ### Description Registers the start of a session and provides initial context to Claude. ### Request Body - **session_id** (string) - Unique identifier for the session - **transcript_path** (string) - Path to the session transcript - **source** (string) - Source identifier - **model** (string) - Model identifier ### Response - **stdout** (string) - Plain text added to Claude's context ``` -------------------------------- ### Targeted Agent Installation Source: https://github.com/bitflight-devops/hallucination-detector/blob/main/README.md Install the hallucination detector plugin for specific agents using the Vercel Skills CLI. This allows for granular control over plugin deployment. ```bash npx skills add bitflight-devops/hallucination-detector -a claude-code ``` ```bash npx skills add bitflight-devops/hallucination-detector -a cursor ``` ```bash npx skills add bitflight-devops/hallucination-detector -a codex ``` ```bash npx skills add bitflight-devops/hallucination-detector -a opencode ``` -------------------------------- ### Hallucination Detection Examples Source: https://github.com/bitflight-devops/hallucination-detector/blob/main/README.md Comparison of LLM behavior without and with the hallucination-detector plugin enabled. ```text You: "Why is the API returning 500 errors?" Claude: "This is probably caused by a database connection timeout. The issue is likely due to connection pool exhaustion, which typically happens under high load." ``` ```text You: "Why is the API returning 500 errors?" Claude: [attempts to respond with speculation] [BLOCKED by hallucination-detector] Claude: "I don't have enough information yet. Let me check: 1. Reading the error logs... 2. Checking database connection configuration... I found in /var/log/app.log at line 1247: 'ConnectionError: max retries exceeded for host db.internal' The logs show the API server cannot reach the database host. Next step: verify network connectivity and database status." ``` -------------------------------- ### Update hallucination-detector plugin Source: https://github.com/bitflight-devops/hallucination-detector/blob/main/README.md Use the plugin update command for standard installations. ```bash /plugin update hallucination-detector ``` -------------------------------- ### Haiku Evaluation Output Examples Source: https://github.com/bitflight-devops/hallucination-detector/blob/main/docs/evaluative-claims-bypass-design-intent__external-observer.md Expected JSON output formats for the Haiku evaluation prompt. ```json {"ok": true} ``` ```json {"ok": false, "reason": "..."} ``` -------------------------------- ### Define Structured Response Contract Source: https://github.com/bitflight-devops/hallucination-detector/blob/main/docs/architecture.md Example of the required format for audit responses, including claim IDs and evidence prefixes. ```text ANSWER - Task acknowledged. Claim c1 identifies the root cause. VERIFIED - [VERIFIED][c1] Evidence: File: scripts/hallucination-claim-structure.cjs:42 INFERRED - [INFERRED][c2] Basis: MEMORY WRITE - Allowed: c1 - Blocked: c2 ``` -------------------------------- ### Define Structured Output Templates Source: https://github.com/bitflight-devops/hallucination-detector/blob/main/docs/fix-sequence-spec.md Templates for tool runs, agent reports, and commit verification to be injected at SessionStart. ```text TOOL RUN Command: [exact command] Observed: [specific output] Scope: [what this covers] Does not cover: [what this does not establish] ``` ```text AGENT REPORT (from: [agent name/task ID]) Reported: [what the agent said] Independently verified: [yes — what was verified | no] ``` ```text COMMITTED [hash] Changes: [what was changed] Validation: [what was run | none run] ``` -------------------------------- ### Define Deployment Sequence and Dependencies Source: https://github.com/bitflight-devops/hallucination-detector/blob/main/docs/fix-sequence-spec.md The order of operations for deploying fixes to the hallucination detector. ```text Fix 1 (loop counter) — no dependencies, do first Fix 2 (compact exemption) — no dependencies, do in parallel with Fix 1 Fix 3 (narrow triggers) — after Fix 1 (ensures loop counter works before testing new suppressions) Fix 4 (completeness + evidence) — after Fix 3 (same codebase area) → Measurement checkpoint: confirm FP rate below 30% before proceeding Fix 5 (template system) — after checkpoint passes ⚠️ NOTE: Fix 5c (ungrounded_behavioral_assertion) shipped active (shadow mode, 2026-04-05) before this checkpoint. FP rate confirmation pending before switching to blocking mode. ``` -------------------------------- ### A/B Protocol for Fix Implementation Source: https://github.com/bitflight-devops/hallucination-detector/blob/main/docs/holistic-design-2026-04-04.md This outlines the step-by-step process for implementing and validating fixes using shadow mode and A/B testing. It emphasizes measurement and classification before enabling blocking mode. ```text 1. Shadow mode: 7 days, new rules log without blocking 2. Classify 20-sample from shadow log: TP / FP / impossible 3. Gate: new FP rate < current AND new TP rate >= current - 5% 4. Enable blocking mode 5. Measure same period after: compare to baseline schema ``` -------------------------------- ### Observation Expression Templates for Agent Communication Source: https://github.com/bitflight-devops/hallucination-detector/blob/main/docs/milestone-roadmap-2026-04-04.md These templates are injected at SessionStart to standardize agent communication, ensuring structured responses for better validation and reduced ambiguity. They help agents adhere to specific formats for tool runs, agent reports, and code commits. ```text TOOL RUN Command: [exact command] Observed: [specific output] Scope: [what this covers] Does not cover: [what this does not establish] ``` ```text AGENT REPORT (from: [agent name/task ID]) Reported: [what the agent said] Independently verified: [yes — what was verified | no] ``` ```text COMMITTED [hash] Changes: [what was changed] Validation: [what was run | none run] ``` -------------------------------- ### Helper function to create multi-turn transcripts Source: https://github.com/bitflight-devops/hallucination-detector/blob/main/plan/architect-absence-claim-detection.md Builds a temporary multi-turn transcript file for testing. Accepts an array of raw JSONL entry objects. Returns the file path, which the caller must clean up. ```javascript /** * Build a temporary multi-turn transcript file. * `turns` is an array of raw JSONL entry objects written in order. * Returns the file path. Caller is responsible for cleanup. */ function makeTempMultiTurnTranscript(turns) { const tmpFile = path.join( os.tmpdir(), `hd-multi-test-${Date.now()}-${Math.random().toString(36).slice(2)}.jsonl`, ); const lines = turns.map((t) => JSON.stringify(t)).join('\n'); fs.writeFileSync(tmpFile, `${lines}\n`, 'utf-8'); return tmpFile; } ``` -------------------------------- ### Test shadow mode integration Source: https://github.com/bitflight-devops/hallucination-detector/blob/main/plan/architect-absence-claim-detection.md Verifies that shadow logging occurs instead of blocking when dryRun is enabled via configuration. ```js // Add to E2E_SESSION_IDS: 'e2e-ab-1', // New describe block: describe('e2e: unsupported_absence shadow mode', () => { it('shadow-logs absence phrase when dryRun: true', () => { tmpDir = makeTmpProjectDir(); // Write a config file that enables shadow mode fs.writeFileSync( path.join(tmpDir, '.hallucination-detectorrc.cjs'), 'module.exports = { dryRun: true };\n', ); const transcriptPath = writeTranscript( tmpDir, 'The function does not exist in this module.', ); const result = runHook( { transcript_path: transcriptPath, session_id: 'e2e-ab-1' }, tmpDir, ); expect(result.status).toBe(0); // Shadow mode exits 0 with no block decision on stdout const decision = parseDecision(result.stdout); expect(decision).toBeNull(); // The hook does not crash — stderr should be empty expect(result.stderr).toBe(''); }); }); ``` -------------------------------- ### TOOL RUN Template for Expression Contracts Source: https://github.com/bitflight-devops/hallucination-detector/blob/main/docs/holistic-design-2026-04-04.md This template is used to structure tool run observations, ensuring specific output, scope, and limitations are clearly defined. It helps prevent over-generalization by requiring a 'Does not cover' field. ```markdown TOOL RUN Command: [exact command] Observed: [specific output] Scope: [what this covers] Does not cover: [what this does not establish] ``` -------------------------------- ### Shadow Mode Configuration Source: https://github.com/bitflight-devops/hallucination-detector/blob/main/docs/holistic-design-2026-04-04.md Enables shadow mode for new detection rules by setting the `dryRun` option to `true`. This logs would-block decisions without actually blocking, allowing for analysis without token cost. ```json { "dryRun": true } ``` -------------------------------- ### Enable Subagent Monitoring with monitorSubagents Source: https://github.com/bitflight-devops/hallucination-detector/blob/main/docs/milestone-roadmap-2026-04-04.md Configuration option to enable `SubagentStop` event monitoring. When set to `true`, detection runs and telemetry is written, but blocking is suppressed for subagent sessions. Defaults to `false`. ```javascript monitorSubagents: true ``` -------------------------------- ### Haiku Evaluation Prompt Template Source: https://github.com/bitflight-devops/hallucination-detector/blob/main/docs/evaluative-claims-bypass-design-intent__external-observer.md The prompt template used by the Haiku model to evaluate assistant messages for baseless design intent changes. ```text You are evaluating whether an AI agent is about to make a baseless design intent change. A baseless design intent change occurs when ALL FOUR of the following are simultaneously present: 1. A system component is described as broken or failing 2. A change to that component is proposed (removal, bypass, or modification) 3. An evaluative label justifies the change ("clean", "simple", "correct", "best", "obvious") 4. No prior statement establishes what the component protects when functioning correctly Evaluate the following assistant message: {last_assistant_message} If all four conditions are present, return: {"ok": false, "reason": "State what [component] protects when functioning correctly before proposing to change it."} If the pattern is not present, return: {"ok": true} Return only valid JSON. Do not explain. ``` -------------------------------- ### Match Object Data Model Source: https://github.com/bitflight-devops/hallucination-detector/blob/main/plan/architect-absence-claim-detection.md Defines the structure for a match object, including its kind and the evidence found. ```javascript { kind: 'unsupported_absence', // string literal evidence: string, // the matched substring from the stripped haystack } ``` -------------------------------- ### Define Session State Path Source: https://github.com/bitflight-devops/hallucination-detector/blob/main/docs/architecture.md The path used for storing temporary session state files to prevent infinite loops. ```javascript ${os.tmpdir()}/claude-hallucination-audit-${sessionId}.json ``` -------------------------------- ### Query Sessions with 3+ Blocks Source: https://github.com/bitflight-devops/hallucination-detector/blob/main/docs/milestone-roadmap-2026-04-04.md Find session IDs that have experienced three or more block events, indicating potential runaway behavior. ```sql -- Sessions with 3+ blocks (runaway indicator) SELECT session_id, COUNT(*) AS block_count FROM blocks GROUP BY session_id HAVING block_count >= 3 ORDER BY block_count DESC; ``` -------------------------------- ### Manual Audit Command Source: https://github.com/bitflight-devops/hallucination-detector/blob/main/README.md Use this command to manually trigger an audit of text for hallucination patterns. ```bash /hallucination-detector:audit ``` -------------------------------- ### Test unsupported_absence phrase detection Source: https://github.com/bitflight-devops/hallucination-detector/blob/main/plan/architect-absence-claim-detection.md Flags common phrases indicating absence or non-existence. Ensure `findTriggerMatches` is imported. ```javascript describe('unsupported_absence', () => { it('flags "does not exist"', () => { const matches = findTriggerMatches('The function does not exist in this module.'); const kinds = matches.map((m) => m.kind); expect(kinds).toContain('unsupported_absence'); }); it('flags "there is no"', () => { const matches = findTriggerMatches('There is no configuration option for this.'); const kinds = matches.map((m) => m.kind); expect(kinds).toContain('unsupported_absence'); }); it('flags "is not documented"', () => { const matches = findTriggerMatches('This behavior is not documented anywhere.'); const kinds = matches.map((m) => m.kind); expect(kinds).toContain('unsupported_absence'); }); it('flags "cannot be found"', () => { const matches = findTriggerMatches('The endpoint cannot be found in the codebase.'); const kinds = matches.map((m) => m.kind); expect(kinds).toContain('unsupported_absence'); }); }); ``` -------------------------------- ### Query Estimated Cost This Month Source: https://github.com/bitflight-devops/hallucination-detector/blob/main/docs/milestone-roadmap-2026-04-04.md Calculate the total estimated cost in USD for block events that occurred within the last 30 days. ```sql -- Estimated cost this month SELECT SUM(cost_usd) AS monthly_cost FROM blocks WHERE timestamp > datetime('now', '-30 days'); ``` -------------------------------- ### Test Update: Key Presence in DEFAULT_WEIGHTS Source: https://github.com/bitflight-devops/hallucination-detector/blob/main/plan/architect-absence-claim-detection.md Adds an assertion to `hallucination-config.test.cjs` to check for the presence of the 'unsupported_absence' key in `DEFAULT_WEIGHTS`. ```javascript expect(DEFAULT_WEIGHTS).toHaveProperty('unsupported_absence'); ``` -------------------------------- ### Query Total Blocks This Week Source: https://github.com/bitflight-devops/hallucination-detector/blob/main/docs/milestone-roadmap-2026-04-04.md Use this SQL query to count the number of block events that occurred within the last seven days. ```sql -- Total blocks this week SELECT COUNT(*) FROM blocks WHERE timestamp > datetime('now', '-7 days'); ``` -------------------------------- ### Post-filter for Unsupported Absence Matches Source: https://github.com/bitflight-devops/hallucination-detector/blob/main/plan/architect-absence-claim-detection.md This code snippet filters out 'unsupported_absence' matches if recent tool use is detected in the entries. It requires `triggerMatches` to be declared with `let` and `entries` to be available. ```javascript if (triggerMatches.some((m) => m.kind === 'unsupported_absence') && entries.length > 0) { const hasRecentToolUse = hasToolUseInRecentEntries(entries, 10); if (hasRecentToolUse) { triggerMatches = triggerMatches.filter((m) => m.kind !== 'unsupported_absence'); } } ``` -------------------------------- ### Implement Absence Claim Matching (JavaScript) Source: https://github.com/bitflight-devops/hallucination-detector/blob/main/plan/architect-absence-claim-detection.md Iterates through matches of the `ABSENCE_CLAIM_RE` in a haystack, filtering by `isIndexWithinQuestion` and adding valid matches to a results array. Ensure `lastIndex` is reset before calling `matchAll()` on a global regex. ```javascript // Reset lastIndex before matchAll() — g-flagged regex retains state. ABSENCE_CLAIM_RE.lastIndex = 0; for (const m of haystack.matchAll(ABSENCE_CLAIM_RE)) { if (isIndexWithinQuestion(haystack, m.index)) continue; matches.push({ kind: 'unsupported_absence', evidence: m[0].trim() }); } ``` -------------------------------- ### Stop Hook Source: https://github.com/bitflight-devops/hallucination-detector/blob/main/docs/architecture.md Triggered to audit the session and decide whether to block Claude's completion. ```APIDOC ## Stop Hook ### Description Audits the session transcript and returns a decision to either block or allow Claude's completion. ### Request Body - **session_id** (string) - Unique identifier for the session - **transcript_path** (string) - Path to the session transcript - **stop_hook_active** (boolean) - Status of the stop hook ### Response #### Success Response (200) - **decision** (string) - "block" to prevent completion, or empty for allow - **reason** (string) - Explanation for the block decision (required if decision is "block") ``` -------------------------------- ### Helper Function: hasToolUseInRecentEntries Source: https://github.com/bitflight-devops/hallucination-detector/blob/main/plan/architect-absence-claim-detection.md Checks if any of the last `lookback` entries contain a tool_use block with a qualifying tool name. It uses a Set for efficient lookup of qualifying tools. ```javascript function hasToolUseInRecentEntries(entries, lookback) { const QUALIFYING_TOOLS = new Set(['Read', 'Grep', 'Glob', 'WebSearch', 'WebFetch', 'Bash']); const window = entries.slice(-lookback); for (const entry of window) { const content = entry?.message?.content; if (!Array.isArray(content)) continue; for (const block of content) { if (block?.type === 'tool_use' && QUALIFYING_TOOLS.has(block.name)) { return true; } } } return false; } ``` -------------------------------- ### Query Blocks by Category Source: https://github.com/bitflight-devops/hallucination-detector/blob/main/docs/milestone-roadmap-2026-04-04.md This SQL query aggregates and counts block events, grouping them by their category to identify the most frequent triggers. ```sql -- Blocks by category SELECT category, COUNT(*) AS n FROM blocks GROUP BY category ORDER BY n DESC; ``` -------------------------------- ### Test absence phrase blocking behavior Source: https://github.com/bitflight-devops/hallucination-detector/blob/main/plan/architect-absence-claim-detection.md Verifies that the hook blocks when an absence phrase is detected and dryRun is false. ```js it('blocks when absence phrase present and no tool use in transcript', () => { const sessionId = `absence-blocks-${Date.now()}`; const transcriptPath = makeTempMultiTurnTranscript([ { type: 'assistant', message: { content: [ { type: 'text', text: 'The function does not exist in this module.' }, ], }, }, ]); try { const result = runHook({ transcript_path: transcriptPath, session_id: sessionId }); expect(result.status).toBe(0); // In shadow mode (default dryRun: false), this should block // OR be shadow-logged if dryRun defaults to true. // The hook exits 0 in both cases; check stdout for decision. // When dryRun: false (default), a block decision is emitted. if (result.stdout.trim()) { const decision = JSON.parse(result.stdout.trim()); expect(decision.decision).toBe('block'); } } finally { try { fs.unlinkSync(transcriptPath); } catch { /* ok */ } } }); ``` -------------------------------- ### Query Top Trigger Phrases Source: https://github.com/bitflight-devops/hallucination-detector/blob/main/docs/milestone-roadmap-2026-04-04.md Identify and count the top 10 most frequent trigger phrases that resulted in block events. ```sql -- Top trigger phrases SELECT trigger_phrase, COUNT(*) AS n FROM blocks GROUP BY trigger_phrase ORDER BY n DESC LIMIT 10; ``` -------------------------------- ### Tool-Use Check Result Data Model Source: https://github.com/bitflight-devops/hallucination-detector/blob/main/plan/architect-absence-claim-detection.md Represents the result of a tool-use check, indicating if recent tool usage was found. ```javascript // boolean — true when at least one qualifying tool_use block found in last 10 entries const hasRecentToolUse = /* boolean */; ``` -------------------------------- ### Full Detection Block for Absence Claims (JavaScript) Source: https://github.com/bitflight-devops/hallucination-detector/blob/main/plan/architect-absence-claim-detection.md Integrates the absence claim detection logic within a conditional block, checking if the category is enabled and if built-in patterns should be used. It includes resetting the regex's `lastIndex` and iterating through matches. ```javascript // 7) Absence claims — "X does not exist", "there is no Y", "cannot be found" etc. // without tool-use evidence in the preceding transcript turns. // Per-match suppression: isIndexWithinQuestion (text-local). // Response-level suppression: hasRecentToolUse post-filter in main() removes all // unsupported_absence matches when tool use found in last 10 entries. if (isCategoryEnabled('unsupported_absence')) { if (useBuiltIn('unsupported_absence')) { ABSENCE_CLAIM_RE.lastIndex = 0; for (const m of haystack.matchAll(ABSENCE_CLAIM_RE)) { if (isIndexWithinQuestion(haystack, m.index)) continue; matches.push({ kind: 'unsupported_absence', evidence: m[0].trim() }); } } runCustomPatterns('unsupported_absence'); } ``` -------------------------------- ### AGENT REPORT Template for Expression Contracts Source: https://github.com/bitflight-devops/hallucination-detector/blob/main/docs/holistic-design-2026-04-04.md This template structures agent reports, requiring attribution and disclosure of independent verification. It addresses the failure mode of agent-report relay by ensuring claims are not presented as the orchestrator's own verified facts. ```markdown AGENT REPORT (from: [agent name / task ID]) Reported: [what the agent said] Independently verified: [yes — what was verified | no] ``` -------------------------------- ### Test Update: Key Count in DEFAULT_WEIGHTS Source: https://github.com/bitflight-devops/hallucination-detector/blob/main/plan/architect-absence-claim-detection.md Modifies an assertion in `hallucination-config.test.cjs` to reflect the addition of a new key to `DEFAULT_WEIGHTS`. ```javascript // Before: expect(Object.keys(DEFAULT_WEIGHTS).length).toBe(6); // After: expect(Object.keys(DEFAULT_WEIGHTS).length).toBe(7); ``` -------------------------------- ### Update DEFAULT_WEIGHTS in hallucination-config.cjs Source: https://github.com/bitflight-devops/hallucination-detector/blob/main/plan/architect-absence-claim-detection.md Adds the `unsupported_absence` category with a weight of 0.7 to the default weights configuration. ```javascript const DEFAULT_WEIGHTS = { speculation_language: 0.25, causality_language: 0.3, pseudo_quantification: 0.15, completeness_claim: 0.2, evaluative_design_claim: 0.4, internal_contradiction: 0.35, unsupported_absence: 0.7, }; ``` -------------------------------- ### Uninstall Hallucination Detector Source: https://github.com/bitflight-devops/hallucination-detector/blob/main/README.md Remove the hallucination detector plugin using the Vercel Skills CLI. This command uninstalls the plugin from all targeted agents. ```bash npx skills remove hallucination-detector ``` -------------------------------- ### JSDoc Update for Return Types Source: https://github.com/bitflight-devops/hallucination-detector/blob/main/plan/architect-absence-claim-detection.md Updates the JSDoc `@returns` tag to include the new 'unsupported_absence' kind for match objects. ```javascript * @returns {Array<{kind: string, evidence: string}>} An array of match objects where `kind` is one of: * `speculation_language`, `causality_language`, `pseudo_quantification`, `completeness_claim`, * `evaluative_design_claim`, `internal_contradiction`, or `unsupported_absence`, * and `evidence` is the matched snippet from the text. ``` -------------------------------- ### Update Trigger Detection Categories in CLAUDE.md Source: https://github.com/bitflight-devops/hallucination-detector/blob/main/plan/architect-absence-claim-detection.md Adds the 'unsupported_absence' category to the list of trigger detection categories in the CLAUDE.md file. ```markdown 6. `unsupported_absence` — absence claims without tool evidence ("does not exist", "there is no", "cannot be found", etc.) ``` -------------------------------- ### Test Update: Weight Sum in DEFAULT_WEIGHTS Source: https://github.com/bitflight-devops/hallucination-detector/blob/main/plan/architect-absence-claim-detection.md Adjusts an assertion in `hallucination-config.test.cjs` to account for the updated sum of weights after adding 'unsupported_absence'. ```javascript // Before (line 35): expect(Math.abs(sum - 1.65)).toBeLessThan(1e-9); // After: expect(Math.abs(sum - 2.35)).toBeLessThan(1e-9); ``` -------------------------------- ### COMMITTED Template for Expression Contracts Source: https://github.com/bitflight-devops/hallucination-detector/blob/main/docs/holistic-design-2026-04-04.md This template is used to commit changes, requiring a validation field to ensure honesty and transparency. It addresses bare behavioral assertions by mandating a validation status, even if it's 'none run'. ```markdown COMMITTED [hash] Changes: [what was changed] Validation: [what was run | none run] ``` -------------------------------- ### Update weighted score calculations Source: https://github.com/bitflight-devops/hallucination-detector/blob/main/plan/architect-absence-claim-detection.md Adjusts expected values and score objects to account for the new unsupported_absence category weight. ```js // Before: // speculation weight = 0.25, weightSum = 1.65 // result = 0.25 / 1.65 ≈ 0.15152 // After: // speculation weight = 0.25, weightSum = 2.35 // result = 0.25 / 2.35 ≈ 0.10638 const expected = 0.25 / 2.35; expect(Math.abs(result - expected)).toBeLessThan(0.001); ``` ```js // Before comment: // Only speculation fires: 0.25 / 1.65 // After: // Only speculation fires: 0.25 / 2.35 ≈ 0.10638 (weightSum = 2.35 with unsupported_absence: 0.7) const expected = 0.25 / 2.35; ``` ```js const scores = { speculation_language: 1, causality_language: 1, pseudo_quantification: 1, completeness_claim: 1, evaluative_design_claim: 1, internal_contradiction: 1, unsupported_absence: 1, // ← add }; expect(aggregateWeightedScore(scores, DEFAULT_WEIGHTS)).toBe(1); ``` -------------------------------- ### Test hallucination suppression with tool use Source: https://github.com/bitflight-devops/hallucination-detector/blob/main/plan/architect-absence-claim-detection.md Checks that absence claims are suppressed when tool use is present in recent transcript entries. Requires `runHook` and `makeTempMultiTurnTranscript`. The session ID should be unique or purged to avoid state bleed. ```javascript it('does not block when tool use found in recent transcript entries', () => { const transcriptPath = makeTempMultiTurnTranscript([ { type: 'assistant', message: { content: [ { type: 'tool_use', name: 'Read', id: 'tu1', input: { file_path: '/tmp/x' } }, ], }, }, { type: 'tool_result', content: [{ type: 'text', text: 'file content here' }], }, { type: 'assistant', message: { content: [ { type: 'text', text: 'The function does not exist in this module.' }, ], }, }, ]); try { const result = runHook({ transcript_path: transcriptPath, session_id: 'absence-suppressed-1' }); expect(result.status).toBe(0); // No block decision emitted — absence claim suppressed by tool use if (result.stdout.trim()) { const decision = JSON.parse(result.stdout.trim()); expect(decision.decision).not.toBe('block'); } } finally { try { fs.unlinkSync(transcriptPath); } catch { /* ok */ } } }); ``` ```javascript it('does not block when tool use found in recent transcript entries', () => { const sessionId = `absence-suppressed-${Date.now()}`; // ... (same as above but with sessionId variable) }); ``` -------------------------------- ### Test absence phrase suppression in code blocks Source: https://github.com/bitflight-devops/hallucination-detector/blob/main/plan/architect-absence-claim-detection.md Verifies that absence phrases within fenced code blocks are ignored. Requires `findTriggerMatches`. ```javascript it('does not flag absence phrase inside a fenced code block', () => { const matches = findTriggerMatches( 'Check the output:\n```\n// does not exist\n```\n', ); const absenceMatches = matches.filter((m) => m.kind === 'unsupported_absence'); expect(absenceMatches.length).toBe(0); }); ``` -------------------------------- ### Module-Scope Regex Constant for Absence Claims (JavaScript) Source: https://github.com/bitflight-devops/hallucination-detector/blob/main/plan/architect-absence-claim-detection.md Declares the `ABSENCE_CLAIM_RE` constant at the module scope, alongside other regex constants. Includes a JSDoc comment explaining its purpose, usage with the `g` flag, and the need to reset `lastIndex`. ```javascript /** * Absence-claim phrases: "X does not exist", "there is no Y", "cannot be found", etc. * Used by the `unsupported_absence` detection category in findTriggerMatches(). * Must use `g` flag; reset lastIndex before matchAll(). */ const ABSENCE_CLAIM_RE = /\b(does?\s+not\s+(?:exist|support|document|mention)|do\s+not\s+(?:exist|support|document|mention)|there\s+(?:is|are)\s+no\b|(?:is|are)\s+not\s+(?:documented|available|supported|mentioned)|cannot\s+be\s+found|no\s+\w+\s+(?:found|exists?|(?:is|are)\s+available))\b/gi; ```