AgentLeak

https://github.com/yagobski/agentleak
Admin
AgentLeak is an open-source privacy-leakage testing tool for AI agents that audits execution...
Tokens:201,482
Snippets:1,394
Trust Score:7.6
License:MIT
Update:2 days ago
Show doc for...
Tokens:
Raw
### Install and run the web GUI

Source: https://github.com/yagobski/agentleak/blob/main/docs/gui.md

Install the GUI extra and start the server. The default URL is http://127.0.0.1:8000. Use --port and --no-browser to customize.

```bash
pip install 'agentleak[gui]'
agentleak serve                 # http://127.0.0.1:8000
agentleak serve --port 9000 --no-browser
```

--------------------------------

### Install and start agentleak serve

Source: https://github.com/yagobski/agentleak/blob/main/docs/platform.md

Installs the GUI-enabled agentleak package and starts the local platform server at http://127.0.0.1:8000.

```bash
pip install 'agentleak[gui]'
agentleak serve            # http://127.0.0.1:8000
```

--------------------------------

### Install GUI and serve locally

Source: https://github.com/yagobski/agentleak/blob/main/docs/cli.md

Installs the GUI extra and starts the server. `--local` restricts to loopback addresses only, because the workspace belongs to whoever can reach the port.

```bash
pip install "agentleak[gui]"
agentleak serve --local          # single-user, no sign-in, 127.0.0.1 only
agentleak serve --port 8080 --no-browser
```

--------------------------------

### Install and run agentleak MCP server

Source: https://github.com/yagobski/agentleak/blob/main/docs/mcp.md

Install the MCP extra and start the server over stdio. Editors typically launch this command for you.

```bash
pip install "agentleak[mcp]"
agentleak mcp          # serves over stdio; editors launch this for you
```

--------------------------------

### Web GUI setup and build commands

Source: https://github.com/yagobski/agentleak/blob/main/CONTRIBUTING.md

Navigate to the frontend directory, install dependencies, run the Vite dev server (which proxies /api to http://127.0.0.1:8000), and build the bundle into ../static. After changing the frontend, run npm run build and commit the updated static files.

```bash
cd agentleak/web/frontend
npm install
npm run dev          # Vite dev server; proxies /api → http://127.0.0.1:8000
                     # In another terminal: `agentleak serve`
npm run build        # type-check + build into ../static
```

--------------------------------

### Python setup and verification commands

Source: https://github.com/yagobski/agentleak/blob/main/CONTRIBUTING.md

Install the package in editable mode with dev dependencies, then run tests, lint, and type-check. All commands are executed from the project root.

```bash
pip install -e ".[dev]"
pytest                       # run the tests
ruff check agentleak/        # lint
mypy agentleak/              # type-check
```

--------------------------------

### Run AgentLeak local web UI

Source: https://github.com/yagobski/agentleak/blob/main/README.md

Install the GUI extra and start the local platform server. The web UI will be available at http://127.0.0.1:8000/app/.

```bash
pip install 'agentleak[gui]'
agentleak serve              # opens http://127.0.0.1:8000/app/
```

--------------------------------

### Install MCP extra and start MCP server

Source: https://github.com/yagobski/agentleak/blob/main/docs/cli.md

Installs the MCP extra and starts the MCP server. Serves privacy_preflight, privacy_scan_code, privacy_check_trace, and privacy_redact as MCP tools for coding agents.

```bash
pip install "agentleak[mcp]"
agentleak mcp
```

--------------------------------

### Start app and check readiness (Path A)

Source: https://github.com/yagobski/agentleak/blob/main/deploy/README.md

Builds and starts the Docker Compose stack, then verifies the app is ready on the loopback interface.

```bash
docker compose up -d --build
curl -fsS http://127.0.0.1:8787/readyz    # {"status":"ready", ...}
```

--------------------------------

### Install AgentLeak with full extras

Source: https://github.com/yagobski/agentleak/blob/main/docs/install.md

Install AgentLeak with all optional extras.

```bash
pip install "agentleak[full]"
```

--------------------------------

### Install AgentLeak with GUI extra

Source: https://github.com/yagobski/agentleak/blob/main/docs/install.md

Install AgentLeak with the GUI extra to enable the local web UI and the 'agentleak serve' command.

```bash
pip install "agentleak[gui]"
```

--------------------------------

### Install and run AgentLeak with a bundled scenario

Source: https://github.com/yagobski/agentleak/blob/main/docs/trace-analysis.md

Installs AgentLeak, initializes it, and runs the healthcare_patient_summary scenario. Reports are written to reports/ as JSON, HTML, and Markdown.

```bash
pip install agentleak
agentleak init
agentleak run --scenario healthcare_patient_summary
```

--------------------------------

### Check AgentLeak availability and install if missing

Source: https://github.com/yagobski/agentleak/blob/main/agentleak/skill/SKILL.md

Run this command to verify AgentLeak is installed. If not, it installs the package via pip.

```bash
agentleak version || pip install agentleak
```

--------------------------------

### Install AgentLeak and optional GUI

Source: https://github.com/yagobski/agentleak/blob/main/docs/redteam.md

Install the agentleak package with pip. The optional 'agentleak[gui]' extra installs a local UI and API server.

```bash
pip install agentleak
# Optional local UI and API server:
pip install 'agentleak[gui]'
```

--------------------------------

### Install AgentLeak from source

Source: https://github.com/yagobski/agentleak/blob/main/README.md

Clone the repository and install in editable mode with development dependencies. Use this when you want to contribute or modify the source code.

```bash
git clone https://github.com/yagobski/agentleak.git
cd agentleak
pip install -e ".[dev]"
```

--------------------------------

### Build and start the container (loopback only)

Source: https://github.com/yagobski/agentleak/blob/main/deploy/APACHE-DEPLOYMENT.md

Builds and starts the container with Docker Compose, then checks readiness on 127.0.0.1:8787. The container publishes only on loopback, so it is never reachable from the internet except through the reverse proxy.

```bash
docker compose build
docker compose up -d
curl -fsS http://127.0.0.1:8787/readyz    # {"status":"ready", ...}
```

--------------------------------

### Install AgentLeak with pip

Source: https://github.com/yagobski/agentleak/blob/main/docs/quickstart.md

Install the core package, optional GUI, Presidio detector, or full installation. The GUI requires FastAPI and React; Presidio adds Tier-2b detection with 12 domain recognizers.

```bash
pip install agentleak
pip install 'agentleak[gui]'              # + local web UI (FastAPI + React)
pip install 'agentleak[presidio]'         # + Tier-2b detector (Presidio + 12 domain recognizers)
pip install 'agentleak[full]'             # gui + presidio
```

--------------------------------

### Install AgentLeak MCP tools for coding agents

Source: https://github.com/yagobski/agentleak/blob/main/agentleak/skill/SKILL.md

Install the MCP tools for coding agents. After installation, point your MCP client at 'agentleak mcp'. The 'privacy_preflight' tool scans the project and reports what is new since the last check.

```bash
pip install "agentleak[mcp]"     # then point your MCP client at: agentleak mcp
```

--------------------------------

### Install AgentLeak with Presidio extra

Source: https://github.com/yagobski/agentleak/blob/main/docs/install.md

Install AgentLeak with the Presidio extra for advanced PII detection with 12 custom recognizers.

```bash
pip install "agentleak[presidio]"
```

--------------------------------

### Install Presidio with agentleak[presidio] and download spaCy model

Source: https://github.com/yagobski/agentleak/blob/main/docs/detection.md

Install the Presidio extra and download the required spaCy language model.

```bash
pip install 'agentleak[presidio]'
python -m spacy download en_core_web_lg
```

--------------------------------

### Install and run agentleak scan

Source: https://github.com/yagobski/agentleak/blob/main/docs/code-scan.md

Install agentleak via pip and run a basic scan on a local directory. The scan accepts a local directory, ZIP archive, or GitHub repository.

```bash
pip install agentleak
agentleak scan ./my-agent
```

--------------------------------

### Run Python example scripts

Source: https://github.com/yagobski/agentleak/blob/main/examples/README.md

These commands execute the runnable Python scripts in the examples directory. Each script demonstrates a different use case: simple_agent.py is an SDK trace builder, multiagent_pipeline.py uses a generic recorder for multi-agent shapes, and claude_code_transcript.py converts a Claude Code sub-agent transcript (JSONL) into a trace.

```bash
python examples/simple_agent.py          # SDK trace builder
python examples/multiagent_pipeline.py   # generic recorder, multi-agent shape
python examples/claude_code_transcript.py  # Claude Code sub-agent transcript (JSONL) -> trace
```

--------------------------------

### Proxy with signing key

Source: https://github.com/yagobski/agentleak/blob/main/docs/cli.md

Starts the proxy with a signing key to sign evidence entries.

```bash
agentleak proxy --sign-key keys/evidence.key -- …
```

--------------------------------

### Install AgentLeak with pip extras

Source: https://github.com/yagobski/agentleak/blob/main/README.md

Install the core agentleak package or add optional features: gui for the local web UI, mcp for MCP tools, presidio for Presidio, and full for everything. Choose the appropriate command based on your needs.

```bash
pip install agentleak               # core
pip install "agentleak[gui]"        # + local web UI
pip install "agentleak[mcp]"        # + MCP tools for coding agents
pip install "agentleak[presidio]"   # + Presidio
pip install "agentleak[full]"       # everything above
```

--------------------------------

### Verify AgentLeak installation

Source: https://github.com/yagobski/agentleak/blob/main/docs/install.md

Check the installed version and run a demo scan on the healthcare scenario. Expected output includes a Risk Index, per-channel breakdown, and report path.

```bash
agentleak version
agentleak scan --scenario healthcare_patient_summary
```

--------------------------------

### agentleak skill command usage

Source: https://github.com/yagobski/agentleak/blob/main/docs/cli.md

Shows all skill subcommands: check installation location, install into detected agents, specify target agents, custom path, dry-run, force overwrite, uninstall, and print skill text. Exit codes: 0 success, 1 conflict or no agent detected, 2 invalid usage. Start a new agent session after installing.

```bash
agentleak skill                       # where is it installed?
agentleak skill --install             # write it into every detected agent
agentleak skill --install -t claude-code -t cursor
agentleak skill --install --path ~/.config/myagent/skills
agentleak skill --install --dry-run   # show the plan, write nothing
agentleak skill --install --force     # overwrite a file that differs from ours
agentleak skill --uninstall
agentleak skill --print               # dump the skill text to stdout
```

--------------------------------

### SDK usage in 6 lines

Source: https://github.com/yagobski/agentleak/blob/main/docs/quickstart.md

Minimal Python example: create a Trace, add events for tool_call and final_output channels, run analysis with AgentLeakRunner, and print the risk index and verdict. The comment shows expected output 0.86 Fail.

```python
from agentleak import Trace, AgentLeakRunner

trace = Trace(run_id="demo")
trace.add_event(channel="tool_call", content={"nam": "TREM12345678"})
trace.add_event(channel="final_output", content="Follow-up scheduled.")

result = AgentLeakRunner().analyze(trace)
print(result.risk_index, result.verdict)      # 0.86 Fail
```

--------------------------------

### Safe mode dry-run install

Source: https://github.com/yagobski/agentleak/blob/main/docs/install.md

Show what would be installed without installing anything. Use for cautious users or important servers.

```bash
pip install agentleak --dry-run          # show what would be installed, install nothing
```

--------------------------------

### Run proxy with config and evidence output

Source: https://github.com/yagobski/agentleak/blob/main/docs/cli.md

Starts the proxy with a configuration file and evidence output, launching the real MCP server as a child. Judges every tools/call against privacy_policy.flows before forwarding.

```bash
agentleak proxy --config agentleak.yaml --evidence evidence.jsonl \
  -- npx -y @modelcontextprotocol/server-github
```

--------------------------------

### Install AgentLeak with pip

Source: https://github.com/yagobski/agentleak/blob/main/docs/install.md

Install the core AgentLeak package. Optional extras are available for GUI, MCP, Presidio, LLM, and full installation.

```bash
pip install agentleak
```

--------------------------------

### CLI: Initialize and run AgentLeak

Source: https://github.com/yagobski/agentleak/blob/main/README.md

Use the CLI to scaffold a project with `agentleak init`, analyze the bundled healthcare scenario with `agentleak run --scenario healthcare_patient_summary`, or analyze your own trace file with `agentleak run --trace traces/example_trace.json --format json,html,markdown`. Outputs a console summary and reports in `reports/<run_id>.{json,html,md}`.

```bash
# scaffold a project (config + folders + a sample trace)
agentleak init

# analyze the bundled healthcare scenario
agentleak run --scenario healthcare_patient_summary

# or analyze your own trace file
agentleak run --trace traces/example_trace.json --format json,html,markdown
```

--------------------------------

### Install agentleak editable

Source: https://github.com/yagobski/agentleak/blob/main/docs/PRODUCT-AUDIT.md

Command to install agentleak in editable mode, which was re-tested and works from anywhere.

```bash
pip install -e .
```

--------------------------------

### Continuous monitoring with Monitor and watch

Source: https://github.com/yagobski/agentleak/blob/main/docs/trust-and-monitoring.md

Python example showing how to set up a Monitor with sampling and drop thresholds, and wrap an agent call with watch to record a run. The monitor runs in-process with no thread or timer, and on_alert is called outside the lock.

```python
from agentleak import Monitor, watch

monitor = Monitor(sample=0.05, drop=10, on_alert=notify_the_team)

def handle(request):
    if monitor.should_sample():
        with watch(project="support-bot") as run:
            answer = agent(request)
        monitor.record(run.report.to_dict())
    else:
        answer = agent(request)
    return answer

```

--------------------------------

### Complete AgentLeak YAML configuration example

Source: https://github.com/yagobski/agentleak/blob/main/docs/configuration.md

Full configuration example showing all top-level sections: project, agent, scenarios, channels, detectors, detection, scoring, vault, privacy_policy, privacy, and reports. Includes a YAML language server schema reference. Note that provider keys and real personal data should never be placed in this file; resolve credentials via environment variables or a secret manager.

```yaml
# yaml-language-server: $schema=https://www.agentleak.org/api/schemas/config
project: {name: support-bot, description: Privacy regression suite}
agent: {name: support-bot, type: generic, endpoint: null}
scenarios:
  - {id: healthcare_patient_summary, enabled: true}
channels: [user_input, tool_call, tool_response, shared_memory, log, generated_file, inter_agent_message, final_output]
detectors: {pii: true, secrets: true, healthcare: true, finance: false, hr: false}
detection:
  mode: fast
  presidio: {enabled: false, score_threshold: 0.5}
  llm_judge: {enabled: false, threshold: 0.7}
scoring: {fail_below: 40, conditional_below: 70, block_on_critical: true, weights: [1, 2, 3, 4]}
vault:
  levels: {"1": 40, "2": 12, "3": 5, "4": 2}
  scope_def: customer records reachable by support-bot
privacy_policy: {max_risk_index: 0.20, max_findings: 0, forbid_levels: [4]}
privacy: {redact_values: true, store_raw_traces: false}
reports: {output_dir: reports, formats: [json, html, markdown]}

```

--------------------------------

### Install agentdojo from upstream

Source: https://github.com/yagobski/agentleak/blob/main/scripts/packs/README.md

Installs the agentdojo package from the upstream GitHub repository into a throwaway virtualenv. Requires Python >= 3.10.

```bash
pip install git+https://github.com/ethz-spylab/agentdojo.git
```

--------------------------------

### Verify deployment with curl

Source: https://github.com/yagobski/agentleak/blob/main/deploy/README.md

Checks the health and meta endpoints, then runs a full agent onboarding flow. The first account created becomes the admin.

```bash
curl -fsS https://www.agentleak.org/api/health     # {"status":"ok","version":"0.7.x"}
curl -fsS https://www.agentleak.org/api/meta | jq .free_tier

# Full agent flow, end to end:
curl -sX POST https://www.agentleak.org/api/agent/onboard \
  -H 'content-type: application/json' \
  -d '{"email":"smoke@fomox.com","agent_name":"SmokeBot"}'
```

--------------------------------

### Onboard agent and create API key

Source: https://github.com/yagobski/agentleak/blob/main/docs/agent-api.md

POST to /api/agent/onboard with an email and agent name. The response creates an account, project, and a project-scoped ak_... key. Store the key in a secret manager immediately.

```bash
curl -sS -X POST "$BASE/api/agent/onboard" \
  -H 'content-type: application/json' \
  -d '{"email":"owner@example.com","agent_name":"SupportBot"}'
```

--------------------------------

### Scaffold a project with agentleak init

Source: https://github.com/yagobski/agentleak/blob/main/docs/quickstart.md

Creates the project structure: agentleak.yaml configuration, scenarios/ directory, reports/ directory, and traces/example_trace.json.

```bash
agentleak init
```

--------------------------------

### Install AgentLeak with LLM extra

Source: https://github.com/yagobski/agentleak/blob/main/docs/install.md

Install AgentLeak with the LLM extra to enable semantic detection using an LLM judge for paraphrased leaks.

```bash
pip install "agentleak[llm]"
```

--------------------------------

### Post-release verification commands

Source: https://github.com/yagobski/agentleak/blob/main/docs/releasing.md

Run these commands from a clean environment to verify the release works. If they succeed, downstream instructions like the skill installer and docs/install.md will work.

```bash
# The install path everyone else follows, from a clean environment.
python -m venv /tmp/check && /tmp/check/bin/pip install agentleak
/tmp/check/bin/agentleak version
/tmp/check/bin/agentleak run --scenario healthcare_patient_summary
```

--------------------------------

### Install AgentLeak with MCP extra

Source: https://github.com/yagobski/agentleak/blob/main/docs/install.md

Install AgentLeak with the MCP extra to enable coding agent integration (e.g., Claude Code, Cursor).

```bash
pip install "agentleak[mcp]"
```

--------------------------------

### Run in local mode without sign-in

Source: https://github.com/yagobski/agentleak/blob/main/docs/gui.md

Start the server in local mode, which creates an implicit owner account on first use with an unusable password. It only listens on loopback and fails closed at multiple layers.

```bash
agentleak serve --local
```

--------------------------------

### POST /api/agent/onboard

Source: https://github.com/yagobski/agentleak/blob/main/docs/agent-api.md

Creates an account, project, and API key.

```APIDOC
## POST /api/agent/onboard

### Description
Creates an account, project, and API key.

### Method
POST

### Endpoint
/api/agent/onboard
```

--------------------------------

### Reproduce encoded leaks benchmark

Source: https://github.com/yagobski/agentleak/blob/main/docs/encoded-leaks.md

Run the benchmark script from this checkout or against the installed version. The first command uses the current source, the second uses the installed package.

```bash
python scripts/encoded_leaks.py                 # this checkout
python scripts/encoded_leaks.py --installed     # whatever version is installed
```

--------------------------------

### Start bundled Caddy profile

Source: https://github.com/yagobski/agentleak/blob/main/deploy/README.md

Starts the Docker Compose stack with the Caddy profile, which automatically obtains and renews the agentleak.org certificate. Use only if ports 80/443 are free.

```bash
docker compose --profile caddy up -d --build
```

--------------------------------

### POST /api/agent/improve response JSON example

Source: https://github.com/yagobski/agentleak/blob/main/docs/selftest-agents.md

Example JSON response from the improve endpoint, showing privacy score, risk index, delta, and prioritized next steps.

```json
{
  "privacy_score": 38, "risk_index": 0.62, "passed": false,
  "delta": {"previous_run_id": "run_...", "delta_score": -4, "direction": "regressed"},
  "next_steps": [
    {"kind": "runtime_leak", "priority": "critical", "channel": "tool_call",
     "data_types": ["ssn"], "action": "Strip or mask sensitive fields…", "code_fix": "..."},
    {"kind": "compliance", "priority": "high", "framework": "gdpr",
     "action": "Resolve 3 at-risk control(s) for GDPR (EU 2016/679)."},
    {"kind": "code_scan", "priority": "high", "action": "Fix 5 static finding(s)…"}
  ]
}
```

--------------------------------

### Agent card JSON example

Source: https://github.com/yagobski/agentleak/blob/main/docs/selftest-agents.md

Example of an AgentCard.json document declaring agent identity and code source for static scanning. Includes capabilities, endpoints, and privacy declarations.

```json
{
  "name": "support-bot",
  "description": "Handles support tickets",
  "capabilities": ["ticket_triage", "crm_lookup"],
  "tags": ["support"],
  "examples": ["triage this ticket"],
  "input_mode": "text",
  "output_mode": "text",
  "agent_protocol_version": "a2a-v1",
  "endpoints": {"/chat": "Chat endpoint", "/health": "Health check"},
  "source": {"type": "github", "repo": "acme/support-bot", "branch": "main"},
  "privacy": {"declared_data_types": ["email", "phone_number"]}
}
```

--------------------------------

### Run build_agentdojo_pack.py

Source: https://github.com/yagobski/agentleak/blob/main/scripts/packs/README.md

Runs the AgentDojo pack builder. Needs the agentdojo package importable.

```bash
python scripts/packs/build_agentdojo_pack.py
```

--------------------------------

### Run build_privacylens_pack.py

Source: https://github.com/yagobski/agentleak/blob/main/scripts/packs/README.md

Runs the PrivacyLens pack builder. Expects privacylens.json in the working directory.

```bash
python scripts/packs/build_privacylens_pack.py
```

--------------------------------

### YAML policy example for the runtime gateway

Source: https://github.com/yagobski/agentleak/blob/main/docs/runtime-gateway.md

Example policy in the shared contextual-integrity grammar. A flow for 'sin' to 'server-github' for 'identity_check' is permitted; a flow for 'health_condition' to any destination is denied.

```yaml
privacy_policy:
  flows:
    - data_type: sin
      to: server-github
      for: identity_check
    - data_type: health_condition
      to: "*"
      deny: true
```

--------------------------------

### Enable required Apache modules

Source: https://github.com/yagobski/agentleak/blob/main/deploy/APACHE-DEPLOYMENT.md

Run this command to enable the Apache modules needed for proxying and WebSocket support: proxy, proxy_http, proxy_wstunnel, headers, rewrite, and ssl.

```bash
sudo a2enmod proxy proxy_http proxy_wstunnel headers rewrite ssl
```

--------------------------------

### Run agentleak proxy with config and evidence

Source: https://github.com/yagobski/agentleak/blob/main/docs/runtime-gateway.md

Launches the proxy in front of an MCP server. The proxy starts the real server as a child and forwards JSON-RPC between them. Requires --config with privacy_policy.flows; without it nothing is refused but calls are still recorded. --evidence defaults to agentleak-evidence.jsonl.

```bash
agentleak proxy \
  --config agentleak.yaml \
  --evidence evidence.jsonl \
  -- npx -y @modelcontextprotocol/server-github
```

--------------------------------

### List scenario packs from CLI

Source: https://github.com/yagobski/agentleak/blob/main/docs/scenarios.md

Lists all available scenario packs from the command line.

```bash
agentleak scenarios --packs
```

--------------------------------

### privacy_policy.flows YAML rule example

Source: https://github.com/yagobski/agentleak/blob/main/docs/contextual-integrity.md

Example YAML for defining flow permissions and prohibitions. Shows a permission for SIN to kyc-vendor for identity_check, a permission for health data to clinician/ehr, and a deny-all rule for analytics.

```yaml
privacy_policy:
  flows:
    - data_type: sin
      to: kyc-vendor
      for: identity_check

    - data_type: [health_condition, medication]
      to: [clinician, ehr]

    - data_type: "*"
      to: analytics
      deny: true
      description: "Nothing personal reaches the analytics sink."
```
Tokens:
Raw