### Run tuning scripts on example files Source: https://github.com/agent-ecosystem/topicsleuth/blob/main/data/examples/README.md Commands to run the tuning tools against the synthetic example files. The first compares a new replay against the example replay; the second compares labels using the model key; the third labels pairs with a limit of one. ```bash python scripts/diff_tuning_runs.py data/examples/replay.example.json tuning_runs/.json python scripts/compare_labels.py --key data/examples/model_key.example.json data/examples/labels.example.json python scripts/label_pairs_openai.py --pairs data/examples/pairs.example.json --limit 1 ``` -------------------------------- ### topicsleuth command usage examples Source: https://github.com/agent-ecosystem/topicsleuth/blob/main/README.md Lists the topicsleuth commands for auditing, merging, splitting, adding, moving, reprocessing, seeding, backfilling embeddings, and cleaning a store. All commands that write accept --dry-run and show the plan first. The store is configured via environment variables (MONGODB_URI, TOPICSLEUTH_DATABASE, TOPICSLEUTH_SETTINGS, optional model names, and collection overrides). ```bash topicsleuth audit # how big the store is and how evidence is spread topicsleuth suggest-merges # pairs of topics that look like the same question topicsleuth merge-topics # fold one into the other; the judge rewrites the survivor topicsleuth split-topic # the judge proposes sub-topics and reassigns the entries topicsleuth add-topic topic.yaml --link # open a page by hand and move named entries onto it topicsleuth move-observation # move one document's entry to another page topicsleuth reprocess-tail --max-obs 1 --older-than-days 60 # clear pages that never got a second entry topicsleuth seed topics.yaml # start an empty store from a list of topics topicsleuth backfill-embeddings # embed topics that have none; refresh well-evidenced ones topicsleuth clean --hours 28 # remove what the last run wrote ``` -------------------------------- ### Install topicsleuth with pip Source: https://github.com/agent-ecosystem/topicsleuth/blob/main/README.md Install the topicsleuth package with all optional dependencies using pip. Requires Python 3.12 or higher. ```bash pip install "topicsleuth[all]" ``` -------------------------------- ### Use topicsleuth backends, record, settings, and synthesize Source: https://github.com/agent-ecosystem/topicsleuth/blob/main/README.md Import default clients from `backends` and use `record`, `settings`, and `synthesize` to make and store observations for your topics. Requires a `settings.yaml` file and a MongoDB URI. Recording is idempotent: retries complete unfinished stages, and a `run_id` lets `clean --run-id` remove a specific run. ```python from datetime import date from topicsleuth.backends.anthropic import AnthropicJudge from topicsleuth.backends.mongo import MongoStore from topicsleuth.backends.voyage import VoyageEmbedder from topicsleuth.record import record from topicsleuth.settings import load_settings from topicsleuth.synthesize import synthesize_items settings = load_settings("settings.yaml") store = MongoStore.from_uri(MONGODB_URI, database="briefing") embedder = VoyageEmbedder(model="voyage-4", dimensions=settings.embedding.dimensions) judge = AnthropicJudge(model="claude-opus-4-8") items = fetch_new_documents() # your dicts: url, title, summary, source_type, tags ... class Eval: # what the library needs per document def __init__(self, url, embedding, admitted): self.url, self.embedding, self.above_threshold = url, embedding, admitted vectors = embedder.embed_documents([f"{d['title']}\n\n{d.get('summary', '')}" for d in items]) evals = [Eval(d["url"], v, True) for d, v in zip(items, vectors)] decisions = synthesize_items(items, evals, store, settings, judge, embedder) result = record(items, evals, decisions, store, settings=settings, embedder=embedder, run_id=f"my-feed-{date.today()}") # so `clean --run-id` can remove this run for obs in result.observations_created: print(f"{obs['item_title']} -> {obs['topic_id']} ({obs['connection_type']})") for obs in result.already_recorded: # held from an earlier run; not judged or written again print(f"{obs['item_title']} was already on {obs['topic_id']}") for failure in result.failures: # retry these next run; do not mark them done print(f"{failure.item_url} failed at {failure.stage}: {failure.error}") ``` -------------------------------- ### Baseline and replay tuning runs Source: https://github.com/agent-ecosystem/topicsleuth/blob/main/README.md Run two baseline samples on 30 documents each with different seeds, then replay one baseline after a change and diff the reports. Each baseline writes tuning_runs/synthesize-