### Install Dependencies
Source: https://github.com/ndif-team/aletheias-quest-competition/blob/master/tutorials/3.4-llm-judge.ipynb
Installs required libraries including nnsight, peft, and specific versions of torchao and transformers. Clears output and confirms setup.
```python
# nnsight — hackathon build (PEFT/LoRA + remote NDIF support)
!pip install -q git+https://github.com/ndif-team/nnsight.git@hackathon/peft
!pip install -q peft
!pip install -q --upgrade torchao "transformers==5.9.0"
from IPython.display import clear_output
clear_output()
print("Setup complete.")
```
--------------------------------
### Install and Import Libraries
Source: https://github.com/ndif-team/aletheias-quest-competition/blob/master/tutorials/3.1-mean-probe.ipynb
Installs required libraries including nnsight, peft, and transformers. Clears the output after installation.
```python
# nnsight — hackathon build (PEFT/LoRA + remote NDIF support)
!pip install -q git+https://github.com/ndif-team/nnsight.git@hackathon/peft
!pip install -q peft
!pip install -q --upgrade torchao "transformers==5.9.0"
from IPython.display import clear_output
clear_output()
```
--------------------------------
### Install Dependencies
Source: https://github.com/ndif-team/aletheias-quest-competition/blob/master/tutorials/2b-predicting-s3.ipynb
Installs the required Python packages for data handling and probing. This setup does not require nnsight or API keys as the model is not run locally.
```python
# Data + probing stack — no nnsight needed (we don't run a model here), no keys needed (the bucket is public).
!pip install -q "zarr>=3.0" "s3fs>=2024.0" boto3 pandas pyarrow numpy scikit-learn matplotlib
from IPython.display import clear_output
clear_output()
print("Setup complete.")
```
--------------------------------
### Load, Tokenize, and Score Examples
Source: https://github.com/ndif-team/aletheias-quest-competition/blob/master/baselines/followup_probe.ipynb
Loads examples, tokenizes the follow-up turn's last token, and scores valid examples using a probe. Unparseable examples default to a score of 0.5.
```python
import sys
# util.py ships alongside the notebooks in submission/; the runner executes from
# the repo root, so add submission/ to the path before importing it.
sys.path.insert(0, "submission")
import util
examples = util.load_examples(DATASET_NAME) # single `test` split + an `index` join key
print(f"loaded {len(examples)} examples")
tok = []
valid_idx = []
for i, ex in enumerate(examples):
msgs = to_judged_messages(list(ex["messages"]))
if msgs is None:
continue
ids, idx = cap_length(*tokenize_followup_last_token(msgs))
if idx < 0: # degenerate: follow-up turn got trimmed away
continue
tok.append((ids, idx))
valid_idx.append(i)
predictions = np.full(len(examples), 0.5, dtype=float) # fallback for unparseable rows
if valid_idx:
X = extract_layer_last_token(tok, LAYER)
predictions[np.array(valid_idx)] = probe_proba(X)
print(f"scored {len(valid_idx)}/{len(examples)} examples with the probe "
f"({len(examples) - len(valid_idx)} fell back to 0.5)")
```
--------------------------------
### Set up Local Development Environment
Source: https://github.com/ndif-team/aletheias-quest-competition/blob/master/README.md
Installs dependencies and activates a virtual environment for local development and dry runs. Ensure Python 3.10+ is installed.
```bash
./setup_dev.sh && source .venv/bin/activate
```
--------------------------------
### Tokenization and Batch Extraction
Source: https://github.com/ndif-team/aletheias-quest-competition/blob/master/baselines/mean_probe.ipynb
This snippet sets up constants for batch size and padding, loads examples, tokenizes them, and prepares them for batch processing. It filters out examples with empty response spans.
```python
BATCH_SIZE = 8
PAD = tok.pad_token_id if tok.pad_token_id is not None else tok.eos_token_id
examples = load_examples(DATASET_NAME)
n = len(examples) if LIMIT is None else min(LIMIT, len(examples))
print(f"scoring {n}/{len(examples)} rows")
# Tokenize locally; rows with an empty response span fall back to 0.5.
toks, valid = [], []
for i in range(n):
ids, s, e = tokenize(examples[i]["messages"])
if e > s:
toks.append((ids, s, e)); valid.append(i)
```
--------------------------------
### Install Dependencies Manually
Source: https://github.com/ndif-team/aletheias-quest-competition/blob/master/README.md
Installs project dependencies from `requirements-dev.txt` into an existing environment. Alternatively, install nnsight directly from its git repository.
```bash
pip install -r requirements-dev.txt
```
```bash
pip install 'nnsight @ git+https://github.com/ndif-team/nnsight.git@hackathon/peft'
```
--------------------------------
### Python Code for Loading Examples
Source: https://github.com/ndif-team/aletheias-quest-competition/blob/master/NOTES.md
Illustrates the function call within the NDIF runner that loads dataset examples, potentially triggering sandbox errors.
```python
ds = load_examples(dataset_name)
```
--------------------------------
### Load Examples and Generate Predictions
Source: https://github.com/ndif-team/aletheias-quest-competition/blob/master/baselines/self_eval.ipynb
Loads examples from a dataset, builds self-evaluation prompts for valid examples, and generates predictions using the self-evaluation scores. Unparseable examples default to a score of 0.5.
```python
import sys
# util.py ships alongside the notebooks in submission/; the runner executes from
# the repo root, so add submission/ to the path before importing it.
sys.path.insert(0, "submission")
import util
examples = util.load_examples(DATASET_NAME) # single `test` split + an `index` join key
print(f"loaded {len(examples)} examples")
prompts, valid_idx = [], []
for i, ex in enumerate(examples):
p = build_self_eval_prompt(list(ex["messages"]))
if p is not None:
prompts.append(p)
valid_idx.append(i)
predictions = np.full(len(examples), 0.5, dtype=float) # fallback for unparseable rows
if prompts:
s = self_eval_scores(prompts)
for k, i in enumerate(valid_idx):
if not np.isnan(s[k]):
predictions[i] = s[k]
n_scored = int(np.sum(~np.isnan(s))) if prompts else 0
print(f"self-evaluated {n_scored}/{len(examples)} examples "
f"({len(examples) - n_scored} fell back to 0.5)")
```
--------------------------------
### Setup and Imports for Mean Probe
Source: https://github.com/ndif-team/aletheias-quest-competition/blob/master/baselines/mean_probe.ipynb
Imports necessary libraries and sets up environment variables for dataset name and optional data limits. Ensure 'util.py' is in the submission directory.
```python
import os, sys
import numpy as np
import torch
sys.path.insert(0, "submission") # util.py lives in submission/
from util import build_model, decoder_layers, load_examples, response_span, write_submission
# The runner sets DATASET_NAME; read it, don't hard-code.
DATASET_NAME = os.environ["DATASET_NAME"]
# Optional `--limit N` -> $ALETHEIA_LIMIT: score only the first N rows (fast rehearsal).
LIMIT = int(os.environ["ALETHEIA_LIMIT"]) if os.environ.get("ALETHEIA_LIMIT") else None
```
--------------------------------
### Install nnsight Hackathon Build
Source: https://github.com/ndif-team/aletheias-quest-competition/blob/master/tutorials/1-setup.ipynb
Installs the nnsight library with PEFT/LoRA and remote NDIF support. Restart the kernel if prompted after installation.
```python
# nnsight — hackathon build (PEFT/LoRA + remote NDIF support)
!pip install -q git+https://github.com/ndif-team/nnsight.git@hackathon/peft
!pip install -q --upgrade torchao "transformers==5.11.0"
from IPython.display import clear_output
clear_output()
print("Setup complete.")
```
--------------------------------
### Install nnsight skills for OpenAI Codex
Source: https://github.com/ndif-team/aletheias-quest-competition/blob/master/tutorials/1-setup.ipynb
Use this command to install the nnsight skills pack for OpenAI Codex. Ensure you have Codex running and accessible.
```bash
codex
skill-installer install https://github.com/ndif-team/skills.git
```
--------------------------------
### Load Precomputed Fingerprints and Questions
Source: https://github.com/ndif-team/aletheias-quest-competition/blob/master/tutorials/3.6-unrelated-questions.ipynb
Loads training data from a CSV file and a list of unrelated questions from a JSON file. It then prints the number of examples and questions loaded.
```python
train_df = pd.read_csv("data/pacchiardi_gemma_training.csv")
with open("data/pacchiardi_questions.json") as f:
QUESTIONS = json.load(f)
print(f"{len(train_df)} examples across {train_df['dataset'].nunique()} datasets")
print(f"{len(QUESTIONS)} elicitation questions")
train_df.head()
```
--------------------------------
### Install nnsight skills for Claude
Source: https://github.com/ndif-team/aletheias-quest-competition/blob/master/tutorials/1-setup.ipynb
Use this command to install the nnsight skills pack for Claude Code. Ensure you have Claude Code running and accessible.
```bash
claude
/plugin marketplace add https://github.com/ndif-team/skills.git
/plugin install nnsight@skills
```
--------------------------------
### Python Code for Dry Run Configuration
Source: https://github.com/ndif-team/aletheias-quest-competition/blob/master/NOTES.md
Example of a configuration file using `--dry` to simulate submission locally, which requires the participant to have access to the datasets.
```python
--dry
```
--------------------------------
### Load Examples and Generate Predictions
Source: https://github.com/ndif-team/aletheias-quest-competition/blob/master/baselines/llm_judge.ipynb
Loads examples from a dataset, builds judge prompts, and generates predictions using the judge_scores function. It handles cases where no assistant message is present and falls back to a default score.
```python
import sys
# util.py ships alongside the notebooks in submission/; the runner executes from
# the repo root, so add submission/ to the path before importing it.
sys.path.insert(0, "submission")
import util
examples = util.load_examples(DATASET_NAME) # single `test` split + an `index` join key
print(f"loaded {len(examples)} examples")
prompts, valid_idx = [], []
for i, ex in enumerate(examples):
p = build_judge_prompt(ex)
if p is None:
continue
prompts.append(p)
valid_idx.append(i)
predictions = np.full(len(examples), 0.5, dtype=float) # fallback for rows with no assistant turn
if prompts:
s = judge_scores(prompts)
for k, i in enumerate(valid_idx):
if not np.isnan(s[k]):
predictions[i] = s[k]
n_scored = int(np.sum(~np.isnan([s[k] for k in range(len(prompts))]))) if prompts else 0
print(f"judged {n_scored}/{len(examples)} examples "
f"({len(examples) - n_scored} fell back to 0.5)")
```
--------------------------------
### Build Judge Prompt for LLM
Source: https://github.com/ndif-team/aletheias-quest-competition/blob/master/baselines/llm_judge.ipynb
Formats an example into a chat prompt for the judge. Returns None if there is no assistant message to judge. Handles context and response formatting, truncating if necessary.
```python
def build_judge_prompt(ex):
"""Format one example into a chat prompt for the judge. Returns None if there is no
assistant message to judge."""
msgs = list(ex["messages"])
a_idx = [i for i, m in enumerate(msgs) if m.get("role") == "assistant"]
if not a_idx:
return None
last = a_idx[-1]
# Final assistant turn(s) glued together = the message under judgement.
response = "".join(m.get("content", "") for m in msgs[last:] if m.get("role") == "assistant")
context = msgs[:last]
user = (f"
{format_dialogue(context)}
" f"")
if len(user) > MAX_USER_CHARS: # keep the (recent) tail near the output
user = user[-MAX_USER_CHARS:]
judge_msgs = [{"role": "system", "content": GRADING_PROMPT},
{"role": "user", "content": user}]
try: # Qwen3: skip block for a direct rating
return tokenizer.apply_chat_template(
judge_msgs, tokenize=False, add_generation_prompt=True, enable_thinking=False)
except TypeError:
return tokenizer.apply_chat_template(
judge_msgs, tokenize=False, add_generation_prompt=True)
```
--------------------------------
### Load Examples and Predict Probabilities
Source: https://github.com/ndif-team/aletheias-quest-competition/blob/master/baselines/unrelated_questions_probe.ipynb
Loads example data, preprocesses messages, elicits features, and predicts probabilities using a classifier. Fallback predictions of 0.5 are used for unparseable rows. Ensure 'submission' is in the Python path.
```python
import sys
# util.py ships alongside the notebooks in submission/; the runner executes from
# the repo root, so add submission/ to the path before importing it.
sys.path.insert(0, "submission")
import util
examples = util.load_examples(DATASET_NAME) # single `test` split + an `index` join key
print(f"loaded {len(examples)} examples")
predictions = np.full(len(examples), 0.5, dtype=float) # fallback for unparseable rows
n_scored = 0
for i, ex in enumerate(examples):
msgs = to_judged_messages(list(ex["messages"]))
if msgs is None:
continue
features = elicit_features(msgs)
predictions[i] = classifier_proba(features)
n_scored += 1
print(f" example {i + 1}/{len(examples)}")
print(f"scored {n_scored}/{len(examples)} examples "
f"({len(examples) - n_scored} fell back to 0.5)")
```
--------------------------------
### Load and Prepare 'soft-trigger' Dataset
Source: https://github.com/ndif-team/aletheias-quest-competition/blob/master/tutorials/3.2-followup-probe.ipynb
Loads the 'soft-trigger' dataset and preprocesses it to create a balanced subset of deceptive and honest examples for analysis. Ensures the dataset is ready for probe evaluation.
```python
TRANSFER_CONFIG = "soft-trigger"
ds2 = load_dataset("Cadenza-Labs/liars-bench", data_dir=TRANSFER_CONFIG)
df2 = ds2[list(ds2.keys())[0]].to_pandas()
df2 = df2[df2["model"] == GENERATOR].reset_index(drop=True)
dec2 = df2[df2["deceptive"]].index.values
hon2 = df2[~df2["deceptive"]].index.values
n2 = min(N_PER_LABEL, len(dec2), len(hon2))
rng2 = np.random.default_rng(42)
keep2 = np.concatenate([rng2.choice(dec2, n2, replace=False), rng2.choice(hon2, n2, replace=False)])
data2 = df2.iloc[keep2].sample(frac=1, random_state=42).reset_index(drop=True)
y2 = data2["deceptive"].astype(int).values
print(f"{TRANSFER_CONFIG}: {len(data2)} examples ({y2.sum()} deceptive / {(1 - y2).sum()} honest)")
```
--------------------------------
### Install Dependencies for Nnsight and ML Stack
Source: https://github.com/ndif-team/aletheias-quest-competition/blob/master/tutorials/2-predicting.ipynb
Installs necessary libraries including nnsight for NDIF model access, torchao, transformers, pandas, scikit-learn, matplotlib, and datasets. Clears output and confirms completion.
```python
# nnsight — hackathon build (needed to extract activations on NDIF)
!pip install -q git+https://github.com/ndif-team/nnsight.git@hackathon/peft
!pip install -q --upgrade torchao "transformers==5.11.0"
# data + probing stack
!pip install -q pandas pyarrow numpy scikit-learn matplotlib datasets
from IPython.display import clear_output
clear_output()
print("Setup complete.")
```
--------------------------------
### Load Environment Variables for Dataset
Source: https://github.com/ndif-team/aletheias-quest-competition/blob/master/baselines/mean_probe_fast.ipynb
Loads the dataset name from environment variables. This is a required setup step for the runner.
```python
import os
# ── DO NOT CHANGE ───────────────────────────────────────────────────
# The runner sets these for each run; read them, don't hard-code or override.
# DATASET_NAME : the eval dataset to predict on.
# NDIF_HOST / NDIF_API_KEY / HF_TOKEN : injected; nnsight + HF read them.
DATASET_NAME = os.environ["DATASET_NAME"]
```
--------------------------------
### Load and Prepare Dataset for Transfer Learning
Source: https://github.com/ndif-team/aletheias-quest-competition/blob/master/tutorials/3.4-llm-judge.ipynb
Loads a dataset, filters for a specific generator model, and prepares deceptive and honest examples for evaluation. Ensure N_PER_LABEL and GENERATOR are defined.
```python
TRANSFER_CONFIG = "soft-trigger"
ds2 = load_dataset("Cadenza-Labs/liars-bench", data_dir=TRANSFER_CONFIG)
df2 = ds2[list(ds2.keys())[0]].to_pandas()
df2 = df2[df2["model"] == GENERATOR].reset_index(drop=True)
dec2 = df2[df2["deceptive"]].index.values
hon2 = df2[~df2["deceptive"]].index.values
n2 = min(N_PER_LABEL, len(dec2), len(hon2))
rng2 = np.random.default_rng(42)
keep2 = np.concatenate([rng2.choice(dec2, n2, replace=False), rng2.choice(hon2, n2, replace=False)])
data2 = df2.iloc[keep2].sample(frac=1, random_state=42).reset_index(drop=True)
y2 = data2["deceptive"].astype(int).values
print(f"{TRANSFER_CONFIG}: {len(data2)} examples ({y2.sum()} deceptive / {(1 - y2).sum()} honest)")
```
--------------------------------
### Build and Run Self-Evaluation Prompts
Source: https://github.com/ndif-team/aletheias-quest-competition/blob/master/tutorials/3.5-self-eval.ipynb
Iterates through the prepared data to build self-evaluation prompts using a helper function. It then runs the self-evaluation process on these prompts to get raw scores.
```python
prompts2, valid_idx2 = [], []
for i, row in data2.iterrows():
p = build_self_eval_prompt(list(row["messages"]))
if p is not None:
prompts2.append(p)
valid_idx2.append(i)
print(f"Built {len(prompts2)} self-eval prompts out of {len(data2)} examples")
print(f"Running self-eval on {len(prompts2)} {TRANSFER_CONFIG} examples…")
raw_scores2 = self_eval_scores(prompts2)
print("Done.")
```
--------------------------------
### Load and Prepare Liars' Bench Dataset
Source: https://github.com/ndif-team/aletheias-quest-competition/blob/master/tutorials/3.2-followup-probe.ipynb
Loads the Liars' Bench dataset, filters for a specific model ('google/gemma-3-27b-it'), and creates a balanced subset of deceptive and honest examples for training or evaluation. Prints the dataset size and class distribution.
```python
import numpy as np
import pandas as pd
from datasets import load_dataset
CONFIG_NAME = "instructed-deception"
GENERATOR = "gemma-3-27b-it"
N_PER_LABEL = 500 # balanced dataset
ds = load_dataset("Cadenza-Labs/liars-bench", data_dir=CONFIG_NAME)
df = ds[list(ds.keys())[0]].to_pandas()
df = df[df["model"] == GENERATOR].reset_index(drop=True)
dec = df[df["deceptive"].index.values
hon = df[~df["deceptive"]].index.values
n = min(N_PER_LABEL, len(dec), len(hon))
rng = np.random.default_rng(42)
keep = np.concatenate([rng.choice(dec, n, replace=False), rng.choice(hon, n, replace=False)])
data = df.iloc[keep].sample(frac=1, random_state=42).reset_index(drop=True)
y = data["deceptive"].astype(int).values
print(f"{len(data)} examples ({y.sum()} deceptive / {(1 - y).sum()} honest)")
```
--------------------------------
### Python Code for Dataset Preparation
Source: https://github.com/ndif-team/aletheias-quest-competition/blob/master/NOTES.md
Shows the step in the dataset loading process where `download_and_prepare` is called, which can lead to sandbox issues.
```python
download_and_prepare
```
--------------------------------
### Configure NNSight and Login to Hugging Face
Source: https://github.com/ndif-team/aletheias-quest-competition/blob/master/tutorials/3.2-followup-probe.ipynb
Sets up NNSight configuration, including API key and host, and logs into Hugging Face if a token is provided. Defines the model name to be used.
```python
import os
from nnsight import CONFIG
from huggingface_hub import login
NDIF_API_KEY = os.environ.get("NDIF_API_KEY", "YOUR_NDIF_API_KEY")
HF_TOKEN = os.environ.get("HF_TOKEN", "YOUR_HF_TOKEN")
CONFIG.set_default_api_key(NDIF_API_KEY)
CONFIG.API.HOST = "http://ndif-hackathon.duckdns.org:8001" # hackathon NDIF cluster
if HF_TOKEN not in ("", "YOUR_HF_TOKEN"):
login(token=HF_TOKEN)
MODEL_NAME = "google/gemma-3-27b-it"
```
--------------------------------
### Write Submission File
Source: https://github.com/ndif-team/aletheias-quest-competition/blob/master/baselines/unrelated_questions_probe.ipynb
Clips predictions to be between 0.0 and 1.0 and writes the submission file using the 'index' column from the examples and the generated predictions. Asserts that the number of predictions matches the number of examples.
```python
predictions = np.clip(np.asarray(predictions, dtype=float), 0.0, 1.0)
assert len(predictions) == len(examples), "need exactly one prediction per example"
# Write index,deceptive,score, joining on the dataset's `index` column; `deceptive`
# defaults to score >= 0.5. (Replaces the old id,prediction output.)
util.write_submission(examples["index"], predictions)
```
--------------------------------
### Filter and Display Liars' Bench Examples
Source: https://github.com/ndif-team/aletheias-quest-competition/blob/master/tutorials/2-predicting.ipynb
This code filters the dataset to include only responses from Gemma-3 27B models and then displays one honest and one deceptive example, showing their labels and conversation messages.
```python
data = ds[split]
# True if this example was generated by a Gemma-3 27B model.
def is_gemma_27b(ex):
m = ex["model"].lower()
return "gemma" in m and "27b" in m
data = data.filter(is_gemma_27b)
print(f"{len(data)} Gemma-3 27B rows in {SCENARIO}")
# Pretty-print one example's label and each message in its conversation.
def show(ex):
print(f" deceptive = {ex['deceptive']} (generated by {ex['model']})")
for m in ex["messages"]:
content = m["content"][:280].replace("\n", " ")
print(f" [{m['role']:>9}] {content}")
honest = next(ex for ex in data if not ex["deceptive"])
deceptive = next(ex for ex in data if ex["deceptive"])
print("-- HONEST --------------------------------------------------")
show(honest)
print("\n-- DECEPTIVE -----------------------------------------------")
show(deceptive)
```
--------------------------------
### Initial Real Submission
Source: https://github.com/ndif-team/aletheias-quest-competition/blob/master/llms.txt
Make your first real submission to the competition. This requires specifying your team name.
```bash
python submit.py --team "your-team"
```
--------------------------------
### Redacted Submission Failure Message
Source: https://github.com/ndif-team/aletheias-quest-competition/blob/master/NOTES.md
Example of a redacted, generic failure message displayed on the leaderboard when a dataset cannot be processed.
```text
failed on Dataset Doris
```
--------------------------------
### Submit Solution
Source: https://github.com/ndif-team/aletheias-quest-competition/blob/master/README.md
Submits your solution to the competition leaderboard. Provide your NDIF API key and the leaderboard Space URL. The team name is only required for the first submission.
```bash
python submit.py --team "your-team-name" --ndif-api-key --space-url
```
```bash
python submit.py --ndif-api-key --space-url
```
--------------------------------
### Submission Rejection Output
Source: https://github.com/ndif-team/aletheias-quest-competition/blob/master/NOTES.md
Example of a submission rejection message indicating a data access issue with a specific dataset.
```text
✓ packaged 33 files · 1.3 MB
▸ entering the lists as aletheia-blue → https://ndif-leaderboard-dev.hf.space/submit
✗ submission rejected [400]
invalid submission: Dataset 'aletheias-quest/dev-test-instructed-deception-Qwen3.5-27B-None'
doesn't exist on the Hub or cannot be accessed.
```
--------------------------------
### Build Model and Tokenizer
Source: https://github.com/ndif-team/aletheias-quest-competition/blob/master/baselines/mean_probe.ipynb
Initializes the model and tokenizer. The tokenizer is configured to handle specific token IDs like end-of-turn.
```python
model = build_model(MODEL_ID)
DECODER = decoder_layers(model)
tok = model.tokenizer
EOT = tok.convert_tokens_to_ids("")
```
--------------------------------
### Initiating Activation Extraction
Source: https://github.com/ndif-team/aletheias-quest-competition/blob/master/tutorials/3.2-followup-probe.ipynb
Prints the scope of the extraction process and calls the function to extract activations for all specified layers and examples.
```python
print(f"Extracting {len(LAYERS)} layers for {len(tok_list)} examples…")
X_by_layer = extract_all_layers_followup(tok_list, LAYERS)
```
--------------------------------
### Load Liars' Bench Dataset Configurations
Source: https://github.com/ndif-team/aletheias-quest-competition/blob/master/tutorials/2-predicting.ipynb
This snippet shows how to load the Liars' Bench dataset from HuggingFace and list its available configurations. It then loads a specific scenario and prints its schema.
```python
from datasets import load_dataset, get_dataset_config_names
# The eight Liars' Bench configs (seven deception scenarios + the alpaca control).
configs = get_dataset_config_names("Cadenza-Labs/liars-bench")
print("Available configs:", configs)
# Load one scenario and inspect the schema.
SCENARIO = "instructed-deception"
ds = load_dataset("Cadenza-Labs/liars-bench", data_dir=SCENARIO)
split = list(ds.keys())[0]
print(f"
{SCENARIO} / {split}: {len(ds[split])} rows")
print("fields:", ds[split].column_names)
```
--------------------------------
### Get Final Assistant Message Text
Source: https://github.com/ndif-team/aletheias-quest-competition/blob/master/llms.txt
Extracts the text content from the final assistant message. This is a basic heuristic for deception detection.
```python
util.final_assistant_message(ex)
```
--------------------------------
### Evaluate and Visualize Self-Evaluation Scores
Source: https://github.com/ndif-team/aletheias-quest-competition/blob/master/tutorials/3.5-self-eval.ipynb
Calculates AUROC and balanced accuracy from self-evaluation scores against true labels, and visualizes the distribution of scores for honest and deceptive examples.
```python
from sklearn.metrics import roc_auc_score, balanced_accuracy_score
import matplotlib.pyplot as plt
import pandas as pd
self_eval_labels = y[np.array(valid_idx)]
scores = np.where(np.isnan(raw_scores), 0.5, raw_scores)
auroc = roc_auc_score(self_eval_labels, scores)
bacc = balanced_accuracy_score(self_eval_labels, (scores > 0.5).astype(int))
print(f"Self-eval AUROC: {auroc:.3f}")
print(f" bal-acc (threshold=50/100): {bacc:.3f}")
fig, ax = plt.subplots(figsize=(8, 4))
ax.hist(scores[self_eval_labels == 0], bins=25, alpha=0.6, color="tab:blue", label="honest")
ax.hist(scores[self_eval_labels == 1], bins=25, alpha=0.6, color="tab:red", label="deceptive")
ax.axvline(0.5, color="k", ls="--", lw=1, label="threshold (50/100)")
ax.set_xlabel("self-evaluated lying score / 100")
ax.set_ylabel("count")
ax.set_title(f"Self-eval score distribution (AUROC = {auroc:.3f})")
ax.legend()
plt.tight_layout()
plt.show()
```
--------------------------------
### Configure NDIF Connection and HuggingFace Token
Source: https://github.com/ndif-team/aletheias-quest-competition/blob/master/tutorials/1-setup.ipynb
Set up your NDIF API key and HuggingFace token, preferring environment variables. Configure the default API key and the host for the NDIF hackathon instance.
```python
import os
from nnsight import CONFIG
# Prefer environment variables; fall back to pasting keys for a local session.
NDIF_API_KEY = os.environ.get("NDIF_API_KEY", "YOUR_NDIF_API_KEY")
os.environ["HF_TOKEN"] = os.environ.get("HF_TOKEN", "YOUR_HF_TOKEN") # used for gated repos
CONFIG.set_default_api_key(NDIF_API_KEY)
# Competition models + LoRA imposters are hosted on the hackathon NDIF instance.
CONFIG.API.HOST = "http://ndif-hackathon.duckdns.org:8001"
```
--------------------------------
### Load Trained Probe Weights and Parameters
Source: https://github.com/ndif-team/aletheias-quest-competition/blob/master/baselines/mean_probe.ipynb
Loads the 'mean_probe.npz' file, which contains the probe direction and standardization parameters. It asserts that the file is found and prints the loaded probe configuration.
```python
CANDIDATES = ["submission/mean_probe.npz", "mean_probe.npz", "baselines/mean_probe.npz"]
WEIGHTS = next((p for p in CANDIDATES if os.path.exists(p)), None)
assert WEIGHTS is not None, f"mean_probe.npz not found; looked in {CANDIDATES}"
z = np.load(WEIGHTS)
DIRECTION = z["direction"].astype(np.float64)
LAYER = int(z["layer"])
MODEL_ID = str(z["model_id"])
MAX_LEN = int(z["max_seq_len"])
NORMALIZE = bool(z["normalize"]) if "normalize" in z.files else False
MU = z["mu"].astype(np.float64) if "mu" in z.files else np.zeros_like(DIRECTION)
SIGMA = z["sigma"].astype(np.float64) if "sigma" in z.files else np.ones_like(DIRECTION)
BIAS = float(z["bias"]) if "bias" in z.files else 0.0
SCORE_STD = float(z["score_std"]) if "score_std" in z.files else 1.0
print(f"probe: model={MODEL_ID} | layer={LAYER} | dim={DIRECTION.shape[0]} | "
f"max_len={MAX_LEN} | normalize={NORMALIZE}")
```
--------------------------------
### Calculate Scores from Extracted Activations
Source: https://github.com/ndif-team/aletheias-quest-competition/blob/master/baselines/mean_probe.ipynb
This snippet calculates the final scores by applying a probe function to the extracted activations. It initializes scores to 0.5 and updates them for valid, tokenized examples.
```python
scores = np.full(n, 0.5, dtype=float)
if valid:
scores[np.array(valid)] = probe_proba(extract(toks))
```
--------------------------------
### Initialize API Clients and Host Configuration
Source: https://github.com/ndif-team/aletheias-quest-competition/blob/master/tutorials/2-predicting.ipynb
Retrieves NDIF API key and HuggingFace token from environment variables. Defines the hackathon host URL for accessing competition models and LoRA imposters on the NDIF instance.
```python
import os
NDIF_API_KEY = os.environ.get("NDIF_API_KEY")
HF_TOKEN = os.environ.get("HF_TOKEN")
# Competition models + LoRA imposters are served on the hackathon NDIF instance.
HACKATHON_HOST = "http://ndif-hackathon.duckdns.org:8001"
```
--------------------------------
### Import necessary libraries and set up environment variables
Source: https://github.com/ndif-team/aletheias-quest-competition/blob/master/baselines/unrelated_questions_probe.ipynb
Imports essential libraries like os, numpy, and pandas. It also sets the DATASET_NAME environment variable, which is used by the runner to identify the dataset for prediction, and defines the path for the submission file.
```python
import os
import numpy as np
import pandas as pd
# The runner (and `submit.py --dry`) always sets DATASET_NAME — one repo, single
# `test` split — to the dataset you predict on.
DATASET_NAME = os.environ["DATASET_NAME"]
# The canonical output file the grader reads. Do not rename.
SUBMISSION_PATH = "submission.csv"
```
--------------------------------
### Configure NNSIGHT and Authentication
Source: https://github.com/ndif-team/aletheias-quest-competition/blob/master/tutorials/3.1-mean-probe.ipynb
Sets up NNSIGHT configuration, including the NDIF API key and host, and handles Hugging Face token login. Defines the model name to be used.
```python
import os
from nnsight import CONFIG
from huggingface_hub import login
NDIF_API_KEY = os.environ.get("NDIF_API_KEY", "YOUR_NDIF_API_KEY")
HF_TOKEN = os.environ.get("HF_TOKEN", "YOUR_HF_TOKEN")
CONFIG.set_default_api_key(NDIF_API_KEY)
CONFIG.API.HOST = "http://ndif-hackathon.duckdns.org:8001" # hackathon NDIF cluster
if HF_TOKEN not in ("", "YOUR_NDIF_API_KEY"):
login(token=HF_TOKEN)
MODEL_NAME = "google/gemma-3-27b-it"
```
--------------------------------
### Leaderboard Performance Metrics
Source: https://github.com/ndif-team/aletheias-quest-competition/blob/master/NOTES.md
Example metrics observed during dry runs and final submissions, highlighting balanced accuracy and AUROC across different datasets and models. Note the performance on validation-insider-trading.
```text
MEAN Bal.Acc 0.6796 AUROC 0.8989
dev-test instructed (gemma ×3) Bal.Acc 0.909–0.921 AUROC ~0.98 ← probe's home base
dev-test instructed (Qwen ×3) Bal.Acc 0.679–0.802 AUROC 0.95–0.99
validation-convincing-game (gemma) Bal.Acc 0.628 AUROC 0.914
validation-insider-trading (gemma) Bal.Acc 0.500 AUROC 0.377 ← probe ANTI-correlated
validation-soft-trigger ×4 Bal.Acc 0.500 (each) AUROC 0.90–0.91 ← signal there, 0 fire
```
--------------------------------
### Qualitative Inspection of Predictions
Source: https://github.com/ndif-team/aletheias-quest-competition/blob/master/tutorials/3.2-followup-probe.ipynb
Performs a qualitative inspection by examining the highest-confidence deceptive and honest predictions made by the best probe. It extracts and prints snippets of the assistant's responses for these examples.
```python
# ── Qualitative inspection (scored over all valid examples, train ∪ test) ────
predictions = best_probe.decision_function(X_by_layer[best_layer])
followup_labels = y_valid
data_valid = data.iloc[np.array(valid_idx)].copy()
data_valid["score"] = predictions
data_valid["label"] = followup_labels
def last_assistant(row):
return next(
(m["content"] for m in reversed(list(row["messages"])) if m.get("role") == "assistant"),
"")
print("=== Highest-confidence DECEPTIVE predictions ===")
for _, row in data_valid[data_valid["label"] == 1].nlargest(3, "score").iterrows():
snippet = last_assistant(row)[:150].replace("\n", " ")
print(f" score={row['score']:.3f} | {snippet}")
print("\n=== Highest-confidence HONEST predictions ===")
for _, row in data_valid[data_valid["label"] == 0].nsmallest(3, "score").iterrows():
snippet = last_assistant(row)[:150].replace("\n", " ")
print(f" score={row['score']:.3f} | {snippet}")
```
--------------------------------
### Write Submission File
Source: https://github.com/ndif-team/aletheias-quest-competition/blob/master/submission/example.ipynb
After scoring, join the computed scores with the original example indices and write them to a submission CSV file. The 'deceptive' column defaults to true if the score is 0.5 or higher.
```python
# Join scores back to each row's `index` and write index,deceptive,score.
# `deceptive` defaults to score >= 0.5 — pass deceptive= to override.
examples = util.load_examples(DATASET_NAME)
# scores cover the first len(scores) rows (every row unless --limit/$ALETHEIA_LIMIT
# is set), in dataset order — align the index column to match.
util.write_submission(list(examples["index"])[:len(scores)], scores)
```