🗄️ The Analytical Language of John Wilkins

📜The task, in plain English

A hidden animal sits behind an oracle. The oracle is a local LLM (Qwen2.5-3B-Instruct) that truthfully-ish answers yes/no questions about the hidden animal, at temperature 0 (fully deterministic). You get at most 15 questions per animal, the questions must come from a fixed pool (questions_pool.txt), and the animal is one of ~1,400 candidates (animals_pool.txt).

🔧The baseline you're given

Two reference points are given:

1. Random baseline (the floor): never asks anything, just guesses random animals 15 times. Expected score ≈ 0 (15/1400 ≈ 1% hit rate). Exists so any real strategy beats it.
2. Fixed-questions reference (deliberately weak): precomputes, with its own LLM copy, the oracle's answer to 12 fixed broad questions ("is it a mammal?", "can it fly?"…) for every animal → one 12-bit vector per animal. At solve time it asks the same 12 questions, then guesses the animals whose stored bit-vector is closest. It works but leaves points everywhere: the questions never adapt to what's already known, and 12 lopsided bits ≪ 10.5 good bits.

The notebook then says outright: your job is (1) the full animal × question table, (2) adaptive, information-gain question selection, (3) robustness to the oracle's occasional wrong answer.

🚀Baseline vs. solution

🧒Explain it like I'm brand new

This is "20 Questions" played perfectly. When you play it with a friend, good players don't ask "is it a platypus?" first — they ask "is it bigger than a breadbox?" because either answer removes half the possibilities. That halving idea is called information gain, and log₂(1400) ≈ 10.5 just means: 1,400 possibilities can be halved down to 1 in about 11 perfect halvings.

The cheat code the task hands you: the oracle is a deterministic robot. You can build an identical robot at home and ask it every question about every animal in advance — for free. On game day you're not guessing what the oracle will say; you're looking answers up in your own table and just narrowing down which row of the table you're in.

The one twist: the oracle is an LLM with occasionally odd beliefs (it might insist a penguin "can fly"). Your table has the same odd beliefs (same model — that's the point!), but edge cases can still mismatch. So never delete a candidate on one disagreement — just make it less likely. Detective work, not courtroom verdicts.

💬The Gemma 4 playthrough (2000-token limit)

Task: identify 1 of 1400 animals via yes/no questions to a deterministic LLM oracle, 15-question budget, score = 1 - 0.02*queries. I can run the same LLM offline free. My plan: (1) precompute full animal-x-question answer table offline, (2) at solve time pick the question that best splits remaining candidates 50/50, (3) soft-downweight instead of eliminating on mismatch, (4) guess when one candidate dominates. Any flaw or improvement? Max 6 lines.

Plan is sound and matches the intended solution. Improvements:
1. Cache the table to disk — don't recompute across runs
2. Weight information gain by current candidate weights, not raw counts
3. Stop early: if best candidate > ~70% of total weight, guess (each extra question costs 0.02)
4. If a guess is wrong, zero that animal's weight and continue — you still have budget

I have Interactor(gold_animal, animals_pool, questions_pool, budget) whose .ask(q) returns 'yes'/'no' using a local LLM. Write ONLY code to build table[animal] = list of 0/1 for every question in questions_pool, for every animal in animals_pool, with tqdm and saving to table.json every 100 animals. Code only, max 30 lines.

import json
from tqdm import tqdm

table = {}
for i, animal in enumerate(tqdm(animals_pool)):
    sim = Interactor(gold_animal=animal, animals_pool=animals_pool,
                     questions_pool=questions_pool, budget=10**9)
    table[animal] = [1 if sim.ask(q) == 'yes' else 0
                     for q in questions_pool]
    if (i + 1) % 100 == 0:
        with open('table.json', 'w') as f:
            json.dump(table, f)
with open('table.json', 'w') as f:
    json.dump(table, f)

Given table (dict animal -> 0/1 list), write ONLY a solver class: keep weights w[animal]=1.0; each turn pick question index j minimizing |weighted_yes_fraction - 0.5| among unasked; ask it; multiply weights of disagreeing animals by 0.15; if top candidate weight > 0.7 * total weight OR budget nearly gone, guess top candidate; on 'wrong', set its weight to 0 and continue. Code only, max 45 lines.

My solver averages 12.8 queries with 96% solve rate on dev. Score formula is solved - 0.02*queries. Is it better to guess earlier? Give the expected-score math for guessing at threshold 0.5 vs 0.7 vs 0.9 assuming my top-candidate accuracy at those thresholds is 82%/94%/98%. Short answer.

E[score] ≈ P(correct at guess) − 0.02·E[queries].
0.5: 0.82 − 0.02·9  = 0.640
0.7: 0.94 − 0.02·11 = 0.720
0.9: 0.98 − 0.02·13 = 0.720
→ 0.7 and 0.9 tie; prefer 0.7 (fewer queries = more headroom for the wrong-guess recovery path). Measure on dev to confirm.

🎯Takeaways & what Day 1 might do with this

• Pattern family: "LLM as a budgeted component" — 2025's Concepts task had a 12,500-call judge API; 2026 has a 15-question oracle. Rule: simulate offline for free, spend the budget adaptively, expect noise.
• The scoring formula is a strategy dial (−0.02/query): compute when to stop asking; don't feel it out.
• Start long-running precomputes immediately and build the rest while they run — contest time management in miniature.
• Likely Day-1 extension: same interactor pattern, different domain (objects? words?), tighter budget, larger pool, or a noisier oracle. Your greedy-info-gain solver is nearly domain-agnostic — keep it.