2025 Contest Days

💡 Concepts (Chameleon's Day-1 extension)

The at-home guessing game returns — now with an official LLM judge API and a call budget.

NLPLLMPromptingBudgeted API★★★☆ shows how Day 1 extends at-home

📜The task, in plain English

Contest Day 1, 2025: the Chameleon word/concept-guessing idea extended — contestants worked with hint generation / harder guessing, and were given an official LLM API proxy ($10 credits ≈ 12,500 judge calls) as part of the task. The exam: combine embeddings with prompt engineering under a budget.

Why this page matters even though you'll get different tasks

It's the clearest historical proof of the At-Home → Day 1 mechanic: same core (semantic similarity), new twist (an LLM component with a budget). 2026 will do the same thing to Night Watch, Robot Delivery, and John Wilkins.

🔧The baseline you're given

A working pipeline from the at-home version (embed + compare) plus starter code for calling the judge API. Limitation: naive use burns the call budget fast, and unprompted LLM calls give noisy results.

🚀Baseline vs. solution

⚠️ The baseline (what you are given)

Spends judge calls freely, one candidate at a time
Vague prompts → inconsistent judgments

✅ The winning approach

Filter cheap, judge expensive: use free embeddings to shortlist candidates, spend LLM calls only on the shortlist
Tight, structured prompts ("Answer only YES or NO") — parseable outputs
Track budget in code; decide spend-per-item up front
Cache every (input → judgment) pair; never pay twice

🧒Explain it like I'm brand new

The budgeted-LLM pattern: when an LLM call costs something (money, queries, time), sandwich it: cheap tools (embeddings) narrow 1,000 options to 10, the expensive LLM picks among the 10, and a cache remembers everything. Recognize this? It's exactly John Wilkins 2026 — precompute what's free, spend the budget only where it decides something.

💬The Gemma 4 playthrough (2000-token limit)

Chat 1 · Budget math first

YOU

I have 12500 LLM judge calls, 400 test items, and a shortlist of 20 candidates per item from embeddings. Judging one candidate = 1 call. What's my per-item budget and a sensible allocation? Short answer, no code.

GEMMA 4

12500/400 ≈ 31 calls/item. Allocation: judge top 10 shortlist candidates (10 calls), keep ~20 calls of headroom for re-judging ties and the hardest items. Reserve ~10% of total budget for the end.

🎯Takeaways & what Day 1 might do with this

Pattern family: LLM as a budgeted component (→ recurs as 2026's John Wilkins oracle).
Day 1 = at-home + one twist. Whoever mastered the at-home version spends Day 1 adapting, not learning.