🌃 Operation Night Watch

📜The task, in plain English

You are given a deployed sound-recognition model — an Audio Spectrogram Transformer (AST) — that already recognizes 16 sound classes (city sounds). Your city now needs it to also recognize 13 new classes: chainsaws, gunshots, and four insect species among them. The catch: after adding the new classes, the model will be tested on all 29 classes together, and old and new count equally.

🔧The baseline you're given

The notebook walks you through: loading the pretrained AST from Hugging Face (ASTForAudioClassification), turning waveforms into 128-band log-mel spectrograms with the feature extractor, and running the standard fine-tuning loop on the new data. Structure:

1. Feature extraction — waveform → spectrogram "image" → 16×16 patches → 12 transformer layers (~86M params) → tiny linear classification head.
2. The naive path (shown on purpose): replace the 16-way head with a 13-way head and fine-tune on fine_tune.csv only. Result: new classes learned, old classes obliterated. The notebook wants you to watch this happen.
3. Diagnostics provided — per-class accuracy, confusion matrix, t-SNE of embeddings, audio players so you can listen to mistakes.

The stated limitations (= your to-do list): the naive approach forgets; the new data is small and imbalanced; training the full 86M-parameter encoder is slow and risky.

🚀Baseline vs. solution

🧒Explain it like I'm brand new

Analogy: imagine a waiter who has memorized 16 regular customers' orders. You now ask him to memorize 13 new customers — but you drill him only on the new ones for a week. When an old regular walks in, he blanks. That's catastrophic forgetting. The fixes are common sense:

• Replay = while drilling new orders, keep quizzing him on a few old ones every session.
• Freezing = tell him "don't restructure your whole memory system, just add new entries."
• Distillation = keep a photo of his old notebook and check his answers against it.

What's a spectrogram? Sound is a wave. A spectrogram slices it into tiny time windows and plots which frequencies are loud in each window — producing a 2-D picture where time runs left→right and pitch runs bottom→top. Once sound is a picture, image models (like this transformer) can classify it. This trick — "turn weird data into an image" — shows up at IOAI every single year.

What's a "head"? Big pretrained models = a huge encoder (understands the input, 86M params here) + a tiny final layer, the head, that maps understanding → class scores. Swapping/expanding heads is cheap surgery and an IOAI yearly ritual: the encoder already "hears" — you just re-wire what it reports.

💬The Gemma 4 playthrough (2000-token limit)

How you'd drive this on contest day with a 2000-token cap (≈120 lines of code per reply) and unlimited fresh chats. Notice: one chat = one job, code requested in pieces, everything verified on a tiny slice.

Task: audio classifier (AST, Hugging Face) knows 16 classes. I must add 13 new classes from a small imbalanced dataset. Metric = 0.5*acc_old + 0.5*acc_new on all 29 classes. Baseline naively fine-tunes on new data only and forgets old classes. List the 5 highest-impact fixes ordered by (score gain ÷ implementation time) for a 6-hour contest. One line each, no code.

1. Expand head to 29 outputs, copy old 16 weight rows (5 min, prevents starting from zero)
2. Experience replay: mix retained old clips into every batch ~50/50 (15 min, biggest anti-forgetting win)
3. Freeze encoder, train head + last transformer block only (5 min, faster + less forgetting)
4. Class-balanced sampling for the 13 imbalanced new classes (10 min)
5. Distillation vs frozen original model on old classes (45 min, do only if time remains)

transformers ASTForAudioClassification, loaded from a local path. Current classifier head: Linear(768, 16). Write ONLY the code to replace it with Linear(768, 29), copying the existing 16 rows of weight and bias into the first 16 outputs. Code only, max 20 lines.

import torch, torch.nn as nn

old_head = model.classifier.dense          # Linear(768, 16)
new_head = nn.Linear(768, 29)
with torch.no_grad():
    new_head.weight[:16] = old_head.weight
    new_head.bias[:16]   = old_head.bias
model.classifier.dense = new_head
model.config.num_labels = 29
print(model.classifier.dense)   # sanity check

I have two pandas DataFrames: old_df (16 classes, cols: path,label) and new_df (13 classes, labels 16..28). Write ONLY a PyTorch Dataset + DataLoader that samples each batch ~50% from old_df and ~50% from new_df using WeightedRandomSampler. Assume a function load_features(path) exists. Code only, max 40 lines.

import pandas as pd, torch
from torch.utils.data import Dataset, DataLoader, WeightedRandomSampler

full = pd.concat([old_df, new_df], ignore_index=True)
is_old = (full.label < 16).values
w = torch.where(torch.tensor(is_old),
                0.5/ is_old.sum(),
                0.5/ (~is_old).sum())

class AudioDS(Dataset):
    def __init__(self, df): self.df = df.reset_index(drop=True)
    def __len__(self): return len(self.df)
    def __getitem__(self, i):
        r = self.df.iloc[i]
        return load_features(r.path), int(r.label)

sampler = WeightedRandomSampler(w.double(), num_samples=len(full))
loader = DataLoader(AudioDS(full), batch_size=16, sampler=sampler)

Freeze all AST encoder params except the last transformer layer and the classifier head, then a training loop: AdamW lr=1e-4, cross-entropy, 3 epochs, print loss per epoch. Variables that exist: model, loader, DEVICE. Code only, max 40 lines.

Write an eval function that computes accuracy separately for labels 0-15 and labels 16-28 given (model, val_loader, DEVICE), then prints 0.5*acc_old + 0.5*acc_new. Code only, max 30 lines.

🎯Takeaways & what Day 1 might do with this

• Pattern family: "frozen encoder + head surgery" (yearly ritual) + continual learning (new for 2026).
• The metric defines the game. Read the scoring cell before the story. 50/50 weighting = protect old knowledge at all costs.
• peft/LoRA is in the contest library list — that's a hint from the organizers, not an accident.
• Likely Day-1 extension: same AST audio setup with a twist — more/fewer retained clips, additional class batches (continual learning in multiple steps), distribution shift (noisy recordings), or a stricter compute budget. If you've tuned replay ratios once, you'll do it in minutes on Day 1.