Hangman: uncovering words, letter by letter
Here the task changes nature: it's not classifying or estimating, it's deciding step by step. Each model gets the same hidden word and picks one letter per round, looking only at what's been revealed. Nobody gets eliminated: we measure how much time, how many guesses and how much money each model needed to get there.
Heads-up: the tasks themselves run in Portuguese (Brazilian news feed, Portuguese words, same prompts for every model). That keeps every run comparable — and the leaderboard in one piece.
Start here: pick the word
We'll draw one of the 264 long words on our list — or you can type your own (7+ letters, no accents). The models never see the word itself: only what's already been revealed.
Now let's kick off the race
Each round, every model that hasn't solved it yet gets the same JSON state and has 5 seconds to send back a letter. If a model doesn't answer, repeats a letter, or makes something up, we log the penalty, add the time and cost, and let it keep trying.