The unsolved problem at hackathons: judging

The format matured. The judging didn't.

The hackathon format itself has matured. 36 hours, strong cases, teams shipping working products in a weekend. That part works. The judging part doesn't.

I've sat in enough pitching rooms now to notice that no two judges work the same way. One reads the documentation. The next flips through it. The next looks at nothing. Some just want the idea. Some decide on the UI in 30 seconds.

One fix that worked

A year into organizing, a participant told us our jury wasn't technical enough. He was right. At the next edition we pulled in a distinctly more technical jury, and the depth of the questions changed everything. Teams got asked about their architecture, not just their story. That was one fix. The rest of the problem is wide open.

Same word, completely different test

The pitch format itself is just as inconsistent. At one hackathon, judging happens science-fair style: judges walk from team to team and listen for a minute each. No real pitch, no Q&A, no demo. At another, teams get six minutes on a main stage and deep Q&A. Same word, completely different test.

Three decision-makers, three lenses

At the largest event I organized, we ran judging in three rounds. The case partner picks their case winner. The public picks the top five. The top five pitch again on the main stage in front of the jury. Three decision-makers, three lenses.

At smaller events we went further: the hackers themselves cast the final vote. No override. Which sounds great until you remember the hackers never see the code. They're voting on the pitch, not the build.

The sweet spot hasn't been found

Every hackathon I've been to does some part of this well. None does the whole thing well. When 36 hours of work gets decided in one minute, the format owes the builders better.

That's one of the problems we take seriously as we design our own sprints. The sweet spot hasn't been found yet. We intend to keep looking, starting this October in Zurich.