I was curious about the variance of the output and I made some runs w/ deepseek v4 flash and found that it was pretty high?
There's also a possibility of strong model memorization on the tasks, I saw a header w/ the authors generated in one of the runs for the cmatrix task (one of 3 of 200 tasks I selected to re-evaluate).
Additionally curious if anyone else thinks that passing in the gold executable as part of the task lowers the usefulness of this benchmark.
caveats: N=5 on my runs and I used my own generalized task prompt
There's also a possibility of strong model memorization on the tasks, I saw a header w/ the authors generated in one of the runs for the cmatrix task (one of 3 of 200 tasks I selected to re-evaluate).
Additionally curious if anyone else thinks that passing in the gold executable as part of the task lowers the usefulness of this benchmark.
caveats: N=5 on my runs and I used my own generalized task prompt