Rendered at 20:55:15 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
andhuman 3 hours ago [-]
This benchmark expects models to not only do the happy path, but also edge cases, even though the ticket they give to the model doesn’t specify if. This to align more to real world tickets. Usually with benchmarks it’s the other way around: only implement what’s asked. So I welcome this type of benchmark because this is how I use the models.
klooney 1 hours ago [-]
Is this a proprietary harness? I feel like harnesses have a huge influence on how models behave