Early Evals for OpenAI O3

by maurycyon 12/20/2024, 6:06:28 PM with 4 comments
  • by macawfishon 12/20/2024, 8:05:28 PM

    Wow, the demo where the user asks for untraceable payments shows some pretty sophisticated reasoning. The word "crafty" comes to mind.

  • by og_kaluon 12/20/2024, 6:13:12 PM

    New SOTA's on:

    SWE-Bench - 71.7

    Competition Code - 2727

    ARC (Semi Private Eval) - 75.7 on low, 87.5% on high compute

    Frontier Math (previous SOTA was 2%) - 25% on high compute