Antigravity Model Benchmark Leaderboard

Automated SWE-Bench coding evaluation & Pareto efficiency matrix • Run run_20260927_091818 • Generated 2026-09-27 09:46:14

Models Evaluated
17
Total Trials Executed
102
Top Resolution Rate
83.3%
Fastest Solved Avg
17.0s

Model Leaderboard & Efficiency Rankings

Model Name ▴▾ Pass Rate ▴▾ Solved / Total ▴▾ Avg Time (s) ▴▾ Avg Total Tokens ▴▾ Avg Thinking Tokens ▴▾ Avg Cost / Task ▴▾ Pareto Frontier ▴▾
gemini-3.7-flash-low 83.3% 5 / 6 17.0s 88,690 523 $0.0142 Pareto Optimal
gemini-3.6-flash-high 83.3% 5 / 6 21.8s 60,865 5,578 $0.0126 Pareto Optimal
gemini-3.8-flash-low 83.3% 5 / 6 26.9s 101,828 258 $0.0163 -
gemini-3.1-pro-low 83.3% 5 / 6 29.9s 23,745 2,029 $0.0412 -
gemini-3.7-flash-medium 83.3% 5 / 6 33.0s 105,110 2,869 $0.0181 -
gemini-3.6-flash-low 83.3% 5 / 6 35.3s 36,408 1,157 $0.0067 Pareto Optimal
gemini-3.1-pro-high 83.3% 5 / 6 40.8s 30,245 3,067 $0.0541 -
gemini-3.7-flash-high 83.3% 5 / 6 41.2s 120,928 4,549 $0.0214 -
gemini-flash-demo 83.3% 5 / 6 43.4s 106,364 4,419 $0.0191 -
gemini-3.6-flash-medium 83.3% 5 / 6 45.6s 53,205 4,306 $0.0107 -
agy-demo 83.3% 5 / 6 45.8s 56,225 920 $0.0792 -
gemini-3.8-flash-medium 83.3% 5 / 6 69.2s 126,340 8,247 $0.0241 -
gemini-3.8-flash-demo 83.3% 5 / 6 85.8s 113,390 7,042 $0.0212 -
gemini-3.8-flash-high 83.3% 5 / 6 107.1s 134,225 9,737 $0.0258 -
gpt-oss-120b-medium 16.7% 1 / 6 74.3s 18,936 0 $0.0098 -
claude-opus-4-6-thinking 0.0% 0 / 6 121.6s 0 0 $0.0000 -
claude-sonnet-4-6 0.0% 0 / 6 122.4s 0 0 $0.0000 -

Task Matrix Results

Task ID Model Status Time Total Tokens Thinking Tokens Cost ($)
task_01_kv_ttl agy-demo PASSED 41.8s 53,212 976 $0.0756
task_02_frontmatter agy-demo PASSED 63.2s 69,395 1,248 $0.0986
task_03_limiter agy-demo PASSED 33.9s 53,616 498 $0.0729
task_04_refactor agy-demo PASSED 49.6s 54,143 512 $0.0749
task_05_emitter agy-demo FAILED 37.7s 51,490 826 $0.0729
task_06_semver agy-demo PASSED 48.3s 55,492 1,461 $0.0805
task_01_kv_ttl claude-opus-4-6-thinking FAILED 124.6s 0 0 $0.0000
task_02_frontmatter claude-opus-4-6-thinking FAILED 126.2s 0 0 $0.0000
task_03_limiter claude-opus-4-6-thinking FAILED 114.1s 0 0 $0.0000
task_04_refactor claude-opus-4-6-thinking FAILED 110.0s 0 0 $0.0000
task_05_emitter claude-opus-4-6-thinking FAILED 122.9s 0 0 $0.0000
task_06_semver claude-opus-4-6-thinking FAILED 132.0s 0 0 $0.0000
task_01_kv_ttl claude-sonnet-4-6 FAILED 118.9s 0 0 $0.0000
task_02_frontmatter claude-sonnet-4-6 FAILED 120.5s 0 0 $0.0000
task_03_limiter claude-sonnet-4-6 FAILED 124.9s 0 0 $0.0000
task_04_refactor claude-sonnet-4-6 FAILED 114.5s 0 0 $0.0000
task_05_emitter claude-sonnet-4-6 FAILED 132.4s 0 0 $0.0000
task_06_semver claude-sonnet-4-6 FAILED 123.5s 0 0 $0.0000
task_01_kv_ttl gemini-3.1-pro-high PASSED 30.4s 37,090 1,486 $0.0557
task_02_frontmatter gemini-3.1-pro-high PASSED 57.4s 40,206 4,001 $0.0746
task_03_limiter gemini-3.1-pro-high PASSED 29.3s 26,155 1,609 $0.0416
task_04_refactor gemini-3.1-pro-high PASSED 33.6s 24,657 2,550 $0.0444
task_05_emitter gemini-3.1-pro-high FAILED 36.2s 24,708 2,504 $0.0443
task_06_semver gemini-3.1-pro-high PASSED 57.8s 28,654 6,253 $0.0638
task_01_kv_ttl gemini-3.1-pro-low PASSED 25.2s 29,443 1,493 $0.0459
task_02_frontmatter gemini-3.1-pro-low PASSED 53.9s 30,533 4,752 $0.0620
task_03_limiter gemini-3.1-pro-low PASSED 24.9s 21,561 1,670 $0.0361
task_04_refactor gemini-3.1-pro-low PASSED 21.2s 21,704 1,028 $0.0340
task_05_emitter gemini-3.1-pro-low FAILED 28.9s 19,535 1,822 $0.0352
task_06_semver gemini-3.1-pro-low PASSED 25.2s 19,691 1,410 $0.0339
task_01_kv_ttl gemini-3.6-flash-high PASSED 15.6s 60,042 2,596 $0.0110
task_02_frontmatter gemini-3.6-flash-high PASSED 38.3s 88,373 13,839 $0.0210
task_03_limiter gemini-3.6-flash-high PASSED 17.9s 51,411 2,298 $0.0095
task_04_refactor gemini-3.6-flash-high PASSED 15.8s 55,031 2,384 $0.0101
task_05_emitter gemini-3.6-flash-high FAILED 20.8s 55,467 3,347 $0.0108
task_06_semver gemini-3.6-flash-high PASSED 22.6s 54,863 9,002 $0.0133
task_01_kv_ttl gemini-3.6-flash-low PASSED 29.1s 38,962 1,223 $0.0070
task_02_frontmatter gemini-3.6-flash-low PASSED 24.0s 26,480 0 $0.0049
task_03_limiter gemini-3.6-flash-low PASSED 45.0s 47,147 966 $0.0083
task_04_refactor gemini-3.6-flash-low PASSED 30.1s 27,218 0 $0.0046
task_05_emitter gemini-3.6-flash-low FAILED 47.4s 43,236 1,275 $0.0078
task_06_semver gemini-3.6-flash-low PASSED 36.2s 35,406 3,477 $0.0076
task_01_kv_ttl gemini-3.6-flash-medium PASSED 37.3s 49,417 2,467 $0.0092
task_02_frontmatter gemini-3.6-flash-medium PASSED 59.3s 71,631 10,813 $0.0169
task_03_limiter gemini-3.6-flash-medium PASSED 55.6s 58,168 1,856 $0.0104
task_04_refactor gemini-3.6-flash-medium PASSED 53.2s 59,227 1,634 $0.0104
task_05_emitter gemini-3.6-flash-medium FAILED 33.0s 34,619 2,997 $0.0072
task_06_semver gemini-3.6-flash-medium PASSED 35.1s 46,166 6,067 $0.0104
task_01_kv_ttl gemini-3.7-flash-high PASSED 32.2s 101,700 1,980 $0.0171
task_02_frontmatter gemini-3.7-flash-high PASSED 57.0s 148,527 7,146 $0.0274
task_03_limiter gemini-3.7-flash-high PASSED 29.6s 109,384 2,548 $0.0185
task_04_refactor gemini-3.7-flash-high PASSED 39.8s 134,994 2,413 $0.0224
task_05_emitter gemini-3.7-flash-high FAILED 44.3s 130,179 4,990 $0.0228
task_06_semver gemini-3.7-flash-high PASSED 44.4s 100,782 8,218 $0.0200
task_01_kv_ttl gemini-3.7-flash-low PASSED 11.3s 80,857 0 $0.0127
task_02_frontmatter gemini-3.7-flash-low PASSED 14.3s 87,874 0 $0.0140
task_03_limiter gemini-3.7-flash-low PASSED 12.9s 83,130 0 $0.0130
task_04_refactor gemini-3.7-flash-low PASSED 14.0s 103,560 0 $0.0162
task_05_emitter gemini-3.7-flash-low FAILED 11.8s 76,431 0 $0.0121
task_06_semver gemini-3.7-flash-low PASSED 37.6s 100,288 3,136 $0.0175
task_01_kv_ttl gemini-3.7-flash-medium PASSED 31.2s 103,862 848 $0.0171
task_02_frontmatter gemini-3.7-flash-medium PASSED 47.6s 140,984 3,811 $0.0243
task_03_limiter gemini-3.7-flash-medium PASSED 16.0s 68,344 1,195 $0.0113
task_04_refactor gemini-3.7-flash-medium PASSED 28.6s 109,321 1,391 $0.0179
task_05_emitter gemini-3.7-flash-medium FAILED 38.5s 111,345 4,646 $0.0199
task_06_semver gemini-3.7-flash-medium PASSED 36.1s 96,805 5,320 $0.0181
task_01_kv_ttl gemini-3.8-flash-demo PASSED 48.7s 120,345 3,163 $0.0207
task_02_frontmatter gemini-3.8-flash-demo PASSED 180.2s 0 0 $0.0000
task_03_limiter gemini-3.8-flash-demo PASSED 45.4s 119,439 5,084 $0.0212
task_04_refactor gemini-3.8-flash-demo PASSED 47.3s 119,806 4,353 $0.0209
task_05_emitter gemini-3.8-flash-demo FAILED 91.5s 172,378 10,851 $0.0324
task_06_semver gemini-3.8-flash-demo PASSED 101.5s 148,372 18,803 $0.0322
task_01_kv_ttl gemini-3.8-flash-high PASSED 49.5s 121,443 3,101 $0.0207
task_02_frontmatter gemini-3.8-flash-high PASSED 180.2s 0 0 $0.0000
task_03_limiter gemini-3.8-flash-high PASSED 66.2s 118,470 5,786 $0.0214
task_04_refactor gemini-3.8-flash-high PASSED 56.2s 137,246 4,173 $0.0235
task_05_emitter gemini-3.8-flash-high FAILED 162.3s 277,078 24,439 $0.0550
task_06_semver gemini-3.8-flash-high PASSED 128.0s 151,115 20,921 $0.0341
task_01_kv_ttl gemini-3.8-flash-low PASSED 19.9s 92,889 0 $0.0147
task_02_frontmatter gemini-3.8-flash-low PASSED 43.4s 138,775 0 $0.0225
task_03_limiter gemini-3.8-flash-low PASSED 31.4s 97,777 1,547 $0.0161
task_04_refactor gemini-3.8-flash-low PASSED 23.7s 105,826 0 $0.0165
task_05_emitter gemini-3.8-flash-low FAILED 18.2s 98,325 0 $0.0155
task_06_semver gemini-3.8-flash-low PASSED 24.6s 77,376 0 $0.0126
task_01_kv_ttl gemini-3.8-flash-medium PASSED 26.3s 83,444 2,152 $0.0144
task_02_frontmatter gemini-3.8-flash-medium PASSED 147.9s 187,266 17,494 $0.0393
task_03_limiter gemini-3.8-flash-medium PASSED 32.3s 105,725 1,596 $0.0174
task_04_refactor gemini-3.8-flash-medium PASSED 45.7s 120,658 2,093 $0.0201
task_05_emitter gemini-3.8-flash-medium FAILED 80.0s 149,255 15,263 $0.0305
task_06_semver gemini-3.8-flash-medium PASSED 83.1s 111,692 10,882 $0.0230
task_01_kv_ttl gemini-flash-demo PASSED 21.6s 66,927 2,293 $0.0118
task_02_frontmatter gemini-flash-demo PASSED 70.7s 138,093 9,399 $0.0269
task_03_limiter gemini-flash-demo PASSED 40.7s 97,859 1,601 $0.0163
task_04_refactor gemini-flash-demo PASSED 38.0s 123,102 2,312 $0.0204
task_05_emitter gemini-flash-demo FAILED 41.7s 118,417 3,944 $0.0206
task_06_semver gemini-flash-demo PASSED 47.5s 93,786 6,962 $0.0185
task_01_kv_ttl gpt-oss-120b-medium FAILED 125.0s 21,143 0 $0.0109
task_02_frontmatter gpt-oss-120b-medium FAILED 60.9s 10,501 0 $0.0053
task_03_limiter gpt-oss-120b-medium PASSED 6.6s 35,847 0 $0.0186
task_04_refactor gpt-oss-120b-medium TIMEOUT 180.2s 0 0 $0.0000
task_05_emitter gpt-oss-120b-medium FAILED 3.3s 22,403 0 $0.0114
task_06_semver gpt-oss-120b-medium FAILED 69.7s 23,719 0 $0.0128