Cascadity Audio
← All streams

#benchmarks

Everything tagged benchmarks, across every stream.

0
MVMaker Views #benchmarks#deepseek#Hacker-News#llm#open-weights

DeepSeek releases V4.1-Flash-1017, an MIT-licensed "minor update" with sizable benchmark gains

The model card has the full benchmark table against GLM-5.3, GPT-5.6 Sol and Opus 5.

DeepSeek published DeepSeek-V4.1-Flash-1017 on Hugging Face, calling it a minor update to V4.1-Flash that brings substantial improvements in reasoning, coding, agentic tasks and visual benchmarks while cutting task time and output length. Its table shows, for example, Terminal-Bench 4.0 rising from 31.2 to 42.3 and DeepSWE v1.1 from 74.2 to 77.0, ahead of GPT-5.6 Sol on several tests and behind Opus 5 on others. The model has a 1M-token context window, weights are MIT-licensed, and DeepSeek released deepseek-recipe, Rust libraries with Python bindings for its prompt format.

On Hacker News, users said the model feels better than its benchmarks because of speed, visible reasoning and fewer refusals; others posted informal head-to-head tests and cautioned that single runs prove little. Some joked about calling big gains a minor update; a side debate asked whether US labs want regulation that would hurt open models.

Cascadic Analysis 2
UndertowBenchmark scores can overstate real-world reliability

What hidden risk could pull against this, even if the news is good?

The gains, such as Terminal-Bench 4.0 from 31.2 to 42.3, come from DeepSeek's own table on its model card. Hacker News users posted head-to-head tests of their own, and others cautioned that single runs prove little; how the model holds up on a team's own tasks still has to be measured there.

0
UndertowLoss of meaningful human control as capability and autonomy increase

What hidden risk could pull against this, even if the news is good?

The weights are MIT-licensed and downloadable, and some Hacker News users liked the model partly for its fewer refusals. Once the weights are out, how the model is fine-tuned, what it refuses and where it runs is up to whoever downloaded it; DeepSeek can't recall or patch copies already in circulation.

0
Rabbit Holes 1
Generated by AI.
0
AIAI #benchmarks#evaluation#jobs#labor#research

Mercor maps AI benchmarks against US jobs and finds most working hours barely covered

The interactive map lets you click into any job family and see which tasks AI tests actually measure.

Mercor, which sells AI evaluation services, mapped 29 AI benchmarks onto the US Labor Department's O*NET list of 923 occupations and 18,797 tasks. Only 208 occupations have a task covered by at least five benchmark prompts, 6 of 22 job families have none, and Mercor estimates 90% of American working hours go to tasks with little or no benchmark coverage, making it hard to know how well AI does the work people actually do.

Coverage is lopsided: computer and math (29 of 36 occupations) and business and finance (33 of 48) fare well, while healthcare practitioners (12 of 89) and education (9 of 62) lag. Mercor says benchmarks favor the largest labor markets; for example, they heavily cover lawyers' tasks but barely touch those of similarly exposed legal secretaries and court clerks.

Cascadic Analysis 2
UndertowBenchmark scores can overstate real-world reliability

What hidden risk could pull against this, even if the news is good?

Mercor's map says 90% of US working hours go to tasks with little or no benchmark coverage, so a model's headline scores say little about most real work. Mercor sells AI evaluation services, which gives it a stake in the conclusion that more evaluation is needed.

0
UndertowRole underspecification / the Doorman Problem

What hidden risk could pull against this, even if the news is good?

Mercor's example: benchmarks heavily cover lawyers' tasks but barely touch those of similarly exposed legal secretaries and court clerks. If those jobs are automated on the strength of lawyer-focused scores, no one will have measured how AI handles the exceptions and judgment calls they actually involve.

0
Rabbit Holes 1

Sign in to upvote

Your votes tell us which stories, analysis and rabbit holes are worth it. We'll bring you right back to this spot and count your vote.

Sign in Create an account