#evals

Local AI Video Models on a DGX Spark: the Scene Renders, the Verb Does Not

“Grade on geometry and object identity, never on vibe. Materials lie convincingly.”

FLUX Can Spell, SDXL Cannot: Local AI Image Models on a DGX Spark

“A caption written after looking will always find something to praise.”

Forty-Four Harness Bugs, Zero Local LLM Limitations: an Accounting

“A model that looks broken is a harness bug until proven otherwise.”

Nine Local LLMs Ranked by Cost Per Solved Task — Seven of Them Have No Price at All

“A repair that doesn't satisfy the contract isn't 80% of a repair. It's a workspace you throw away.”

Nine Lines of Verification That Beat a Six-Agent AI Swarm

“It tried to stop 52 times. It was refused 51. That is the whole mechanism.”

1,200 Lines of Multi-Agent Orchestration, Beaten by One Local LLM Agent

“The orchestration didn't lose because multi-agent is a bad idea. It lost because it was insurance against a blindness that no longer existed.”

Twenty-Two Behavioural Criteria: What Replaced the Structural Grader My AI Agents Kept Fooling

“Scoring went from 28 seconds to 2, and got strictly harder to fool.”

The AI Grader That Passed the Same Bug Three Times: 229 Checks, Three Agent Workspaces, One Blind Spot

“A perfect score, three times, over the same bug. The checks weren't wrong. They were pointed at the wrong thing.”

A Scorecard, Not a Vibe: What I'd Need Before an AI Coding Agent Touches My Codebase

“An agent asked to judge its own completion will do what anything does in that position.”

Roadmap reviews when half the work is non-deterministic

“I stole four columns from my therapist; my engineers stopped sighing.”

An open-source eval starter kit for product managers

“Vibes are a feature flag without a kill switch.”

Building a multi-tenant LLM platform, 0 → 1

“The rubric outranked the org chart. That's the only way a call like that ever survives.”