The Test Was Green, the Page Was Unusable

This report is also available in German.

How we got here

In March we tested two AI models on our own servers (in German), one day each. One question stayed open: how good are self-hosted models actually?

The answer convinced us enough not to stop at a test. Since then we have been hosting models permanently ourselves. They are in daily use, and we have accumulated substantial operating experience with them.

A split has emerged along the way that matters for this article:

• When developing — an agent writing code — we still use Claude Code heavily.
• Inside the applications we ship that use AI themselves, it is essentially DeepSeek only.

The reason for the second is not thrift, it is scope. Those applications do not run open-ended chats with many loose, complex ends. They operate inside fixed, in places almost deterministic rules — and for that, DeepSeek is entirely sufficient.

That is the actual rule of thumb, and it runs through everything below: the narrower the frame, the less model you need.

So what follows is not a lab experiment with an unfamiliar technology. It is a snapshot taken in the middle of ongoing operations. The open question was no longer “does this work at all,” but: where exactly is the gap to a commercial frontier model — and what does that mean for who in the company gets an agent?

What we did

We had two AI agents work simultaneously on the same codebase for one workday: community.programmieren.de, a public competition platform where schools present their programming projects and the public votes on them. Real software, sitting on the open internet with no VPN in front of it, presented to a conference of school principals the next day.

• Lane A — “Rules, permissions, data”: the data layer, the day’s only database migration, the rules that money depends on. Handled by Claude Code with Opus 5.
• Lane B — “What visitors see”: interface, accessibility, the public reading pages. Handled by a self-hosted DeepSeek V4 Flash, served through vLLM on Seibert-owned GPUs, driven by the open-source tools Reasonix and First Mate — both third-party projects, not Seibert software.

Apart from the task list and the file boundaries, the two briefs were word for word identical: same review gate, same sub-agent rules, same merge condition. Separate files were the only collision protection — each lane got a list of “this is yours” and “do not touch this.” Both used the same CI and the same automated review agent.

The numbers

The method belongs with the table, or the table is worthless: lines as the sum of each merge commit against its main parent (git diff --numstat <merge>^1 <merge>), times from API timestamps, verdicts from PR comments. All of it recomputable.

And a disclosure that belongs here: the evaluation itself was produced by Opus 5, in a separate session that took part in neither lane. So one of the models is, among other things, grading its own lane. We did not paper over that: the self-assessment was recomputed independently against merge commits and timestamps, and it matches to the line (see We audited the auditor below). Even so, anyone who distrusts these numbers has a legitimate reason to, and the raw data is in the repository.

And the most important caveat up front: the slices were unequal. Lane A had the data layer, the migration and the money rules — that weighs more per line than interface and reading pages. “Application code per hour” is therefore not a ranking; it is only readable together with the slice. Anyone turning this table into “model A beats model B” has misread it.

The result that actually occupied us is not in the table anyway. It is five lessons about what “done” is measured by — and about the fact that the usual measures do not measure it.