Research
Can a model find lettering faults on a translated page?
- Measured
- 2026-11-18
- Sample
- 10 pages, 7 faults, 7 models, 2 runs
- Data
- 2 CSV files, CC BY 4.0
Partly. In our November 2026 test, 7 models checked 10 translated webtoon pages with 7 known faults, twice each. The best, Gemini 3.8 Flash, found 6 of 7 in both runs for $0.16 to $0.17 a run. The other six found 1 to 5. No cheaper model found the emptied speech bubble reliably, so a person still checks every page.
Table of contents
What did we test?
After Bublate letters a chapter, a model looks at every rendered segment next to the source and flags faults for the editor. This check is the largest single line in a chapter's model cost, 30% of it in our measured jobs. We tested whether cheaper models find the same faults.
How did we test it?
- Pages. A Bublate render of episode 1 of the Korean webtoon True Beauty: 10 pages with 36 text blocks. We publish no images or text from it.
- Faults. 7 known faults. We found 3 real ones by eye and planted 4 in copies of the render.
- Runs. Each model checked all 10 pages twice, with no cache, through Bublate's production check code and prompt. Only the model changed.
- Scoring. For each run we counted how many of the 7 known faults the model flagged.
| Fault | Kind | What was wrong |
|---|---|---|
| 1 | real | A shop sign lost its letters under a sound effect (MURMUR) |
| 2 | real | A red emphasis word was printed black |
| 3 | real | A sound effect (SWISH) was drawn far heavier than the source, with the hair under it smeared |
| 4 | planted | A speech bubble was emptied |
| 5 | planted | Korean source text was pasted back into a bubble |
| 6 | planted | A flat patch was painted on the art |
| 7 | planted | A caption was shrunk to 30% of its size |
Which models found the faults?
| Model | Run 1 | Run 2 | Cost per run (10 pages) | Missed |
|---|---|---|---|---|
| Gemini 3.8 Flash, high (production) | 6 of 7 | 6 of 7 | $0.1614, $0.1685 | the SWISH size, both runs |
| DeepSeek V4.1 Flash 1017 | 5 of 7 | 4 of 7 | $0.0437, $0.0362 | the sign, SWISH |
| GPT-6 Luna, high | 4 of 7 | 5 of 7 | $0.0142, $0.0103 | the empty bubble both runs, the sign |
| Meta Muse Ember | 4 of 7 | 4 of 7 | $0.0195, $0.0134 | the sign, SWISH, the empty bubble |
| Tencent Hy4 Flash | 4 of 7 | 4 of 7 | $0.0096, $0.0077 | the sign, SWISH, the empty bubble once |
| MiniMax M3.1 Flash | 4 of 7 | 4 of 7 | $0.0104, $0.0076 | 23 block flags a run, most of them extra |
| Gemma 4 31B | 1 of 7 | 1 of 7 | $0.0134, $0.0026 | almost all |
Most models also flagged a real fault that we had not logged: Korean traces left beside one translated line. We did not check the other extra flags.
What do the results show?
- The production checker stays. Gemini 3.8 Flash found 6 of 7 faults in both runs. No other model matched it in either run.
- An empty bubble is easy to miss. A bubble with no text looks like a clean bubble. No cheaper model found it reliably.
- Cheaper checkers miss more. The other models cost $0.0026 to $0.0437 a run, against $0.1614 and $0.1685 for Gemini 3.8 Flash, and each run missed 2 or more faults.
- A person still checks. Even the best run missed one of 7 faults. The editor shows every flag, and the final look at each page is a person's.
Download the data
Each file is a CSV with a header row, comma separated, UTF-8. Measured: 2026-11-18. The files hold numbers and descriptions only, no art or text from the test chapters.
Faults found per model and run
One row per model and run: faults found, pages checked, cost and flag counts.
page-check-2026-11.csv · 14 rows · CSV
Columns
| Column | Meaning |
|---|---|
model | Model id on OpenRouter |
run | 1 or 2 |
faults_found, faults_in_test | Known faults the run found, out of 7 |
pages_checked | Pages sent to the model |
cost_usd | Model cost of the run on OpenRouter, US dollars |
blocks_flagged | Text blocks the model flagged, right or wrong |
unmarked_text_flagged | Places outside the known blocks where the model reported text left untranslated |
malformed_replies | Replies that were not valid JSON |
The 7 known faults
Each known fault, real or planted, in words.
page-check-faults-2026-11.csv · 7 rows · CSV
Columns
| Column | Meaning |
|---|---|
fault | Fault number |
kind | real (found by eye) or planted (made in a copy of the render) |
description | What was wrong |
Licence and how to cite
The data is free to reuse under the Creative Commons Attribution 4.0 International (CC BY 4.0) licence. You can copy, change and publish it, also for commercial use, if you credit Bublate and link to this page. Suggested citation:
Bublate team (2026). Comic lettering fault check test (November 2026). Bublate. https://bublate.com/research/page-check-test. Data licensed CC BY 4.0.First published . If we correct or extend the data, the files and this page change together and the updated date above moves.
Frequently asked questions
Which model finds lettering faults best?
In our test, Gemini 3.8 Flash on high reasoning found 6 of 7 known faults in both runs, for $0.16 to $0.17 per run of 10 pages. DeepSeek V4.1 Flash 1017 found 5 and then 4. The other cheaper models found 1 to 5.
Which fault was the hardest to find?
The emptied speech bubble. A bubble with no text looks like a clean bubble, so a checker has to compare it with the source page to see that text is missing. No cheaper model found it reliably, and GPT-6 Luna missed it in both runs.
Does an automatic check replace a person?
No. The best run in our test still missed one of 7 faults, and every model flagged things that were not faults. In Bublate the check marks blocks for the editor, and a person makes the final call on each page.
Can I reuse this data?
Yes. The CSV files are under the Creative Commons Attribution 4.0 licence. You can copy, change and publish them, also for commercial use, if you credit Bublate and link to this page.
Sources
- OpenRouter, models and prices · checked
- Creative Commons, Attribution 4.0 International · checked