Research

Can a model find lettering faults on a translated page?

Measured
2026-11-18
Sample
10 pages, 7 faults, 7 models, 2 runs
Data
2 CSV files, CC BY 4.0

Partly. In our November 2026 test, 7 models checked 10 translated webtoon pages with 7 known faults, twice each. The best, Gemini 3.8 Flash, found 6 of 7 in both runs for $0.16 to $0.17 a run. The other six found 1 to 5. No cheaper model found the emptied speech bubble reliably, so a person still checks every page.

Table of contents
  1. What did we test?
  2. How did we test it?
  3. Which models found the faults?
  4. What do the results show?
  5. Download the data
  6. Licence and how to cite

What did we test?

After Bublate letters a chapter, a model looks at every rendered segment next to the source and flags faults for the editor. This check is the largest single line in a chapter's model cost, 30% of it in our measured jobs. We tested whether cheaper models find the same faults.

How did we test it?

  • Pages. A Bublate render of episode 1 of the Korean webtoon True Beauty: 10 pages with 36 text blocks. We publish no images or text from it.
  • Faults. 7 known faults. We found 3 real ones by eye and planted 4 in copies of the render.
  • Runs. Each model checked all 10 pages twice, with no cache, through Bublate's production check code and prompt. Only the model changed.
  • Scoring. For each run we counted how many of the 7 known faults the model flagged.
FaultKindWhat was wrong
1realA shop sign lost its letters under a sound effect (MURMUR)
2realA red emphasis word was printed black
3realA sound effect (SWISH) was drawn far heavier than the source, with the hair under it smeared
4plantedA speech bubble was emptied
5plantedKorean source text was pasted back into a bubble
6plantedA flat patch was painted on the art
7plantedA caption was shrunk to 30% of its size

Which models found the faults?

ModelRun 1Run 2Cost per run (10 pages)Missed
Gemini 3.8 Flash, high (production)6 of 76 of 7$0.1614, $0.1685the SWISH size, both runs
DeepSeek V4.1 Flash 10175 of 74 of 7$0.0437, $0.0362the sign, SWISH
GPT-6 Luna, high4 of 75 of 7$0.0142, $0.0103the empty bubble both runs, the sign
Meta Muse Ember4 of 74 of 7$0.0195, $0.0134the sign, SWISH, the empty bubble
Tencent Hy4 Flash4 of 74 of 7$0.0096, $0.0077the sign, SWISH, the empty bubble once
MiniMax M3.1 Flash4 of 74 of 7$0.0104, $0.007623 block flags a run, most of them extra
Gemma 4 31B1 of 71 of 7$0.0134, $0.0026almost all

Most models also flagged a real fault that we had not logged: Korean traces left beside one translated line. We did not check the other extra flags.

What do the results show?

  • The production checker stays. Gemini 3.8 Flash found 6 of 7 faults in both runs. No other model matched it in either run.
  • An empty bubble is easy to miss. A bubble with no text looks like a clean bubble. No cheaper model found it reliably.
  • Cheaper checkers miss more. The other models cost $0.0026 to $0.0437 a run, against $0.1614 and $0.1685 for Gemini 3.8 Flash, and each run missed 2 or more faults.
  • A person still checks. Even the best run missed one of 7 faults. The editor shows every flag, and the final look at each page is a person's.

Download the data

Each file is a CSV with a header row, comma separated, UTF-8. Measured: 2026-11-18. The files hold numbers and descriptions only, no art or text from the test chapters.

Faults found per model and run

One row per model and run: faults found, pages checked, cost and flag counts.

page-check-2026-11.csv · 14 rows · CSV

Columns
ColumnMeaning
modelModel id on OpenRouter
run1 or 2
faults_found, faults_in_testKnown faults the run found, out of 7
pages_checkedPages sent to the model
cost_usdModel cost of the run on OpenRouter, US dollars
blocks_flaggedText blocks the model flagged, right or wrong
unmarked_text_flaggedPlaces outside the known blocks where the model reported text left untranslated
malformed_repliesReplies that were not valid JSON

The 7 known faults

Each known fault, real or planted, in words.

page-check-faults-2026-11.csv · 7 rows · CSV

Columns
ColumnMeaning
faultFault number
kindreal (found by eye) or planted (made in a copy of the render)
descriptionWhat was wrong

Licence and how to cite

The data is free to reuse under the Creative Commons Attribution 4.0 International (CC BY 4.0) licence. You can copy, change and publish it, also for commercial use, if you credit Bublate and link to this page. Suggested citation:

Bublate team (2026). Comic lettering fault check test (November 2026). Bublate. https://bublate.com/research/page-check-test. Data licensed CC BY 4.0.

First published . If we correct or extend the data, the files and this page change together and the updated date above moves.

Frequently asked questions

Which model finds lettering faults best?

In our test, Gemini 3.8 Flash on high reasoning found 6 of 7 known faults in both runs, for $0.16 to $0.17 per run of 10 pages. DeepSeek V4.1 Flash 1017 found 5 and then 4. The other cheaper models found 1 to 5.

Which fault was the hardest to find?

The emptied speech bubble. A bubble with no text looks like a clean bubble, so a checker has to compare it with the source page to see that text is missing. No cheaper model found it reliably, and GPT-6 Luna missed it in both runs.

Does an automatic check replace a person?

No. The best run in our test still missed one of 7 faults, and every model flagged things that were not faults. In Bublate the check marks blocks for the editor, and a person makes the final call on each page.

Can I reuse this data?

Yes. The CSV files are under the Creative Commons Attribution 4.0 licence. You can copy, change and publish them, also for commercial use, if you credit Bublate and link to this page.

Sources

  1. OpenRouter, models and prices · checked
  2. Creative Commons, Attribution 4.0 International · checked

Try Bublate for free

Continue with Google

or