Research
Which models translate Korean webtoon dialogue best, and at what cost?
- Measured
- 2026-11-18
- Sample
- 38 lines, 10 models, 760 scores
- Data
- 2 CSV files, CC BY 4.0
In our November 2026 blind test on 38 Korean webtoon lines, Claude Opus 5.5 scored 8.77 of 10 and DeepSeek V4.1 Flash 1017 scored 8.68 at one 36th of the cost. Seven of the ten models scored between 7.59 and 8.25. Every model scored lower on sound effects than on dialogue. At this size, differences under about 0.3 points are noise.
Table of contents
What did we test?
Translation and the edit pass after it are the two most expensive model calls in a Bublate chapter. We tested whether cheaper models can do them as well as the model in production, Claude Opus 5.5, with everything else held the same.
How did we test it?
- Lines. Two sets from episode 0 of the Korean webtoon Mercenary Enrollment: a sample set of 15 lines and a held-out set of 23 lines. 28 lines are dialogue (speech, shouts and thoughts) and 10 are sound effects. We publish no text from the chapter.
- Pipeline. Bublate's production code for translate and polish, with the settings of the quality profile. Only the model changed, with no fallback model. All models got one shared scene brief.
- Polish. Each model polished its own drafts. Models that read images polished with the pages in view. Three text-only models polished without them.
- Judging. Two blind judges per set scored every line from 0 to 3 against a key of meaning and style marks, paraphrased from the official English release. The judges saw the models under shuffled letters.
- Score. The mean line score, divided by 3 and multiplied by 10. Each model has 76 scores: 38 lines, 2 judges.
- Cost. Each model's real tokens on OpenRouter for both sets, thinking included.
Which models scored best?
| Model | Score of 10 | Dialogue | Sound effects | Scores at 3 | Scores at 0 or 1 | Test cost | Cheaper than Opus |
|---|---|---|---|---|---|---|---|
| Claude Opus 5.5 (production) | 8.77 | 8.99 | 8.17 | 67% | 3 | $0.2919 | 1x |
| DeepSeek V4.1 Flash 1017 | 8.68 | 9.11 | 7.50 | 68% | 6 | $0.0081 | 36x |
| Meta Muse Ember | 8.25 | 8.63 | 7.17 | 54% | 5 | $0.0206 | 14x |
| GPT-6 Luna | 8.16 | 8.51 | 7.17 | 49% | 3 | $0.0070 | 42x |
| Tencent Hy4 Flash | 8.16 | 8.57 | 7.00 | 53% | 6 | $0.0080 | 37x |
| Gemma 4 31B | 7.94 | 8.10 | 7.50 | 49% | 8 | $0.0038 | 77x |
| Solar Pro 4 (text only) | 7.89 | 8.33 | 6.67 | 47% | 8 | $0.0089 | 33x |
| MiniMax M3.1 Flash | 7.72 | 7.74 | 7.67 | 47% | 11 | $0.0088 | 33x |
| Hy-MT2 30B (text only) | 7.59 | 8.04 | 6.33 | 38% | 8 | $0.0026 | 113x |
| Mi:dm K 3.0 Pro (text only) | 6.80 | 6.85 | 6.67 | 26% | 13 | $0.0147 | 20x |
Mi:dm K 3.0 Pro left 2 lines empty. Tencent Hy4 Flash wrote 49,294 output tokens and took 325.5 seconds for both sets, the slowest in the test.
What do the results show?
- One cheap model matched the production model. DeepSeek V4.1 Flash 1017 came within 0.09 points of Claude Opus 5.5 and scored higher on dialogue. Its weak lines were of the same kind: an idiom, a sound effect choice.
- Sound effects are the hard part. Every model scored lower on the 10 sound effect lines than on dialogue. The best sound effect score was 8.17.
- Price is not quality. The cheapest model, Hy-MT2 30B, scored 7.59. Meta Muse Ember cost more than every model except Claude Opus 5.5 and scored third.
- Low scores tell the story. Claude Opus 5.5 and GPT-6 Luna had 3 scores of 0 or 1 out of 76. Mi:dm K 3.0 Pro had 13.
Bublate still translates with Claude Opus 5.5. A model change needs a full chapter and more than one run per model first. For how the series bible keeps names and voice the same across chapters, read inside the series bible.
Download the data
Each file is a CSV with a header row, comma separated, UTF-8. Measured: 2026-11-18. The files hold numbers and descriptions only, no art or text from the test chapters.
Score and cost per model
One row per model: overall, dialogue and sound effect scores, score counts, test cost, tokens and time.
translation-models-summary-2026-11.csv · 10 rows · CSV
Columns
| Column | Meaning |
|---|---|
model | Model id on OpenRouter |
score_of_10, dialogue_score_of_10, sfx_score_of_10 | Mean line score divided by 3 and multiplied by 10: all lines, dialogue lines, sound effect lines |
judgements | Scores the model got (38 lines, 2 judges) |
pct_judgements_at_3, judgements_at_0_or_1 | Share of scores at 3, and count of scores at 0 or 1 |
test_cost_usd, times_cheaper_than_opus | Model cost of both sets on OpenRouter, and Claude Opus 5.5 cost divided by it |
tokens_in, tokens_out, seconds | Tokens sent and received, thinking included, and time for both sets |
polish_saw_pages | yes when the edit pass saw the page images |
lines_left_empty | Lines the model returned no text for |
Every score
One row per line, model and judge: 38 lines, 10 models, 2 judges.
translation-models-scores-2026-11.csv · 760 rows · CSV
Columns
| Column | Meaning |
|---|---|
set | sample (15 lines) or heldout (23 lines) |
line | Line id: segment and block number in the chapter |
line_kind | speech, shout, thought or sfx |
model | Model id on OpenRouter |
judge | judge_1 or judge_2 of that set |
score_0_to_3 | The judge's score for the line |
Licence and how to cite
The data is free to reuse under the Creative Commons Attribution 4.0 International (CC BY 4.0) licence. You can copy, change and publish it, also for commercial use, if you credit Bublate and link to this page. Suggested citation:
Bublate team (2026). Korean webtoon translation model test (November 2026). Bublate. https://bublate.com/research/translation-model-test. Data licensed CC BY 4.0.First published . If we correct or extend the data, the files and this page change together and the updated date above moves.
Frequently asked questions
Which model translates Korean webtoons best?
In our test of 38 lines, Claude Opus 5.5 scored highest at 8.77 of 10, and DeepSeek V4.1 Flash 1017 scored 8.68. A gap under about 0.3 points is noise at this size, so the two are level on this test. One chapter and one run cannot settle it for every series.
Is a cheaper model good enough?
One was on this test. DeepSeek V4.1 Flash 1017 scored within 0.09 points of Claude Opus 5.5 at about one 36th of the cost. The other cheap models scored 6.80 to 8.25. Test on a full chapter of your own series before you switch.
Why are sound effects scored lower?
A sound effect has no single right word. The translator must pick the English sound that fits the scene, the size of the art and the house style. Every model in the test scored lower on the 10 sound effect lines than on its dialogue lines.
Who were the judges?
Two blind judges per set scored each line from 0 to 3 against a key of meaning and style marks, paraphrased from the official English release. They saw the candidate lines under shuffled letters, so they could not tell which model wrote which line.
Can I reuse this data?
Yes. The CSV files are under the Creative Commons Attribution 4.0 licence. You can copy, change and publish them, also for commercial use, if you credit Bublate and link to this page.
Sources
- OpenRouter, models and prices · checked
- Creative Commons, Attribution 4.0 International · checked