Four Local Voice-Cloning Tools, One 576 KB Failure File

What I Was Actually Testing

I run a Japanese TTS side project and wanted to know whether free local voice-cloning tools could replace the paid cloud API I'd been using. Not in theory — I wanted a single session, same test sentences, same pass/fail criteria applied to every engine. I tested GPT-SoVITS v2 and v3, RVC with a Kokoro-prosody corpus, and seed-vc with a separate candidate reference. Each generated a phonetic coverage set plus a longer news-style sentence. I expected the local tools to be rougher but usable. That expectation was wrong in a specific, embarrassing way.

The 70-Second Runaway

Three of six generated takes were flat-out unusable. That alone would have settled the question. But one take didn't just fail — it ran completely out of control. The expected output was a few seconds long. The file I got back was 70 seconds and 576,873 bytes. I still have it sitting in a REJECT folder. A bad take that sounds wrong is annoying. A take that won't stop generating, in an unattended pipeline, is a different category of problem.

The 50% failure rate across local engines wasn't surprising in retrospect. Individual demos of GPT-SoVITS and seed-vc can sound genuinely good. What those demos don't show is what happens on your third run at 2 AM when the engine quietly decides to generate until you kill the process.

The Cloud Result and What I Did With It

The paid Japanese voice-cloning cloud API scored 5/5 passes on the identical test set with zero runaway outputs. That gap made the call straightforward. But I didn't stop there, because 5/5 on a small test set isn't a guarantee — it's an optimistic data point.

To harden that path, I added an ASR round-trip check using faster-whisper (base model, int8, CPU) and a hard duration-ratio ceiling. Any future take that runs long or transcribes badly gets caught and retried before it touches anything downstream. The 576 KB file is exactly the case that gate was designed to catch retroactively.

I also ran a manual A/B listening test on prosody settings. Stripping tone markers entirely — bare output, no annotation — produced the most natural result for neutral and bright presets. It also reduced byte count, which matters because this API bills per byte. Smaller and better is a rare combination; I'll take it.

Where Local Still Lives

I kept GPT-SoVITS installed. "Discard local entirely" isn't the right framing here. If the cloud API goes down, or pricing changes, I want an offline fallback I've already tested. What I won't do is run it as a primary path without a QC gate that catches runaway generation — which I didn't have during this test, and which is exactly why I have that file.

The honest read on local open-source voice cloning right now: individual takes can be acceptable, sometimes genuinely good. Unattended batch use is a different story. A tool that fails half the time and occasionally generates a 70-second file instead of a 4-second one needs a hard ceiling on output duration before it goes anywhere near automation.

If you're evaluating local TTS engines for a pipeline rather than a demo, measure failure rate on a boring coverage set, not just your best cherry-picked sentence — the catastrophic cases only show up when you stop watching.

Editor's note: AI tools were used to help draft parts of this story. The techniques it describes come from work originally written by human practitioners, and the story was reviewed and edited by a human before publishing.