Building a Self-Hosted Voice Cloning App: The Boring Problems Nobody Mentions

I’ve been building Timbre — a self-hosted text-to-speech and voice cloning app, Next.js on top, a C++ inference engine (CrispASR) underneath. The pitch is the usual self-hosted one: no subscription, no API key, no per-request billing, just your own hardware. What I didn’t expect was how little of the actual work was TTS-related. Almost all of it was “how do I get a binary and four gigabytes of weights onto someone else’s machine without making them suffer.”

1. Build vs. download is a bigger decision than it looks

First instinct: ship a setup script that clones the engine repo and runs cmake --build. It works, it’s “correct,” and it’s wrong for almost everyone who’ll actually run this.

# what I started with
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j

This assumes git, cmake, and a working C++ toolchain are already on the machine. On my dev box, sure. On a teammate’s fresh Windows install, that’s an afternoon of Visual Studio Build Tools before they’ve generated a single sample. The moment I checked, the engine’s CI was already publishing platform binaries as release assets — so the setup script became a download-and-extract, not a build:

function assetName() {
  const plat = os.platform();
  if (plat === "win32") return "crispasr-windows-x64.zip";
  if (plat === "darwin") return "crispasr-macos.tar.gz";
  if (plat === "linux") return "crispasr-linux-x86_64.tar.gz";
}

Same outcome, zero toolchain requirement. The lesson wasn’t really about CMake — it’s that “how do I get this dependency” is worth a five-minute check against the upstream repo before writing any install logic, because someone else has usually already solved the packaging problem for you.

2. Where the download happens matters more than how

The other early instinct was to lazy-download models on first API request — check if the file exists, fetch it if not, then run inference. It technically works and it means one less setup step. It’s also a bad idea, for a reason that only shows up once you think about when someone hits that code path:

// don't do this in a request handler
if (!existsSync(modelPath)) {
  await downloadModel(modelPath); // multi-GB, multi-minute
}
const result = await runInference(modelPath, input);

Your first real user’s first request now blocks on a multi-gigabyte download, inside a request that already has a generation timeout. Worse on a dev server, where file-watcher restarts can re-trigger the check mid-download. Model fetching belongs in a setup step that runs once, is cacheable in a Docker layer, and fails loudly before anyone’s waiting on an API response — not inline in a route handler:

async function assertReady(binPath: string, files: string[]) {
  await access(binPath, constants.X_OK).catch(() => {
    throw new Error("crispasr binary missing — run `bun run setup`");
  });
  // ...same for each model file
}

The route just checks and fails fast now. No download logic anywhere near request handling.

3. “Cross-platform script” usually means “not a shell script”

The setup script started as bash. Fine on Linux and macOS, dead on arrival on Windows without WSL or git-bash — and telling users to install a POSIX shell just to run bun run setup defeats the point of a one-command setup. Rewriting it in Node using only built-ins solved this without adding a dependency, since Node was already required to run the app at all:

Nothing exotic — just picking the runtime that’s already a hard requirement (Node, via bun) instead of assuming a shell that half your users won’t have.

None of these three problems were about text-to-speech, voice cloning, or even Next.js — they were all “how does a piece of software get from GitHub onto a stranger’s laptop without them opening an issue.” That turned out to be most of the actual engineering work. If you’re building anything that wraps a native binary or ships large model weights, budget real time for distribution — it’ll outweigh the model code by a wide margin.