Vincent/FlightTube
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

perf: run yt-dlp from a zipapp on a portable Python

Browse Source 
Replaces the official yt-dlp_macos sidecar. That build is PyInstaller
one-file and unpacks 37MB on every invocation, costing ~8s per call on
this machine — ruled out signing, xattrs and thinning the universal
binary as causes, and it is 8% CPU over 8s, so it is the unpack itself.
Every stream resolve paid it.

The 3MB zipapp on a trimmed portable CPython starts in ~0.45s, a 17x
improvement; a real stream resolve goes from ~10s to ~2.4s. macOS ships
only Python 3.9 and yt-dlp needs 3.10+, hence bundling an interpreter.

yt-dlp is now resolved as an argv prefix rather than a path, so a system
install (a plain script, instant) still takes precedence.

Bundle grows 153MB -> 219MB, which buys back the responsiveness.
This commit is contained in:
vincent
2026-08-29 12:07:09 +02:00
parent 50ad908f1e
commit 9e9f71b31b
5 changed files with 130 additions and 45 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitignore
+3 -1
View File
@@ -28,5 +28,7 @@ dist-ssr
*.sln
*.sw?
# Bundled helper binaries — ~140MB, fetched by scripts/fetch-binaries.sh
# Bundled helper tools — ~130MB, fetched by scripts/fetch-binaries.sh
src-tauri/binaries/
src-tauri/resources/python/
src-tauri/resources/yt-dlp.pyz
README.md
+14 -3
View File
@@ -50,9 +50,17 @@ The binaries are not in git (~140 MB together). Fetch them once before building:
./scripts/fetch-binaries.sh
```
That pulls yt-dlp from its official GitHub release and static arm64 ffmpeg/ffprobe from
osxexperts.net — Homebrew's ffmpeg links a dozen dylibs and cannot be relocated into an
app bundle. Note that static ffmpeg builds are GPL, which matters if you redistribute.
That pulls static arm64 ffmpeg/ffprobe from osxexperts.net (Homebrew's ffmpeg links a
dozen dylibs and cannot be relocated into a bundle), the yt-dlp **zipapp**, and a portable
CPython from astral's python-build-standalone.
The interpreter is bundled deliberately. The official `yt-dlp_macos` binary is a
PyInstaller one-file build that unpacks 37 MB on *every* invocation — about eight seconds
per call, which every stream resolve would pay. The 3 MB zipapp on a portable interpreter
starts in under half a second. macOS ships only Python 3.9 and yt-dlp requires 3.10+,
which is why the interpreter has to come too.
Static ffmpeg builds are GPL, which matters if you redistribute.
## Running it
@@ -88,6 +96,9 @@ system by default; Settings offers System / Light / Dark.
- The bundled `yt-dlp` is frozen at build time and YouTube changes often. Re-run
`./scripts/fetch-binaries.sh` and rebuild, or install a newer one on your system —
the app prefers a system copy when it finds one.
- yt-dlp warns that extraction without a JavaScript runtime (deno) is deprecated. Format
lists are currently identical either way, but that may not hold; installing deno, or a
system yt-dlp that depends on it, is the escape hatch.
- Downloading videos is contrary to YouTube's Terms of Service.
## Tests
scripts/fetch-binaries.sh
+45 -21
View File
@@ -1,35 +1,59 @@
#!/usr/bin/env bash
# Fetches the helper binaries FlightTube ships as Tauri sidecars.
# Fetches the helper tools FlightTube ships inside its bundle.
#
# They are not in git — together they are about 140MB. Run this once before
# building, or after changing the target architecture.
# They are not in git (~130MB together). Run this once before building.
#
# ffmpeg / ffprobe static arm64 binaries, shipped as Tauri sidecars
# python + yt-dlp a portable interpreter plus the yt-dlp zipapp
#
# yt-dlp is deliberately NOT the official yt-dlp_macos build: that is a
# PyInstaller one-file binary which unpacks ~37MB on every invocation, costing
# about eight seconds per call. The 3MB zipapp on a portable interpreter starts
# in under half a second. macOS ships only Python 3.9 and yt-dlp needs 3.10+,
# which is why the interpreter has to come along.
set -euo pipefail
TRIPLE="${1:-aarch64-apple-darwin}"
DEST="$(cd "$(dirname "$0")/.." && pwd)/src-tauri/binaries"
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
BIN="$ROOT/src-tauri/binaries"
RES="$ROOT/src-tauri/resources"
TMP="$(mktemp -d)"
trap 'rm -rf "$TMP"' EXIT
mkdir -p "$DEST"
mkdir -p "$BIN" "$RES"
echo "Fetching yt-dlp (official GitHub release)…"
curl -fsSL -o "$TMP/yt-dlp" \
https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp_macos
# Static arm64 builds; Homebrew's ffmpeg links a dozen dylibs and cannot be
# relocated into an app bundle.
echo "Fetching ffmpeg and ffprobe (static arm64)…"
# Homebrew's ffmpeg links a dozen dylibs and cannot be relocated into a bundle.
for tool in ffmpeg ffprobe; do
echo "Fetching $tool (static, osxexperts.net)…"
curl -fsSL -A "Mozilla/5.0" -o "$TMP/$tool.zip" "https://www.osxexperts.net/${tool}9arm.zip"
unzip -o -q "$TMP/$tool.zip" -d "$TMP"
install -m 0755 "$TMP/$tool" "$BIN/$tool-$TRIPLE"
done
for tool in yt-dlp ffmpeg ffprobe; do
install -m 0755 "$TMP/$tool" "$DEST/$tool-$TRIPLE"
echo " -> $DEST/$tool-$TRIPLE"
done
echo "Fetching the yt-dlp zipapp…"
curl -fsSL -o "$RES/yt-dlp.pyz" \
https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp
echo "Done. Verifying they run:"
for tool in yt-dlp ffmpeg ffprobe; do
"$DEST/$tool-$TRIPLE" -version 2>/dev/null | head -1 || \
"$DEST/$tool-$TRIPLE" --version 2>/dev/null | head -1
done
echo "Fetching a portable Python…"
PY_TAG="$(curl -fsSL -H 'User-Agent: flighttube-build' \
https://api.github.com/repos/astral-sh/python-build-standalone/releases/latest \
| python3 -c 'import json,sys; print(json.load(sys.stdin)["tag_name"])')"
PY_ASSET="cpython-3.12.14+${PY_TAG}-${TRIPLE}-install_only_stripped.tar.gz"
curl -fsSL -o "$TMP/python.tar.gz" \
"https://github.com/astral-sh/python-build-standalone/releases/download/${PY_TAG}/${PY_ASSET// /%20}"
rm -rf "$RES/python"
tar -xzf "$TMP/python.tar.gz" -C "$TMP"
mv "$TMP/python" "$RES/python"
echo "Trimming the interpreter to what yt-dlp needs…"
P="$RES/python"
rm -rf "$P/include" "$P/share" "$P/lib/pkgconfig" \
"$P/lib/python3.12/test" "$P/lib/python3.12/idlelib" \
"$P/lib/python3.12/tkinter" "$P/lib/python3.12/lib2to3" \
"$P/lib/python3.12/ensurepip" "$P/lib/python3.12/turtledemo" \
"$P/lib/python3.12/config-3.12-darwin"
find "$P" -name "libpython*.a" -delete
find "$P" -name "__pycache__" -type d -prune -exec rm -rf {} + 2>/dev/null || true
echo "Verifying:"
"$BIN/ffmpeg-$TRIPLE" -version | head -1
"$P/bin/python3.12" "$RES/yt-dlp.pyz" --version
src-tauri/src/commands.rs
+63 -18
View File
@@ -43,6 +43,18 @@ pub struct AppState {
/// Resolved stream URLs, keyed by video and quality cap. YouTube's signed
/// URLs last hours, so replaying a video should not pay for yt-dlp again.
pub streams: Arc<Mutex<HashMap<(String, Option<u32>), (String, std::time::Instant)>>>,
/// argv prefix that runs yt-dlp: either a system binary, or the bundled
/// Python interpreter followed by the zipapp.
pub yt_dlp: Vec<String>,
}
impl AppState {
/// A ready-to-configure yt-dlp process.
fn yt_dlp(&self) -> tokio::process::Command {
let mut cmd = tokio::process::Command::new(&self.yt_dlp[0]);
cmd.args(&self.yt_dlp[1..]);
cmd
}
}
/// Comfortably inside the ~6h lifetime of YouTube's signed URLs.
@@ -107,6 +119,35 @@ fn bin(name: &str) -> String {
name.to_string()
}
/// Works out how to run yt-dlp.
///
/// A system install wins when present — it is a plain Python script, starts in
/// milliseconds, and is easier to keep current than a bundled copy. Otherwise
/// the app runs its own interpreter against the yt-dlp zipapp. The 3MB zipapp
/// plus a portable Python starts in about half a second; the official
/// PyInstaller binary took eight, because it unpacks 37MB on every call.
fn resolve_yt_dlp(app: &AppHandle) -> Vec<String> {
for prefix in ["/opt/homebrew/bin", "/usr/local/bin", "/usr/bin"] {
let candidate = format!("{prefix}/yt-dlp");
if std::path::Path::new(&candidate).exists() {
return vec![candidate];
}
}
if let Ok(res) = app.path().resource_dir() {
// python3 and python are symlinks; name the real file so the bundle
// does not depend on symlinks surviving the copy.
let python = res.join("python/bin/python3.12");
let zipapp = res.join("yt-dlp.pyz");
if python.exists() && zipapp.exists() {
return vec![
python.to_string_lossy().to_string(),
zipapp.to_string_lossy().to_string(),
];
}
}
vec!["yt-dlp".to_string()]
}
/// True when the binary we resolved is the one inside the app bundle.
fn is_bundled(name: &str) -> bool {
std::env::current_exe()
@@ -115,18 +156,18 @@ fn is_bundled(name: &str) -> bool {
.unwrap_or(false)
}
/// Whether yt-dlp is the app's own copy rather than one found on the system.
fn yt_dlp_is_bundled(argv: &[String]) -> bool {
argv.len() > 1
}
/// `flag` differs per tool: yt-dlp takes `--version`, ffmpeg only accepts
/// `-version` (it exits non-zero on `--version` and writes to stderr), so the
/// flag is passed in and both streams are consulted.
async fn version_of(name: &str, flag: &str) -> Option<String> {
// Must be the async Command: yt-dlp is a PyInstaller bundle that unpacks
// ~37MB on its first run, so a blocking call here would stall a runtime
// worker for the better part of a minute.
let out = tokio::process::Command::new(bin(name))
.arg(flag)
.output()
.await
.ok()?;
async fn version_of(argv: &[String], flag: &str) -> Option<String> {
let mut cmd = tokio::process::Command::new(&argv[0]);
cmd.args(&argv[1..]);
let out = cmd.arg(flag).output().await.ok()?;
if !out.status.success() {
return None;
}
@@ -159,8 +200,11 @@ pub async fn check_prereqs(state: State<'_, AppState>) -> Result<Prereqs, String
}
let fresh = Prereqs {
yt_dlp: version_of("yt-dlp", "--version").await.map(|v| label(&v, "yt-dlp")),
ffmpeg: version_of("ffmpeg", "-version").await.map(|v| {
yt_dlp: version_of(&state.yt_dlp, "--version").await.map(|v| {
let origin = if yt_dlp_is_bundled(&state.yt_dlp) { "bundled" } else { "system" };
format!("{v} ({origin})")
}),
ffmpeg: version_of(&[bin("ffmpeg")], "-version").await.map(|v| {
// ffmpeg's first line is long; keep the useful head of it.
let head = v.split_whitespace().take(3).collect::<Vec<_>>().join(" ");
label(&head, "ffmpeg")
@@ -260,6 +304,7 @@ pub async fn resolve_stream(
// The HLS master playlist. Every m3u8 format shares the same manifest_url,
// so any one of them yields the master.
if let Some(master) = yt_dlp_print(
&state,
&["-f", "bv*[protocol^=m3u8]", "--print", "%(manifest_url)s", &url],
)
.await
@@ -287,7 +332,7 @@ pub async fn resolve_stream(
}
// Rare fallback: an old-style progressive muxed MP4.
if let Some(u) = yt_dlp_print(&[
if let Some(u) = yt_dlp_print(&state, &[
"-f",
"b[ext=mp4][acodec!=none][vcodec!=none]",
"--print",
@@ -366,8 +411,8 @@ fn resolution_height(stream_inf: &str) -> Option<u32> {
}
/// Runs yt-dlp and returns its first non-empty stdout line, or None.
async fn yt_dlp_print(args: &[&str]) -> Option<String> {
let mut cmd = tokio::process::Command::new(bin("yt-dlp"));
async fn yt_dlp_print(state: &State<'_, AppState>, args: &[&str]) -> Option<String> {
let mut cmd = state.yt_dlp();
cmd.args(["--no-warnings", "--no-playlist", "--simulate"]);
cmd.args(args);
let out = cmd.output().await.ok()?;
@@ -545,14 +590,13 @@ pub async fn download_video(
args.push("--ffmpeg-location".into());
args.push(bin("ffmpeg"));
let mut child = tokio::process::Command::new(bin("yt-dlp"))
let mut child = state
.yt_dlp()
.args(&args)
.stdout(std::process::Stdio::piped())
.stderr(std::process::Stdio::piped())
.spawn()
.map_err(|e| {
format!("Could not start yt-dlp: {e}. Install it with: brew install yt-dlp ffmpeg")
})?;
.map_err(|e| format!("Could not start yt-dlp: {e}"))?;
let stdout = child.stdout.take().ok_or("yt-dlp produced no stdout")?;
let stderr = child.stderr.take().ok_or("yt-dlp produced no stderr")?;
@@ -810,6 +854,7 @@ pub fn build_state(app: &AppHandle) -> Result<AppState, String> {
playlists: PlaylistServer::start()?,
prereqs: Arc::new(Mutex::new(None)),
streams: Arc::new(Mutex::new(HashMap::new())),
yt_dlp: resolve_yt_dlp(app),
})
}
src-tauri/tauri.conf.json
+5 -2
View File
@@ -45,9 +45,12 @@
"icons/icon.ico"
],
"externalBin": [
"binaries/yt-dlp",
"binaries/ffmpeg",
"binaries/ffprobe"
]
],
"resources": {
"resources/python": "python",
"resources/yt-dlp.pyz": "yt-dlp.pyz"
}
}
}