Projects

ProjectProduct Engineering · 2024

Yoink

A media extraction platform with a browser extension, a web dashboard, Lambda streaming downloads, and one backend for auth, billing and analytics.

  • WXT
  • Next.js
  • NestJS
  • Go
  • AWS Lambda
  • PostgreSQL
  • Stripe
An extension and a dashboard reach one backend that owns every write, while Go on Lambda streams the download to the browserExtensionsends X-API-KeyDashboardproxies, holds no secretNestJS backendthe only writerauthorizelimitsticketsbillingLambda · Goyt-dlp · ffmpegPostgreSQLidentity · access · keysusage · download historysubscriptions · couponsBrowserno staged fileAPI KEYPROXYONE WRITERTICKETRESULTSTREAMSA SIGNED TICKET KEEPS THE KEY OUT OF THE URLTHREE AUTH MODES, ONE SHARED SECRET

Product Engineering · Project

Chapter
Web products
Year
Availability
Private

Yoink extracts and downloads video from YouTube, LinkedIn, X, Instagram, Threads, Pinterest and Facebook. It is four services rather than one application. A browser extension handles the in-page action and a Next.js dashboard handles the account. A NestJS backend owns every persistent decision, and Go on Lambda does the extraction itself.

The split is not about scale. It is about which parts are allowed to fail. Extraction breaks whenever a platform quietly changes something. Billing must never be wrong. Putting both in one process means the first one takes out the second.

# extension: user hits Download on a YouTube pagePOST /api/auth/download-ticket   signed, short ttlGET  /download?downloadTicket=…   (lambda)lambda -> backend /api/internal/authorize   plan okdownload_history: startedyt-dlp + ffmpeg, streaming 1080p to the browserdownload_history: completed, 42.1 MB
Yoink in use

Why four services

Most download tooling keeps extraction, authentication, billing and user state in a single runtime, and that is the thing that breaks. The four surfaces here are a WXT extension, a Next.js dashboard, a NestJS backend and Go handlers behind a Lambda Function URL. Only the Lambda handlers run yt-dlp and ffmpeg, and they stream the response back instead of assembling a file first. The backend shells out to nothing. A yt-dlp change can break extraction now without going anywhere near a subscription record.

Three trust boundaries

There are three authentication modes, each with exactly one job. The extension sends X-API-Key, because it runs inside a browser I do not control. The dashboard never holds a backend secret. It proxies through /api/backend/[...path], which injects Bearer BACKEND_SERVICE_TOKEN and X-User-* context headers, blocks internal paths for non-admins, and streams responses back including download payloads. Lambda presents that same service token against internal endpoints only. It holds no database credentials at all. It authorizes through the backend, logs through the backend, and writes nothing itself.

A download in flight

An extension download begins with the options call. The extension hits Lambda /options, Lambda calls backend /api/internal/authorize, and the backend checks plan, rate and monthly limits before updating key activity. Usage is logged asynchronously, so the format list is not waiting on a write. The download itself needs a plain URL, because that is what a browser download manager understands. A plain URL would put the API key in a query string. So the extension mints a short-lived signed ticket at POST /api/auth/download-ticket, the browser calls Lambda with the ticket, and Lambda exchanges it through the backend before running anything. The backend opens a download_history row, Lambda streams the bytes, and completion or failure is posted back afterwards.

Web Downloader Interface

One writer

PostgreSQL holds identity and access in users, api_keys and device_bindings. Telemetry lives in usage_logs, download_history and extension_installations. The billing lifecycle sits in user_subscriptions, billing_coupons and billing_coupon_redemptions. The backend is the only writer, including for the events Lambda starts. Consistency stays in one place and the admin analytics queries stay predictable. Extension installs and versions report into the same store, so I can see which client build is actually in the field when a platform changes underneath it.

Costs of the split

  • Four services means four deployments and one shared secret. BACKEND_SERVICE_TOKEN has to match in three places at once, so rotating it is a coordinated change rather than a config edit.
  • Routing every Lambda write through the backend removes a class of consistency bug. It also adds a network hop inside each download, and a backend outage stops extraction that would otherwise have worked.
  • Signed tickets keep the long-lived key out of URLs, and they expire. Ticket expiry is what I would revisit first, because the TTL is tuned for mint-then-download and not for a tab left open over lunch.
  • Streaming from Lambda gives the native download experience with no intermediate storage. A failure halfway through therefore arrives as a truncated file rather than as a retry.

Running it

Local ports are fixed: extension on 3000, web on 3001, backend on 3002, Lambda on 3003. Two secrets have to line up, BACKEND_SERVICE_TOKEN and DOWNLOAD_TICKET_SECRET, and Neon gets a pooled URL at runtime with an unpooled one for migrations. The runbook covers stale key deactivation, ticket expiry, monthly limit violations and stuck download_history rows. Yoink is in active development. The source is not public, so the hosted product is the only way to use it.

Next projectProductivity

Perch

View project