Zip Thousands of Files Without Melting Anything

A ZIP feature that works beautifully for 50 files meets a customer with 19,000, and things start failing in places that never failed before. This page is the checklist of what breaks at thousands-of-files scale and the pattern that handles each failure mode server-side.

What breaks, in the order it breaks

1. The URL list itself gets heavy. A presigned S3 URL runs ~500 bytes; 10,000 of them is ~5 MB of JSON. Shipping that to a browser before any work starts is why client-side approaches degrade first. Server-side, the list goes straight from your endpoint to the job API and the browser never carries it:

const files = await listPrefixAsSignedUrls('bucket', 'big-export/'); // 10k entries
const response = await fetch('https://api.eazip.io/jobs', {
  method: 'POST',
  headers: {
    'X-API-Key': process.env.EAZIP_API_KEY,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    files,
    zip_filename: 'export.zip',
    max_zip_size_bytes: 10 * 1024 ** 3,
    fail_on_url_error: false,
  }),
});

(If the trigger must stay in browser code, a backend-created session achieves the same: the browser carries a session handle, never the list.)

2. Listing and signing stop being free. Storage listings paginate (1,000 keys per page on S3/R2), and presigning 10,000 objects is 10,000 signature computations — fast, but no longer instant. Batch where the platform allows (Supabase signs an array in one createSignedUrls call) and keep the signing loop server-side where it can stream through pages.

3. Some URLs will fail. At 10,000 objects, a deleted file, an archived-tier object, or an expired signature is a statistical certainty, not an edge case. Decide the policy up front: fail_on_url_error: false completes the archive and reports the casualties in errors; the default true fails fast so nothing ships incomplete. For big exports, false plus a review of errors is almost always the right trade.

4. Signature lifetimes meet job duration. Fetching thousands of objects takes real time, and every object must still have a valid signature when its turn comes. Sign immediately before creating the job and give expiresIn comfortable headroom — an hour, not five minutes. (And on AWS, mind role-credential expiry.)

5. The output outgrows one archive. Thousands of files often means tens of gigabytes. One 40 GB ZIP is hostile to whoever downloads it — and archives cap at 50 GB regardless. max_zip_size_bytes splits output into numbered parts, each with its own resumable download link; total output reaches 500 GB per job on the largest plan.

6. Plan ceilings. Files per job are plan-bound — 100 on the free tier up to 20,000 on the largest. Past that, shard by prefix or manifest into several jobs; per-customer or per-period sharding usually matches the product need anyway (see the report-bundling pattern).

What you never deal with

For contrast, the failure modes this path simply doesn't have: browser memory ceilings and tab lifetime (the job runs server-side), your server's bandwidth (bytes go storage → Eazip → user), response timeouts (the job outlives any HTTP request, and delivery is an expiring link with Range/resume), and ZIP64 (archives past 4 GB or 65k entries are handled). The general flow is documented in Create a ZIP from URLs with One API Call.

FAQ

How long does a 10,000-file job take?

It scales with total bytes and source latency, not file count per se — 10,000 thumbnails finish far faster than 500 RAW photos. Design for asynchrony (webhook, not a held request) and duration stops mattering.

Should I send one huge job or several smaller ones?

One job per logical export (one customer, one period) is the right boundary — it keeps failure isolation, filenames, and delivery links aligned with the product. Shard only when a logical export exceeds plan limits.

What happens to the files that failed?

With fail_on_url_error: false they are listed per-URL in the job's errors while everything else ships. Fix the sources and either re-run the job or deliver the completed archive as-is — partial delivery is usually better than blocked delivery.

Can the browser kick this off if the list lives on my server?

Yes — that is exactly what backend-created sessions are for: your endpoint builds the list, the browser receives only a session handle.