Back to results for “Performance Optimization & Profiling”
Node.js › Async Patterns

How would you implement a concurrency-limited task queue in Node.js to process many async tasks without overwhelming a downstream resource?

Advanced

Answer

Running Promise.all() over thousands of items sends every request essentially simultaneously, which can overwhelm a downstream database or API and exhaust connections -- a concurrency-limited queue instead processes only N tasks at a time, starting a new one immediately as each one completes, keeping throughput high while capping the maximum simultaneous load; libraries like p-limit or a hand-rolled worker-pool pattern both implement this.
async function processWithConcurrencyLimit(items, limit, fn) {
  const results = [];
  const executing = new Set();
  for (const item of items) {
    const p = fn(item).then(r => { executing.delete(p); return r; });
    executing.add(p);
    results.push(p);
    if (executing.size >= limit) await Promise.race(executing);
  }
  return Promise.all(results);
}
Real-world example A data-migration script updating 100,000 records via an external API uses a concurrency-limited queue capped at 10 simultaneous requests, avoiding the rate-limit errors and connection exhaustion that occurred when an earlier version of the script fired all 100,000 requests via a single Promise.all().

Common follow-ups: How do you choose an appropriate concurrency limit for a given downstream dependency?;How does this pattern compare to using a proper job queue backed by Redis or RabbitMQ for the same problem at larger scale?