Promise queue with concurrency control
p-queue is a Promise-based task queue library that provides fine-grained control over asynchronous operation execution. With over 16 million weekly downloads, it's a battle-tested solution for preventing resource exhaustion by limiting how many async operations run concurrently. Instead of firing hundreds of API requests or processing tasks simultaneously, p-queue lets you specify exactly how many should execute at once.
The library emerged from a common problem in JavaScript development: managing async operations at scale without overwhelming system resources or hitting rate limits. When you need to process thousands of files, make hundreds of API calls, or handle user-generated tasks, native Promise.all() executes everything simultaneously. p-queue gives you the throttle control that Promise.all() lacks.
Developers use p-queue in production systems ranging from web scrapers and data processing pipelines to API clients and build tools. It works across Node.js 14+ and modern browsers, making it suitable for both server-side and client-side applications. The library supports priority-based task scheduling, dynamic pause/resume functionality, and sliding-window rate limiting to handle complex real-world scenarios.
import PQueue from 'p-queue';
const queue = new PQueue({
concurrency: 3,
interval: 1000,
intervalCap: 5
});
const fetchUser = async (userId) => {
const response = await fetch(`https://api.example.com/users/${userId}`);
return response.json();
};
const userIds = Array.from({ length: 50 }, (_, i) => i + 1);
const results = await Promise.all(
userIds.map(id =>
queue.add(() => fetchUser(id), { priority: id < 10 ? 1 : 0 })
)
);
console.log(`Processed ${results.length} users`);
console.log(`Queue size: ${queue.size}, Pending: ${queue.pending}`);
queue.on('active', () => {
console.log(`Working on task. Remaining: ${queue.size + queue.pending}`);
});
await queue.onIdle();
console.log('All tasks completed');API rate limiting compliance: When integrating with third-party APIs that enforce rate limits (e.g., 10 requests per second), p-queue ensures you never exceed these limits by setting concurrency and interval constraints. This prevents 429 errors and potential API key bans.
Batch image/video processing: Processing large media files consumes significant memory and CPU. By limiting concurrency to 2-3 operations, you can process thousands of files without crashing the system or degrading performance for other operations.
Database migration scripts: When migrating millions of records between databases, executing all operations simultaneously would overwhelm connections and memory. p-queue allows controlled batching with monitoring of completion status.
Web scraping with politeness: Responsible web scraping requires limiting concurrent requests to avoid overwhelming target servers. Combine concurrency limits with interval-based rate limiting to implement polite crawling behavior.
Background job processing: In applications with user-generated tasks (report generation, email sending, file conversions), p-queue manages task execution order based on priority while preventing resource starvation.
npm install p-queuepnpm add p-queuebun add p-queue