JSPM

RedisSMQ is a Node.js library for queuing messages (aka jobs) and processing them asynchronously with consumers. Backed by Redis, it allows scaling up your application with ease of use.

Features

• High-performance message processing
• Multi-Queue Producers & Consumers: Offering very flexible models which make RedisSMQ an ideal message broker for your microservices.
• Scalable: You can run multiple Consumer/Producer instances concurrently in the same host, or in different hosts.
• Supporting both at-least-once/at-most-once delivery: In case of failures, while delivering or processing a message, RedisSMQ can guaranty that the message will be not lost and redelivered again. When configured to do so, RedisSMQ can ensure that the message is delivered at-most-once.
• Message expiration: A message will not be delivered if it has been in a queue for longer than a given amount of time, called TTL (time-to-live).
• Message consume timeout: Timeout for consuming messages.
• Delaying and scheduling message delivery: Messages can be configured to be delayed, delivered for N times with an optional period between deliveries, and to be scheduled using CRON expressions.
• Reliable Priority Queues: Supports priority messaging.
• HTTP API: an HTTP interface is provided to interact with the MQ.
• Web UI: RedisSMQ can be managed also from your web browser.
• Logging: Comes with a built-in JSON logger. But you can also use your own logger instance.
• Highly optimized: Strongly-typed and implemented using pure callbacks, with small memory footprint and no memory leaks. See callbacks vs promises vs async/await benchmarks.
• Configurable: Many options and features can be configured.
• Rigorously tested: With 100+ tests and code coverage no less than 80%.
• Supports both redis & ioredis: RedisSMQ can be configured to use either redis or ioredis to connect to Redis server.

RedisSMQ Use Case: Multi-Queue Producers & Consumers

Table of Content

1. What's new?
2. Installation
3. Configuration
4. Usage
   1. Basics
      1. Message Class
      2. Producer Class
      3. Consumer Class
   2. Advanced Topics
      1. Scheduling Messages
      2. Priority Queues
      3. Message Manager
      4. Queue Manager
      5. HTTP API
      6. Web UI
      7. Logs
5. RedisSMQ Architecture
6. Performance
7. Contributing
8. License

What's new?

2022.01.03

• Release v6 is almost ready. This release includes new features such as multi-queue consumers and multi-queue producers, message rate time series, complete integration with the Web UI, as well as many improvements and bug fixes. If you are upgrading your installation, take a look at the migration guide before proceeding.

2021.11.22

• Starting with RedisSMQ v5, you can now manage your queues and messages from the Web UI. Also, many changes and improvements have been made, allowing for better user experience and system stability.

See CHANGELOG for more details.

Installation

npm install redis-smq --save

Considerations:

• RedisSMQ is targeted to be used in production environments. Therefore, only active LTS and maintenance LTS Node.js releases (v12, v14, and v16) are supported. The latest stable Node.js version is recommended.
• Minimal Redis server version is 2.6.12. The latest stable Redis version is recommended.

Configuration

See Configuration for more details.

Usage

Basics

RedisSMQ provides 3 classes: Message, Producer, and Consumer in order to work with the message queue.

Message Class

Message class is responsible for creating and manipulating messages.

const { Message } = require('redis-smq');
const message = new Message();
message
    .setBody({hello: 'world'})
    .setTTL(3600000) // in millis
    .setQueue('test_queue');

let messageTTL = message.getTTL();

See Message Reference for more details.

Producer Class

Producer class is in turn responsible for publishing messages.

You can use the same producer instance for publishing messages to multiple queues. The same producer instance can also produce messages with priority.

// filename: ./examples/javascript/producer.js

'use strict';
const {Message, Producer} = require('redis-smq');

const message = new Message();

message
    .setBody({hello: 'world'})
    .setTTL(3600000)
    .setQueue('test_queue');

message.getId() // null

const producer = new Producer();
producer.produce(message, (err) => {
    if (err) console.log(err);
    else {
      const msgId = message.getId(); // string
      console.log('Successfully produced. Message ID is ', msgId);
    }
});

See Producer Reference for more details.

Consumer Class

In the same manner as a producer, you can use a single consumer instance to consume messages from different queues, including messages from priority queues.

To consume messages from a given queue, you need to define and register a message handler.

A message handler is a function which get called once a message is received. For a given consumer, a queue can have only 2 message handlers. One for consuming messages without priority, and the second one for consuming messages with priority.

To register a message handler, the consume() method is provided and can be used as shown in the example bellow. Message handlers can be registered at any time, before or after you have started your consumers. A consumer can be started using the consumer.run() method.

To shut down and remove a message handler from your consumer, use the cancel() method.

To shut down completely your consumer, use the shutdown() method.

// filename: ./examples/javascript/consumer.js
'use strict';

const { Consumer } = require('redis-smq');

const consumer = new Consumer();

consumer.consume('test_queue', false, (msg, cb) => {
  const payload = msg.getBody();
  console.log('Message payload', payload);
  cb(); // acknowledging the message
});

consumer.consume('another_queue', true, (msg, cb) => {
   const payload = msg.getBody();
   console.log('Message payload', payload);
   cb(); // acknowledging the message
});

consumer.run();

From your message handler, when you receive a message, in order to acknowledge it, you can invoke the callback function, without arguments as shown the example above.

Message acknowledgment informs the MQ that a given message has been successfully consumed.

If an error occurred while processing a message, you can unacknowledge it by passing the error to the callback function.

By default, unacknowledged messages are re-queued and delivered again unless message retry threshold is exceeded. Then the messages are moved to dead-letter queue (DLQ).

A dead-letter queue is a system generated queue that holds all messages that couldn't be processed or can not be delivered to consumers.

By default, RedisSMQ does not store acknowledged and dead-lettered messages for saving disk and memory space, and also to increase message processing performance. If you need such feature, you can enable it from your configuration object.

See Consumer Reference for more details.

Advanced Topics

• Scheduling Messages

• Priority Queues

• Message Manager

• Queue Manager

• HTTP API

• Web UI

• Logs

RedisSMQ Architecture

• See Architecture Overview.

Performance

See Performance for more details.

Contributing

So you are interested in contributing to this project? Please see CONTRIBUTING.md.

License

MIT