Messaging and Queuing - Concepts

A Dead Letter Queue (DLQ), or undelivered-message queue, receives and holds messages that cannot be delivered to their destination queues. You may define an existing queue as a Dead Letter Queue.

Fanout is a type of messaging pattern. A fanout exchange broadcasts messages to all queues/consumers it is aware of. This allows the same published message to be consumed by different consumers, who will process it in different ways.

In a topic-based system, where topics handle the logic, filtering is similar to routing. Messages are sent to defined topics, which can be thought of as filters in so far as subscribers can subscribe only to the topics they are interested in. In a content-based system, filtering is carried out more directly by subscribers, who define filters for messages based on the content/attributes they want to receive.

Long polling is a technology where the client requests information from the server without expecting an immediate response. In SQS, this enables clients to wait for the system to get messages that are not immediately available.

A message broker is a piece of software that allows applications, systems and services to communicate with each other and send/receive data. It facilitates the exchange of information by receiving messages from a producer, and transmitting them to a consumer. All communication with producers and consumers uses a protcol. There are two basic models of communication for message brokers: publish/subscribe and queuing.

Scaleway’s Messaging and Queuing product is a message broker tool that allows you to transfer messages between different microservices and platforms. This enables them to “talk” to each other effectively even if they are not otherwise compatible. Messaging and Queuing enables and simplifies microservices application development and allows you to build highly scalable, reliable, distributed applications.

A messaging protocol defines a structured way for users / platforms / services / applications to exchange data and messages, even if normally they do not “speak the same language”. Protocols also describe how messages should be processed, prioritized, managed and routed. Messaging protocols currently supported by Scaleway Messaging and Queuing include NATS, SQS and SNS.

A Scaleway Messaging and Queuing namespace enables systems, applications, and services to communicate with each other and exchange information. A namespace sets a scope for your queues, topics and credentials. For each namespace, you can generate one or multiple sets of credentials, which allow the bearer to perform actions within that namespace, dependent on their access rights. Actions could include creating topics or queues, publishing messages, reading messages etc. Namespaces therefore allow you to isolate queues and topics from each other, as credentials only give access to queues and topics within that namespace. Each namespace has a type, currently either NATS or SQS/SNS which defines the messaging protocol to be used.

The Neural Autonomic Transport System, or NATS, is an open-source messaging system written in Go. It is part of the Cloud Native Computing Foundation (CNCF) and has more than 40 client language implementations. The application has been designed with performance, scalability, and ease of use in mind.

Also known as “pub/sub”, the publish/subscribe model provides a pattern or framework for the exchange of messages between publishers and subscribers. It contrasts with the queuing model. The key feature of publish/subscribe is that messages are not sent to defined recipients. Instead, subscribers define the types of message they are interested in, and only receive messages matching their criteria. The publisher sends the message without knowing exactly who will receive it. The process of selecting which messages to receive is called filtering, which can be topic-based or content-based. The publish/subscribe model relies on a message broker to relay messages between publishers and subscribers.

The message queuing model provides a pattern or framework for sending messages, which contrasts with the publish/subscribe model. Queuing is a form of asynchronous service-to-service communication. Whereas with the publish/subscribe model multiple subscribers can receive each message, with the queuing model, messages have just one destination. Messages are stored in the queue until they are processed and delivered, and they are deleted once consumed. This model is used in serverless and microservices architectures.

In topic-based messaging, topics allow messages to be routed to the correct subscribers. Topics act as labels for each message, and the broker routes messages to subscribers who match the topic.

The Simple Notification Service, or SNS, is a publish/subscribe notification service for the mass delivery of messages. SNS acts as a single message bus that can be sent to a variety of devices and platforms through a single code interface. It is also possible to adapt message formats to the particular needs of each platform.

The Simple Queue Service, or SQS, is a distributed message queuing service that supports programmatic sending of messages via web service applications. The SQS protocol offers two types of message queues:

• Standard queues offer maximum throughput, best-effort ordering, and at-least-once delivery.
• FIFO queues are designed to guarantee that messages are processed exactly once, in the exact order that they are sent.

SNS/SQS is one of the namespace types available at Scaleway. You can use the SNS and/or SQS protocols with this type of namespace.

Distinct from traditional message brokers where messages are deleted once received/consumed, streams retain records of their events. A streaming broker is therefore often likened to a distributed append-only logs file, where every new message is added at the end of the persistent log. Each message can be delivered to one or more consumers.

In publish/subscribe systems such as SNS, a subscriber is the entity (e.g. SQS queue, function, URL) that messages from topics are pushed to. Subscribers can filter messages based on their topic or content.

Topic-based messaging systems are a subset of the publish/subscribe model, and contrast with content-based systems. In a topic-based system, messages are published to “topics” or named logical channels. Subscribers then receive all messages published to the topics to which they subscribe. The publisher is responsible for defining the possible topics which can be subscribed to.

A topic (or ‘exchange’ in AMQP) is a named resource to which messages are sent by publishers.