Jump toUpdate content

Messaging and Queuing - Concepts

Reviewed on 08 June 2023Published on 04 January 2023

AMQP

Advanced Message Queuing Protocol (AMQP) is an open standard messaging protocol that allows the sending of transactional messages between servers and devices via a message broker. The advantages of AMQP are reliable communication and interoperability.

Content-based

Content-based messaging systems are a subset of the publish/subscribe model, and contrast with topic-based systems in terms of the way messages are filtered/routed. In a content-based messaging system, the subscriber specifies the properties of the messages they want to receive, based on the message’s attributes or content. Message delivery is therefore selective, and messages are only delivered to a subscriber if the attributes or content match the constraints they set.

Content based deduplication

Content based deduplication is a setting available for FIFO queues. Enable content based deduplication if the message body is guaranteed to be unique for each message. A unique hash value is generated from the body of each message, which is used as its deduplication ID. This avoids the need to set a deduplication ID when sending messages. Read more in our dedicated documentation on creating queues.

Credentials

Credentials give services and platforms access to your Messaging namespace, enabling them to connect to the host system. Credentials are protocol-specific: for SQS/SNS credentials different levels of permissions can be defined, whereas NATS credentials give full access to the messaging namespace. Refer to our Messaging and Queuing Reference Content for more information.

Dead Letter Queue

A Dead Letter Queue (DLQ), or undelivered-message queue, receives and holds messages that cannot be delivered to their destination queues. You may define an existing queue as a Dead Letter Queue.

Fanout

Fanout is a type of messaging pattern. A fanout exchange broadcasts messages to all queues/consumers it is aware of. This allows the same published message to be consumed by different consumers, who will process it in different ways. Each message is processed in the order in which it arrives

FIFO queue

FIFO stands for First In First Out, and represents a type of queue where the exact order of messages is preserved, and duplicate messages are not tolerated. As well as these specificities, FIFO queues support all the same features as Standard queues. Consider using FIFO queues for any use cases where the order of messages is critical, such as e-commerce order management systems, systems where one action should not happen until another has been completed, or first-come-first-served ticketing systems.

Filtering

In a topic-based system, where topics handle the logic, filtering is similar to routing. Messages are sent to defined topics, which can be thought of as filters in so far as subscribers can subscribe only to the topics they are interested in. In a content-based system, filtering is carried out more directly by subscribers, who define filters for messages based on the content/attributes they want to receive.

Long Polling

Long polling is a technology where the client requests information from the server without expecting an immediate response. In SQS, this enables clients to wait for the system to get messages that are not immediately available.

Message broker

A message broker is a piece of software that allows applications, systems and services to communicate with each other and send/receive data. It facilitates the exchange of information by receiving messages from a producer, and transmitting them to a consumer. All communication with producers and consumers uses a protocol. There are two basic models of communication for message brokers: publish/subscribe and queuing.

Message retention period

The message retention period is a setting that can be configured for an SQS queue. It represents the length of time (in seconds) that messages are kept in the queue before being deleted. Setting a longer message retention period allows for a longer interval between a message being sent and it being received. Read more in our dedicated documentation on creating queues.

Messaging and Queuing

Scaleway’s Messaging and Queuing product is a message broker tool that allows you to transfer messages between different microservices and platforms. This enables them to “talk” to each other effectively even if they are not otherwise compatible. Messaging and Queuing enables and simplifies microservices application development and allows you to build highly scalable, reliable, distributed applications.

Messaging protocol

A messaging protocol defines a structured way for users / platforms / services / applications to exchange data and messages, even if normally they do not “speak the same language”. Protocols also describe how messages should be processed, prioritized, managed and routed. Messaging protocols currently supported by Scaleway Messaging and Queuing include NATS, SQS and SNS.

Namespace

A Scaleway Messaging and Queuing namespace enables systems, applications, and services to communicate with each other and exchange information. A namespace sets a scope for your queues, topics, and credentials. For each namespace, you can generate one or multiple sets of credentials, which allow the bearer to perform actions within that namespace, dependent on their access rights. Actions could include creating topics or queues, publishing messages, reading messages etc. Namespaces therefore allow you to isolate queues and topics from each other, as credentials only give access to queues and topics within that namespace. Each namespace has a type, currently either NATS or SQS/SNS which defines the messaging protocol to be used.

NATS

The Neural Autonomic Transport System, or NATS, is an open-source messaging system written in Go. It is part of the Cloud Native Computing Foundation (CNCF) and has more than 40 client language implementations. The application has been designed with performance, scalability, and ease of use in mind.

Publish/Subscribe

Also known as “pub/sub”, the publish/subscribe model provides a pattern or framework for the exchange of messages between publishers and subscribers. It contrasts with the queuing model. The key feature of publish/subscribe is that messages are not sent to defined recipients. Instead, subscribers define the types of message they are interested in, and only receive messages matching their criteria. The publisher sends the message without knowing exactly who will receive it. The process of selecting which messages to receive is called filtering, which can be topic-based or content-based. The publish/subscribe model relies on a message broker to relay messages between publishers and subscribers.

Queue

Queues facilitate asynchronous communication between different microservices, applications and platforms. You can create a queue, configure its delivery and message parameters, and then start sending messages to it. Messages are stored in the queue until they are processed and delivered, and deleted once consumed.

Queue types

When creating queues in a Scaleway Messaging and Queuing SQS/SNS namespace, two queue types are available. Standard queues provide at-least-once delivery, while FIFO queues offer first-in-first-out delivery, and (unlike Standard queues) guarantee that messages are delivered in order and without duplication. Find out more about creating queues with our dedicated documentation.

Queuing

The message queuing model provides a pattern or framework for sending messages, which contrasts with the publish/subscribe model. Queuing is a form of asynchronous service-to-service communication. Whereas with the publish/subscribe model multiple subscribers can receive each message, with the queuing model, messages have just one destination. Messages are stored in the queue until they are processed and delivered, and they are deleted once consumed. This model is used in serverless and microservices architectures.

Routing

In topic-based messaging, topics allow messages to be routed to the correct subscribers. Topics act as labels for each message, and the broker routes messages to subscribers who match the topic.

SNS

The Simple Notification Service, or SNS, is a publish/subscribe notification service for the mass delivery of messages. SNS acts as a single message bus that can be sent to a variety of devices and platforms through a single code interface. It is also possible to adapt message formats to the particular needs of each platform.

SQS

The Simple Queue Service, or SQS, is a distributed message queuing service that supports programmatic sending of messages via web service applications. The SQS protocol offers two types of message queues:

  • Standard queues offer maximum throughput, best-effort ordering, and at-least-once delivery.
  • FIFO queues are designed to guarantee that messages are processed exactly once, in the exact order that they are sent.

SQS/SNS

SQS/SNS is one of the namespace types available at Scaleway. You can use the SNS and/or SQS protocols with this type of namespace.

Standard queue

Standard queues represent the default queue type, and offer an at-least-once message delivery system. Unlike FIFO queues, Standard queues provide only best-effort attempts to deliver messages in order. At-least-once delivery means that it is possible under rare circumstances that the same message may be received more than once.

Stream

Distinct from traditional message brokers where messages are deleted once received/consumed, streams retain records of their events. A streaming broker is therefore often likened to a distributed append-only logs file, where every new message is added at the end of the persistent log. Each message can be delivered to one or more consumers.

Subscriber

In publish/subscribe systems such as SNS, a subscriber is the entity (e.g. SQS queue, function, URL) that messages from topics are pushed to. Subscribers can filter messages based on their topic or content.

Topic-based

Topic-based messaging systems are a subset of the publish/subscribe model, and contrast with content-based systems. In a topic-based system, messages are published to “topics” or named logical channels. Subscribers then receive all messages published to the topics to which they subscribe. The publisher is responsible for defining the possible topics which can be subscribed to.

Topic/Exchange

A topic (or ‘exchange’ in AMQP) is a named resource to which messages are sent by publishers.

Visibility timeout

Visibility timeout is a setting that can be configured for an SQS queue. It represents the length of time (in seconds) during which, after a message is received, the queue hides it, so it cannot be received again by the same or other consumers. This is useful as the queue itself does not automatically delete messages once they are received, and so prevents consumers from receiving the same message many times before they have finished processing it. Read more in our dedicated documentation on creating queues.