Jump toUpdate content

Messaging and Queuing - Concepts

Published on 04 January 2023

AMQP

Advanced Message Queuing Protocol (AMQP) is an open standard messaging protocol that allows the sending of transactional messages between servers and devices via a message broker. The advantages of AMQP are reliable communication (data is received exactly once at the receiving end) and interoperability.

Content-based

Content-based messaging systems are a subset of the publish/subscribe model, and contrast with topic-based systems in terms of the way messages are filtered/routed. In a content-based messaging system, the subscriber specifies the properties of the messages they want to receive, based on the message’s attributes or content. Message delivery is therefore selective, and messages are only delivered to a subscriber if the attributes or content match the constraints they set.

Credentials

Credentials give services and platforms access to your Messaging namespace, enabling them to connect to the host system. Credentials are protocol-specific: for SQS/SNS credentials different levels of permissions can be defined, whereas NATS credentials give full access to the messaging namespace. Refer to our Messaging and Queuing Reference Content for more information.

Dead Letter Queue

A Dead Letter Queue (DLQ), or undelivered-message queue, receives and holds messages that cannot be delivered to their destination queues. You may define an existing queue as a Dead Letter Queue.

Fanout

Fanout is a type of messaging pattern. A fanout exchange broadcasts messages to all queues/consumers it is aware of. This allows the same published message to be consumed by different consumers, who will process it in different ways.

Filtering

In a topic-based system, where topics handle the logic, filtering is similar to routing. Messages are sent to defined topics, which can be thought of as filters in so far as subscribers can subscribe only to the topics they are interested in. In a content-based system, filtering is carried out more directly by subscribers, who define filters for messages based on the content/attributes they want to receive.

Long Polling

Long polling is a technology where the client requests information from the server without expecting an immediate response. In SQS, this enables clients to wait for the system to get messages that are not immediately available.

Message broker

A message broker is a piece of software that allows applications, systems and services to communicate with each other and send/receive data. It facilitates the exchange of information by receiving messages from a producer, and transmitting them to a consumer. All communication with producers and consumers uses a protcol. There are two basic models of communication for message brokers: publish/subscribe and queuing.

Messaging and Queuing

Scaleway’s Messaging and Queuing product is a message broker tool that allows you to transfer messages between different microservices and platforms. This enables them to “talk” to each other effectively even if they are not otherwise compatible. Messaging and Queuing enables and simplifies microservices application development and allows you to build highly scalable, reliable, distributed applications.

Messaging protocol

A messaging protocol defines a structured way for users / platforms / services / applications to exchange data and messages, even if normally they do not “speak the same language”. Protocols also describe how messages should be processed, prioritized, managed and routed. Messaging protocols currently supported by Scaleway Messaging and Queuing include NATS, SQS and SNS.

Namespace

A Scaleway Messaging and Queuing namespace enables systems, applications, and services to communicate with each other and exchange information. A namespace sets a scope for your queues, topics and credentials. For each namespace, you can generate one or multiple sets of credentials, which allow the bearer to perform actions within that namespace, dependent on their access rights. Actions could include creating topics or queues, publishing messages, reading messages etc. Namespaces therefore allow you to isolate queues and topics from each other, as credentials only give access to queues and topics within that namespace. Each namespace has a type, currently either NATS or SQS/SNS which defines the messaging protocol to be used.

NATS

The Neural Autonomic Transport System, or NATS, is an open-source messaging system written in Go. It is part of the Cloud Native Computing Foundation (CNCF) and has more than 40 client language implementations. The application has been designed with performance, scalability, and ease of use in mind.

Publish/Subscribe

Also known as “pub/sub”, the publish/subscribe model provides a pattern or framework for the exchange of messages between publishers and subscribers. It contrasts with the queuing model. The key feature of publish/subscribe is that messages are not sent to defined recipients. Instead, subscribers define the types of message they are interested in, and only receive messages matching their criteria. The publisher sends the message without knowing exactly who will receive it. The process of selecting which messages to receive is called filtering, which can be topic-based or content-based. The publish/subscribe model relies on a message broker to relay messages between publishers and subscribers.

Queuing

The message queuing model provides a pattern or framework for sending messages, which contrasts with the publish/subscribe model. Queuing is a form of asynchronous service-to-service communication. Whereas with the publish/subscribe model multiple subscribers can receive each message, with the queuing model, messages have just one destination. Messages are stored in the queue until they are processed and delivered, and they are deleted once consumed. This model is used in serverless and microservices architectures.

Routing

In topic-based messaging, topics allow messages to be routed to the correct subscribers. Topics act as labels for each message, and the broker routes messages to subscribers who match the topic.

SNS

The Simple Notification Service, or SNS, is a publish/subscribe notification service for the mass delivery of messages. SNS acts as a single message bus that can be sent to a variety of devices and platforms through a single code interface. It is also possible to adapt message formats to the particular needs of each platform.

SQS

The Simple Queue Service, or SQS, is a distributed message queuing service that supports programmatic sending of messages via web service applications. The SQS protocol offers two types of message queues:

  • Standard queues offer maximum throughput, best-effort ordering, and at-least-once delivery.
  • FIFO queues are designed to guarantee that messages are processed exactly once, in the exact order that they are sent.

SQS/SNS

SNS/SQS is one of the namespace types available at Scaleway. You can use the SNS and/or SQS protocols with this type of namespace.

Stream

Distinct from traditional message brokers where messages are deleted once received/consumed, streams retain records of their events. A streaming broker is therefore often likened to a distributed append-only logs file, where every new message is added at the end of the persistent log. Each message can be delivered to one or more consumers.

Subscriber

In publish/subscribe systems such as SNS, a subscriber is the entity (e.g. SQS queue, function, URL) that messages from topics are pushed to. Subscribers can filter messages based on their topic or content.

Topic-based

Topic-based messaging systems are a subset of the publish/subscribe model, and contrast with content-based systems. In a topic-based system, messages are published to “topics” or named logical channels. Subscribers then receive all messages published to the topics to which they subscribe. The publisher is responsible for defining the possible topics which can be subscribed to.

Topic/Exchange

A topic (or ‘exchange’ in AMQP) is a named resource to which messages are sent by publishers.