NATS, Queues, and Topics and Events - Concepts

Content-based

Content-based messaging systems are a subset of the publish/subscribe model, and contrast with topic-based systems in terms of the way messages are filtered/routed. In a content-based messaging system, the subscriber specifies the properties of the messages they want to receive, based on the message’s attributes or content. Message delivery is therefore selective, and messages are only delivered to a subscriber if the attributes or content match the constraints they set.

Content-based deduplication

Content-based deduplication is a setting available for FIFO queues. Enable content-based deduplication if the message body is guaranteed to be unique for each message. A unique hash value is generated from the body of each message, which is used as its deduplication ID. This avoids the need to set a deduplication ID when sending messages. Read more in our dedicated documentation on creating queues.

Credentials

Credentials give services and platforms access to Scaleway NATS, Queues, and Topics and Events, enabling them to connect to the host system. Credentials are product-specific: for Queues and Topics ad Events, different levels of permissions can be defined to write, read, or manage queues/topics. NATS credentials give full read-write-manage access. Refer to our additional content for more information.

Dead Letter Queue

A Dead Letter Queue (DLQ), or undelivered-message queue, receives and holds messages that cannot be delivered to their destination queues. This concept is not yet implemented for Scaleway Queues.

Fanout

Fanout is a type of messaging pattern. A fanout exchange broadcasts messages to all queues/consumers it is aware of. This allows the same published message to be consumed by different consumers, who will process it in different ways. Each message is processed in the order in which it arrives.

FIFO

FIFO stands for First In First Out, and represents a type of queue or topic where the exact order of messages is preserved, and duplicate messages are not tolerated. As well as these specificities, FIFO queues and topics support all the same features as the Standard type. Consider using FIFO queues and topics for any use cases where the order of messages is critical, such as e-commerce order management systems, systems where one action should not happen until another has been completed, or first-come-first-served ticketing systems.

Filtering

In a topic-based system, where topics handle the logic, filtering is similar to routing. Messages are sent to defined topics, which can be thought of as filters in so far as subscribers can subscribe only to the topics they are interested in. In a content-based system, filtering is carried out more directly by subscribers, who define filters for messages based on the content/attributes they want to receive.

Long Polling

Long polling is a technology where the client requests information from the server without expecting an immediate response. For Queues, this enables clients to wait for the system to get messages that are not immediately available.

Message broker

A message broker is a piece of software that allows applications, systems and services to communicate with each other and send/receive data. It facilitates the exchange of information by receiving messages from a producer, and transmitting them to a consumer. All communication with producers and consumers uses a protocol. There are two basic models of communication for message brokers: publish/subscribe and queuing.

Message retention period

The message retention period is a setting that can be configured for a queue. It represents the length of time (in seconds) that messages are kept in the queue before being deleted. Setting a longer message retention period allows for a longer interval between a message being sent and it being received. Read more in our dedicated documentation on creating queues.

Messaging and Queuing

Previously, Scaleway Messaging and Queuing was a single product that grouped together three different messaging protocols. It has now been split into three distinct products: NATS, Queues, and Topics and Events.

Messaging protocol

A messaging protocol defines a structured way for users / platforms / services / applications to exchange data and messages, even if normally they do not “speak the same language”. Protocols also describe how messages should be processed, prioritized, managed and routed. Scaleway NATS is based on the NATS protocol, Queues on the SQS protocol, and Topics and Events on the SNS protocol.

NATS

The Neural Autonomic Transport System, or NATS, is an open-source messaging system written in Go. It is part of the Cloud Native Computing Foundation (CNCF) and has more than 40 client language implementations. The application has been designed with performance, scalability, and ease of use in mind.

Check our our NATS quickstart to get started with Scaleway NATS, or our tutorial on creating a serverless architecture to process large messages with NATS, to get an idea of how to go further.

NATS account

A NATS account sets a scope for any NATS credentials, messages, queues and streams held within it. You can create one or multiple NATS accounts with Scaleway NATS.

Protocol

See messaging protocol.

Publish/Subscribe

Also known as “pub/sub”, the publish/subscribe model provides a pattern or framework for the exchange of messages between publishers and subscribers. It contrasts with the queuing model. The key feature of publish/subscribe is that messages are not sent to defined recipients. Instead, subscribers define the types of message they are interested in, and only receive messages matching their criteria. The publisher sends the message without knowing exactly who will receive it. The process of selecting which messages to receive is called filtering, which can be topic-based or content-based. The publish/subscribe model relies on a message broker to relay messages between publishers and subscribers.

Queue

Creating a queue with Scaleway Queues facilitates asynchronous communication between different microservices, applications, and platforms. You can create a queue, configure its delivery and message parameters, and then start sending messages to it. Messages are stored in the queue until they are processed and delivered, and deleted once consumed. Read more about creating and configuring queues, or check our tutorial on creating a serverless scraping architecture using a queue to get an idea of what you can do with message queues.

Queues

Scaleway Queues is a product for creating managed messaging queues based on the SQS protocol. Previously, it was part of the Messaging and Queuing product.

Queue types

When creating queues with Scaleway Queues, two queue types are available. Standard queues provide at-least-once delivery, while FIFO queues offer first-in-first-out delivery, and (unlike Standard queues) guarantee that messages are delivered in order and without duplication. Content-based deduplication is only available for FIFO queue types. Find out more about creating queues with our dedicated documentation.

Queuing

The message queuing model provides a pattern or framework for sending messages, which contrasts with the publish/subscribe model. Queuing is a form of asynchronous service-to-service communication. Whereas with the publish/subscribe model multiple subscribers can receive each message, with the queuing model, messages have just one destination. Messages are stored in the queue until they are processed and delivered, and they are deleted once consumed. This model is used in serverless and microservices architectures.

Queue volume

Queue volume is one of the factors affecting the billing of Scaleway Queues. Queue volume is calculated as the number of messages in a queue, multiplied by the message size. Or, the sum of the size of all messages in a queue.

Region

NATS, Queues, and Topics and Events are available in multiple regions. A region designates the geographical area where the service is hosted. Refer to the product availability table to check which regions are available for NATS, Queues, and Topics and Events.

When creating a NATS account or creating queues or topics, you need to do this on a region-by-region basis. The region drop-down in the console allows you to switch between available regions.

Routing

In topic-based messaging, topics allow messages to be routed to the correct subscribers. Topics act as labels for each message, and the broker routes messages to subscribers who match the topic.

SNS

The Scaleway Topics and Events product is based on the SNS protocol. Simple Notification Service, or SNS, is a publish/subscribe notification service for the mass delivery of messages. SNS acts as a single message bus that can be sent to a variety of devices and platforms through a single code interface. It is also possible to adapt message formats to the particular needs of each platform.

SQS

The Scaleway Queues product is based on the SNS protocol. Simple Queue Service, or SQS, is a distributed message queuing service that supports programmatic sending of messages via web service applications.

Standard

Standard-type queues and topics represent the default queue/topic type, and offer an at-least-once message delivery system. Unlike FIFO queues and topics, standard queues provide only best-effort attempts to deliver messages in order. At-least-once delivery means that it is possible under rare circumstances that the same message may be received more than once.

Stream

Distinct from traditional message brokers where messages are deleted once received/consumed, streams retain records of their events. A streaming broker is therefore often likened to a distributed append-only logs file, where every new message is added at the end of the persistent log. Each message can be delivered to one or more consumers.

Stream volume

Stream volume is one of the factors affecting the billing of Scaleway NATS. Stream volume is calculated as the number of messages in a stream, multiplied by the message size. Or, the sum of the size of all messages in a stream.

Stream persistence

Stream persistence is one of the factors affecting the billing of Scaleway NATS. Stream persistence is calculated as the total amount stored in a stream, multiplied by the duration it is stored for.

Subscriber

In publish/subscribe systems such as Topics and Events, a subscriber is the entity (e.g. a queue, function, or URL) that messages from topics are pushed to. Subscribers can filter messages based on their topic or content.

Subscription

A subscription is a connection between a client or endpoint, and a topic. By creating a subscription, the subscribed endpoint receives messages and notifications published to the topic. You can create subscriptions for HTTP/S endpoints, Scaleway queues, and Scaleway Serverless Functions and Containers.

Subscription protocols and endpoints

A subscription protocol refers to the communication method used to deliver messages to a subscriber. Different types of subscriber require different protocols. When you create a new subscription to a topic with Scaleway Topics and Events, the following options are available:

Topic

A topic is a communication channel used to send messages and notifications to subscribed endpoints or clients. Publishers send messages to topics, and those messages are received by subscribers. Subscribers can include Serverless Functions, Scaleway queues and HTTP/HTTPS endpoints. As such, topics decouple the publishing and the receiving of messages, allowing for flexibility and scalabilty in building loosely-coupled systems.

Topic types

When creating topics with Scaleway Topics and Events, two topic types are available. Standard topics provide at-least-once delivery, while FIFO topics offer first-in-first-out delivery, and (unlike Standard topics) guarantee that messages are delivered in order and without duplication. Content-based deduplication is only available for FIFO topic types. Find out more about creating topics with our dedicated documentation.

Topic-based

Topic-based messaging systems are a subset of the publish/subscribe model, and contrast with content-based systems. In a topic-based system, messages are published to “topics” or named logical channels. See topic for more information.

Topic volume

Topic volume is one of the factors affecting the billing of Scaleway Topics and Events. Topic volume is calculated as the total sum of the sizes of all messages sent from a topic to its subscriptions.

Topics and Events

Scaleway Topics and Events is a product for creating managed messaging topics based on the SNS protocol. Previously, it was part of the Messaging and Queuing product.

Visibility timeout

Visibility timeout is a setting that can be configured for a Scaleway queue. It represents the length of time (in seconds) during which, after a message is received, the queue hides it, so it cannot be received again by the same or other consumers. This is useful as the queue itself does not automatically delete messages once they are received, and so prevents consumers from receiving the same message many times before they have finished processing it. Read more in our dedicated documentation on creating queues.