Understanding Messaging and Queuing GA and migration
Overview
A new version of Messaging and Queuing has been released. In the new version, the concept of namespaces has been removed, to streamline your workflows, offer you a better developer experience, and simplify the management of your Messaging and Queuing resources.
The release of this new version in the Scaleway console coincides with the NATS and SQS protocols leaving Public Beta and going into General Availability (GA). This means that the Messaging and Queuing product, when used with these protocols, is now billable. SNS will remain in Public Beta for now while we continue testing and improving the product as you use it.
Your NATS resources have been automatically migrated to the new version of the product. However, for SQS and SNS you will need to migrate, by recreating your resources in the new version. You can do this via the API or the console.
Migrating to the new version is very important, as on the 1st of December 2023, the old product version will be retired. All resources still present in the old version will be deleted.
For all protocols, the endpoints of your brokers has changed in the new version, and you will need to update these endpoints in any code, applications or architectures using your message brokers.
Read on to find out more about the new no-namespace version of the product, and how migration works for the protocols you use.
The previous situation: v1alpha1 and namespaces
Whether you create and manage your NATS or SQS/SNS namespaces and credentials via our API or our console, they both pointed towards the same underlying version of the product: v1alpha1, which held your resources.
In this previous version of the product, you created namespaces, setting the protocol for each namespace as either NATS or SQS/SNS. Each namespace gave you a URL (or endpoint) representing a Messaging and Queuing broker, and also allowed you to create credentials to access that particular namespace. When you created messages, queues, streams, topics and other resources, they were scoped to that namespace.
What’s changed: v1beta1 and no more namespaces
v1alpha1 is now deprecated, and v1beta1 is in production, representing the new and improved Messaging and Queuing product.
In this new version of the product, the concept of namespaces has been removed:
- For NATS namespaces have been replaced by accounts. You can create one or multiple NATS accounts, and credentials for each account. Each account will have its own URL/endpoint.
- SQS and SNS have been separated into two distinct protocols. Instead of creating a namespace, you simply activate each protocol. You can then create credentials for each protocol. If you previously used multiple SQS or SNS namespaces to separate resources, you should now use Scaleway Projects to separate your resources.
From Public Beta to General Availability: impact on billing
The change to the new “no-namespace” version of Messaging and Queuing coincides with the NATS and SQS protocols moving into General Availability. This means that the beta testing phase is over, and you are now billed for your use of Messaging and Queuing with NATS and SQS protocols.
Billing is based on:
- For NATS: Stream volume (the number of messages sent * the message size) and stream persistence (the total amount of data stored * duration)
- For SQS: Queue volume (the number of messages sent * the message size)
Full details of price points and how the billing model works are displayed in the console when you create a NATS account or an SQS queue, or can be viewed on our website.
The precise start of the billing period for NATS and SQS is defined as the time of General Availability, not as the time of the migration to the new v1beta1 “no-namespace” version of the product. This means that messages sent using both the old and new versions are billed from GA onwards.
Note that SNS has not yet moved into General Availability, but will instead stay in Public Beta while we continue to work on testing and improving this product. This means that you will not yet be billed for using Messaging and Queuing with the SNS protocol, even though the v1beta1 version is also now used for SNS.
Migration: timescales and action to take
Below, we explain for each protocol how to migrate from the old to the new Messaging and Queuing.
NATS
Scaleway has automatically migrated each of your NATS namespaces and their credentials from the old version of Messaging and Queuing to the new one. In the new version, each NATS namespace you had is now a NATS account (each with its corresponding credentials).
The migration should have been seamless: if you logged into the Scaleway console before migration/GA you saw the “old” version of Messaging and Queuing with your existing resources showing as namespaces. After migration/GA, you now see the “new” version with your newly migrated resources now showing as accounts.
In terms of the API, the old v1alpha1 version continues to be available until the 1st of December 2023. After this date it will be retired and you must use the new v1beta1 version. The documentation for both is accessible via the Scaleway Developers website.
Internally, NATS namespaces (and their credentials) and NATS accounts (and their credentials) are the same resources, visible through both APIs. During the migration period, both APIs’ endpoints are essentially accessing the same resources. Do not attempt to delete your NATS resources from the v1alpha1 API, as this will also delete them from the new v1beta1 API and console. You will not be billed “twice” for resources in the old and new API because these are essentially the same resources.
Note that the URL/endpoint by which you access your NATS resources is not be the same in the two versions. Before the 1st of December 2023, you must update any code, scripts, applications, and architectures using the old endpoint, to update to the new endpoint. After this time, the old endpoints will no longer be recognized.
The new URL/endpoint is visible in the console from your Messaging > NATS > NATS Account dashboard, and also from the new API.
If you are a Serverless Functions/Containers user, and have created NATS triggers with the v1alpha1 version of the Messaging and Queuing API, you must recreate these triggers.
Summary: NATS
✅ Migration from the old to new API has been automatically performed for you
✅ Billing of NATS resources began at the time of General Availability
✅ You must update your code, applications and architectures so they use the new endpoint by the 1st of December 2023
✅ You must recreate your Serverless Functions/Containers NATS triggers if you have some
See our dedicated documentation, How to migrate to the new Messaging and Queuing - NATS.
SQS
From the moment Messaging and Queuing with SQS went into General Availability, a 30 day period began, by the end of which you must migrate your SQS resources from the old to the new version. This period ends on the 1st of December 2023. You must migrate your resources yourself before this date: this will not be automatic like it will for NATS.
To migrate, you must recreate your resources (credentials and queues) in the new version. Remember that the new version of the API has no namespaces: you will simply activate the SQS protocol and then directly create your credentials and queues. SQS and SNS have been separated, so if you want to use both you will need to activate both separately and create separate credentials for both.
If you previously had multiple namespaces to separate your SQS resources, we recommend that you use Scaleway Projects with the new version to handle this. By creating multiple Projects, activating SQS in each one, and then creating SQS credentials in each Project, you will be able to achieve separation of resources, and credentials scoped to the SQS broker of each Project.
You can recreate your credentials and queues either via the console or via the new v1beta1 API directly. In terms of the API, the current v1alpha1 version will continue to be available until the 1st of December 2023.
Console documentation:
API documentation:
Note the following important information about the migration/crossover period, during which both versions of the API exist:
Before GA
- Only the
v1alpha1API was available. All your resources existed in this API. - The console was based on this API, so when you viewed your resources in the console, you were viewing resources in the
v1alpha1API.
At the time of GA
- The
v1alpha1API is still available. Your resources continue to exist in this API until you delete them. - The
v1beta1API also becomes available. You should recreate your resources (credentials and queues) in this API. - The console now uses the
v1beta1API. You are no longer able to see or manage your old resources from thev1alpha1API via the console. Any resources now created via the console exist in thev1beta1API. - Messaging and Queuing with SQS is now billable. Whether you are using the old version or the new one, via the API or other devtools or the console, you are now billed for the messages you send.
From the 1st of December 2023
- The
v1alpha1API will be decommissioned. Any resources still existing on this API are deleted.
The URL/endpoints for your Messaging and Queuing SQS service and for your queues are not the same in the two versions. Make sure that when you recreate your resources with the new version, you remember to update the endpoint in any code, scripts, applications and architectures that were using the old endpoint. If you do not do so, they will continue to access the “old” resources rather than the new ones you have created for the new version.
The new URLs/endpoints are visible in the console from your Messaging > SQS dashboard, and also from the new API.
If you are a Serverless Functions/Containers user, and have created SQS triggers with the v1alpha1 version of the Messaging and Queuing API, you must recreate these triggers.
Summary: SQS
✅ You must migrate from the old version to the new by recreating your resources and credentials
✅ You must update your code, applications and architectures so they use the endpoints of the new version once you migrate
✅ Billing of SQS resources began at the time of General Availability, and applies to both the old and new versions
✅ All resources still existing on the old version will be deleted on the 1st of December 2023
✅ You must recreate your Serverless Functions/Containers SQS triggers if you have some
See our dedicated documentation, How to migrate to the new Messaging and Queuing - SQS.
SNS
SNS with Messaging and Queuing is not going into General Availability the way that NATS and SQS are. However, it has still moved from the existing old version (v1alpha1) to the new version (v1beta1) without namespaces. Therefore, even though you will not be billed for your SNS resources in the immediate future while it remains in Public Beta, there is still a migration to carry out.
At the time of the release of the new version (which coincided with SQS and NATS going into General Availability) a 30 day period began. You must migrate your SNS resources from the old to the new version within this time period, which ends on the 1st of December 2023.
To migrate, you must recreate your resources in the new version. Remember that the new version of the API has no namespaces: you should simply activate the SNS protocol and then directly create your credentials. SQS and SNS have been separated, so if you want to use both you will need to activate both separately and create separate credentials for both.
If you previously had multiple namespaces to separate your SNS resources, we recommend that you use Scaleway Projects with the new version to handle this. By creating multiple Projects, activating SNS in each one, and then creating SNS credentials in each Project, you will be able to achieve separation of resources, and credentials scoped to the SNS broker of each Project.
You can recreate your credentials either via the console (once the new version is in production) or via the new v1beta1 API directly. In terms of the API, the current v1alpha1 version will continue to be available until the 1st of December 2023. The documentation for both versions is accessible via the Scaleway Developers website.
Console documentation:
API documentation:
Note the following important information about the migration/crossover period, when both versions of the API will exist:
Before release of new version
- Only the
v1alpha1API was available. All your resources existed in this API. - The console was based on this API, so when you viewed your resources in the console, you were viewing resources in the
v1alpha1API.
At the time that SQS and NATS went into GA
- The
v1alpha1API is still available. Your resources continue to exist in this API until you delete them. - The
v1beta1API also becomes available. You should recreate your resources in this API. - The console now uses the
v1beta1API. You are no longer able to see or manage your old resources from thev1alpha1API via the console. Any resources now created via the console exist in thev1beta1API.
From the 1st of December 2023
- The
v1alpha1API will be decommissioned. Any resources still existing on this API will be deleted.
The URL/endpoints for your Messaging and Queuing SNS service and for your topics are not the same in the two versions. Make sure that when you recreate your resources with the new version, you remember to update the endpoint in any code, scripts, applications and architectures that were using the old endpoint. If you do not do so, they will continue to access the “old” resources rather than the new ones you have created for the new version.
The new URLs/endpoints are visible in the console from your Messaging > SNS dashboard, and also from the new API.
Summary: SNS
✅ You must migrate from the old version to the new by recreating your resources and credentials
✅ You must update your code, applications and architectures so they use the endpoints of the new version once you migrate
✅ SNS will remain in Public Beta (in both old and new versions) and will not yet be billed
✅ All resources still existing on the old version will be deleted on the 1st of December 2023
See our dedicated documentation, How to migrate to the new Messaging and Queuing - SNS.
Terraform
There is a new Terraform provider for the new version of Messaging and Queuing. The link will be published shortly.
If you manage Messaging and Queuing using the old Terraform provider, you will need to update your Terraform files to use the new provider, in line with the migration timelines laid out above.
Further support and next steps
Do not hesitate to join us on the Scaleway Community Slack where we will keep you up to date and answer any questions on the #messaging-queuing-beta channel.
Messaging and Queuing documentation Messaging and Queuing developers documentation