Leverage NATS Messaging and Queuing pub/sub capabilities to offload your Managed Database

terraform
scraping-architecture

Introduction

In this tutorial, we show how to set up a system that receives a message from a publisher and sends it to a database through a buffer subscriber. For this, we use Scaleway Serverless products and deploy two applications:

A publisher application, that sends a message to a NATS stream created with Scaleway Messaging and Queuing.
A subscriber application, that consumes messages published to the stream and then writes the data into a Scaleway Managed Database.

We show how to provision all the required Scaleway resources using code examples and the Scaleway console. The code for the functions is written in Python.

This project exemplifies a decoupled architecture, where producer and consumer perform specific tasks independently. This kind of design is modular and allows for flexibility and scalability. It also adheres to the principles of microservices and serverless architectures, where individual functions or scripts focus on specific tasks.

To properly follow this tutorial, we recommend you create a dedicated Project in your Scaleway account. You can name it RDB and NATS Tutorial. It will hold all your Scaleway resources created for this tutorial, so they will be easier to find.

Before you start

To complete the actions presented below, you must have:

A Scaleway account logged into the console
Owner status or IAM permissions allowing you to perform actions in the intended Organization
Installed the NATS CLI on your local machine
Created a Messaging and Queuing NATS account
Generated NATS credentials
Created a mySQL Managed Database

Create a NATS stream

Refer to the dedicated documentation to create your NATS stream. We recommend keeping your stream specifications as simple as possible. There is no need for replication or specific parameters for this tutorial.

When your NATS server is up and running, check you did not forget anything for the Scaleway context by running the following code in a new window of your terminal:

nats context info scaleway

The output should mention “OK” for the credentials path, and confirm that your NATS server is ready. If not, go back to the NATS CLI documentation to properly create a Scaleway context.

Install dependencies

Open a new terminal window from your project directory.

Install the following dependencies for the tutorial:

sudo apt-get install mysql-server
pip install mysql-connector-python 
pip install pynacl
pip install nkeys
pip install nats-python

Connect to your Scaleway Managed Database

Open a new terminal window.
Log in to the Scaleway console and access your Project.
Click PostgreSQL and MySQL in the console side menu.
Click the relevant database, and go to its Overview tab.
Copy the snippet under Connection, in the Network block.
Paste the snippet into the terminal window and update it with your created username.
Follow the instructions in the terminal to validate your password. You should now have access to the mysql environment.

Create the database and table with the following code:

CREATE DATABASE nats_messages;
USE nats_messages;
CREATE TABLE messages (
    id INT AUTO_INCREMENT PRIMARY KEY,
    subject VARCHAR(255),
    data TEXT,
    received_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

Create the publisher

We start by creating the publisher application.

Create a new file in your project directory named nats_publisher.py.

Paste the following Python code into the file:

import asyncio
from nats.aio.client import Client as NATS
async def run_publisher():
    # Connect to the NATS server
    nc = NATS()
    await nc.connect("NATS ACCOUNT URL", user_credentials="NATS CREDENTIALS PATH")
    # Publish a message to the subject
    await nc.publish("foo", b'Hello, World!')
    # Flush to ensure messages are processed
    await nc.flush()
    # Close the connection
    await nc.close()
if __name__ == '__main__':
    asyncio.run(run_publisher())

Do not forget to replace NATS ACCOUNT URL and NATS CREDENTIALS PATH with your own NATS account details.

This small Python application will publish a “Hello World!” message under the “foo” subject.

Create the subscriber

Next we create the subscriber application.

Create a new file in your project directory named nats_subscriber_with_mysql.py.

Paste the following Python code into the file:

import asyncio
import mysql.connector
from nats.aio.client import Client as NATS
async def run_subscriber():
    # Connect to the NATS server
    nc = NATS()
    await nc.connect("NATS ACCOUNT URL", user_credentials="NATS CREDENTIALS PATH")
    async def message_handler(msg):
        subject = msg.subject
        data = msg.data.decode()
        print(f"Received a message on '{subject}': {data}")
        # Store the message in the MySQL database
        conn = mysql.connector.connect(
            host="PUBLIC ENDPOINT",
            port="PORT",
            user="NAME",
            password="PASSWORD",
            database="nats_messages"
        )
        cursor = conn.cursor()
        cursor.execute('INSERT INTO messages (subject, data) VALUES (%s, %s)', (subject, data))
        conn.commit()
        cursor.close()
        conn.close()
    # Subscribe to a subject
    await nc.subscribe("foo", cb=message_handler)
    # Keep the subscriber running to receive messages
    while True:
        await asyncio.sleep(1)
if __name__ == '__main__':
    asyncio.run(run_subscriber())

Do not forget to replace the subscriber code with the correct configuration details for your Scaleway Managed Database information, and NATS ACCOUNT URL and NATS CREDENTIALS PATH with your own NATS account details.

You will note that they are the same as the publisher’s. This is normal as both applications are communicating through the same NATS server.

Put it all in motion

Your architecture is now ready:

A NATS server to handle the pub/sub messaging pattern
A publisher application that will send a message through the NATS server
A subscriber application that will retrieve the message from the NATS server and write it to the mySQL database
A database that can be read by the rest of your architecture’s clients

To see the flow running, you can follow these steps:

Open your terminal window dedicated to the subscriber and run the Python file:

python nats_subscriber_with_mysql.py

Open your terminal window dedicated to the publisher and run the python file:

python nats_publisher.py

Go back to your terminal window of the subscriber. You can see that the message has been received by the application and sent to the database.

Go back to your database terminal window (the one with >sql) and run the following code:

USE nats_messages
SELECT * FROM messages;

You should see your message in the table with the timestamp.

How to remove all resources used in the tutorial

To be sure that you are not running resources without using them, you can delete your database by following this documentation.

You can also delete your NATS server with the following command in your terminal:

nats-server --signal stop

Messaging and Queuing NATS is a Serverless product and is only billed when you are moving messages through the broker. This eliminates the need to worry about unexpected billing when you are not using it.

Summary, going further, key takeaways

We have shown how to asynchronously decouple a producer and a mySQL database using NATS, adhering to serverless patterns.

While the volume of data processed in this example is quite small, thanks to the Messaging and Queuing NATS streams, your database will be able to process messages at an adapted pace. You can adapt this example to manage larger workloads.

Here are some possible extensions to this basic example:

Use a continuous flow of incoming messages
Use a Serverless Database to embrace further the Serverless architecture
Adding multiple subscribers and publishers thanks to the scalability of NATS pub/sub patterns