Leverage NATS Messaging and Queuing pub/sub capabilities to offload your Managed Database
- terraform
- scraping-architecture
Introduction
In this tutorial, we show how to set up a system that receives a message from a publisher and sends it to a database through a buffer subscriber. For this, we use Scaleway Serverless products and deploy two applications:
- A publisher application, that sends a message to a NATS stream created with Scaleway Messaging and Queuing.
- A subscriber application, that consumes messages published to the stream and then writes the data into a Scaleway Managed Database.
We show how to provision all the required Scaleway resources using code examples and the Scaleway console. The code for the functions is written in Python.
This project exemplifies a decoupled architecture, where producer and consumer perform specific tasks independently. This kind of design is modular and allows for flexibility and scalability. It also adheres to the principles of microservices and serverless architectures, where individual functions or scripts focus on specific tasks.
To properly follow this tutorial, we recommend you create a dedicated Project in your Scaleway account. You can name it RDB and NATS Tutorial
. It will hold all your Scaleway resources created for this tutorial, so they will be easier to find.
Before you start
To complete the actions presented below, you must have:
- A Scaleway account logged into the console
- Owner status or IAM permissions allowing you to perform actions in the intended Organization
- Installed the NATS CLI on your local machine
- Created a Messaging and Queuing NATS account
- Generated NATS credentials
- Created a mySQL Managed Database
Create a NATS stream
Refer to the dedicated documentation to create your NATS stream. We recommend keeping your stream specifications as simple as possible. There is no need for replication or specific parameters for this tutorial.
When your NATS server is up and running, check you did not forget anything for the Scaleway context by running the following code in a new window of your terminal:
nats context info scaleway
The output should mention “OK” for the credentials path, and confirm that your NATS server is ready. If not, go back to the NATS CLI documentation to properly create a Scaleway context.
Install dependencies
- Open a new terminal window from your project directory.
- Install the following dependencies for the tutorial:
sudo apt-get install mysql-serverpip install mysql-connector-pythonpip install pynaclpip install nkeyspip install nats-python
Connect to your Scaleway Managed Database
-
Open a new terminal window.
-
Log in to the Scaleway console and access your Project.
-
Click PostgreSQL and MySQL in the console side menu.
-
Click the relevant database, and go to its Overview tab.
-
Copy the snippet under Connection, in the Network block.
-
Paste the snippet into the terminal window and update it with your created username.
-
Follow the instructions in the terminal to validate your password. You should now have access to the
mysql
environment. -
Create the database and table with the following code:
CREATE DATABASE nats_messages;USE nats_messages;CREATE TABLE messages (id INT AUTO_INCREMENT PRIMARY KEY,subject VARCHAR(255),data TEXT,received_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP);
Create the publisher
We start by creating the publisher application.
-
Create a new file in your project directory named
nats_publisher.py
. -
Paste the following Python code into the file:
import asynciofrom nats.aio.client import Client as NATSasync def run_publisher():# Connect to the NATS servernc = NATS()await nc.connect("NATS ACCOUNT URL", user_credentials="NATS CREDENTIALS PATH")# Publish a message to the subjectawait nc.publish("foo", b'Hello, World!')# Flush to ensure messages are processedawait nc.flush()# Close the connectionawait nc.close()if __name__ == '__main__':asyncio.run(run_publisher())
Do not forget to replace NATS ACCOUNT URL
and NATS CREDENTIALS PATH
with your own NATS account details.
This small Python application will publish a “Hello World!” message under the “foo” subject.
Create the subscriber
Next we create the subscriber application.
-
Create a new file in your project directory named
nats_subscriber_with_mysql.py
. -
Paste the following Python code into the file:
import asyncioimport mysql.connectorfrom nats.aio.client import Client as NATSasync def run_subscriber():# Connect to the NATS servernc = NATS()await nc.connect("NATS ACCOUNT URL", user_credentials="NATS CREDENTIALS PATH")async def message_handler(msg):subject = msg.subjectdata = msg.data.decode()print(f"Received a message on '{subject}': {data}")# Store the message in the MySQL databaseconn = mysql.connector.connect(host="PUBLIC ENDPOINT",port="PORT",user="NAME",password="PASSWORD",database="nats_messages")cursor = conn.cursor()cursor.execute('INSERT INTO messages (subject, data) VALUES (%s, %s)', (subject, data))conn.commit()cursor.close()conn.close()# Subscribe to a subjectawait nc.subscribe("foo", cb=message_handler)# Keep the subscriber running to receive messageswhile True:await asyncio.sleep(1)if __name__ == '__main__':asyncio.run(run_subscriber())Do not forget to replace the subscriber code with the correct configuration details for your Scaleway Managed Database information, and
NATS ACCOUNT URL
andNATS CREDENTIALS PATH
with your own NATS account details.You will note that they are the same as the publisher’s. This is normal as both applications are communicating through the same NATS server.
Put it all in motion
Your architecture is now ready:
- A NATS server to handle the pub/sub messaging pattern
- A publisher application that will send a message through the NATS server
- A subscriber application that will retrieve the message from the NATS server and write it to the mySQL database
- A database that can be read by the rest of your architecture’s clients
To see the flow running, you can follow these steps:
-
Open your terminal window dedicated to the subscriber and run the Python file:
python nats_subscriber_with_mysql.py -
Open your terminal window dedicated to the publisher and run the python file:
python nats_publisher.py -
Go back to your terminal window of the subscriber. You can see that the message has been received by the application and sent to the database.
-
Go back to your database terminal window (the one with >sql) and run the following code:
USE nats_messagesSELECT * FROM messages;You should see your message in the table with the timestamp.
How to remove all resources used in the tutorial
To be sure that you are not running resources without using them, you can delete your database by following this documentation.
You can also delete your NATS server with the following command in your terminal:
nats-server --signal stop
Messaging and Queuing NATS is a Serverless product and is only billed when you are moving messages through the broker. This eliminates the need to worry about unexpected billing when you are not using it.
Summary, going further, key takeaways
We have shown how to asynchronously decouple a producer and a mySQL database using NATS, adhering to serverless patterns.
While the volume of data processed in this example is quite small, thanks to the Messaging and Queuing NATS streams, your database will be able to process messages at an adapted pace. You can adapt this example to manage larger workloads.
Here are some possible extensions to this basic example:
- Use a continuous flow of incoming messages
- Use a Serverless Database to embrace further the Serverless architecture
- Adding multiple subscribers and publishers thanks to the scalability of NATS pub/sub patterns