Transactions that span over multiple physical systems or computers over the network, are simply termed Distributed Transactions. In the world of microservices, a transaction is now distributed to multiple services that are called in a sequence to complete the entire transaction. For detailed explanation refer Handling Distributed Transactions in the Microservice world
Scalability: The system should be scalable as per the requirements else it could fail when load increases.
Error handling: The system should be able to catch errors and handle it appropriately.
Idempotency: Sometimes the system may fail due to any error and restart again. So the system may send the same emails again creating duplication. So the system must be able to perform deduplication.
Let’s take an example :
There are various ways to implement the system. We tried below mentioned implementations:
Problems:
Problems:
Note : Here the sendEmail processes the single row update and sends email accordingly.
key | userId | data1 | data2 | data3 | data4 | timestamp |
---|---|---|---|---|---|---|
1 | abc-256 | 1 | 2 | 4 | 3 | 00:50 |
2 | xyz-023 | 4 | 8 | 7 | 2 | 01:20 |
Problems:
What are Step Functions :
AWS Step Functions lets you coordinate multiple AWS services into serverless workflows. Workflow into a state machine diagram that is easy to understand, easy to explain to others, and easy to change.
(Main Advantage) Step Functions automatically triggers and tracks each step, and retries when there are errors, so your application executes in order and as expected.
Example of workflow design in Step Functions:
We wanted to send the device usage of a customer, every month in the form of templated email. The system has to be completely serverless and automated.
You can create the Step Function for the following workflow using StepFunctions console https://console.aws.amazon.com/states/ :
Step Function definition:
{
"Comment": "sending Email",
"StartAt": "sendEmail",
"States": {
"sendEmail": {
"Type": "Task",
"Resource": "paste the sendEmail lambda ARN",
"End": true
}
}
}
Workflow will be created as per definition:
This lambda function runs on the trigger of ‘sendEmailStepFunction’ execution. Here it uses AWS SES api to send a single mail as per the event created by sendEmailStepFunction. The event contains the email data.
Sample code for sending email using Simple email service:
const params = {
Source: "xyz@gmail.com",
Template: "emailTemplate",
Destination: {
ToAddresses: [emailId]
},
TemplateData: "{\"usage\":\"99\”}”
};
//send email
var sendTemplatedEmailPromise = (new aws.SES()).ses.sendTemplatedEmail(params).promise();
await sendTemplatedEmailPromise;
The mainFunction extracts the data from the athena table using SQL query.
userId | data1 | data2 |
---|---|---|
abc-256 | 1 | 2 |
xyz-023 | 4 | 8 |
The data obtained is in the form of JSON format.
Sample athena response:
athenaQueryOutput = {
Items:
[{ userid: 'abc-256', data1: '1', data2: '2' },
{ userid: 'xyz-023', data1: '4', data2: '8' }],
Data Scanned In MB: 0,
QueryCostInUSD: 0.0000,
EngineExecutionTimeInMillis: 0,
Count: 0,
QueryExecutionId: 'testExecutionID',
S3 Location:
'Test_s3_location'
}
var params = {
stateMachineArn: 'sendEmailStepFunction ARN',
input: JSON.stringify(emailData),
name: key //you can set any name but it should be unique for all step functions
};
var stepFunctionPromise = (new aws.StepFunctions())stepFunctions.startExecution(params).promise();
await stepFunctionPromise;
Solution to previous example:
So how to prevent duplication?
Cons of using AWS Step Functions:
We will come back soon with the blog on Implementation of above email system using Apache Airflow.
WhatsApp us