As the nation celebrates Republic day, republic day sale events are live on all the major e-commerce platforms, with exclusive offers and discounts to attract customers. These sale events including the ones held during the festive season are a testing time for e-commerce and payment companies as buyers hit the platform in great numbers to shop causing huge spikes in the in transaction volumes.
PayU, the global payment processing company, for example, saw a 31% increase in the number of transactions last year, compared to the festive period in 2021. During Diwali, they processed a record 11 million transactions in one day, which was the highest single-day transaction volume to date. PayU ensures that its systems are prepared to handle such massive spikes in transaction volume during the online sale events, and scale up to 3x to 4x in comparison to their day-to-day capacity.
“Ensuring that our merchants succeed along with us is very important for us. We do a lot of preparations in the background to make sure that we meet the scaled-up demands for our merchants. A payment transaction contains multiple stages. To effectively scale, we break down the processing into multiple steps. For example, fraud check is one step, data preparation based on the payment method is the next, and finally finishing the transaction with the actual payment entity (bank, credit card network, etc.) is the final step,” explained Narendra Babu, CTO, PayU payments.
Henceforth, the company has optimised their systems to ensure they are running efficiently. This includes regular maintenance and updates to ensure that their systems can handle the increased volume during the sale festive season, which is one of the major crackpoints.
Scaling up successfully
“In any company, there is a certain amount of human capital that can handle exceptions and manage all of these processes. However, you cannot simply go out and hire people just for the sale events or festive big billion-day sales. So, you have to make sure that all of this is managed efficiently,” said Babu. “We use MySQL Aurora for our transaction DB and Redshift for our warehouse requirements. We have a multi-stage data management pipeline to sanitize, index, and serve our data according to the requirements,” he added.
To scale up during the sale time, PayU has built each of the stages as different databases and created a copy of these records in their data warehouse (Redshift). This enables them to address any subsequent querying in the warehouse and further reduce the load on their main database. This is the strategy of breaking complex functionality into microservices and moving data to different databases offline for efficient processing.
Therefore, a performant relational database is key for PayU to manage high-scale applications. A relational database is a collection of information that organises data in predefined relationships where data is stored in one or more tables (or “relations”) of columns and rows, making it easy to see and understand how different data structures relate to each other.
“These are useful to PayU because relationships are a logical connection between different tables; once established based on interaction among these tables, it is easier to draw inferences, predictions, or actionable insights. We rely on an Amazon Web Services (AWS) solution called Aurora as a database solution. PayU stores all the transaction data in this database to manage the payment workflow. Naturally, the scalability of the database is key to PayU’s overall scalability,” he added.
Preventing fraudulent transactions
With high-volume data pouring with every traction, it’s vital for PayU to address the noise in the system such as fraudulent transactions. They implement advanced security measures such as two-factor authentication, biometric verification, and machine learning-based fraud detection systems. These measures help to identify and prevent fraudulent transactions, ensuring that the payment process is secure for both merchants and customers.
Babu explains, “We have a combination of rule-based and ML-based systems to detect fraud. We use Wibmo’s Trident system to check for transactions that violate our fraud rules and run an offline ML-based system to detect spurious transactions.”
Rule-based systems are systems that detect specific patterns in transactions within PayU systems. e.g., they can see if any merchant who exceeds a certain money limit on the daily transaction is paused for further transactions until our expert looks at these transactions. ML-based systems are intelligent systems that auto-detect any strange patterns in their transactions.
“If there is a high frequency of transactions within a short amount of time for a given merchant ( the threshold can be different for different merchants, and the systems themselves infer these thresholds) we have a combination of rule-based systems and ML systems to reduce the noise-to-signal ratio in detecting erring transactions,” he adds.
Furthermore, anomaly detection is a key strategy for PayU to employ, detect issues of fraud, and manage data effectively within their systems. Anomaly detection is a crucial aspect of managing data. It allows organisations to identify and address potential issues before they become major problems.
One such system is time series pattern analysis. This method uses metrics such as latencies, error codes, and success rates to build patterns for each application. The system then looks at historical patterns and gives more weight to recent signals while giving less weight to prior data signals. This allows the system to compute variations in the data and detect anomalies.
He explains, “Even though there are many hundreds of integrations and permutations possible, the system can narrow down on the specific issue and log into the application for further analysis and corrective action. But to overcome issues with smaller sample sizes, we utilize tools such as statistical analysis and machine learning to identify patterns and detect anomalies in the data.”
However, preparing for the sale season is not just about scaling systems. PayU also has to balance the need for stability with the demand for new features, as merchants want to offer new and improved experiences for their customers during this period. “Balancing both can always be a challenge, while remaining confident in the system’s ability to function properly,” says Babu.
Serving the merchants better
PayU plans to scale its operations and support a growing number of small and medium-sized business merchants. The company is focusing on leveraging the latest technology to reduce the time and effort required for merchant onboarding, transaction processing, and post-processing.
One of the key areas of focus for PayU is open-source technology. They plan to use open-source solutions for automated chatbots, which will help understand and service customer queries. They are also exploring new technologies such as explainable artificial intelligence (XAI) for enhancing the customer experience.
He says, “As we add more and more merchants, it’s very important that we train our systems to automatically support our merchants in terms of any queries they might have. Leveraging open-source technology is a key part of our strategy to become more self-sufficient and automated to scale our operations and better serve our merchants.”