Why spend £10M replacing your sales order processing system when CQE can fix your current system for a fraction of the cost?
Situation
Black Friday and its surge in consumer spending present a significant opportunity for retailers, accompanied by substantial threats. In 2022, Black Friday in the UK witnessed a total consumer expenditure of £8.71 billion, with £4.81 billion being spent online. Despite the online trend, there was still a 9.3% increase in UK shoppers at physical retail locations including retail parks, shopping centres, and high streets. The previous year, in 2021, online spending had reached even higher figures at £5.74 billion, contributing to the total spending of £9.42 billion. This heightened activity puts immense strain on retailers' IT systems, leading to dire consequences when orders cannot be fulfilled, and customer expectations are not met.
It was in 2010 when Amazon introduced the UK to the concept, but it wasn’t until 2013 when Asda was purchased by Walmart, that the mainstream retailers really started to take notice.
Challenge
IT systems when faced with unexpected demand can sometimes be unpredictable. A high street retailer faced significant challenges preparing for their first Black Friday event due to heavy discounts on specific stock items, causing a surge in footfall at both physical stores and online platforms. The retailer needed to ensure its systems were equipped to handle the subsequent rush in both realms. Ideally, their new IT system with upgraded hardware and a generous £10 million budget would be the perfect solution. However, the timescales of the new system didn’t align with the rapidly approaching Black Friday, so a team was mobilised to conduct a deep-dive review of the existing system to try and identify and remove any bottlenecks to system performance.
Setting the Stage
Upon initial assessment, high-risk areas within the system were pinpointed. We undertook hardware upgrades and implemented an open-source system and infrastructure monitoring to facilitate testing. A custom test harness was developed, enabling simulation of a configurable virtual fulfillment centre (VFC). We devised scripts and scenarios to accurately replicate customer behaviour, including micro-bursts characterized by small, low-cost transactions.
Our Approach
Exploration and Analysis
With the testing environment in place, we executed tests and closely monitored key performance indicators (KPIs). Our approach to performance testing emphasised exploration: tests were conducted to scrutinise system behaviours, and metrics were analysed to isolate issues. Problem areas were identified through iterative cycles, leading to focused attention on specific system components.
Challenges and Achievements
The project centred on addressing poor system performance that hindered order processing, especially during peak periods like Black Friday. Despite the challenge, this scenario offered the opportunity for significant improvements. Our team efficiently cycled through goal setting, testing, analysis, and implementation, allowing us to pinpoint problem areas swiftly.
Collaborative Decision-Making
During the project, collaborative technology teams were formed, comprising specialists from both the client and our project team. This group evaluated KPIs and made informed tuning decisions. The complexity of system behaviour analysis often rendered tuning decisions intricate; however, we recognized the potential for both performance degradation and improvement.
Resolution and Findings
For this particular client, performance issues were largely attributed to the Java Virtual Machines (JVMs). Our analysis revealed inefficient garbage collection, thread contention, and inappropriate heap sizes as culprits. Despite the complexity introduced by the system's intricacies and diverse opinions, our methodical approach enabled us to confidently focus on the JVMs as the primary area requiring attention.
To speak to us and learn more about our services, fill out the form here and we will be in touch
Results
Our comprehensive and highly configurable environment was key to the rapid identification and resolution of performance issues. The existing order processing ceiling of 70 orders/hour was increased to 700 and then de-tuned back to 500 as other parts of the system couldn’t cope with that level of throughput.
The success of the tuning project led the client to cancel the upgrade program that was budgeted to cost them £10 million.
Lastly, our comprehensive virtual fulfilment centre proved to alleviate environment contention for the teams performing functional testing. The logistics provider only had a single test environment, the client had many, so performing end to end tests was limited to a single environment at a time. Configuring multiple VFCs meant that these test teams had significantly more flexibility and ability to operate concurrently than they had ever had before.
Conclusion
This venture underscores two pivotal insights. First, while testing might be deemed a cost, its returns can be exponential. Here, a £250k investment not only bolstered sales but also saved a potential £10 million. Second, being pro-active is key to preparing IT systems for peak sales events. As this endeavour illustrates, engaging with a partner you can trust, who have a proven track record of successful client delivery, can empower retailers to seize market opportunities as they arise.