AI vs Traditional Algorithms: A Practical Example

This issue is illustrated by a case study involving a U.S.-based finance company with an ambitious goal: quadrupling its client capacity from 50 to 200. Achieving this objective has been difficult due to tight deadlines and a limited project budget.

We needed to overcome the existing platform’s limitations and related maintenance and operational processes. The platform was based on an on-premises infrastructure, significantly limiting scalability.

The system struggled to efficiently manage the increasing number of new clients, as onboarding each new company required code changes.

Additionally, data was submitted in various formats that weren’t consistent from one upload to the next. Given that the company operates in a demanding finance market, data accuracy was paramount, yet reconciling the data in an efficient way was becoming increasingly challenging.

Due to the unique nature of the company’s business processes, standard off-the-shelf solutions were unsuitable. The client needed a customized solution that could integrate seamlessly and scale according to their specific needs.

Our primary goal was to resolve the client’s issues within the specified timeframe and budget while minimizing risks.

After completing the initial recon phase, the idea of using artificial intelligence to tackle the problem seemed very attractive. We began exploring whether integrating an AI module for data analysis into the client’s current on-premises setup could be a viable solution. After all, is there anything that AI can’t handle?

However, the company’s specialized processes and the need to comply with regulations required a careful examination. That’s why, with our consulting team, I had to determine if AI could meet the client’s needs while maintaining high data security and process accuracy standards. This meant taking a closer look at how AI could fit into the existing system, which specific tasks it could handle, and what the overall benefits of its use would be.

To decide whether using AI is the right path, we considered our selection criteria:

1. Problem definition:
   Defining the business problem, we aim to solve and determine whether AI is necessary or if traditional solutions are sufficient.
2. Business justification:
   Estimating the expected return on investment (ROI), such as increased capabilities, cost reductions, or improved customer experiences, to develop a strong business case.
3. Analysis of costs and benefits:
   Assessing the potential benefits of AI against the financial costs, resource needs, and possible disruptions to existing processes.
4. Legal and ethical considerations:
   Addressing data privacy, regulatory compliance, and ethical issues such as bias and fairness.
5. Success metrics:
   Defining clear indicators of the success of the AI-powered project, like accuracy, performance improvements, and cost savings.
6. Infrastructure assessment:
   Evaluating the current technical infrastructure to ensure it can support the required AI technologies, including hardware, software frameworks, and technical expertise.
7. Organizational readiness:
   Determining the organization’s readiness for adopting new technologies, including staff adaptability and the cultural fit for innovation.

By focusing on these key areas, we ensure that our solutions are not only effective but also sustainable and align with the client’s strategic goals and operational capabilities. 

The process was quite thorough, and in the course of discovery, we went through several possible options.

We explored the availability of AI models that meet our industry and regulatory standards. We also evaluated our capacity to develop a custom AI model, factoring in aspects such as time, costs, expertise needed, and data availability. Additionally, we considered the possibility of using two models: one for data organization and another for processing.

Furthermore, we assessed the current infrastructure’s capability to support AI solutions, particularly focusing on scalability. We also reviewed the client’s in-house capabilities to maintain an AI solution and considered using discriminative AI for structuring and processing large datasets.

Lastly, we examined the project’s budget and timeframe to determine the feasibility of experimentation and development, as well as the long-term viability and sustainability of potential solutions.

Ultimately, our conclusions came down to the following:

• There are no suitable, ready-made AI models on the market.
• Building a custom AI model is not feasible due to high costs, required expertise, and data limitations.
• Using multiple models is impractical due to insufficient data and frequent changes.
• The existing infrastructure cannot act solely as a relay for AI in the cloud, as it doesn’t solve scalability issues.
• The client lacks the necessary in-house competencies for AI maintenance but can maintain a classical system.
• Discriminative AI does not fit the solution requirements.
• Limited budget and time constraints leave no room for experimentation.
• Incremental improvements based on ROI are necessary, with the potential for future AI integration.

A significant component of deciding on the use of AI included a risk analysis. We had to be mindful of several crucial requirements and limitations:

1. Data quality and complexity: AI requires high-quality, consistent data, but the project’s data arrived in various formats and with different naming conventions, complicating the processing and cleaning tasks. This could potentially lead to a project delay.
2. Security and data integrity: Financial transaction security and data integrity are top priorities. Off-the-shelf AI solutions might create security gaps and data breaches, especially under strict regulations.
3. Domain-specific challenges: Implementing AI in specialized areas is challenging. Additionally, customizing AI for unique business processes increases costs and extends timelines.
4. Accuracy and oversight: Relying on AI without proper oversight is risky. Even minor inaccuracies in financial operations could lead to significant pitfalls and costs.
5. Integration and transformation: Integrating AI involves more than just technology – it’s a comprehensive transformation requiring training, integration, and ongoing maintenance.
6. Accountability: Clearly defining who is responsible for AI-generated outcomes is essential, especially in regulated industries.

Given these risks, we decided to go with a well thought-out and written, classical architecture. A cloud-based, modular design using traditional algorithmics would allow us to improve reliability, achieve our objectives, and avert potential delays and additional risks resulting from premature AI implementation.

The selected solution involved creating a robust, adaptable architecture using established cloud services and well designed, solid data cleansing and processing algorithms, written in line with the best practices of classical algorithmics. This method guaranteed the platform’s predictability, dependability, and potential for future growth – including AI components, if the need ever arises.

To achieve this goal, we migrated the platform to Microsoft Azure. We also developed a configurable and intelligent data import module to streamline client onboarding and data integration.

Additionally, Python-based ETL pipelines were created to automate and optimize the data cleaning process, ensuring high-quality data for processing.

The redesign involved updating trade matching and reconciliation algorithms to eliminate performance bottlenecks and enhance accuracy and efficiency. We also added built-in monitoring to simplify compliance with regulatory requirements and integrated comprehensive reporting features to provide valuable insights for decision-making.

The critical business modules – Import, Clean, and Reconcile – were redesigned to allow full reconfiguration as needed, using clear and transparent algorithms to maintain data integrity and quality. This redesign considered industry-specific requirements and ensured robust security measures.

The client has experienced significant improvements and benefits from implementing a new cloud-based solution. Here is what we achieved:

• Improved scalability: The client has been able to expand their operation, resulting in a 400% increase in scale, which means they can now handle a much larger volume of work or transactions.
• Flexible processing: Cloud technology enables handling more transactions and users while maintaining performance and speed. The system can scale up as the client’s demand increases.
• Eliminating bottlenecks: Performance bottlenecks are being eliminated by addressing and removing inefficiencies caused by poorly written algorithms.
• Cost reduction: The new approach brought significant cost savings, reducing overall expenses by 60%, including savings on infrastructure, maintenance, and human resource costs.
• Improved data processing and security: The client has gained advanced data processing capabilities. Additionally, improved security measures were implemented to protect sensitive financial data, ensure compliance with regulations, and safeguard against breaches.
• Eliminating technical debt: The issue of accumulated technical debt has been addressed and resolved. (The term refers to the future costs and complications that arise from choosing quick, easy solutions instead of more thorough, longer-term approaches).
• Business continuity and growth readiness: Our client can continue their business operations without interruptions and is well-prepared for future growth.
• Modular and open architecture: The proposed architecture is modular and open, making it easier to develop and add new functionalities without risking the system’s stability. For example, the system can be expanded with AI components when needed, allowing the client to integrate advanced AI features to further develop their operations.

This project required us to make a decision between the use of AI and traditional programming for this specific scenario. The basis has been a thorough understanding of the difference between AI and traditional approaches. This is what we learned while considering various solutions:

1. The key elements of the strategic decision-making process involve creating a strong business case by clearly defining the problem, understanding market needs, and justifying the investment based on potential benefits.
2. It is crucial to determine the return on investment (ROI) to demonstrate how the solution will generate financial returns and support overall profitability. It’s equally important to align with long-term business goals to ensure the company can make decisions supporting the business vision.
3. Moreover, it is important to assess the technical feasibility and evaluate the economic and strategic benefits of implementing new technologies, such as cloud solutions, to ensure that the solution is both practical and advantageous for the business.

For this reason, here are my recommendations for similar projects:

1. Conduct a robust discovery phase to ensure the solution effectively addresses the business problem, taking into account existing requirements and constraints.
2. Focus on well-structured, thought-through architecture design that allows for future component additions while solving current issues.
3. Prioritize a solution that works, generates value, and provides ROI, ensuring it serves the customer and secures business operations.
4. Once a solution that meets business needs is in place, you can further develop it and experiment with new technologies, such as AI.

Choosing between artificial intelligence and classical algorithmics is a significant decision for any business wanting to leverage its data. Understanding the difference between these approaches can help in making an informed choice.

Traditional algorithms offer reliability and predictability, while AI models bring advanced capabilities. By carefully evaluating relevant factors, businesses can select the approach that best aligns with their goals and ensures long-term success.

If you’re facing a similar dilemma or need expert guidance, contact us for a consultation. Our team is ready to help you make the best choice for your business needs.