How Machine Learning Models Enhance Anomaly Detection

Howdy AI friends, 

Some beloved dishes arise from unexpected mistakes. Take tarte Tatin, the classic French dessert. Legend says the Tatin sisters accidentally overcooked apples in butter and sugar. To salvage the caramelized apples, they covered them with pastry and baked them, creating a favorite upside-down tart.  

This delightful mistake shows how kitchen (or data) anomalies can lead to surprising discoveries, though they can also ruin a dish or become harmful if not appropriately handled. 

Consider data breaches in security systems. The average cost of a data breach soared to $4.88 million in 2024, with breaches taking an average of 194 days to identify. Meanwhile, global eCommerce fraud has ballooned from $41 billion in 2022 to a projected $48 billion in 2023. These staggering numbers highlight how unchecked anomalies can drain resources, erode trust, and create operational downtime. 

The real question is: Do we have the tools and processes to spot and act on these anomalies before they affect our bottom line? 

The tools’ corner: Machine Learning models and their exciting potential to revolutionize anomaly prediction 

Anomaly detection models identify patterns and deviations in data, categorized into 3 types: 

• Point Anomalies: Occur when individual data points differ significantly from the rest (e.g., a sudden spike in network traffic) — like a sour grape in a bunch
  
• Context Anomalies: Data points anomalous only in a specific context (e.g., high sales during an off-season) — like ketchup in dessert
  
• Collective Anomalies: Groups of related data points showing unusual behavior together (e.g., multiple failed login attempts) — like a row of sushi rolls that all taste off

To address these varied anomaly types, we can employ different models: 

• Statistical Models rely on fixed thresholds or distribution assumptions, making them suitable for identifying point anomalies in simpler datasets
  
• Machine Learning Models: Algorithms such as Isolation Forests, clustering, or Support Vector Machines adapt dynamically to data distributions, effectively detecting point and context anomalies across complex datasets
  
• Deep Learning Models: Advanced techniques like autoencoders, Generative Adversarial Networks (GANs), and recurrent neural networks (RNNs) are well-suited for detecting collective anomalies in high-dimensional or sequential data, such as video streams or sensor outputs

Selecting the appropriate model depends on the nature of the data, the type of anomalies to detect, and the system’s operational constraints.  

I believe the possibilities of deep learning models, when coupled with federated learning, are truly exciting. This collaborative approach allows these advanced techniques to be trained across multiple decentralized devices or nodes without sharing the data, thus preserving privacy and security—see this Nvidia blog. It can bring invaluable advancements in healthcare, financial services, cybersecurity, and many other fields, where different organizations can collaborate to train a shared model without needing to share sensitive data directly. 

Where the anomaly detection machine learning technique shines and scales up 

However, predicting anomalies is not just about finding the right tool. Often, existing prediction mechanisms must be modernized to maintain accuracy, scalability to multiple use cases, and the ability to process an increasing amount of data. This is the case for NEXI Croatia and its collaboration with CROZ. This example stands out for three core points: 

• Advanced Infrastructure: Tools for efficient long-term feature computation and faster data processing
  
• Enhanced Models: State-of-the-art machine learning models with optimized recall, precision, and fraud detection rates
  
• Real-Time Processing: A scalable architecture using Kafka and Flink for near real-time detection

The integration of OpenShift AI enabled self-service capabilities for data scientists and automated operational tasks and transitioned the system to a microservices architecture. These upgrades boosted fraud detection accuracy, reduced processing times, and ensured scalability. This case is inspiring, as it lets us think about the complexity of such anomaly prediction models in enterprise environments and the many variables to consider. 

What is new on the frontier of anomaly machine learning prediction models? 

One of the core questions behind such machine learning models is how to improve them to produce more robust and interpretable results. My colleagues Damir Kopljar and Vjekoslav Drvar have just published a novel methodology that integrates Active Anomaly Discovery (AAD) with the Isolation Forest algorithm, resulting in a more robust, interpretable detection framework. This new methodology has the potential to significantly enhance the accuracy and usability of anomaly detection systems, aligning their outputs with real-world expertise. 

Their key advancements include: 

• Efficient High-Dimensional Data Handling: Isolation Forest isolates anomalies with fewer splits, optimizing performance on complex datasets
  
• Active Learning for Precision: AAD incorporates user feedback on ambiguous instances, refining model accuracy
  
• Explainability Metrics: The integration of AWS explainability metrics ensures clear post-feedback insights into the model’s behavior

By leveraging user-labeled samples, this approach aligns anomaly detection outputs with real-world expertise, enhancing accuracy and usability. As the first active learning-enhanced tree-based anomaly detection system, it demonstrates significant utility and scalability across diverse applications. 

Are you inspired to dig deeper into machine learning models?  

As we wrap up our exploration of anomaly detection, it is clear that this field is as broad and fascinating as a culinary adventure. There is much to consider, from the scalability of solutions to the trade-offs between computational resources and methods. And the cherry on top? Generative AI (GenAI) is here to spice things up, using embeddings for dimensionality reduction and few-shot learning with tailored prompts.  

As explained in this Databricks community blog, large language models (LLMs) can detect anomalies and explain why specific data points are flagged, adding an extra layer of insight. If you’re inspired to dig deeper, I encourage you to explore these topics further and share your insights with the community. 

This March in Frankfurt, my colleagues and I are hosting AI Business Innovation Sprint in collaboration with IBM. Past editions have spun the AI journeys of banks such as PBZ in Croatia and machinery manufacturers like AGCO in the USA. We aim to help you pick the right tools and techniques for your AI journey, ensuring your projects sizzle with innovative ideas. AI Business Innovation Sprint can help organizations navigate these complexities by providing expert guidance and support. 

Whether you’re a data scientist, engineer, developer, or just curious, remember that anomaly detection is full of surprises — just like a delicious Tarte Tatin.  
 
Bon appétit!