Neural networks, with their ability to learn behavioural patterns from arbitrary data, seem like a natural way to deal with intrusion detection. There are many academic papers on the topic which report good performance and an even better potential.
The question is, are there any real-life implementations? Is there a single intelligent firewall, or a firewall module, or some other sort of an intelligent intrusion detector that actually uses NNs?
There has been an enormous amount of research into using machine learning techniques for anomaly detection, i.e., to scan network traffic and detect intrusions. However, this research has had very little practical impact. These techniques have seen little deployment and are rarely used in practice.
Why not? There are a number of reasons.
First, these systems tend to have a high false alarm rate. They often raise multiple spurious alarms per day (sometimes even dozens per day), which takes up system administrators' time. This is a fundamental challenge for anomaly detection systems, because they suffer from the "needle in a haystack" problem: billions of packets traverse your network every day, and almost all of them are benign. If the algorithm has a false alarm rate as low as 0.1%, that's still thousands of packets being spuriously flagged. To be practical, the anomaly detection algorithm needs to have an exceptionally low false alarm rate, which is very challenging to do well -- for the same reason that it is very difficult to detect terrorists in airport screening, without introducing a lot of false alarms that cause everyday folks to have to be searched.
Second, anomaly detection systems tend to be not very robust. They focus on detect unusual or novel patterns in your network traffic: anything out of the ordinary. The consequence is that, any time something changes about your network, no matter how benign, they tend to raise alarms. Did your website just get slashdotted? Blam, spurious alarms go crazy. Did some user install a new application that plays novel NAT traversal games? Blam, here come the spurious alarms. Did someone just install IPv6 for the first time? Blam. Someone connect a new mobile phone with a wonky TCP/IP stack, that sends out broken packets? Blam. You get the idea.
If you want to read more about the challenges of innovation in this area, I would recommend the following research papers:
Outside the Closed World: On Using Machine Learning For Network Intrusion Detection. Robin Sommer and Vern Paxson. IEEE Security & Privacy 2010.
The Base-Rate Fallacy and its Implications for the Difficulty of Intrusion Detection. Stefan Axelsson. RAID 1999.
The problem with ML is with the training. Over-training leads to matching the exact training set, making the learning non-generic.
Since my current employer develops scientific computing libraries for Python, I can point you to map-reduce on Disco for finding common clusters in log files: https://github.com/JensRantil/disco-slct
That's not a NN, but it is a way to analyze the data.
With the help of SLCT, one can quickly build a model of logfile(s), and also identify rare lines that do not fit the model (and are possibly anomalous).
I doubt you can find any commercial product since this domain is highly commercialized and their is almost no open source implementation available and most of the work is done in is closed ecosystem. There has been a discussion on this topic you can find at link The only open source tool i found was OSSEC it is a Host-Based Intrusion Detection and there is a recent research on integrating it with AI techniques and there is also a book on it you can find interesting.
As AI is my primary job, I can tell you straight away that the NN alone is useless solution for dynamic architecture.
What you are looking for, is the automation between NN <=> Operator.
This works the following way, for example:
There are no existing applications like you say because it's not done this way, it doesnt have future and it's inherently bad design to use NN to make dynamic architectures.
What is used is actually Bayesian classification as well other heuristic methods. Example implementation is Cloud Exchange by microsoft. It is also using multi-layer architecture.
If you really want to try hard and go the scientific way, you need to analyse each security layer in real-time and make the workflow with the operator.
This forum is not either good place to ask such questions, as there are no experts in AI, or even not engineers, you can try actually MSDN which is better place.
Found yet another paper on the topic: "Anomaly-based network intrusion detection: Techniques, systems and challenges", by García-Teodoro et al, 2008.
