Intrusion Detection by Different Machine Learning Techniques

Intrusion detection is an important part of modern computer security systems. An intrusion detection system should be able to decide if a certain user operation is abnormal. Then it should be able to decide if such abnormal behavior is an intrusion attempt according to different features such as protocol used and connection length.

Our project object is to compare and contrast the performanc (speed and accuracy) of different machine learning algorithms to detect intrusion attempts. We also want to explore optimization options for those algorithms. The algorithms we chose for this projects are: Logistic Regression, Linear SVM, rbf Kernel SVM, and MLP. For each of the algorithms, we run both binary classification and multi-class classification.

The data we used was the KDD(Knowledge Discovery and Data Mining) Cup data by Special Interest Group in 1999. The dataset contained 41 features and 22 types of attack. We used 10 percent of the training set for a total of 500,000 samples and split the dataset into 75% training data and 25% testing data.

For binary classification, all we cared was that if the action was an intrusion attempt or not. The algorithms will only tell if an action was safe and will not decide what kind of intrusion attempt it was. Our results were mainly represented by confusion matrices. Class 0 means normal activity and class 1 means intrusion attempt.

We can see that all methods have very high accuracy. We also listed other interesting results for discuss.

As we can see from the table, all algorithms did great job for binary classification. This means that the dataset is likely to be linearly separable. Non linear classifiers performs consistently better than linear classifiers but also has much longer training time, especially rbf-SVM.

In multi-class classification, the algorithms will try to find out what the actual intrusion attempt was instead of just telling if the behavior is an intrusion. The accuracy in this section is calculated according to whether the algorithm returns the correct intrusion method. Confusion matrix is not as helpful for multi-class classification and is thus not provided.

Because our dataset has a rather big feature space of 41 dimensions, we decided to use Principle Component Analysis to reduce the dimension. The goal of PCA is to find k dimensional representation that preserves maximal variance. Thus we will be able to speed up the training process without losing too much accuracy.

We set our target variance capture rate to 99% and we managed to reduce the dimension from 41 to 17 while keeping 99% variance.

From the table, we can see couple of interesting changes and improvements. First of all, algorithms accuracy were affected by the PCA but the change wasmostly nominal. Second, rbf-SVM still has the best accuracy and PCA greatly reduced its training time to 1/20 of its time before optimization. This shows promise for PCA optimization and we will discuss PCA's effectiveness on multi-class classification in next section.

The table provided similar results as the binary classification: slightly lower accuracy, faster training (rbf-SVM sped up to 1/20). Another interesting point: linear SVM had much lower accuracy and much higher training time. This is likely due to the reduction in dimension: linear SVM was unable to separate the data efficiently.

We found that all four models we used yield very high accuracy while nonlinear models yield slightly higher accuracy than linear ones. rbf kernel had the highest accuracy but had the slowest training time. We also found that PCA can help reduce the training time with minimal negative affect on accuracy.

The algorithms and dataset can also be used for binary classification based on attack types (i.e. identify a specific kind of intrusion attempts). We can also remove the labels from the dataset and us it for unsupervised learning such as clustering (e.g. K nearest neighbors).