英文原文

Chapter 93

Implementation of an Integrated Log Analysis System Through Statistics-Based Prediction Techniques

Kwangman Ko

Abstract Integrated log analysis systems, which could collect, store and analyze a large volume of log and big data in real time by analyzing firewall logs, continue to expand their applications to a variety of fields such as abnormal network behavior detection, use pattern analysis with web log analysis, fraudulent order analysis and detection for internet shopping malls, inside information leakage analysis and detection. This paper presents a result of designing and implementing an prediction engine applying statistics-based log analysis(regression analysis, time-series analysis, cluster analysis and discriminant analysis etc.) technologies, which could overcome problems of trying to implement with GNUR, mathematical and statistical libraries, for finding preemptive action through concentrated

guard during an expected security accident time period by analyzing and predicting security-related infra logs.

Keywords Integrated log analysis _ Statistical prediction analysis _ Security _ Big-data analysis

93.1 Introduction

As the computer communication environment is advanced at a rapid pace and its performance is improved, a diversity of security-related issues has been presented as a serious problem. In particular, integrated log analysis systems, which could collect, store and analyze a large volume of log and big data in real time by

analyzing firewall logs, continue to expand their applications to a variety of fields such as abnormal network behavior detection, use pattern analysis with Web log analysis, fraudulent order analysis and detection for Internet shopping malls, inside information leakage analysis and detection [1, 2].

The log analysis products are divided into domestic simple security management ones [3], foreign-made log analysis ones and the existing ESM solution ones to supply at home and abroad, and it is expected that the ESM, which could not store raw logs, would be losing its existence value in the market as the laws and regulations requiring to store raw logs are increased [4].

In addition, it is considered that domestic simple security management products, which could store raw logs but their correlation analysis functions are

insufficient, could not also get out of low-price centered markets. Domestic logrelated market is divided into high-priced integrated log analysis one and lowpriced simple log management one, for the former, domestic and foreign-made

products are competing, and for the latter, domestic companies are also facing a competitive situation.

Anymon Plus [3], an integrated log analysis product, overwhelms its competitors’ products in PoC thanks to the world’s top real-time big data collection,

storage and analysis performance (processing 40,000 events per second), and is

emerging as the best product in the market. In addition, Anymon Plus is an ESM product storing raw logs, and is growing to a product that could conquer both log analysis and ESM market at the same time because it provides dynamic analysis functions as excellent as foreign-made products.

This paper presents a result of designing and implementing an prediction engine applying statistics-based log analysis (regression analysis, time-series analysis, cluster analysis and discriminant analysis [5] etc.) technologies, which could overcome problems of trying to implement [6] with GNU R [7, 8], mathematical and statistical libraries, for finding preemptive action through concentrated guard during an expected security accident time period by analyzing and predicting

security-related infra logs. This could help actively use to develop solutions which could preemptively respond to security accidents and threats and be introduced general-purposely by small and medium sized businesses.

This paper is organized as follows. Section 93.2 introduces the design of a

system model forming the basis of a statistics-based integrated log analysis system to be developed in this paper and a statistics-based prediction technique to be implemented. Section 93.3 describes how to implement detailed component modules’ specific functions and core prediction algorithms of the integrated log

system, and then presents the result of an experiment. Section 93.4 draws a conclusion and explains about the future study.

844 K. Ko

93.2 Based Works

93.2.1 Integrated Log Analysis System

The existing typical integrated log analysis system is mainly composed of a center manager and a site manager as Fig. 93.1.

The collection engines of center and site managers gather various logs from

security equipment, servers and applications that are clients’ logs to be collected. For this case, agents are loaded, and the site manager has a structure capable of distributed collection, storage and analysis because it could be expanded in parallel unlimitedly.

93.2.2 Design of the Integrated Log Analysis System

The statistics-based integrated log analysis system, which is ultimately designed and implemented in this study, has a structure as Fig. 93.2, in which how to develop statistical prediction engines is practically classified into four main methods.

The first method to develop the statistical prediction engine is an attempt to

connect or share interfaces with the existing products to interoperate with GNU R, which has an advantage of using a statistically and completely proven tool, but involves inefficiency of real-time processing due to a problem of interworking and operation speed. In addition, it excessively occupies system resources such as CPU and main memory, and there are insufficient interfaces for source data and output, so it involves difficulties of development.

Fig. 93.1 General integrate-log analysis system

93 Implementation of an Integrated Log Analysis System 845

内容需要下载文档才能查看

The second method is an attempt using mathematical and statistical libraries, which has an advantage of generality in addition to developing the prediction engine.

However, it should accept an expensive price policy for each server, and a difficulty of implementing additional logic is presented as a disadvantage. The third method is an attempt to purely develop a new statistical prediction engine, which could secure all the sources related to the implementation and it has an advantage of optimized modification considering the characteristics such as operating systems and development languages as necessary. However, there is a disadvantage of requiring considerable resources and time needed for development. Finally, an attempt to develop it based on a mathematical standard library could minimize verification by using a standardized library and the standard

library could be substituted for parallel process etc. In addition, it has a feature that could be concentrated in developing the prediction engine to minimize the

development period. However, it should be solved the consideration for the license issue that may be encountered when selecting an open-source library and the problem of expense that may be arisen when using additional libraries.

Fig. 93.2 Overview of statistic-based prediction engine system

内容需要下载文档才能查看

846 K. Ko

93.3 Statistics-Based Prediction Engine System

93.3.1 Generation of the Prediction Model Based

on Statistics

In order to periodically analyze a large volume of the collected log data to generate a model for analyzing and predicting abnormal data, it was applied the analysis flow as Fig. 93.3.

For the basic log data, it was applied the whole data in which the level that

could determine changes of data by the period is at least three weeks, and it was targeted at the log data for each specific IP and the specific time zone logs in order to decide whether or not to be abnormal data.

93.3.2 Integrated Log Analysis System Modules

This paper developed a mathematical standard library, in which modules could be self-replaced gradually based on open-source libraries, to design the statisticsbased prediction engine as Fig. 93.4 and to implement respective modules.

? Data import module: Import data with xls, csv and text formats into the internal memory

? Data editing module: Remove nulls of the data imported, convert characters into

numbers, and set additional variables

? Basic stat module: Compute basic statistics for the data processed (average, maximum, minimum, standard deviation and variance etc.)

93 Implementation of an Integrated Log Analysis System

Fig. 93.3 Flow of statistic-based prediction model generation

? Matrix module: Compute inverse matrix required in the regression analysis process

? Statistics module: Compute basic statistics (use in parameter estimating) ? Parameter select module: Select variables to be statistically analyzed

? Linear regression module: Compute linear regression analysis

? Parameter estimating module: Compute statistics such as ANOVA and R^2 to evaluate the regression analysis results

? Variables transform module: Transform linear variables into various functions (x -[1/x, log x, x ^ n etc.)

? Residual analysis module: Understand whether there is cross correlation (normal, equal variance or independence) or not between the parameters obtained

? Plot module: Output graphs for various statistics

? Distribution function module: Compute distribution functions needed when finding ANOVA etc.

? Outlier analysis module: Extract/remove outlier data through various algorithms