Large-scale log compressing system based on differential

Large-scale log compressing system based on differential

36 Z1 2015 11 Vol.36 No.Z1
November 2015
Journal on Communications
doi:10.11959/j.issn.1000-436x.2015300
1,2,3 1,2 1,2
(1. , 100093; 2. , 100093; 3. 100049)
!"#$%
&'
()*+,-
./012345
678
9:;6<=>
6?@
ABCDE+FG2HIJ
KLMNOP:Q,RSTUVWCXY
KLMN
OZ[+\]9^_` gzip aRCbcF6M>Fd2 2~10 efb gzip g
hTij 10%+
kNOkKLMkJ
TP302
A
Large-scale log compressing system based on
differential compression
TANG Qiu1,2,3, JIANG Lei1,2, DAI Qiong1,2
(1. National Engineering Laboratory for Information Security Technologies, Beijing 100093, China;
2. Institute of Information Engineering, Chinese Academy of Sciences, Beijing 100093, China;
3. University of Chinese Academy of Sciences, Beijing 100049)
Abstract: The scale of log data produced by the large scale information system is growing rapidly. It leads to the big
challenge of line-speed compressing and saving the large scale log data. By analysis on massive network log data, it is
found that the log data has redundant pattern in terms of log structure and time local similarity in terms of log content. A
differential log compression architecture based on template is proposed. Fine-grained differential compressive strategies
in the architecture can be configured for a special log data. Experimental results show that, compared with gizp, the
proposed log compressing architecture improves 2~10 times’ compressive speed and gain a better compressing ratio
approaching to 10%.
Key words: log; differential compression; fine grain; template
1
!"#$%
!"&'()*+,-./01234
5678 Web 9:45;<=>45;?@AB
!"45C#4567D9:EF;!"G;H
IJK;67LMCNO67P[1~4]QR
SOTUVW !"4567#X67;
YZ[\]^_` !"ab4cde
fg[\hi9: !"jklmnoT
pq`!"4567rstuvwxoT#y
z{|}%~!"€SO12z‚ƒ}
„…†‡*zˆ‰Š67‹F.VWJKŒ
l4567{|45!"D}ŽX
#QR‘’“”;ˆ‰‡VWX•€45
67Dt^ !"–„S—#
˜67•€™š›œUžŸ ¡4567
¢£¤œ¥¦ˆ"“”§¨UF.45
67“”VW8 Linux 45©ª§¨ logrotate
2015-11-11
lmn*[EompqrHsturvwXDA06031000x
Foundation Item: Special Pilot Research of the Chinese Academy of Sciences (XDA06031000)
2015300-1
198
«¦ gzip U{4567F.“”¬­VW#
®D¦ˆ"¯“”°±²4567VW
³'V˜’´²kµ#}¶fˆ"“”°
¯·¸¹z“”´²Q3¯“”°º
‘»J¼4567,½¾·¿À}¶f¯
“”°lXÁ¥ÂêX•€456
7#ÄÅ`Æ{X•€4567“”VWÇ
ÈÉXlÊ˧†#ÌÍ[5]¼ Web 456
7˜Î϶fÐV˜XlÊÑ·¾·h
u}Ò4567aÓÔ*ÕÖ45‹Ö}
×45ÊØ67ÙÓ3ÊØ67TÚÛ¿
&ÊØ67ÜÝÞ¿ßà{aÓà4567
᦯“”°âF}㓔¿ä¶°
&åD{|&Øæç4567èØé“
”°#ÌÍ[6~8]êàëìíîï‡;&Øíîð
ñò“”óôC{ Apache Web 45F.ð
J“”õÌhu¶°ö÷‘}ÊÑp
îDäÌ͋¶°øùœúûüý·q
›DþÆ{| Apache Web 45øù{
æç4567¯·#Kimmo Chu
|4567‹Nd67€x<=
>45“”°[9]#ÌÍ[10]¯
JK DNS 456
7¾·Æ{ DNS 4567‚]';IP
);æç 4 æ67JF.&ؓ”¶
°#ÌÍ[11]hu|¯“”°“”456
7ê°¯
갋J6*
4567JØ45homogeneous
buckets
ßà{&ØØ454567¦
ˆ"“”° bzip2 gzip F.“”#|Ø
45Ï45ÎÚÊÑQR¦
bzip2 gzip “”“”#ÌÍ[12]¦ ! FPGA
"t LZ4 °{È#$%!"‹&:45F.“
”˜Ý' LZ4 “”´¢£¤(bþ !
)`“”Ú*+#,‘45“”§†±²45
VW³'®DV˜¥¤-.&å/1¯·™ð
01|¨23}æ45¿2&¨œ4·&5
6“”óôœúû¿378ړ”óôÆ{&
Ø67æçèÊؓ”óô¿4&569xð
JÁ¥:å˜Â"]45“”S—#
õÌÆ{¥;&åhu}Ò56X•€4
5679xðJ“”<#ä<=ê»J¼
4567ê>?@*}æ4567‹A&a
67hB3€C45“”]žŸ€C‹
36 D67EVW}€CFÆ#˜G“]H
7€CÏIœdJ4567¿ßàÆ{456
7‹&Øæç67ÔzðJ“”óô#õÌ
huX•€45“”!"&01|¾45æ
ç{|&Øæç4567œ¥úû&ØK8
ÚðJ“”óôä!"è9xðJ“”<
Lq¦õÌhu45“”!"¨‘MN“
”´“”Ú*+#
2
4567Oxœ¥D!"P,D¦
Q ¦ syslog 4 5 O x [13] à P RFC3164 •RD#«•}×45 S*ê
TU
;
SVUS 2U3 JWz‹*ê
T D} 6 í « ^ X ƒ W 45 Y Z € 
facility[N·severity¿V\]]'
^¿ 2D¨245Ï#| syslog 4
5•RED}_`•Raz 2\]
Xl45b·®$%ÏcdOx‡º‘e
D!"P,.D#‚¥f4
5!"D*z†3}Ò56OxD{«
F.4#8<=>HIgD¦ NetIQ
hihu WELF 45Ox#Lj"!"‹
èk 21 GB j"4567$67kj"
!"‹<=>lm;ön45;$Óo
345?@AB!"IDS45;VPN 45
pqAB!"45W#¯
{ä67kJK
trÖ45!"V˜¥¤ 3 Ns¾t#
1) 45Ox•uÊÑ#rÖWvwx456
7OxæÑ| syslog RFC3164 •ROxP
y•R45Ox8 WELF#ÖP8ön<=>
45Ox8X 1 ‚z
¿àP}×45
¯{DS<íîíîÛ>U|Û{W
&Ø45!"}˜|Û~|Û{p'J
€;‘Kð8X 1 ‹‚à 3 ×456
7lm<=>45|Û{Jƒ3³O
|Û~3C„q…† IDS 45Jƒ3
J„|Û~3J„#
2) 4567V˜‡Î€x#LX 1 ‹œ
?Ø}æ4567ˆ‰¥ŠÒOxXz˜
Î;‹V˜XlÊ؀x8ön<=>
106017 æç45X 1 ‹Œ 1;Œ 2 .Ð]‘
SASA*Deny IP due to Land Attack from*to*U€
2015300-2
Z1 x¿lm<=>jkÔ45X 1 Œ 3;
Œ 4 .Ð]‘ Sid=*time=*fw=*pri=*ŽU€x#
z‹€x‹„Xzœaϐ„í3‘
&aÏI‡Î€x
#
3) 4567‹b·ÛæçDA›¨‘f
’]'“·ÊÑ·#I{|Ø}æ4567
I¦˜67cdOx;‘ðñ®‚”•~
DÊØ#8&Øi–<=>ƒWj
kÔ4567Ð\]3}—˜™<=>~
š›c;]';MAC );ƒW45;
<=>óô ID „C ¿˜}¨245!"
‹$%b·ÛæçDe›˜œ]'$
%b·ÛD“ÊÑ8]'b·ÛED6
ðñ;ù6 IP )˜š›c‹NdutC#
1 !"#$%&'()*+,-./
!
Apr 29 2015 23:49:55 ASA5585: %ASA-2-106017: Deny
IP due to Land Attack from 10.0.1.5 to10.0.1.6
Apr 29 2015 23:00:09 ASA5585 : %ASA-2-106017: Deny
IP due to Land Attack from 10.0.1.7 to10.0.1.8
id=tos time="2015-04-30 08:01:49" fw=TopsecOS pri=6
type=conn recorder=session src=10.0.1.5 dst=10.0.1.6
proto=tcp sport=49726 dport=10050 inpkt=9
"#
!
outpkt=10
Ž
id=tos time="2015-04-30 15:01:49" fw=TopsecOS pri=6
type=conn recorder=session src=10.0.1.5 dst=10.0.1.6
proto=tcp sport=2450 dport=88 inpkt=5 outpkt=4
$% IDS
Ž
time:2015-04-30
12:11:02;danger_degree:1;breaking_sighn:0;
event:[30061]DNS &'()&'*+,-./01;
src_addr: 10.0.1.5;src_port:58729;dst_addr: 10.0.1.6
3
3.1
Ž
|Œ 2 žË| 21 GB 4567JKΉ
õžhu}Ò|€C45ԟ45
67‹V˜‡Î€xŸxòD8¤#
1íîfield
#íîDíîfld
íîÛvalW|Û{fldλval
z‹SλU
3íîíîÛp'~#}íîXz}
¨245b·8X 1 ‹lm<=>jk
Ô45‹P IP )íî3Ssrc= x.x.x.xU
z
‹ src 3íîP IP
Sx.x.x.xU3íîÛIP
)íîíîÛ~3C„#
245 , log_msg
#}×45 X
z}&!«D‘ íîWíîp'‘
199
êà¡Z;íîJƒθÊ~¿I
log_msg=field1θfield2θŽfieldNz‹fieldi=fldiλvali#
3 45log#45D×45
Wkïlog={log_msgi | ię1,2,3,Ž,M; log_
msgi=field1iθfield2iθ Ž fieldNi; fieldji=fldjiλvalji, j¢1,
2,Ž,N}#
Ö}žJKœ?,‘45OxõÌ
ŸxòD45OxfÊÑQRES{£“”
4567F.¤¥Ãª*4567ÙÓ3•
u45Ox(log)#4567‚‘í‹Dœ¦
§í&]ÔæíQRœ¥¨©Ôæí
†3 λ θ#{|Ø}æ4567œ¥¯
ª
ÜX«x¬úyÌõê­õ®W45ê#
Ø}!"45V˜‡Î€x/
45 íîcWÎDA›+íî
DÊØI log_msg1 log_msg2 ‹\]ÊØ
Ï / fld1λ,θfld2λ Ž θfldN ¿ & Ø J D / val1,
val2,Ž,valN#|R¾tõÌhu45“”!
"*45‹‡Î€x¥íî3¥¯hB3
€Cßà*€CD‡€xL45 ‹ž
Ÿ‡˜45 ‹VW€CFÆ#¨2€CŸ
xò°±8¤#
4 €Ctemplate#}€Cíî
íîÛ~;íîJ€;+íî¥
ä€C ID „
tid
W
I template={tid, λ, θ, fld1,
fld2,Ž, fldN}#
QR | €C 45 ²N Ô^ Oã ³
8¤#
Step1 45ê#
Step2 {´}×45 log_msgi F.8¤
…†#
1) €Cµ‹¶· log_msgi ʬú€C
templatek¿
2) * templatek DíîL log_msgi ‹ž
Ÿ‡* templatek €C ID ¸?45 ‹‚
àb_|€CaÓà log_msgi 3Stidkθval1iθval2 i
θŽθvalN iU
#
{|“”à4567H7€CFƯ
¹
.|€Cº²N…†Iœ»dJ¼4567#
3.2 |€C4567Nà±²45VW
³'®½V˜Xl ‡yzDœ]'
Øæ45 íîp'V˜“ÊÑ·#3F
2015300-3
200
}㓔4567³'õÌ45ðJ<
{|€C²Nà4567âF}ãK8Ú
ðJ“”#=êD-æ¯ðJ“”óô
diff_strgy¿ßàH7&ØíîÛ¾·¨©
+íîۂ¾ïðJóô{|€C²Nà
4567F.íîTðJ“”#3564
5 “”45ðJ“”E¨©Ö}×
Øæç ¡45 F.ðJ¿|45
¨‘]'“ÊÑ·‚¥äðJ“”óô
Ý'45“”]'³'’´#
3.2.1 ´  í î K 8 Ú ð J “ ” FFDE, finegrained field differential encodingœ¥°±3}
š›c/ffde=( fld, fld_type, diff_strgy, initVal, size)
š›c‹+b·D8¤#
fld/íî#
fld_type/íîÛæçJ3í¿;u6 2 X
æz‹u6J3 8 ¯16 ¯32 ¯64 ¯‘
„ˆ„u6¿À.6Ùò3F6Á6 2
u6Xz#
diff_strgy/íîÛðJóôõ45“”!"
D 4 æðJóô¨2DÂX 2{|&Ø
¾·íîè&ØðJóôrœ4ÃðJó
ô¥56Ãæçíî#
2
23
4
5
678const9
:7;<=6>?@ABCD@A7EF6G
HI@AJBKLMEFNO@A7PQLM
RISEFTU
UV8copy9
WX@A78val9YXZ[\]^_`ab
@A78val'9c\MPEFNOWX@A7de
fghD@A7
78delta9
ijWX@A78val9YXZ[\]^_`a
b@A78val'978∆9
BakP∆=val i –val'd
a@lm7n:c\op@lmPq ∆ r@
lst>\@lumBvaMw@lm7P
val=20150501001324, val'=20150501002326Pf
∆=val i –val' =1224
qx8other9
yz{:|6G23@A7}~BS
€‚ƒd„qx…†‡Pˆ‰Š@A
‹ŒŽ
initVal/íîļÛ#Ռ}×
ðJ“
”]* initVal †3 ¡Ûval'ÕÖíîÛ
âðJ-#ÕíîðJóô3SÛU]Ü initVal
b·ÛD3äíîÛ#
size/Å@¦ðJóôb_íîðÛ¦
TÆÇÆÇ3 size ížÈDaTÆÇ
36 #TÆÇIÝ6J¼Û¿aTÆÇè
size=0
Q¦ LEB128[14]aTÆÇI}í
ž 7 bit Xz"É67‚¾¯Å@ÕÖíž
DÊ3aT67‚à}íž8 0 XzÎË#
{|ðJ“”àíîÛBÛRÌA8 IP
)}îÛRÌ3 0~255
›C| 1 íž
X| 4 ížTÛèTÆÇÍpèa
TÆÇ#$霥¦VW³'«_‚*#8}ˆ
„ 32 3u6ÛèaTÆÇ]ÕzÛ
'3[0, 127]ÎÏ}Û]ESO}ížV
WXzq&D 4 íž#
3.2.2 ðJóô¸¹íîÛ¨‘ 3 ÒSÛV˜U/
0/V˜;&V˜;×!V˜#SV˜UXzíî
S&
ðJpàÐD‘Û8SðÛUíîðJóô¿
V˜UXzíîðJàÐDº‘Û8SÛUð
Jóô¿S×!V˜UXzíîðJà07×!œ
ёÛrœÑˆÛ#8íîðJóô3SdÔU
ÕÖ}×45 ÛÊØ]ÕÖ45ž
ŸäÛðJàíîˆÛ¿ÍpÜÝÞäÛð
Jàíî‘Û#QR}×ðJà45 ‚\
]íîÛ6r&A#3ªeGÇ45 }×ðJ“”45 SOVW}íîÛV
˜¯ÒglFVPB, field value presence bitmap
FVPB ‹´}¾{45‹}¨‘S×
!V˜U/0íîû¯XzíîÛV˜¿d¯
XzíîÛ&V˜#{|SV˜US&V˜U
íîÛÜ&SO FVPB Åz#
3.3 Îï 3.2 ž 3.3 žõžÓu|€C45
67K8ÚðJ“”Ð2<#45ê¯
€C²
Nßàá{45íîÛF.K8ÚðJ“”#
$ÔPНD˜45ðJ“”€C‹D8¤#
5K8ÚðJ€Ctemplate′
#}K
8ÚðJ€CíîÛV˜¯Ògl;+íî
K8ÚðJš›c;€C ID „;íîíîÛ
~íîJ€WI template'={ffdetid,λ,θ,
ffde1,ffed2,Ž,ffedN }#z‹€C ID „r†345
Õy}Ö¯íîffdetid¦SdÔUð
Jóôœ×ù~ØØæç45Ÿ‹€C ID VW#|K8ÚðJ€C45ðJ“”°
°±3° 1#
íîðJ“”DÕÖÛvalØæçÖ}
2015300-4
Z1 ×4567{íîÛ(val')¹.ðJ“”QR
u45ðJ“”
YSOÙÚ}ðJíÛ
|12Ö}×Øæç4567íîÛ#° 1
Œ 1 .3´€CD}ðJíÛdict[k][0Ž
N]zļÛ3{€CíîļÛinitVal
¿
´¹.®}×45 ðJ-ÕÖ45
íîÛÃäðJíÛ° 1 Œ 6 .
#{|´
}×45 =êSOe¾ïz€Cßà
H7€C{ä45 F.K8ÚðJ“”°
1 ‹Œ 3~7 .
#
1 |K8ÚðJ€C45“”°
‰?/45 log={log_msgi |i=1,2,Ž,M}; ðJ“
”€Ckï temp_set={template'k|k=1,2,Ž,K};z‹
template'k={ffdetidk,λ,θ, ffde1k,Ž,ffedNk}, ffedjk= (fldjk,
fld_typejk, diff_strgyjk, initValjk, sizejk), j=1,2,Ž, N };
‰u/ðJ“”4567;
1) ļò€CíîðJíÛ dict[k][0ŽN]={k,
initVal1k,Ž,initValN k };
2) {|´45 log_msgi¹.8¤…†¿
3) H745 ζ·zÊ{€
C template'k¿
4) {€C ID „k log_msgi ´íîÛ
i
valj ¹.8¤…†¿
5) 07 diff_strgyjk { valji dict[k][j]F.ð
J-¿8ÜÕíîðJÆÇ/03S×!V˜U
›ðJà‘ÛÜ FVPB[j']=1;ÊÜ FVPB[j']=0¿/*j'
3ÕÖíî˜íîÛVW¯Ò‹¾¯*/¿
6) ÃíîÖÛíÛ dict[k][j]=valj;
7) ‰u FVPB +íîðJàÛ#
4
3
201

[
/KB
fw_log(1w9
1‘
607
fw_log(10w)
10 ‘
6 071
fw_log_(20w)
20 ‘
12 143
fw_log(150w)
150 ‘
94 831
{$ 4 67kJ¦õÌhu|€C
K8ÚðJ“”<¯“”§¨ gzip F.
“”B쨩 gzip F.äåDQ3¦É
P45©ª§¨ logrotate ¦æD gzip †3z4
5“”°¿q›æÑ45“”ÌÍ[6~9]">
{rD gzip ʙ#2 “”§¨{|+6
7k“”]'“”´J8Ò 1 Ò 2 ‚z
| gzip œ¥ûz“”ä6Û眥û3
;“”´€x
“”€xÚ;²“”´
Úç;“”´Ö¯“”“”€x“”Ú
‚¥˜"ÉBì‹
“”´è|Ö 2 Ҁxp'
{|Ø}67kJ¦ 3 Ò&Ø gzip €xF
.“”Bì#LÒ 1 œ?õÌhu|€Cð
J45“”!"˜“”Ú;Ö¯€x
gzip 2~5 铔´€x gzip }6
lTÒ 1 ‹]'êë3{6ìÚ
#
õž¯
Bìj"
!"4567JK
õÌhu56X•€45“”!"’´#^O
ÝÞz“”´“”Ú#
">ßà/êá3 Intel Core I3-3240V
2 GBFedora-14 …†!"#
|huðJ<D9x§†¶xIE
Ö}×45 âðJQR“”Ð]â4
5•€WªË!š•€67kBìIœãM
ä<“”Ú#3b_Bì67k“
”´“”ÚõžLj"lm<=>jk
Ô4567k‹huJíî
cW 4 67k
67k¾·°±8X 3 ‚z#
’1
Mwa“
LÒ 2 œ?õÌhu|€C45ðJ
“”<“”´Ð²| 3 Ҁx¤ gzip “”
´›“”_«3 10.5%#
Ò 3 ° ± õ Ì ‚ h ¶ ° gzip { |
fw_log(150w)67k˜“”Ú“”´ 2 ¶f
u2{#z‹í/ÒXz“”´îïðñŸÒ
Xz“”]'#Ò 3 XMõÌhu45ðJ“
”< gzip ʨ‘MNÚ*+ 2~10
éØ]“”´r*| gzip#$DïªQ3
2015300-5
202
|€CðJ“”»J¼4567,½
¾·˜“”Öæ(b4567ê>?@€
Cq¯ gzip §¨º‘$%ê>?@#
36 International Conference on Advanced Research in Computer Science
Engineering Technology (ICARCSET 2015)[C]. 2015. 1-6.
[5]
SKIBIŃSKI P, SWACHA J. Fast and efficient log file compression[A].
Proceedings of CEUR Workshop of 11th East-European Conference on
Advances in Databases and Information Systems(ADBIS 2007)[C]. 2007.
[6]
GRABOWSKI S, DEOROWICZ S. Web log compression[J]. Automatyka/Akademia Górniczo-Hutniczaim Stanisława Staszicaw Krakowie, 2007, (11): 417-424.
[7]
DEOROWICZ S, GRABOWSKI S. Efficient preprocessing for Web log
compression[J]. International Journal of Computing, 2008, 7(1): 35-42.
[8]
DEOROWICZ S, GRABOWSKI S. Sub-atomic field processing for
improved Web log compression[A]. Proceedings of IEEE International
Conference on Modern Problems of Radio Engineering, Telecommunications and Computer Science[C]. 2008.551-556.
[9]
HÄTÖNEN K. et al. Comprehensive log compression with frequent
patterns[A]. Data Warehousing and Knowledge Discovery[C]. 2003.
360-370.
’2
[10] œž, œŸ, g¡. Z¢£¤ DNS ¥~[J]. ¦¥
”a“
§¨‹, 2010, 36(15): 32-35.
WANG Y F, WANG Z, YAN B P. High efficient DNS log compression
algorithm[J]. Copular Engineering, 2010, 36(15): 32-35.
[11] CHRISTENSEN R. Improving compression of massive log data[EB/OL].
http://www.erg.utal.edu, 2013.
[12] JANG J H, et al. Accelerating forex trading system through transaction
log compression[A]. SoC Design Conference (ISOCC), 2014 International[C]. IEEE, 2014. 24-75.
[13] LONVICK C. RFC 3164: The BSD Syslog Protocol[S]. Network
Working Group.
[14] LEB128 [EB/OL]. http://en.wikipedia.org/wiki/LEB128, 2015.
’3
IY gzip H”8•–’9Y
—p8˜™š9›a“
5
|€C9xðJ45“”!"=êŸ4
567‹‡Î€x#z—¼4567˜]
';“ÊÑ·¯
úû¾|45Ïb·
íîðJóô¹.ðJ“”F}ã±²4
567VW³'#|è9xðJ“”¦õÌ
‚h45“”<¨‘Ns“”Ú*+#ðJ
óôœúû·¦ä<¨‘¯·œ4·ä
¶°œ¥|}ò45“”#
1985-,
!"#$%&'()*+
1984-,-./
01
2 !"#$%&
'()*+
[1]
[2]
[3]
[4]
YEN T F, et al. Beehive: large-scale log analysis for detecting suspicious activity in enterprise networks[A]. Proceedings of the 29th
Annual
Computer
Security
Applications
Conference[C].
2013.199-208.
BREIER J, BRANIŠOVÁ J. Anomaly detection from log files using
data mining techniques[A]. Information Science and Applications[C].
2015.449-457.
DUMAIS S, et al. Understanding user behavior through log data and
analysis[A]. Ways of Knowing in HCI[C]. 2014. 349-372.
SRIVASTAVA M, GARG , MISHRA P K. Analysis of data extraction
and data cleaning in Web usage mining[A]. Proceedings of the 2015
2015300-6
1975-34567
8
9
2&':;<=>
1%&'?@=ABC*+