Hadoop과 R을 이용한 분산처리 시스템 구축 및 예제

Hadoop과 R을 이용한 분산처리
시스템 구축 및 예제
한상우, 지현웅, 이수지., 박새롬, 장희수, 이재욱
[email protected]
This document is confidential and is intended solely for the use
<1>
목차
I. Hadoop 소개
II. Rhadoop
III. MapReduce
IV.Examples
<2>
배경
기존의 문제점
1. 저장 불가능
2. 비경제적
3. 엄청난 분석 시간 소요
Hadoop
- 방대한 양의 데이터
- 분산 처리
- 빠른 시간 내에 결과 제공
- Open source
<3>
Hadoop
Hadoop
=
Hadoop
Distributed
File
System
(HDFS)
+
Hadoop
MapReduce
- HDFS : Distributed File System
수천 대의 서버를 네트워크로 묶어 하나의 서버가 보유
하고 있는 파일 시스템 처럼 사용
- MapReduce : Distributed Processing System
각 서버가 저장하고 있는 데이터를 동시에 병렬로 처리
<4>
Hadoop Structure
MASTER
Back-up
Master
Master
SLAVE
Slave 1
Slave 2
Slave 3
Slave 4
- Master Node : Slave node 관리
Namenode(HDFS), Job Tracker(MapReduce) 역할
- Slave Node
: 데이터 저장 및 전달
Datanode(HDFS), Task Tracker(MapReduce) 역할
<5>
Rhadoop?
1. Rhdfs – R and HDFS
2. Rhbase – R and HBASE
3. RMR – R and MapReduce
Rhadoop을 통해서 R 사용자는 Hadoop으로 데이터를 관리, 분석이 가능
<6>
MapReduce
•
Map function processes a key/value pair to generate a set of intermediate
key/value pairs
•
Reduce function merges all intermediate values associated with the same
intermediate key
<7>
Getting Started with RHadoop
• With RHadoop rmr package we could use ‘mapreduce’ function to implement
same calculations to a list of data
• Simple example to double all the numbers from 1 to 100 :
ints = to.dfs(1:100)
calc = mapreduce(input = ints,
map = function(k, v) cbind(v, 2*v))
from.dfs(calc)
$val
[1,]
[2,]
[3,]
[4,]
[5,]
.....
v
1
2
3
4
5
2
4
6
8
10
<8>
Rhadoop Examples
• Map and Reduce functions are defined with respect to data
structured in (key, value) pairs
nums = to.dfs(rnorm(100, 100, 10))
sort.map.fn <- function(k,v) {
key <- ifelse(v < 100, "less", "greater")
keyval(key, 1)
}
count.reduce.fn <- function(k,v) {
keyval(k, length(v))
}
$key
"𝒍𝒆𝒔𝒔"
"𝒈𝒓𝒆𝒕𝒆𝒓"
⋮
"𝒍𝒆𝒔𝒔"
count <- mapreduce (input= nums,
map = sort.map.fn,
reduce = count.reduce.fn)
from.dfs(count)
>$key
[1] "greater" "less"
$val
[1] 45 55
Reduce functions handle data
separately for each key value
<9>
$value
𝟏
𝟏
⋮
𝟏
Simple Simulations for Option Pricing
Call
option
value
•
We assume that asset 𝑺(𝒕) follows the
stochastic differential equation under the
risk-neutral probability:
𝒅𝑺 𝒕 = 𝒓𝑺 𝒕 + 𝝈𝑺 𝒕 𝒅𝑾
Exercise
price
where 𝑾 is the Brownian motion under
Stock
price
the risk-neutral probability
•
We reproduce the future prices of the
underlying asset, and then the future
payoffs to be obtained
•
The sample mean of the discounted
payoffs is the value of the option contract
< 10 >
Example - Option Pricing
• An example for European option pricing :
inp = cbind(S0*rep(1,nTraj), rep(0,nTraj));
inp = to.dfs(inp);
buildTraj <- function(k, v ){
deltaT = T/nPas;
Data-specific quantities are desired
for (i in 1:nPas){
to be stated in terms of data
dW = sqrt(deltaT)*rnorm(length(v[,1]));
v[,2] = v[,1] + r*v[,1]*deltaT + sigma*v[,1]*dW;
v[,1] = v[,2];
}
key <- ifelse(v[,1]-K>0, "call", "put");
value <- ifelse(v[,1]-K>0, exp(-r*T)*(v[,1]-K), exp(-r*T)*(K-v[,1]));
}
keyval(key,value)
price.reduce.fn <- function(k,v) {
keyval(k, mean(v)*(length(v)/nTraj))
}
>
[1] "call" "put"
$val
[1] 6.038343 10.676495
call <- mapreduce(input = inp , map = buildTraj, reduce = price.reduce.fn);
< 11 >
Rhadoop Examples
• Ellapsed time :
-R
#Timestep
\ #traj
100,000
1,000,000
5,000,000
100
0.8
9.8
130.3
500
3.8
48.3
201.3
#Timestep
\ #traj
100,000
1,000,000
5,000,000
100
41.9
55.5
130.6
500
44.3
82.4
237.9
- RHadoop
< 12 >
Example – k-means Clustering
k- Means Clustering
• Simple partitional clustering
• Chooses the number of clusters k
Iterate {
Compute distance from all points to all k-centers
MAP Assign each point to the nearest k-center
Compute the average of all points assigned to all specific kcenters
REDUCE Replace the k-centers with the new averages
}
< 13 >
Example – k-means Clustering
• In Map function..
Input Data
𝑥11
𝑥21
.
.
.
𝑥𝑛1
𝑥12
𝑥22
.
.
.
𝑥𝑛2
dim1 dim2
distance (x1,c1) distance (x2,c2) distance (x3,c3)
distance (x2,c1) distance (x2,c2) distance (x2,c3)
distance (xn,c1) distance (xn,c2) distance (xn,c3)
Output
$key
1
3
cluster #
.
.
.
2
< 14 >
$value
𝑥11
𝑥21
.
.
.
𝑥𝑛1
𝑥12
𝑥22
.
.
.
𝑥𝑛2
Example – k-means Clustering
• In Reduce function..
$key
$value
𝑥𝑎12
1 𝑥𝑎1 1
𝑥𝑎22
1 𝑥𝑎2 1
.
$key
$value
𝑥𝑏12
2 𝑥𝑏11
𝑥𝑏22
2 𝑥𝑏21
.
.
.
.
.
.
.
.
1
𝑥𝑎𝑙 1
𝑥𝑎𝑙 2
.
.
.
.
.
.
.
.
.
2
𝑥𝑏𝑚1
𝑥𝑏𝑚2
.
$key
New
centers
$key
$value
𝑥𝑐11
𝑥𝑐12
3
𝑥𝑐21
𝑥𝑐22
3
.
.
.
3
𝑥𝑐𝑝1
𝑥𝑐𝑝2
.
$value
1
𝑐11
𝑐12
2
𝑐21
𝑐22
3
𝑐31
𝑐32
< 15 >
.
.
.
.
Example – k-means Clustering
< 16 >
Reference
- Jeffrey Dean and Sanjay Ghemawat, MapReduce: Simpli ed
Data Processing on Large Clusters, Google, Inc.
< 17 >