Homework 3: Map-Reduce, Frequent Itemsets, LSH, Streams

Virginia Tech.
Computer Science
CS 5614 – (Big) Data Management Systems
Fall 2014, Prakash
Homework 3: Map-Reduce, Frequent Itemsets, LSH, Streams
(due October 29 , 2014, 2:30pm, in class—hard-copy please)
th
Reminders:
a.
b.
c.
Out of 100 points. Contains 4 pages.
Rough time-estimates: 7-10 hours.
Please type your answers. Illegible handwriting may get no points, at the discretion of the grader.
Only drawings may be hand-drawn, as long as they are neat and legible.
d. There could be more than one correct answer. We shall accept them all.
e. Whenever you are making an assumption, please state it clearly.
f. Each HW has to be done individually, without taking any help from non-class resources (e.g.
websites etc).
Q1. Map-Reduce [45 points]
In this question, we will use Map Reduce to figure out the number of 2-grams in a large
text corpus given the all the distinct 4-grams from the text corpus. The idea is to
convince you that using Hadoop on AWS has now really become a low-enough
cost/effort proposition (compared to setting up your own cluster). You can use one of
Java/Python/Ruby to implement this question. You are free to use Hadoop Streaming
as well if you want.
Familiarize yourself with AWS (Amazon Web Services). Read the set-up guidelines
posted on the website to set up your AWS account and redeem your free credit ($100)--do this early!
Link: http://people.cs.vt.edu/~badityap/classes/cs5614-Fall14/homeworks/hw3/AWS-setup.pdf
The pricing for various services provided by AWS can be found at
http://aws.amazon.com/pricing/. The services we would be primarily using for this
assignment are the Amazon S3 storage, the Amazon Elastic Cloud Computing (EC2)
virtual servers in the cloud and the Amazon Elastic MapReduce (EMR) managed
Hadoop framework. Play around with AWS and try to create MapReduce job flows (not
required, or graded) or try the sample job flows on AWS.
The questions in this assignment will ideally use up only a very small fraction of your
$100 credit. AWS allows you to use up to 20 instances total (that means 1 master
instance and up to 19 core instances) without filling out a “limit request form”. For this
assignment, you should not exceed this quota of 20 instances. You can learn about
these instance types by going through the extensive AWS documentations. Of course,
after you are done with the HW, feel free to use your remaining credits for any other
fun computations/applications you may have in mind! These credits are applicable
more generally for AWS as a whole, not just MapReduce.
1 We will use data from the Google Books n-gram viewer corpus. N-grams are fixed size
tuples of items. In this case the items are words extracted from the Google Books corpus.
The n specifies the number of elements in the tuple, so a 5-gram contains five words.
This data set is freely available on Amazon S3 in a Hadoop friendly file format and is
licensed under a Creative Commons Attribution 3.0 Unported License. The original
dataset is available from http://books.google.com/ngrams/.
The subset we will be using for this assignment is the 4-gram English 1M dataset in the
following S3 bucket (directory) and is freely accessible to all:
s3://datasets.elasticmapreduce/ngrams/books/20090715/eng-1M/4gram/data
IMP: Note that the dataset is stored in a SequenceFile format (which is not plain text)--so make sure you look-up on how to read/parse this file in Hadoop (feel free to search
online to figure this out). Some helpful links are given below:
https://aws.amazon.com/datasets/8172056142375670
http://stackoverflow.com/questions/18882197/processing-lzo-sequence-files-with-mrjob
Refer to our setup guidelines to see how to set this data as input to your MapReduce job
(Section 6 in the guidelines). We have provided a screenshot to configure the EMR
cluster, which demonstrates how to access input data from some given bucket (here our
bucket is the one given above).
Q1.1. (10 points) The dataset contains all the 4-grams in the Eng-1M dataset. What is
the total number of distinct 4-grams in the dataset? Write a simple MapReduce
job to compute this.
Q1.2. (5 points) Plot the frequency distribution for the occurrence counts of the 4grams i.e. a plot where the x-axis is the occurrence count (say k), and y-axis is
the number of 4-grams which occur k times. Just paste the figure as the answer.
Hint: It will be easiest if you write a simple MR job to pull out just the
occurrence information from the dataset, and then compute the distribution
locally on your machine.
Q1.3. (25 points) Write a MapReduce job to compute the total number of distinct 2grams using the same dataset.
Q1.4. (5 points) Write down the total number of the 2-grams you get (you may need
another separate MR job for this---no need to show us the code for this---just
write down the number).
Code Deliverables: For Q1.1: Give the mapper and reducer files (in addition to the
number). For Q1.3: Give the mapper and reducer for computing the 2-grams from the
2 dataset. Zip all of these as YOUR-LASTNAME.zip and send it to Vanessa (email:
[email protected]) with the subject ‘HW3-Code-Q1’. Also copy-paste these in your hard
copy.
Q2. Finding Similar Items [30 points]
Q2.1. (15 points) In class, we saw how to construct signature matrices using random
permutations. Unfortunately, permuting rows of a large matrix is prohibitive.
Section 3.3.5 of your textbook (in Chapter 3) gives a fairly simple method to
‘simulate’ this randomness using different hash functions. Please read through
that section before attempting this question.
Now consider matrix below.
a. (9 points) Compute the minhash signature for each column using the
method given in Sec 3.3.5 of your textbook, if we use the following three
hash functions: (A) h1(x) = (2x + 1) mod 6; (B) h2(x) = (3x + 2) mod 6; and (C)
h3(x) = (5x + 2) mod 6. So you will finally get a 3x4 matrix as the signature
matrix. Just show the initially computed hash function values and the
final signature matrix.
b. (6 points) How close are the estimated Jaccard similarities for the six pairs
of columns to the true Jaccard similarities (i.e. give the ratio of the
estimated/true for each of the pairs)?
Q2.2. (15 points) Recall that in LSH, given b and r, the probability that a pair of
documents having Jaccard similarity s will be a candidate pair is given by the
function 𝑓 𝑠 = 1 − 1 − 𝑠 ! ! .
a. (6 points) Show that 𝑠 ∗ =
! !/!
!
is a good approximation to the value of s
when the slope of f(s) is the maximum. Hint: Feel free to use Mathematica if
you want---will save you some time ☺
b. (7 points) Recall that the threshold is the value of s at which the probability
of becoming a candidate pair is ½. Given b=32, r=8, plot a graph of f(s) vs s
(of course s should vary from 0 to 1) and numerically estimate the value of
s when f(s) is ½ (call this value s1). Show the plot here, demonstrating your
computation.
c. (2 points) How does s1 compare to s* (for the same values of b=32 and r=8)?
3 Q3. Stream Mining [20 points]
Q3.1. (10 points) Bloom Filters: Suppose we have n bits of memory available, and our
set S has m members. Instead of using k hash functions, we could divide the n
bits into k arrays, and hash once to each array. As a function of n, m, and k,
what is the probability of a false positive? How does it compare with using k
hash functions into a single array?
Q3.2. (5 points) AMS algorithm: Compute the surprise number (second moment) for
the stream 3, 1, 4, 1, 3, 4, 2, 1, 2. What is the third moment of this stream?
Q3.3. (5 points) DGIM algorithm: Suppose the window is as shown below. (Most
recent bit is on the right)
Estimate the number of 1’s in the last k positions, for k = (a) 5 (b) 15. In each case,
how far off the correct value is your estimate?
Q4. Frequent Itemsets [5 points]
Let there be I items in a market-basket data set of B baskets. Suppose that every basket
contains exactly K items. As a function of I, B, and K, how much space does the
triangular-matrix method take to store the counts of all pairs of items assuming four
bytes per array element?
4