CS534 — Written Assignment 4 — Due Nov 18th

CS534 — Written Assignment 4 — Due Nov 18th
1. L2 SVM
Given a set of training examples {(xi , yi )}N
i=1 , where yi ∈ {1, −1} for all i. The following is the primal
formulation of L2 SVM, a variant of the standard SVM obtained by squaring the hinge loss:
min wT w + λ
w,b,ξ
N
X
ξi2
i=1
s.t. yi (wT xi + b) ≥ 1 − ξi , i ∈ {1, · · · , N }
xi ≥ 0, i ∈ {1, · · · , N }
a. Show that removing the second constraint ξi ≥ 0 will not change the solution to the primal problem. In other words, let (w∗ , b∗ , ξ ∗ ) be the optimal solution to the problem without this set of
constraints, show that ξi∗ ≥ 0.
b. After removing the second set of constraints, we have a simpler problem with only one set of
constraints. Now provide the lagrangian of this new problem.
c. Derive the dual of this problem. How is it different from the standard SVM with hinge loss?
2. Neural network expressiveness
In class, we have discussed that neural network can express any arbitrary boolean functions. Please
answer the following question about neural networks.
a. It is impossible to implement a XOR function y = x1 ⊕ x2 using a single unit (neuron). However,
you can do it with a neural net. Use the smallest network you can. Draw your network and show
all the weights.
b. Create a neural network with only one hidden layer that implements (x1 ∨ ¬x2 ) ⊕ (¬x3 ∨ ¬x4 ).
c. Explain how can we construct a neural network to implement a Naive Bayes Classifier with
Boolean features.
d. Explain how can we construct a neural network to implement a decision tree classifier with boolean
features.
3. PAC learnability. Consider the concept class C of all conjunctions (allowing negations) over n boolean
features. Prove that this concept class is PAC learnable. Note that we have gone through this example
in class but not in full detail. For this problem, you should work out the full detail, including showing
that there exists a poly-time algorithm for finding a consistent hypothesis.
4. Consider the class C of concepts of the form (a ≤ x ≤ b) ∧ (c ≤ y ≤ d), where a, b, c, and d are
integers in the interval [0, 99]. Note that each concept in this class corresponds to a rectangle with
integer-valued boundaries on a portion of the (x, y) plane. Hint: Given a region in the plane bounded
by the points (0, 0) and (n − 1, n − 1), the number of distinct rectangles with integer-valued boundaries
2
n(n − 1)
.
within this region is
2
(a) Give an upper bound on the number of randomly drawn training examples sufficient to assure
that for any target concept c in C, any consistent learner using H = C will, with probability 95%,
output a hypothesis with error at most 0.15.
(b) Now suppose the rectangle boundaries a, b, c, and d take on real values instead of integer values.
Update your answer to the first part of this question.
1