Homework 5

EE103, Fall 2014-15
S. Boyd
Homework 5
Don’t forget to include a print copy of your Julia code where required.
1. Weighted least-squares. In least squares, the objective (to be minimized) is
2
kAx − bk =
m
X
(˜
aTi x − bi )2 ,
i=1
where a
˜Ti are the rows of A. In the weighted least-squares problem, we minimize the
objective
m
X
wi (˜
aTi x − bi )2 ,
i=1
where wi are given positive weights. The weights allow us to assign different weights
to the different components of the residuals.
(a) Show that the weighted least-squares objective can be expressed as kD(Ax − b)k2
for an appropriate diagonal matrix D.
(b) The least-squares approximate solution is given by xˆ = (AT A)−1 AT b. Give a
similar formula for the solution of the weighted least-squares problem.
2. Hello World for LinearLeastSquares. (‘Hello World’ is the standard name for the first
program you write using a computer language or package. It typically prints the
message ‘Hello World’ to the screen.) Generate a random 10 × 5 matrix A and a 10vector b. Use Julia to compute the solution xˆ of the associated least-squares problem
using the methods listed below, and verify that the solutions found are the same (or
more accurately, very close to each other). Be sure to submit your code, including the
code that checks if the solutions are close to each other.
• Using the formula xˆ = (AT A)−1 AT b.
• Using the pseudo inverse: xˆ = A† b.
• Using the Julia backslash operator.
• Using the Julia package LinearLeastSquares.
3. Another left inverse. Suppose the m×n matrix A is tall, and has independent columns.
One left inverse of A is A† , its pseudo-inverse.
In this problem we explore another one.
A1
Write A as the block matrix A =
, where A1 is n × n. We will assume that A1 is
A2
invertible (which need not happen in general).
1
0n×m−n is a left inverse of A.
(a) Show that A˜ = A−1
1
˜ What can you say about the associated
(b) Let b be an m-vector, and let x˜ = Ab.
residual vector A˜
x − b?
4. Moore’s law. The figure and table below show the number of transistors N in 13
microprocessors, and the year of their introduction.
9
10
8
10
7
10
n
year N (transistors)
1971
2,250
1972
2,500
5,000
1974
29,000
1978
1982
120,000
275,000
1985
1989
1,180,000
1993
3,100,000
7,500,000
1997
1999
24,000,000
42,000,000
2000
2002
220,000,000
410,000,000
2003
6
10
5
10
4
10
3
10
1970
1975
1980
1985
1990
1995
2000
2005
t
These numbers are available in the Julia file moore_data.jl In this data file, t is the
first column of the table (introduction year) and N is the second column (number of
transistors).
The plot gives the number of transistors on a logarithmic scale. Find the least-squares
straight-line fit of
(log10 N, t − 1970),
where t is the year and N is the number of transistors, from the given data. This gives
an approximation of the form
log10 N ≈ θ1 + θ2 (t − 1970).
(a) Find the coefficients θ1 and θ2 that minimize the RMS error on the data, and give
the RMS error on the data. Plot the model you find along with the data points.
(b) Use your model to predict the number of transistors in a microprocessor introduced in 2015.
(c) Compare your result with Moore’s law, which states that the number of transistors
per integrated circuit roughly doubles every one and a half to two years.
Hints. In Julia, the function log10 is log10. Plotting code is also given in moore_data.jl.
Call plot_log_fit(t, N, theta) where t and N are values from the data file and
theta is a 2-vector with your values for θ.
2
5. Saving TA time using midterm score prediction. The TAs very carefully graded all
problems on all students’ midterms. But suppose they had just graded the second half
of the exam, i.e., problems 7–12, and used a regression model to predict the total score
on each exam.
The 52 × 12 matrix of midterm scores is available online, but for your convenience, we
have created the data file midterm_data.jl for you.
(a) Find the average and standard deviation of the total midterm scores.
(b) Here is a very simple way to predict the total score based on the 6 scores for
second half of the exam: Sum the 6 second half scores, and double the result.
What RMS prediction error does this simple method achieve?
(c) Find a regression model that predicts the total score based on the 6 scores for
problems 7–12. Give the coefficients and briefly interpret them. What is the RMS
prediction error of your model?
(Just for fun, evaluate the predictor on your own exam score. Would you rather
have your actual score or your predicted score?)
Remark. Your dedicated EE103 teaching staff would never do anything like this.
Really.
6. Julia timing test for least-squares. Determine how long it takes for your computer to
solve a least-squares problem with m = 100000 equations and n = 100 variables. (You
can use the backslash operator.)
Remark. Julia compiles just in time, so you should run the code a few times to get the
correct time.
7. Optimizing a linear function subject to constraints. Consider a linear function f :
Rn → R, given by f (x) = aT x where a 6= 0.
(a) How would you choose x to maximize f (x), subject to the constraint (requirement)
that kx − bk ≤ r? (Here the m-vector b and positive scalar r are given.) The
constraint means that we must choose x to be no farther than r from the vector
b.
(b) How would you choose x to maximize f (x), subject to the constraints that Li ≤
xi ≤ Ui , i = 1, . . . , n? (Here the numbers Li and Ui , which satisfy Li ≤ Ui , are
given.) The constraint limits each xi to lie between Li (the lower limit) and Ui
(the upper limit).
In each case, describe the optimal x, give the associated value of f , and briefly justify
why the choice is the best one possible (among vectors that satisfy the constraints).
8. Orthogonality principle.
3
(a) Suppose that A has independent columns, and xˆ is the least-squares approximate
solution of Ax = b. Let rˆ = Aˆ
x − b be the associated residual. Show that for any
x, (Ax) ⊥ rˆ. This is sometimes called the orthogonality principle: every linear
combination of the columns of A is orthogonal to the least-squares residual.
(b) Now consider a data fitting problem, with first basis function φ1 (x) = 1, and data
set (x1 , y1 ), . . . , (xN , yN ). Assume the matrix A in the associated least-squares
problem has independent columns, and let θˆ denote the parameter values that
minimize the mean square prediction error over the data set. Let the N -vector
ˆ Show that
rˆ denote the prediction errors using the optimal model parameter θ.
avg(ˆ
r) = 0. In other words: With the least-squares fit, the mean of the prediction
errors over the data set is zero.
Hint. Consider part (a), with x = e1 .
4