# Outline R Crash Course

## Transcription

Outline R Crash Course

Outline R Crash Course Introduction Part 1: Working with Data How to represent data? Christophe {Lalanne, Pallier} A simple tossing experiment http://pallier.org/cours/stats.2011/ From vectors to matrix and dataframe Cogmaster 2011 Import/export facilities 1 / 50 What is R? 2 / 50 How to interact with R? You will need a “decent” text editor during this course. Let’s not kid ourselves: the most widely used piece of software for statistics is Excel. —Brian Ripley What do we expect from it? Basically, we want good syntax highlight support, to send command to an R process; an internal help display system is a plus. R is a statistical programming language for computing with data. Why R? Not just because it is free and available on all platforms, but because it is developed by an active community of statisticians and end-users enclined to reropducible research. This means good on-line help (mailing-list, wiki, package maintainer, the StackExchange community), continuously updated packages, top-notch algorithms, etc. The general idea will be to write R code, test it in the console, and keep related parts of our statistical progress in dedicated R files. The penultimate idea is to learn to manage a statistical project. See How to efficiently manage a statistical analysis project? on CrossValidated to learn more. 3 / 50 Emacs + ESS 4 / 50 RStudio http://ess.r-project.org/ http://rstudio.org/ 5 / 50 6 / 50 Additional packages Additional resources • Tutorials, textbooks, etc.: See the CRAN docs section, The core R packages might be sufficient in almost all the case study we will encounter. Some recommend add-ons are listed below: http://cran.r-project.org/other-docs.html • Getting help specific to R programming: R mailing, http://stackoverflow.com • Hmisc, a large set of utilities for importing/formatting • Getting help specific to stats (language agnostic): data, power analysis, clustering, tabular output. • rms, everything we ever need for regression models. • plyr, for data manipulation (reshaping, aggregating, etc.). • ggplot2, a high-level visualization backend. http://crossvalidated.com Please read carefully the relevant FAQ before asking questions. To install and update packages, the commands are: install.packages and update.packages (default to CRAN, but take a look at http://r-forge.r-project.org/). 7 / 50 Aims and scope of this course 8 / 50 Let’s get started The objective is to teach you how to do statistical analysis on your own. This means knowing a little bit of the R language itself, as well as having a good understanding of statistical theory and common techniques of data analysis. You’ll need to practice a lot! This session is all about the R language, but with some incursion into statistical concepts like sampling distribution, likelihood theory, parameter estimation. 9 / 50 R knows basic arithmetic 10 / 50 Tail or Head? We can use R as a “simple” calculator. Imagine a simple experiment, one consisting in tossing a coin. > seq(1, 9, by=2)^2 > x <- c(0,1,1,0,1,0,1,0,0) > x[3] [1] 1 9 25 49 81 [1] 1 > x <- seq(-2, 2) > t(x) %*% x * 1/(length(x)-1) [1,] > x[length(x)] [,1] 2.5 [1] 0 > x[2:4] > rm(x) > x <- sample(1:10, 20, replace=TRUE) > x[ceiling(length(x)/2)] [1] 1 1 0 > x[c(2,4)] [1] 6 [1] 1 0 > x[seq(1, length(x), by=2)] [1] 0 1 1 1 0 11 / 50 12 / 50 R vectors More on addressing vector elements 1 Events were stored in a vector that consitutes the base R object. We’ve already seen how to address its values or slicing patterns. What about their internal properties? There are three fundamental types: numeric, character, and logical. Consider the following “named” vector, x: > > > > > is.vector(x) [1] TRUE > is.numeric(x) x <- seq(1, 6) names(x) <- letters[1:6] y <- rep(c(T,F), 3) x[x == 2] == x["b"] [1] TRUE b TRUE > cat(class(x), typeof(x), "\n") > x[y] numeric double a c e 1 3 5 > x <- c(0L,1L,1L,0L,1L,0L,1L,0L,0L) > cat(class(x), typeof(x), "\n") > which(x %in% y) integer integer [1] 1 13 / 50 More on addressing vector elements 2 Missing values We’ll see more dictionnary-like facilities in the sequel of this tutorial. We can use boolean operators too. Some data may be missing. R use NA to denote them. > xc <- x > xc[sample(1:6, 2)] <- NA > is.na(xc) > z1 <- sample(c(T,F), 6, replace=TRUE) > z2 <- sample(c(T,F), 6, replace=TRUE) > z2 [1] TRUE TRUE TRUE FALSE TRUE a TRUE TRUE b c d e f TRUE FALSE FALSE FALSE FALSE > which(is.na(xc)) > z1 & z2 [1] FALSE FALSE 14 / 50 a b 1 2 TRUE FALSE FALSE FALSE > xc[is.na(xc)] <- "." > print(xc) > x[x < 5 & z2] a b c 1 2 3 a b c d e f "." "." "3" "4" "5" "6" 15 / 50 Sorting 16 / 50 Tail or Head again 1 It is also possible to sort vectors, and work with ordered indexes. A better way to simulate this process might be > x <- sample(c(0,1), 100, rep=T) > table(x) > xs <- sample(x) > sort(xs, decreasing=TRUE) f e d c b a 6 5 4 3 2 1 x 0 1 48 52 > xs[order(xs)] > head(ifelse(x==0, "T", "H")) a b c d e f 1 2 3 4 5 6 [1] "T" "T" "H" "T" "T" "T" > rank(rev(xs)) [1] 52 b f c a e d 2 6 3 1 5 4 Is there another way to generate such artificial data? > sum(as.logical(x)) 17 / 50 18 / 50 Probability distribution 1 Probability distribution 2 We could use a very simple probability law, namely the uniform distribution (“all outcomes equally probable”) on the [0, 1] interval and filter those results that are above or below its centre point. Here’s a possible solution: Lookup the on-line R help: > > > > > x <- runif(100) > print(length(x[x < .5])) [1] 52 > apropos("unif") > help.search("unif") > if (require(sos)) findFn("unif") # do it again, we'll get a different result x <- runif(100) xb <- ifelse(x < .5, 0, 1) table(xb) xb 0 1 57 43 19 / 50 Probability distribution 3 20 / 50 The tossing experiment revisited 1 But, we know that such events each follow a Bernoulli distribution, and provided the trials are independent the expected number of tails will follow a Binomial distribution, see Tossing a coin Failure Success 5 ● ● ● ● ● ● ● ● ● ● 6/10 4 ● ● ● ● ● ● ● ● ● ● 7/10 3 ● ● ● ● ● ● ● ● ● ● 5/10 2 ● ● ● ● ● ● ● ● ● ● 6/10 1 ● ● ● ● ● ● ● ● ● ● 3/10 1 2 3 4 5 6 7 8 9 10 > help(rbinom) Sequence Now, we can rewrite our little experiment. > > > + > > n <- seq(10, 100, by=10) res <- numeric(length(n)) for (i in seq_along(n)) res[i] <- table(rbinom(n[i], 1, .5))["1"] names(res) <- n print(round(res/n, 2)) 10 20 30 40 50 60 70 80 90 100 0.60 0.55 0.43 0.42 0.54 0.40 0.50 0.46 0.43 0.46 Trial 21 / 50 The tossing experiment revisited 2 22 / 50 The tossing experiment revisited 3 0.20 Here are two theoretical distributions: 0.15 Please note that these are the results for each single run. Do those results allow to draw a reliable conclusion about the expected number of heads? 0.00 0.00 0.05 0.05 0.10 Density 0.10 0.15 How about increasing n? How about trying this experiment again? How about running it 100 times and calculating the average number of tails? 5 10 No. success 15 20 5 10 No. success 15 20 You tossed a coin 20 times; you got 13 success. Would you say this is a fair coin? Are those results likely to come from one of the above distribution? 23 / 50 24 / 50 Grouping instructions together 1 Grouping instructions together 2 How about reusing the same code several times? It would be easier to call a single command with varying arguments. For that, we need to write a function. Here is a function for the tossing experiment: We’ve already used a small simulation. Let’s look at the following illustration of Marsaglia’s polar method: > > > + + + + + + + > n <- 500 v <- w <- z <- numeric(n) for (i in 1:n) { t <- runif(1) u <- runif(1) v[i] <- 2 * t - 1 w[i] <- 2 * u - 1 a <- v[i]^2 + w[i]^2 z[i] <- ifelse(a <=1 , a, NA) } round(summary(z), 2) Min. 1st Qu. 0.00 0.23 Median 0.46 Mean 3rd Qu. 0.49 0.74 > get.tails <- function(n=5, p=0.5) sum(rbinom(n, 1, p)) with some examples of use: Max. 1.00 > > > > > > > NA's 111.00 get.tails() get.tails(20) get.tails(20, 0.75) get.tails(p=0.75) replicate(10, get.tails(n=100)) sapply(10:20, get.tails) as.numeric(Map("get.tails", n=10:20)) 26 / 50 25 / 50 Writing your own function Some of R RNGs Below are some probability laws that we will use throughout the course. Their parameters are reported in the first column on the left. A function in R is composed of a list of arguments (with or without default value) and a body, which is a series of R instructions. Usually, a value is returned, either the last statement or the one enclosed in a call to return(). Here is an example (what does the function do actually doesn’t matter): > odds.ratio <- function(x, alpha=.05) { + if (!is.table(x)) x <- as.table(x) + pow <- function(x, a=-1) x^a + or <- (x[1,1] * x[2,2]) / (x[1,2] * x[2,1]) + or.se <- sqrt(sum(pow(x))) + or.ci <- exp(log(or) + c(-1,1) * qnorm(1-alpha/2) * or.se) + return(list(OR=or, SE=or.se, CI=or.ci)) + } Binomial B(n, p) rbinom pbinom Uniform U[a,b] runif punif Gaussian N (µ; σ 2 ) rnorm pnorm Student T (ν) rt pt Chi square χ2 (ν) rchisq pchisq qbinom dbinom qunif dunif qnorm dnorm qt dt qchisq dchisq 27 / 50 Your turn 1 2 3 4 28 / 50 Recap’ Convert the Marsaglia’s code to a function (and simplify the code, when possible). Using the code related to the tossing experiment, let’s take n = 50 and generate 100 experiments, store the results, calculate the mean and variance. Simulate a Binomial experiment (0/1), and lookup the index of the first “one”. Using the same n and p, reiterate a large number of times, say 1,000; check whether the variance is approximately 1−p . p2 Simulate data from a Poisson process with a large n and λ < 0.1. Compare to what you would get using a Binomial distribution of parameter λ/n. You should now be familiar with • How to create and manipulate vectors, how to access their elements. • How to access the help system. • How to generate random draws from a discrete or continuous distribution. • How to run simple experiment with a for loop. • How to write little functions. 29 / 50 30 / 50 Beyond vectors Working with matrix Sometimes we need more than a single vector to represent our data. Then, we can rely on matrix. All ways to address vector values works here. > m <- matrix(1:16, nc=4) > m[3,2] > x1 <- rpois(5, 3.8) > x2 <- rpois(5, 3.8) > rbind(x1, x2) [1] 7 > m[,2] [,1] [,2] [,3] [,4] [,5] x1 3 6 6 3 2 x2 3 4 4 2 12 [1] 5 6 7 8 > m[c(1,3), c(2,4)] Or [1,] [2,] > matrix(c(x1, x2), nrow=2, byrow=TRUE) [1,] [2,] [,1] [,2] [,3] [,4] [,5] 3 6 6 3 2 3 4 4 2 12 [,1] [,2] 5 13 7 15 > dim(m) # i.e., nrow(m) x ncol(m) [1] 4 4 32 / 50 31 / 50 Row- and colwise operations Yet another famous distribution We can generate gaussian variates (i.e., realizations from the “normal distribution”) using rnorm(). Say we want to fill a 100 × 10 matrix owith such values (10 draws of 100 observations). Matrix are interesting because we can apply a lot of operations by row or column. For example, > m <- matrix(runif(100), nc=10) > round(apply(m, 1, mean), 2) > res <- matrix(nr=100, nc=10) > for (i in 1:10) + res[,i] <- rnorm(100, mean=12, sd=2) [1] 0.47 0.54 0.54 0.38 0.59 0.58 0.43 0.50 0.46 0.61 > apply(m, 2, function(x) round(mean(x), 2)) A more efficient way to generate those data is to use the replicate() function. [1] 0.57 0.55 0.47 0.36 0.53 0.51 0.61 0.44 0.52 0.52 Matrix objects will only hold values of the same type (e.g., all character or all numeric). Compute the following five-number summary for each column: min, max, 1st and 3rd quartile, median, mean. See help(summary). 33 / 50 Dataframe 34 / 50 More on dataframes 1 Dataframe are matrix-like object (more exactly, they are list) but can hold vectors of different types. Accessing elements from a dataframe might be done like for a matrix, e.g. > d <- data.frame(x=1:2, y=4:1, g=rep(c("m","f"),2)) > dim(d) > d[2,2] [1] 3 [1] 4 3 We can also use named columns: > str(d) > d$y 'data.frame': 4 obs. of 3 variables: $ x: int 1 2 1 2 $ y: int 4 3 2 1 $ g: Factor w/ 2 levels "f","m": 2 1 2 1 [1] 4 3 2 1 > d$g[3] [1] m Levels: f m They can be used to store results of an experiment according to certain conditions, some outcomes together with subjects’ characteristics, etc. There are special operations available for this kind of R object like > summary(d) 35 / 50 36 / 50 More on dataframes 2 x Min. :1.0 1st Qu.:1.0 Median :1.5 Mean :1.5 3rd Qu.:2.0 Max. :2.0 y Min. :1.00 1st Qu.:1.75 Median :2.50 Mean :2.50 3rd Qu.:3.25 Max. :4.00 Factors 1 g f:2 m:2 Factors are categorical variables allowing to establish a distinction between statistical units, experimental conditions, etc. They induce a partition of the dataset. > # class(d$g) > levels(d$g) [1] "f" "m" This is known as the long-format representation. Compare to, e.g. > unique(d$g) [1] m f Levels: f m > m <- matrix(d$y, nc=2) > colnames(m) <- rev(levels(d$g)) > rownames(m) <- unique(d$x) > relevel(d$g, ref="m") [1] m f m f Levels: m f There are many ways to generate factors: 38 / 50 37 / 50 Factors 2 Illustration > print(f1 <- factor(rep(1:2, length=4))) Here are two series of height measurements from two classes: [1] 1 2 1 2 Levels: 1 2 > d <- data.frame(height=rnorm(40, 170, 10), + class=sample(LETTERS[1:2], 40, rep=T)) > d$height[sample(1:40,1)] <- 220 > print(f2a <- as.factor(rep(c("a1", "a2"), each=2))) [1] a1 a1 a2 a2 Levels: a1 a2 220 ● 210 Height (cm) > print(f2b <- gl(2, 2, labels=c("a1","a2"))) [1] a1 a1 a2 a2 Levels: a1 a2 200 ● 190 ● ● 170 ● ● 160 ● ● ● ● ● ● ● ● ● ● ● 10 ● ● ● ● ● ● 0 ● ● ● ● ● ● ● ● ● ● ● ● ● ● 180 20 30 ● ● ● 40 Subject Does the taller come from class A or B? Which class shows more variability with respect to height? 39 / 50 Internal datasets 40 / 50 Importing data 1 R comes with a lot of example datasets. They can be loaded using the data() command. Some of them are directly available from the base packages, others can be imported from additional packages. To list all available datasets, you can try: R can read data from a variety of proprietary format (SPSS, Stata, SAS), “connections”, but also simple tab-delimited or csv file. Here is an example with a tab-delimited dataset: > colcan <- read.table("./data/colon-cancer.txt", header=TRUE) > head(colcan, 3) > data() If you already know the name of the dataset, you can use it directly, e.g. 1 2 3 > data(sleep) > help(sleep) > data(ISwR::intake) stages local local local year n 1974 5 1975 8 1976 10 > table(colcan$stages) local regional 12 12 > with(colcan, tapply(n, stages, sum)) 41 / 50 42 / 50 Importing data 2 Writing data 1 local regional 112 32 Sometimes, it might be interesting to save data or results to disk. We can use quite the same commands, e.g. write.table(). However, here are some possible alternatives to save results to a file: And here is an example with an Excel file converted to csv: > perinatal <- read.csv("./data/perinatal.csv") > str(perinatal) 'data.frame': 74 obs. of 5 variables: > n <- 100 $ weight : Factor w/ 37 levels "<800",">4300",..: 1 36 37 3 4 5 6 7 8 9 ... > d <- data.frame(age=sample(20:35, n, rep=T), $ infants: Factor w/ 2 levels "black","white": 1 1 1 1 1 1 1 1 1 1 ... + y=rnorm(n, 12, 2), $ deaths : int 533 65 40 30 29 21 19 19 20 24 ... + gender=gl(2, n/2, labels=c("male", "female"))) $ births : int 618 131 122 131 137 143 143 165 167 219 ... > cat("Mean of y is:", mean(d$y), "\n", file="out1.txt") $ prob : num 862 496 328 229 212 ... Check your working directory with getwd(). Using save() and load(), you can work with R objects saved in a binary image. Read carefully the default options for read.table(), read.csv(), and read.csv2(). 44 / 50 43 / 50 Writing data 2 Your turn Or, 1 > sink("out2.txt") > with(d, tapply(y, gender, mean)) male female 11.77841 11.92955 2 > sink() 3 Or, > capture.output(summary(d), file="out3.txt") 4 To check whether those files were saved, you can use: > list.files(dir(), pattern="*.txt") 5 The odds.ratio() function expects a 2 × 2 table or matrix. Check that these conditions are fullfilled at the beginning of the function body. Write a summary-like function including the SD in addition to the five-number summary. Load the sleep dataset and summarize the difference in hours of sleep for both groups. Load the USArrests dataset and summarize all numeric variables. Reorder the dataset by numbr of murders. Hint: See help(do.call). Create a fake dataset comprised of two columns of gaussian variates and a factor. Export it as a csv file. 45 / 50 Recap’ 46 / 50 Some words of caution Be careful with floating point arithmetic. (1,2) You should now be familiar with • How to work with matrix and dataframe objects. > .1 == .3 / 3 • How R represents qualitative or (ordered) discrete [1] FALSE variables. • How to avoid looping for row-, col-, and groupwise computations. • How to read external data files. • How to write intermediate results to your HD. > any(seq(0, 1, by=.1) == .3) [1] FALSE Factors are internally stored as numeric values and levels are recorded in lexicographic order. > as.numeric(f <- factor(1:2, levels=2:1)[1]) [1] 2 > as.numeric(levels(f))[f] [1] 1 47 / 50 48 / 50 References 1 List of R commands 1. P Burns. The R Inferno, 2011. URL http://www.burns-stat.com. &, 15 any, 48 apply, 33 apropos, 21 c, 11 capture.output, 45 cat, 13 data, 41 data.frame, 35 factor, 39 function, 26 gl, 39, 45 head, 18 help, 26 help.search, 21 ifelse, 21 2. D Goldberg. What every computer scientist should know about floating-point arithmetic. ACM Computing Surveys, 23(1), 1991. URL http://www.validlab.com/goldberg. 49 / 50 is.na, 16 length, 11 levels, 39 Map, 26 matrix, 31 mean, 33 names, 17 numeric, 26 order, 17 rank, 17 rbind, 31 rbinom, 26 read.csv, 43 read.table, 43 relevel, 39 rep, 15 replicate, 34 rev, 17 rnorm, 34 round, 26 rpois, 31 runif, 21 sample, 11 sapply, 26 seq, 11 sink, 45 sort, 17 sum, 18 t, 11 table, 18 tapply, 43 unique, 39 Last updated: September 28, 2011. LATEX + pdfTEX and R 2.13.1. 50 / 50