Exploring data frames
Table of Contents
At this point, you've see it all - in the last lesson, we toured all the basic data types and data structures in R. Everything you do will be a manipulation of those tools. But a whole lot of the time, the star of the show is going to be the data.frame - that table that we started with that information from a CSV gets dumped into when we load it. In this lesson, we'll learn a few more things about working with data.frame.
We learned last time that the columns in a data.frame were vectors, so that our data are consistent in type throughout the column. As such, if we want to add a new column, we need to start by making a new vector:
newCol <- c(2,3,5,12)
cats
coat weight likes_string
1 calico 2.1 TRUE
2 black 5.0 FALSE
3 tabby 3.2 TRUE
We can then add this as a column via:
cats <- cbind(cats, newCol)
Error in data.frame(..., check.names = FALSE): arguments imply differing number of rows: 3, 4
Why didn't this work? Of course, R wants to see one element in our new column for every row in the table:
cats
coat weight likes_string
1 calico 2.1 TRUE
2 black 5.0 FALSE
3 tabby 3.2 TRUE
newCol <- c(4,5,8)
cats <- cbind(cats, newCol)
cats
coat weight likes_string newCol
1 calico 2.1 TRUE 4
2 black 5.0 FALSE 5
3 tabby 3.2 TRUE 8
Our new column has appeared, but it's got that ugly name at the top; let's give it something a little easier to understand:
names(cats)[4] <- 'age'
Now how about adding rows - in this case, we saw last time that the rows of a data.frame are made of lists:
newRow <- list("tortoiseshell", 3.3, TRUE, 9)
cats <- rbind(cats, newRow)
Warning in `[<-.factor`(`*tmp*`, ri, value = "tortoiseshell"): invalid
factor level, NA generated
Another thing to look out for has emerged - when R creates a factor, it only allows whatever is originally there when our data was first loaded, which was 'black', 'calico' and 'tabby' in our case. Anything new that doesn't fit into one of its categories is rejected as nonsense, until we explicitly add that as a level in the factor:
levels(cats$coat)
[1] "black" "calico" "tabby"
levels(cats$coat) <- c(levels(cats$coat), 'tortoiseshell')
cats <- rbind(cats, list("tortoiseshell", 3.3, TRUE, 9))
Alternatively, we can change a factor column to a character vector; we lose the handy categories of the factor, but can subsequently add any word we want to the column without babysitting the factor levels:
str(cats)
'data.frame': 5 obs. of 4 variables:
$ coat : Factor w/ 4 levels "black","calico",..: 2 1 3 NA 4
$ weight : num 2.1 5 3.2 3.3 3.3
$ likes_string: logi TRUE FALSE TRUE TRUE TRUE
$ age : num 4 5 8 9 9
cats$coat <- as.character(cats$coat)
str(cats)
'data.frame': 5 obs. of 4 variables:
$ coat : chr "calico" "black" "tabby" NA ...
$ weight : num 2.1 5 3.2 3.3 3.3
$ likes_string: logi TRUE FALSE TRUE TRUE TRUE
$ age : num 4 5 8 9 9
We now know how to add rows and columns to our data.frame in R - but in our work we've accidentally added a garbage row. We can ask for a data.frame minus this offender:
cats[-4,]
coat weight likes_string age
1 calico 2.1 TRUE 4
2 black 5.0 FALSE 5
3 tabby 3.2 TRUE 8
5 tortoiseshell 3.3 TRUE 9
Notice the comma with nothing after it to indicate we want to drop the entire fourth row.
Alternatively, we can drop all rows with NA values:
na.omit(cats)
coat weight likes_string age
1 calico 2.1 TRUE 4
2 black 5.0 FALSE 5
3 tabby 3.2 TRUE 8
5 tortoiseshell 3.3 TRUE 9
In either case, we need to reassign our variable to persist the changes:
cats <- na.omit(cats)
The key to remember when adding data to a data.frame is that columns are vectors or factors, and rows are lists.
We can also glue two dataframes together with rbind:
cats <- rbind(cats, cats)
cats
coat weight likes_string age
1 calico 2.1 TRUE 4
2 black 5.0 FALSE 5
3 tabby 3.2 TRUE 8
5 tortoiseshell 3.3 TRUE 9
11 calico 2.1 TRUE 4
21 black 5.0 FALSE 5
31 tabby 3.2 TRUE 8
51 tortoiseshell 3.3 TRUE 9
But now the row names are unnecessarily complicated. We can ask R to re-name everything sequentially:
rownames(cats) <- NULL
cats
coat weight likes_string age
1 calico 2.1 TRUE 4
2 black 5.0 FALSE 5
3 tabby 3.2 TRUE 8
4 tortoiseshell 3.3 TRUE 9
5 calico 2.1 TRUE 4
6 black 5.0 FALSE 5
7 tabby 3.2 TRUE 8
8 tortoiseshell 3.3 TRUE 9
You can create a new data.frame right from within R with the following syntax:
df <- data.frame(id = c('a', 'b', 'c'), x = 1:3, y = c(TRUE, TRUE, FALSE), stringsAsFactors = FALSE)
Make a data.frame that holds the following information for yourself:
- first name
- last name
- lucky number
Then use rbind to add an entry for the people sitting beside you. Finally, use cbind to add a column with each person's answer to the question, "Is it time for coffee break?"
So far, you've seen the basics of manipulating data.frames with our cat data; now, let's use those skills to digest a more realistic dataset. Lets read in some real data now. For the remainder of the workshop we will play with some data which contains details of the passengers aboard the Titanic when it sunk, which was sourced from the data science competition website Kaggle:
https://www.kaggle.com/c/titanic
The data is stored in a CSV on the GitHub repository used for these training materials, and R can read the file directly from there:
titanic <- read.csv("https://goo.gl/4Gqsnz")
Another type of file you might encounter is tab-separated format. To specify a tab as a separator, use
"\t".You can also read in files from the Internet using a web address, or from a local file using the local directory.
You can read directly from excel spreadsheets without converting them to plain text first by using the
xlsxpackage.
Let's investigate titanic data a bit; the first thing we should always do is check out what the data looks like with str:
str(titanic)
'data.frame': 891 obs. of 12 variables:
$ PassengerId: int 1 2 3 4 5 6 7 8 9 10 ...
$ Survived : int 0 1 1 1 0 0 0 0 1 1 ...
$ Pclass : int 3 1 3 1 3 3 1 3 3 2 ...
$ Name : Factor w/ 891 levels "Abbing, Mr. Anthony",..: 109 191 354 273 16 555 516 625 413 577 ...
$ Sex : Factor w/ 2 levels "female","male": 2 1 1 1 2 2 2 2 1 1 ...
$ Age : num 22 38 26 35 35 NA 54 2 27 14 ...
$ SibSp : int 1 1 0 1 0 0 0 3 0 1 ...
$ Parch : int 0 0 0 0 0 0 0 1 2 0 ...
$ Ticket : Factor w/ 681 levels "110152","110413",..: 524 597 670 50 473 276 86 396 345 133 ...
$ Fare : num 7.25 71.28 7.92 53.1 8.05 ...
$ Cabin : Factor w/ 148 levels "","A10","A14",..: 1 83 1 57 1 1 131 1 1 1 ...
$ Embarked : Factor w/ 4 levels "","C","Q","S": 4 2 4 4 4 3 4 4 4 2 ...
We can also examine individual columns of the data.frame with our typeof function:
typeof(titanic$PassengerId)
[1] "integer"
typeof(titanic$Name)
[1] "integer"
typeof(titanic$Age)
[1] "double"
str(titanic$Pclass)
int [1:891] 3 1 3 1 3 3 1 3 3 2 ...
We can also interrogate the data.frame for information about its dimensions; remembering that str(titanic) said there were 418 observations of 11 variables in the titanic data, what do you think the following will produce, and why?
length(titanic)
[1] 12
A fair guess would have been to say that the length of a data.frame would be the number of rows it has (418), but this is not the case; remember, a data.frame is a list of vectors and factors:
typeof(titanic)
[1] "list"
When length gave us 11, it's because the titanic data is built out of a list of 11 columns. To get the number of rows and columns in our dataset, try:
nrow(titanic)
[1] 891
ncol(titanic)
[1] 12
Or, both at once:
dim(titanic)
[1] 891 12
We'll also likely want to know what the titles of all the columns are, so we can ask for them later:
colnames(titanic)
[1] "PassengerId" "Survived" "Pclass" "Name" "Sex"
[6] "Age" "SibSp" "Parch" "Ticket" "Fare"
[11] "Cabin" "Embarked"
At this stage, it's important to ask ourselves if the structure R is reporting matches our intuition or expectations; do the basic data types reported for each column make sense? If not, we need to sort any problems out now before they turn into bad surprises down the road, using what we've learned about how R interprets data, and the importance of strict consistency in how we record our data.
Once we're happy that the data types and structures seem reasonable, it's time to start digging into our data proper. Check out the first few lines:
head(titanic)
PassengerId Survived Pclass
1 1 0 3
2 2 1 1
3 3 1 3
4 4 1 1
5 5 0 3
6 6 0 3
Name Sex Age SibSp
1 Braund, Mr. Owen Harris male 22 1
2 Cumings, Mrs. John Bradley (Florence Briggs Thayer) female 38 1
3 Heikkinen, Miss. Laina female 26 0
4 Futrelle, Mrs. Jacques Heath (Lily May Peel) female 35 1
5 Allen, Mr. William Henry male 35 0
6 Moran, Mr. James male NA 0
Parch Ticket Fare Cabin Embarked
1 0 A/5 21171 7.2500 S
2 0 PC 17599 71.2833 C85 C
3 0 STON/O2. 3101282 7.9250 S
4 0 113803 53.1000 C123 S
5 0 373450 8.0500 S
6 0 330877 8.4583 Q
To make sure our analysis is reproducible, we should put the code into a script file so we can come back to it later.
Read the output of str(titanic) again; this time, use what you've learned about factors, lists and vectors, as well as the output of functions like colnames and dim to explain what everything that str prints out for titanic means. If there are any parts you can't interpret, discuss with your neighbors!