Crashing Hard Drives with data.table

Nov 11, 2017 · 1612 words · 8 minutes read R Docker Linux

A few weeks ago, I accidentally generated 126TB worth of data. Before going home on a Thursday, I submitted a few R jobs to the cluster. When I checked up on them Friday after lunch, I discovered they had generated 126TB worth of text files.

After I had killed the running jobs and deleted the files, I launched a mini-investigation. By some miracle I avoided getting a stern email from IT, but I figured I should avoid making hard drive hogging a habit.

The culprit turned out to be some data.table code I had written and never properly tested. In my eagerness to get results, I had run the same script on several datasets before testing it on one. The script in question was one of the first ones I’d written using the data.table package, and some of my assumptions about its syntax turned out to be horribly, horribly wrong.

To avoid similar incidents in the future, I have gone back to basics and learned the syntax properly. In the process I also recreated my mistake for fun, further driving home the point.

Introduction to data.table

data.table is an R package for handling large datasets. It extends the built in data.frame structure to enable faster operations and updating columns by reference.

The introduction vignette is a great guide for getting started. There are also several vignettes that focus on summarizing and manipulating data, but for this short introduction I’m going to focus on selecting rows and columns.

While the data.table syntax seems similar to data.frame one, there are a few obvious differences. Firstly, data.table treats column names as variables within the [...] operator. Instead of using the dollar sign to select columns, we can refer to them directly by name. Secondly, row selection takes precedence over column selection. For a data.frame, the statement [1:4] selects the first four columns, whereas for a data.table it selects the first four rows.

To see these differences more clearly, consider the NYC flights data from the vignette. The example below shows how to select all flights out of JFK with data.table and data.frame.

library(data.table);

dt <- fread('~/data.fun/flights14.csv');
df <- read.csv('~/data.fun/flights14.csv');

jfk.dt <- dt['JFK' == origin];
jfk.df <- df['JFK' == df$origin, ];

print(jfk.dt);
##        year month day dep_time dep_delay arr_time arr_delay cancelled
##     1: 2014     1   1      914        14     1238        13         0
##     2: 2014     1   1     1157        -3     1523        13         0
##     3: 2014     1   1     1902         2     2224         9         0
##     4: 2014     1   1     1347         2     1706         1         0
##     5: 2014     1   1     2133        -2       37       -18         0
##    ---                                                               
## 81479: 2014    10  31     1705        -4     2024       -21         0
## 81480: 2014    10  31     1827        -2     2133       -37         0
## 81481: 2014    10  31     1753         0     2039       -33         0
## 81482: 2014    10  31      924        -6     1228       -38         0
## 81483: 2014    10  31     1124        -6     1408       -38         0
##        carrier tailnum flight origin dest air_time distance hour min
##     1:      AA  N338AA      1    JFK  LAX      359     2475    9  14
##     2:      AA  N335AA      3    JFK  LAX      363     2475   11  57
##     3:      AA  N327AA     21    JFK  LAX      351     2475   19   2
##     4:      AA  N319AA    117    JFK  LAX      350     2475   13  47
##     5:      AA  N323AA    185    JFK  LAX      338     2475   21  33
##    ---                                                              
## 81479:      UA  N596UA    512    JFK  SFO      337     2586   17   5
## 81480:      UA  N568UA    514    JFK  SFO      344     2586   18  27
## 81481:      UA  N518UA    535    JFK  LAX      320     2475   17  53
## 81482:      UA  N512UA    541    JFK  SFO      343     2586    9  24
## 81483:      UA  N590UA    703    JFK  LAX      323     2475   11  24

Conveniently, data.table also has a different way of printing. By default, only the top and bottom five rows are printed to screen, effectively obliterating the need to use the head function.

The learning curve gets steeper when you start manipulating columns. data.table has a column assignment operator for adding, updating, and deleting columns by reference. The operator is used directly within the [...], and there’s no need to reassign the object. For example, we can use the following code to add a column indicating if the flight occurred in the morning or afternoon.

dt[, ampm := ifelse(hour < 12, 'am', 'pm')];

Syntax can also be different when selecting columns – and this turned out to be my 126TB downfall. When selecting a single column of a data.frame, base R will simplify to a vector by default. With data.table… it depends.

If you select a column using list notation (dt$year or dt[[ 'year' ]]), data.table will return a vector. Similarly, dt[, year] will simplify to a vector. However, dt[, 'year'] will return a single column data table.

str( dt[, year] );
##  int [1:253316] 2014 2014 2014 2014 2014 2014 2014 2014 2014 2014 ...
dt[, 'year'];
##         year
##      1: 2014
##      2: 2014
##      3: 2014
##      4: 2014
##      5: 2014
##     ---     
## 253312: 2014
## 253313: 2014
## 253314: 2014
## 253315: 2014
## 253316: 2014

Unfamiliar with the syntax, I inadvertently used paste to join two single-column data table objects together. The result is a monster of a character string. Rather than pasting the vectors element wise, the whole columns are pasted together into a single string. In the dummy example below, combining three columns of a ~250,000 row data table results in a string with over 3 million characters.

test <- paste0(dt[, 'year'], '-', dt[, 'month'], '-', dt[, 'day']);

print( nchar(test) );
## [1] 3252502

To make matters worse, I assigned the resulting string to a column. Instead of storing it once, it was stored as every element of a column in a large data.table. To data.table’s credit, it was able to deal with my accidental monster. R ran as normal, and I only noticed the problems when I checked the available space on disk.

Recreating my Bug

Intrigued by the large consequences of a tiny bug in notation, I wanted to see the effect it would have on a smaller dataset. The flights14.csv data from the vignette is 15MB. I used this file and the bug shown above as a minimal example of my original problem.

To run this experiment without actually crashing my computer, I used a size-restricted Docker container. Docker can be run with different storage drivers and supporting backend filesystems. In order to use the --storage-opt size option, we need to use the overlay storage driver with the XFS filesystem.

The overlay storage driver can be used for Docker on both Mac and Linux, but the XFS filesystem cannot be mounted on Mac. I had to run Linux on a VirtualBox virtual machine to get it to work on my laptop. I installed CentOS on the virtual machine. Its default filesystem is XFS, and I wanted to minimize setup for the Docker container. In the end I had to do three things to get Docker up and running:

  1. Download the minimal CentOS distribution and install on virtual machine
  2. Install Docker community edition by following the official instructions. I skipped the part about devicemapper drivers as I needed to use overlay anyways.
  3. Add pquota mount option to XFS. This was the trickiest part, and I had a few unsuccessful attempts. In the end I got it to work by following these instructions.


Afterwards, I was ready to launch Docker containers with the --storage-opt size option. An easy way to check if everything works as expected is to run df -h after launching a container. When I did not restrict the size, I got the following output (the virtual machine itself is restricted to 20GB).

Unrestricted Docker container
Setting --storage-opt size=5G restricts the space available to 5GB.

5G Docker container
Just what we wanted! With my safe space set up, I moved on to setting up the container itself. I first wrote a minimal version of the culprit R script that could be run from the container, and saved it as crash.R.

library(data.table);

dt <- fread('flights14.csv');

bug <- paste0(dt[, 'year'], '-', dt[, 'month'], '-', dt[, 'day']);
dt[, bug := bug];

write.table(dt, 'monster.txt');

To execute this within a Docker container, we need to create a Dockerfile with setup instructions. The Dockerfile details what base image the new container should be based on, and what software should be installed. The r-base Docker image contains the latest version of R. Additionally, we need to install the data.table package from CRAN. The COPY command copies the script and data file to the container, and CMD sets a default command to be executed when the container is launched.

FROM r-base
MAINTAINER Erle Holgersen

RUN Rscript -e "install.packages('data.table', repos='https://cloud.R-project.org');"

COPY flights14.csv crash.R src/
WORKDIR src/

CMD ["Rscript", "crash.R"]

From here on out things are fairly straightforward. We first need to build an image from our Dockerfile by running

docker build -t crash .

This will most likely take a few minutes, depending on what Docker images you have built in the past. Afterwards we can launch the container in detached mode and with a 3GB size restriction with the command

docker run --storage-opt size=3G  -d -t crash

Detached mode runs the container in the background, which allows us to keep monitoring the state of the container through docker ps. The -s flag displays details on the size of the container. My container took about two and a half minutes to grow to 3.22GB, and then exited. When I tried to enter the re-enter the container, I got a “disk limit exceeded” error. On the bright side, deleting the Docker container was trivial, and instantly recovered the 3GB of space!

Docker crash

Conclusion

Reading documentation and testing your code are good ideas. And if you really want to overload a hard drive, you can use a Docker container.