Introduction
Sometimes, a large dataframe has one or more variables with a small number of unique combinations. E.g. a dataframe with one or more factor variables. Storing the entire dataframe as a single text file requires storing lots of replicated data. Each row stores the information for every variable, even if a subset of these variables remains constant over a subset of the data.
In such a case we can use the split_by
argument of write_vc()
. This will store the large dataframe over a set of tab separated files. One file for every combination of the variables defined by split_by
. Every partial data file holds the other variables for one combination of split_by
. We remove the split_by
variables from the partial data files, reducing their size. We add an index.tsv
containing the combinations of the split_by
variables and a unique hash for each combination. This hash becomes the base name of the partial data files.
Splitting the dataframe into smaller files makes them easier to handle in version control system. The total size depends on the amount of replication in the dataframe. More on that in the next section.
When to Split the Dataframe
Let’s set the following variables:
\(s\): the average number of bytes to store a single line of the
split_by
variables.\(r\): the average number of bytes to store a single line of the remaining variables.
\(h_s\): the number of bytes to store the header of the
split_by
variables.\(h_r\): the number of bytes to store the header of the remaining variables.
\(N\): the number of rows in the dataframe.
\(N_s\): the number of unique combinations of the
split_by
variables.
Storing the dataframe with write_vc()
without split_by
requires \(h_s + h_r + 1\) bytes for the header and \(s + r + 1\) bytes for every observation. The total number of bytes is \(T_0 = h_s + h_r + 1 + N (s + r + 1)\). Both \(+ 1\) originate from the tab character to separate the split_by
variables from the remaining variables.
Storing the dataframe with write_vc()
with split_by
requires an index file to store the combinations of the split_by
variables. It will use \(h_s\) bytes for the header and \(N_s s\) for the data. The headers of the partial data files require \(N_s h_r\) bytes (\(N_s\) files and \(h_r\) byte per file). The data in the partial data files require \(N r\) bytes. The total number of bytes is \(T_s = h_s + N_s s + N_s h_r + N r\).
We can look at the ratio of \(T_s\) over \(T_0\).
\[\frac{T_s}{T_0} = \frac{h_s + N_s s + N_s h_r + N r}{h_s + h_r + 1 + N (s + r + 1)}\]
Let’s simplify the equation by assuming that we need an equal amount of character for the headers and the data (\(h_s = s\) and \(h_r = r\)).
\[\frac{T_s}{T_0} = \frac{s + N_s s + N_s r + N r}{s + r + 1 + N (s + r + 1)}\]
\[\frac{T_s}{T_0} = \frac{s + N_s s + N_s r + N r}{s + r + 1 + N s + N r + N}\]
Let assume that \(s = a r\) with \(0 < a\) and \(N_s = b N\) with \(0 < b < 1\).
\[\frac{T_s}{T_0} = \frac{a r + N a b r + N b r + N r}{a r + r + 1 + N a r + N r + N}\]
\[\frac{T_s}{T_0} = \frac{(a + N a b + N b + N) r}{(N + 1) (a r + r + 1)}\]
\[\frac{T_s}{T_0} = \frac{a + N a b + N b + N}{(N + 1) (a + 1 + 1 / r)}\] \[\frac{T_s}{T_0} = \frac{a + (a b + b + 1) N }{(N + 1) (a + 1 + 1 / r)}\]
When \(N\) is large, we can state that \(a \lll N\) and \(N / (N + 1) \approx 1\).
\[\frac{T_s}{T_0} \approx \frac{a b + b + 1}{a + 1 + 1 / r}\]
The figure illustrates that using split_by
is more efficient when the number of unique combinations (\(N_s\)) of the split_by
variables is much smaller than the number of rows in the dataframe (\(N\)). The efficiency also increases when the storage for a single combination of split_by
variables (\(s\)) is larger than the storage needed for a single line of the remain variables (\(r\)). The storage needed for a single line of the remain variables (\(r\)) doesn’t influence the efficiency.
Benchmarking
library(git2rdata)
root <- tempfile("git2rdata-split-by")
dir.create(root)
library(microbenchmark)
mb <- microbenchmark(
part_1 = write_vc(airbag, "part_1", root, sorting = "X"),
part_2 = write_vc(airbag, "part_2", root, sorting = "X", split_by = "airbag"),
part_3 = write_vc(airbag, "part_3", root, sorting = "X", split_by = "abcat"),
part_4 = write_vc(
airbag, "part_4", root, sorting = "X", split_by = c("airbag", "sex")
),
part_5 = write_vc(airbag, "part_5", root, sorting = "X", split_by = "dvcat"),
part_6 = write_vc(
airbag, "part_6", root, sorting = "X", split_by = "yearacc"
),
part_15 = write_vc(
airbag, "part_15", root, sorting = "X", split_by = c("dvcat", "abcat")
),
part_45 = write_vc(
airbag, "part_45", root, sorting = "X", split_by = "yearVeh"
),
part_270 = write_vc(
airbag, "part_270", root, sorting = "X", split_by = c("yearacc", "yearVeh")
)
)
mb$time <- mb$time / 1e6
Splitting the dataframe over more than one file takes more time to write the data. The log time seems to increase quadratic with log number of parts.
mb_r <- microbenchmark(
part_1 = read_vc("part_1", root),
part_2 = read_vc("part_2", root),
part_3 = read_vc("part_3", root),
part_4 = read_vc("part_4", root),
part_5 = read_vc("part_5", root),
part_6 = read_vc("part_6", root),
part_15 = read_vc("part_15", root),
part_45 = read_vc("part_45", root),
part_270 = read_vc("part_270", root)
)
mb_r$time <- mb_r$time / 1e6
A small number of parts does not seem to affect the read timings much. Above ten parts, the required time for reading seems to increase. The log time seems to increase quadratic with log number of parts.