process a table in chunks
Usage
process_chunks(
file,
process_fn,
streamable_table = NULL,
lines = 50000L,
encoding = Sys.getenv("encoding", "UTF-8"),
...
)
Arguments
- file
path to a file
- process_fn
a function of a
chunk
- streamable_table
interface for serializing/deserializing in chunks
- lines
number of lines to read in a chunk.
- encoding
encoding to be assumed for input files.
- ...
additional arguments to
streamable_table$read
method.
Examples
con <- system.file("extdata/mtcars.tsv.gz", package = "arkdb")
dummy <- function(x) message(paste(dim(x), collapse = " x "))
process_chunks(con, dummy, lines = 8)
#> Importing /usr/local/lib/R/site-library/arkdb/extdata/mtcars.tsv.gz in 8 line chunks:
#> 8 x 11
#> 8 x 11
#> 8 x 11
#> 8 x 11
#> ...Done! (in 0.002531767 secs)