Returns a reader that mimics the Arrow RecordBatchStreamReader, which can then be consumed to process batches of rows in a streaming fashion. By streaming data, you can avoid the need to load more than a small chunk of data into memory at a time. This approach is highly scalable, since it won't be limited by available memory or disk.
The batch reader exposes a $read_next_batch() method, which should be repeatedly called to fetch the next Arrow RecordBatch. If next() returns NULL, this signals that there is no more data left to read.
Parameters:
max_results : int, default NULL
The max number of records to load into the dataframe. If not specified, the entire table will be loaded.
variables : list(str) | character vector
The specific variables to return, e.g., variables = c("name", "date") . If not specified, all variables in the table will be returned.
Returns:
class RecordBatchReader()
Methods:
$read_next_batch()→ Arrow RecordBatch | NULL
Read the next batch of records. If NULL is returned, indicates that there is nothing left to read.
$close()→ void
Close the underlying connection and releases all resources.
Notes:
The returned instance largely mirrors the behavior of the Arrow RecordBatchStreamReader, though only the $read_next_batch() method is implemented.
Make sure to call on.exit(batch_reader$close()) to ensure resources are released if your application prematurely exits.
batch_reader = redivis$table("test_scores")$to_arrow_batch_reader()
# Make sure the batch_reader gets closed when we exit this function,
# including if an error occurs
on.exit(batch_reader$close())
count <- 0
total <- 0
# Streaming works especially well when performing various aggregations
while (!is.null(batch <- reader$read_next_batch())){
# batch is an instance of an Arrow RecordBatch -> https://arrow.apache.org/docs/r/reference/record_batch.html
scores <- batch$scores
count <- count + length(scores)
total <- sum(scores)
}
print(str_interp("The average of all test cores was ${total/count}"))
