Table.to_arrow_batch_iterator

Table.to_arrow_batch_iterator(max_results=None, *, variables=None, progress=True) → pyarrow.RecordBatch iterator

Returns an iterator that can be used to consume a table in chunks of PyArrow RecordBatches. Allows for streaming workflows where only a small portion of the table is read into memory at a time.

Parameters:

max_results : int, default None The maximum number of rows to return. If not specified, all rows in the table will be read.

variables : list<str>, default None A list of variable names to read, improving performance when not all variables are needed. If unspecified, all variables will be represented in the returned rows. Variable names are case-insensitive, though the names in the results will reflect the variable's true casing. The order of the columns returned will correspond to the order of names in this list.

progress : bool, default True Whether to show a progress bar.

Yields:

pyarrow.RecordBatch

See also

• Table.to_arrow_dataset()

• Table.to_arrow_table()

• Table.to_geopandas_dataframe()

• Table.to_dask_dataframe()

• Table.to_pandas_dataframe()

• Table.to_polars_lazyframe()