Programmatic uploads

Overview

In addition to uploading data through the browser interface, you can leverage the redivis-python and redivis-r libraries, as well as the generic REST API, to automate data ingest and data release pipelines. These libraries can be used for individual file uploads similar to the interface, as well as for streaming data ingest pipelines.

Basic examples are provided below. Consult the complete client library documentation (Python, R) for more details and additional examples.

Python

import redivis

# Could also create a dataset under an organization:
# dataset = redivis.organization("your_organization").dataset("Dataset name")
dataset = redivis.user("your_username").dataset("Dataset name")

# public_access_level can be one of ('none', 'overview', 'metadata', 'sample', 'data')
dataset.create(public_access_level="overview")

# Create a table on the dataset. Datasets may have multiple tables
table = (
    dataset
    .table("Table name")
    .create(description="Some description")
)

# Upload a file to the table:
upload = table.upload().create(
    "./data.csv",           # Path to file, data frame, raw bytes, etc
    type="delimited",       # Inferred from file extension if not provided
    ...                     # See documentation for all parameters
)

# Optional: add more uploads to this table, or create other tables.
# Multiple uploads on the same table will be appended together.

# Release version
dataset.release()

R

library(redivis)

# Could also create a dataset under an organization:
# dataset <- redivis$organization("your_organization")$dataset("Dataset name")
dataset <- redivis$user("your_username")$dataset("Dataset name")

# public_access_level can be one of ('none', 'overview', 'metadata', 'sample', 'data')
dataset$create(public_access_level="overview")

# Create a table on the dataset. Datasets may have multiple tables
table = (
    dataset
    $table("Table name")
    $create(description="Some description")
)

# Upload a file to the table. 
# You can create multiple uploads per table, in which case they'll be appended together.
upload = table$upload()$create(
    "./data.csv",           # Path to file, data.frame, raw vector, etc
    type="delimited",       # Inferred from file extension if not provided
    ...                     # See documentation for all parameters
)

# Optional: add more uploads to this table, or create other tables.
# Multiple uploads on the same table will be appended together.

# Release version
dataset$release()