Comment on page
Work with data in notebooks
Redivis notebooks a performant, flexible environment for analysis that allow you to analyze and visualize data in projects in Python, R, Stata, or SAS. With the notebook computation happening on Redivis, you don't need to configure an environment on a local machine or server, or export data from Redivis. This makes for easy iteration and collaboration, not to mention ensuring better security and data throughput.
Before working with a notebook you'll want to get started first by creating a project and adding data. You can then create a notebook off of any table in your project.
If you are working with larger tables (~1GB is a good rule of thumb), you'll experience better performance if you first reshape and reduce the data via transforms, when possible.
Once you have a table you're ready to analyze, you can create a notebook by clicking the + Notebook button at any time. You'll need to name it and choose a kernel (Python, R, Stata, or SAS).
Notebooks can only reference tables within their project, so we recommend keeping all related work together in the same project.
Python notebooks come pre-installed with a variety of common scientific packages for python. Learn more about working with python notebooks.
R notebooks come pre-installed with a variety of common scientific packages for R. Learn more about working with R notebooks.
Stata notebooks are based off of python notebooks, but offer affordances for moving data between Python and Stata. Learn more about working with Stata notebooks.
SAS notebooks are based off of python notebooks, but offer affordances for moving data between Python and SAS. Learn more about working with SAS notebooks.
All notebooks come with a number of common packages pre-installed, depending on the notebook type. But if there is something specific you'd like to include, you can add versioned packages or write a pre-/post- install script by clicking the Dependencies button in the toolbar.
Notebook nodes need to be started in order to edit or execute cells. When first clicking on a notebook node, you will see a read-only view of its contents (including cell outputs). Click the Start notebook button in the toolbar to connect this notebook to compute resources.
When you create a notebook for the first time it will start automatically.
Notebooks offer access to 2 CPUs and 32GB working memory, alongside a 50GB hard disk. These values may fluctuate based on system utilization.
To do meaningful work in your notebook, you'll want to bring in the tabular and/or non-tabular data that exists in your project into your notebook.
Notebooks come pre-populated with templated code that pulls in data from the notebook's source table. You will need to run this cell to pull the data into the notebook, and you can see that it worked because this code will print a preview of the loaded data.
You can reference any other tables in this project by replicating this script and executing it with a different table reference. As a rule of thumb, notebooks will easily support interactive analysis of tables up to ~1GB; if your table is larger, try reducing it first by creating a transform, or make sure to familiarize yourself with the tools for working with larger tables in the notebook's programming language.
Python
R
Stata
SAS
import redivis
# The source table of this notebook can always be referenced as "_source_"
table = redivis.table("_source_")
# Load table as a pandas dataframe.
# Consult the documentation for more load options.
df = table.to_pandas_dataframe()
# We can also reference any other table in this project by name.
df2 = redivis.table("my_other_table").to_pandas_dataframe()
print(df)
print(df2)
# The source table of this notebook can always be referenced as "_source_"
redivis_table <- redivis::table("_source_")
# Load table as a tidyverse tibble.
# Consult the documentation for more load options.
df <- redivis_table$to_tibble()
# We can also reference any other table in this project by name.
df2 <- redivis::table("my_other_table")$to_tibble()
print(df)
print(df2)
# In order to load data into Stata, we first have to bring it into Python.
# This code loads the "_source_" table in the python variable `df`
# We can then pass this variable as our stata dataset.
import redivis
# The source table of this notebook can always be referenced as "_source_"
# Reference any other table in this project by name.
table = redivis.table("_source_")
df = table.to_pandas_dataframe(dtype_backend="numpy")
%%stata -d df -force
/*
# Use the %%stata magic to load our dataframe, specified by the -d parameter
# The -force flag replaces the the current working dataset in Stata
# The rest is just Stata code!
*/
describe
import saspy
sas = saspy.SASsession(results='HTML')
# We first load the table via python, and then pass the dataframe into SAS
df = redivis.table("_source_").to_pandas_dataframe(dtype_backend="numpy")
sas_data = sas.df2sd(df, '_df') # second argument is the name of the table in SAS
sas_data.heatmap('msrp', 'horsepower')
Any non-tabular files stored in Redivis tables can be referenced by their globally unique
file_id. You can also reference these file_id's in any derivative tables, allowing you to query and download specific subsets of files.When working with large files, you'll want to consider saving the files to disk and/or working with the streaming interfaces to reduce memory overhead and improve performance.
Python
R
redivis_file = redivis.file("rnwk-acs3famee.pVr4Gzq54L3S9pblMZTs5Q")
# Download the file
download_location = redivis_file.download("./my-downloads")
f = open(download_location, "r")
# Read the file into a variable
file_content = redivis_file.read(as_text=True)
print(file_content)
# Stream the file as bytes or text
with redivis_file.stream() as f:
f.read(100) # read 100 bytes
with TextIOWrapper(redivis_file.stream()) as f:
f.readline() # read first line
# We can also iterate over all files in a table
for redivis_file in redivis.table("_source_").list_files():
# Do stuff with file
redivis_file <- redivis::file("s335-8ey8zt7bx.qKmzpdttY2ZcaLB0wbRB7A")
# Download a file
redivis_file$download("/path/to/dir/", overwrite=TRUE)
# Read a file
data <- redivis_file$read(as_text=TRUE)
# Stream a file (callback gets called with each chunk)
data <- redivis_file$stream(function(x) {
print(length(x))
})
# We can also iterate over all files in a table
for (redivis_file in redivis::table("_source_")$list_files()){
# Do stuff with file
}
At this point, you have all the tools you need to work with your data in your chosen language. The Python, R, Stata, and SAS ecosystems contain myriad tools and libraries for performing sophisticated data analysis and visualization.
The notebook interface is based off of Jupyter notebooks, and has similar capabilities. You can also export a read-only copy of your notebook as an .ipynb, PDF, or HTML file.
Notebooks can produce an output table, which you can sanity check and further analyze in your project by including in other notebooks or exporting to other systems.
Python
R
Stata
SAS
# Read table into a pandas dataframe
df = redivis.table('_source_').to_pandas_dataframe()
# Perform various data manipulation actions
df2 = df.apply(some_processing_fn)
# Create an output table with the contents of this dataframe
redivis.current_notebook().create_output_table(df2)
# Read table into a tibble
tbl = redivis::table('_source_')$to_tibble()
# Perform various data manipulation actions
tbl2 = tbl %>% mutate(...)
# Create an output table with the contents of this dataframe
redivis::current_notebook()$create_output_table(tbl2)
%%stata -doutd df2
/*
Once this cell executes, the current dataset will be pushed
to the python variable df2
*/
rename v* newv*
# Via python, pass this dataframe to the output table
redivis.current_notebook().create_output_table(df2)
# Convert a SAS table to a pandas dataframe
df = sas_table.to_df()
# Create an output table with the contents of this dataframe
redivis.current_notebook().create_output_table(df)
All Redivis notebooks support real-time collaboration, allowing multiple editors to edit and run cells in a running notebook. When another editor is active in a notebook, you will see a colored cursor associated with them (much like a Google Doc).
Share your project to work with collaborators in real time, and make it public so that others can fork off of and build upon your work.
If the work you're doing leads to a publication, make sure to reference the dataset pages from datasets you've used for information from the data administrators on how to correctly cite it.
Last modified 1mo ago