> For the complete documentation index, see [llms.txt](https://docs.redivis.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.redivis.com/reference/workflows/notebooks/notebook-concepts.md). # Notebook concepts ## Overview Notebooks provide a highly flexible compute environment for working with data on Redivis. In a notebook, you can [reference any table](#referencing-tables) in your workflow, [install dependencies](#dependencies), perform analyses in [Python, R, Stata, or SAS](#supported-languages), store and download files, and generate an output table for downstream analysis. {% hint style="info" %} **Transforms vs. notebooks?** There are two mechanisms for working with data in workflows: [transforms](/reference/workflows/transforms.md) and [notebooks](/reference/workflows/notebooks.md). Understanding when to use each tool is key to taking full advantage of the capabilities of Redivis, particularly when working with big datasets. Transforms are better for: * Reshaping and combining tabular and geospatial data * Working with large tables, especially at the many GB to TB scale * Preference for a no-code interface, *or* preference for programming in SQL * Declarative, easily documented data operations Notebooks are better for: * Interactive exploration of any data type, including unstructured data files * Working with smaller tables (though working with bigger data is possible) * Preference for Python, R, Stata, or SAS * Interactive visualizations and figure generation {% endhint %} ## Working with data #### Loading data From within a notebook, you can load any data available in your workflow. You can reference the primary source table of the notebook via the special `_source_` identifier, or reference any other table in the workflow by its name. To ensure that your notebook doesn't break when tables get renamed, make sure to use the [qualified reference](/api/referencing-resources.md) for non-primary tables. For example: {% tabs %} {% tab title="Python" %} ```python # Reference the source table with the special "_source_" identifier: df = redivis.table("_source_").to_pandas_dataframe() # Reference any other table via its name: # The last 4 characters are the reference id. This is optional, # but recommended to ensure the notebook works as tables get renamed. df2 = redivis.table("daily_observations:vdwn").to_pandas_dataframe() # If our table is a file index table, we can load and process those files for f in redivis.table("_source_").list_files(): data = f.read() ``` {% endtab %} {% tab title="R" %} ```r # Reference the source table with the special "_source_" identifier: df <- redivis$table("_source_")$to_tibble() # Reference any other table via its name: # The last 4 characters are the reference id. This is optional, # but recommended to ensure the notebook works as tables get renamed. df2 = redivis$table("daily_observations:vdwn")$to_tibble() # If our table is a file index table, we can load and process those files for (f in redivis$table("_source_")$list_files()){ data = f.read() } ``` {% endtab %} {% tab title="Stata" %} ```python # Reference the source table with the special "_source_" identifier: import redivis redivis.table("_source_").to_stata() # We can also reference any table in the workflow by name redivis.table("transform_1_output").to_stata() # Or run a SQL query and load its results into Stata redivis.query("SELECT * FROM _source_ LIMIT 10").to_stata() ``` ```stata %%stata /* Run stata code! All stata cells must be prefixed with %%stata */ describe() ``` {% endtab %} {% tab title="SAS" %} ```python # Reference the source table with the special "_source_" identifier: import redivis redivis.table("_source_").to_sas("ds") # "ds" is the SAS Data Set name # We can also reference any table in the workflow by name redivis.table("transform_1_output").to_sas("ds") # Or run a SQL query and load its results into Stata redivis.query("SELECT * FROM _source_ LIMIT 10").to_sas("ds") ``` ```sas %%SAS /* # SAS code can be executed inside any cell prefixed with the %%SAS command */ proc print data=df(obs=5); run; ``` {% endtab %} {% endtabs %} #### Analyzing data Redivis notebooks support the following kernels (programming languages). For more details and examples on how to use notebooks in each language, consult the language-specific documentation: {% content-ref url="/pages/rt7BqmPWkd6eLSwKt07E" %} [Python notebooks](/reference/workflows/notebooks/python-notebooks.md) {% endcontent-ref %} {% content-ref url="/pages/BflZUB0L9wZcfqo4nZFC" %} [R notebooks](/reference/workflows/notebooks/r-notebooks.md) {% endcontent-ref %} {% content-ref url="/pages/xJgUXs0s21BIh1epJvNG" %} [Stata notebooks](/reference/workflows/notebooks/stata-notebooks.md) {% endcontent-ref %} {% content-ref url="/pages/gbTSiSjA8TO3wJjGnOET" %} [SAS notebooks](/reference/workflows/notebooks/sas-notebooks.md) {% endcontent-ref %} #### Outputting tables A notebook can generate an output table as the result of its execution. This output table is created programmatically: {% tabs %} {% tab title="Python" %} 
# Multiple types for "df" are supported
# Consult the language-specific docs for more info
df = get_dataframe_somehow()

redivis.current_notebook().create_output_table(df)
{% endtab %} {% tab title="R" %} 
# Multiple types for "df" are supported
# Consult the language-specific docs for more info
df <- get_dataframe_somehow()

redivis$current_notebook()$create_output_table(df)
{% endtab %} {% tab title="Stata" %} ```stata %%stata -doutd df2 /* Once this cell executes, the current dataset will be pushed to the python variable df2 */ rename v* newv* ``` ```python # Via python, pass this dataframe to the output table redivis.current_notebook().create_output_table(df2) ``` {% endtab %} {% tab title="SAS" %} 
# Reference the table table named "some_table" in SAS
sas_table = sas_session.sasdata("some_table")

# Convert the sas_table to a pandas dataframe
df = sas_table.to_df()

redivis.current_notebook().create_output_table(df)
{% endtab %} {% endtabs %} #### Storing files As you perform your analysis, you may generate files that are stored on the notebook's hard disk. There are two locations that you should write files to: `/out` for persistent storage, and `/scratch` for temporary storage. Any files written to persistent storage will be available when the notebook is stopped, and will be restored to the same state when the notebook is run again. Alternatively, any files written to temporary storage will only exist for the duration of the current notebook session. To write files to these directories, use the standard tools of your programming language for writing files. {% tabs %} {% tab title="Python" %} ```python df = get_dataframe_somehow() # Write temporary files to /scratch, and files you want to persist to /out: df.to_csv("/out/data.csv") ``` {% endtab %} {% tab title="R" %} ```r df <- get_dataframe_somehow() write.csv(df, "/out/data.csv", na="") ``` {% endtab %} {% tab title="Stata" %} 
%%stata
save "/out/my_dataset.dta"
{% endtab %} {% tab title="SAS" %} ```sas %%SAS sas_session proc export data=datasetname outfile='/out/filename.csv' dbms=csv replace; run; ``` {% endtab %} {% endtabs %} You can [inspect and download](#files) these files anytime. ## Notebook management ### Creation Create a notebook by choosing a [table node](/reference/workflows/tables.md) in a workflow and selecting **+ Notebook**. This table will become the default source table for your new notebook and will have pre-generated code that references the table's data. ### Compute configuration By default notebooks are provisioned with 32GB memory and 2 CPU cores, providing a free compute environment suitable for most analyses. When additional recources are needed, notebooks can scale to powerful viritual machines with hundreds of CPU cores, terabytes of memory, and high performance GPUs for computationsally intensive workloads. You can view and alter the notebook's [compute resources](/reference/workflows/notebooks/compute-resources.md) in the **More** menu. ### Starting and stopping Notebook nodes need to be started in order to edit or execute cells. Select **Start notebook** to start it and provision compute resources. You can also elect to **Clear outputs and start**, which will remove all outputs and reset any [referenced tables](#referencing-tables) in the notebook. ### Run notebooks in the background When starting a notebook, you can select the option to **Run in background**. This will run a notebook similar to a [transform](/reference/workflows/transforms.md), where all code is executed in series, and the notebook stops once all cells have been run, or an error occurs. This can be helpful for quickly re-running notebooks after upstream changes have been made. ### Server-side execution (alpha) When starting a notebook, you will see an option to enable server-side execution. This allows the Jupyter notebook to keep receiving outputs, even if your browser is closed or disconnected, which can be helpful for long-running operations. This is a new feature within Jupyter notebooks, and still has a few rough edges (progress bars often don't display), so it is currently an opt-in feature. In the future, this will become the default behavior for all notebooks. ### Persistence All notebooks are automatically saved as you go. Every time a notebook is stopped, all cell inputs are saved to the notebook version history, giving you a historical record of all code that was run. Additionally, all cell outputs from the last notebook session will be preserved, as will any files written to the `/out` directory. ### Clearing outputs When starting a notebook, you'll be presented with the option to **Clear all outputs and start**. This can be helpful in that it will reset all access rules associated with the notebook, since there is no longer any data associated with the notebook. Choosing this option will clear all output cells in your notebook, any files saved in the `/out` directory, and any output tables from the notebook. ### Logs You can select the three-dot **More** menu to open the logs for this notebook. Opening the logs when a notebook is stopped will show the logs from the notebook's previous run. ### Lifecycle The default notebooks have a maximum lifetime of 6 hours, and after 30 minutes of inactivity, any running notebook will automatically be stopped. If you are use a notebook with [paid custom compute](/reference/workflows/notebooks/compute-resources.md#custom-compute-configurations), these values can be modified. Activity is determined based on the Jupyter kernel and edits to the notebook. As long as the Jupyter kernel remains active or an edit is made, the idle timeout clock resets. You do not need to keep your browser open to prevent an active computations for timing out. ### Collaboration All Redivis notebooks support real-time collaboration, allowing multiple editors to edit and run cells in a running notebook. When another editor is active in a notebook, you will see a colored cursor associated with them. Workflow viewers will see a read-only version of the notebook. ### **Changing the source table** To change a notebook's primary source table, either right-click on the notebook or select the three-dot (`ⵗ`) icon and select the **Change source table** option. #### **Limitations** Notebooks are subject to certain [concurrency and duration limits](/reference/quotas-and-limits.md). ## Dependencies All notebooks come with a number of common packages pre-installed. You can install additional packages by selecting **Edit dependencies** in the notebook start modal or toolbar. For more detailed information about the default dependencies and adding new packages, consult the documentation for your notebook type: * [Python base image & dependencies](/reference/workflows/notebooks/python-notebooks.md#base-image-and-dependencies) * [R base image & dependencies](/reference/workflows/notebooks/r-notebooks.md#base-image-and-dependencies) * [Stata base image & dependencies](/reference/workflows/notebooks/stata-notebooks.md#base-image-and-dependencies) * [SAS base image & dependencies](/reference/workflows/notebooks/sas-notebooks.md#base-image-and-dependencies) {% hint style="info" %} For notebooks that reference restricted data, internet will be disabled while the notebook is running. This means that the dependencies interface is the *only* place from which you can install dependencies. In other words running `pip install` for python or `devtools::install()` for R within your notebook will fail. Moreover, it is strongly recommended to always install your dependencies through the dependencies interface, regardless of whether your notebook has internet access, as this provides better reproducibility and documentation for future use. {% endhint %}
## Secrets Secrets are simple key/value pairs that are securely stored within an organization or under your account. These secrets can then be loaded in a notebook. A common use case is for storing external API tokens that then enable you to interface with these APIs from within your notebook. Secrets are accessed via the [Python](/api/client-libraries/redivis-python.md) or [R](/api/client-libraries/redivis-r/reference/redivis.md) client libraries: {% tabs %} {% tab title="Python" %} ```python # secrets are case-sensitive secret = redivis.user("my_username").secret("EXTERNAL_API_TOKEN") # Or for organization secrets: # secret = redivis.organization("organization_name").secret("...") # Avoid printing secrets, but rather pass them to other methods as needed make_http_request("https://example.com/api", auth_token=secret.get_value()) ``` {% endtab %} {% tab title="R" %} ```r # secrets are case-sensitive secret <- redivis$user("my_username")$secret("EXTERNAL_API_TOKEN") # Or for organization secrets: # secret <- redivis$organization("organization_name").secret("...") # Avoid printing secrets, but rather pass them to other methods as needed make_http_request("https://example.com/api", auth_token=secret$get_value()) ``` {% endtab %} {% endtabs %} ## Files Notebooks offer special capabilities for [files written to specific directories](#storing-files) on the notebook's hard disk. Any files you've stored in a notebook's `/out` and `/scratch` directories are available in the files modal. This modal can allow you to preview and download specific file outputs from a notebook. Moreover, files written to the `/out` directory are always available, and will persist across notebook sessions. This allows for workflows where you can cache certain results between notebook sessions, avoiding the need to rerun time-intensive computations. The files in the `/scratch` directory are only available when the notebook is running, and will be cleared once it is stopped. The default "working directory" of all notebooks is `/scratch`. This is where files are written if you do not specify another location. You can view the files in either directory by selecting **Files** in the top right of the notebook.
You can list files in either directory by choosing the corresponding tab, and selecting any file to view it. Redivis supports interactive previews for many [file types](broken://pages/fIPy3eidqDbxtxGd3BYB#inspecting-files) in the file inspecto. You can also download the file for further inspection and analysis. To download all files in a directory, select **Download all** in the files modal. ## Version history Every time you stop your notebook, all cell inputs (your code and markdown) are saved and associated with that notebook session. You can view the code from all previous sessions by selecting **History** in the top right of a notebook, allowing you to view and share your code as it was any previous point in time.
## Access rules #### Determining notebook access Your access to a notebook is determined by your corresponding access to all tables (and their antecedent datasets) referenced by the notebook. These linkages persist across notebook sessions, as a future session could reference data from a previous session. In order to reset the tables referenced by your notebook, which will also clear all outputs in the notebook, you can choose to **Clear outputs and start** when [starting the notebook](#clearing-outputs). #### Access levels In order to view a notebook, you must first have view access to the corresponding workflow, and in order to run and edit the notebook, you must also have edit access to that workflow. Additionally, your access to a notebook is governed by your [access](/reference/data-access/access-levels.md) to its source tables. In order to run a notebook and see its outputs, you must have data access to all source tables. If you have metadata access, you will be able to see cell inputs in a notebook (that is, the code), but not outputs. If you only have overview (or no) access to the source tables, you will not be able to see notebook contents. #### External internet access If a notebook contains data with export restrictions, access to the external internet will be disabled while the notebook is running. When the internet is disabled in a notebook, you can still specify packages and other startup scripts in the **Dependencies** modal that is installed on notebook start. Additionally, if any of your packages require internet access to run, you'll need to "preload" any content using a post-install script. For example, if you're using the `tidycensus` package in R, you could preload the content as follows: 
R -e '
  library(tidycensus)
  library(tidyverse)

  census_api_key("YOUR API KEY GOES HERE")
  get_decennial(geography = "state", 
                 variables = "P13_001N", 
                 year = 2020,
                 sumfile = "dhc")
'
#### Downloading files Typically, you will be able to download any files written to the notebooks `/out` or `/scratch` directories. However, if a notebook references data with export restrictions, you will not be able to download these files, unless the file size is smaller than the relevant [size-based export restrictions](/reference/data-access/usage-rules.md#export-environments) specified on source datasets. ## Exporting notebooks Notebooks can be downloaded as PDF, HTML, and .ipynb files by selecting the three-dot **More** button at the top right of the notebook. You will be given the option of whether to include cell outputs in your export. It is important that you ensure the outputs displayed in your notebook do not contain sensitive data, and that your subsequent distribution is in compliance with any data use agreements. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.redivis.com/reference/workflows/notebooks/notebook-concepts.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.