Redivis is a platform for data distribution and analysis that prioritizes open access and FAIR data practices. We believe that supporting open science principles drives better research outcomes, collaboration, and scientific impact.

Redivis is creating tools not just for data distribution and analysis, but also to help drive forward standards for transparent research that is accessible to all.

Reproducibility

Every Redivis dataset has a versioned history that is automatically generated while the data owner uploads and edits data contents. This history is fully exposed to end users and the dataset can be accessed at any version. Citations are generated for every version of a dataset which include the ORCID iDs for all creators. Data owners can opt to create Digital Object Identifier (DOI) for all versions of their dataset, which is available at a persistent URL along with the required metadata describing the content at that point in time.

As users analyze data in a Redivis workflow each step they take is recorded in a visual tree that can be navigated sequentially. Every step has a full history which can be viewed at any point in time.

Availability

Datasets in any format can be deposited into Redivis and made available immediately to the world via open access licenses at no cost to the end user.

All Redivis datasets and analysis workflows are available to anyone on the internet without a Redivis account unless the owner decides to add restrictions. In this case someone wishing to view the work would need to create a free Redivis account to request access from the owner.

Longevity

The stable technical infrastructure and dedicated Redivis team is funded by a coalition of academic institutions paying license fees for large scale data distribution. This diverse group of funders ensures the long-term availability of the datasets.

We've started building a library of common workflow types and analytic tasks that might be useful when you're getting started working with data.

Please let us know if there are other actions or concepts you would like us to provide examples for by reaching out or emailing [email protected]

Overview

Retaining data is vital to establishing long-lasting data linkages and making research available. Redivis' data retention policy is aligned with the goal of keeping all data available and accessible whenever possible.

Owner-initiated deletion

Data owners always have the ability to remove their data from Redivis or make it inaccessible if they choose to do so.

Redivis also has policies in place in case of an accidental deletion:

• All data is fully backed up, with point-in-time recovery, over a 7 day rolling window. Deletions and modifications to data is permanent after 7 days.

• All metadata is backed up with point-in-time recovery over a 7 day rolling window, with daily backups of metadata extending to 1 year.

Redivis-initiated deletion

Data storage that exceeds the free tier must be paid for by the data owner or other sponsor. Upon payment failure, every effort will be made to contact the data owner and re-establish payment.

If unsuccessful, data may be destroyed after 30 days of non-payment, though all metadata, and permanent landing pages will be persisted.

We've started building a library of common actions that you might take when administering an organization.

Please let us know if there are other actions or concepts you would like us to provide examples for by reaching out or emailing [email protected]

What is Redivis?

Redivis is an online data platform for research.

Redivis offers data distributors – from research centers and institutions to labs and individual investigators – the tools to securely host and distribute data in alignment with FAIR data practices.

Redivis provides researchers the means to easily discover, access, and analyze data in a collaborative and reproducible manner.

Whether you are working with terabytes of high-risk data or lightweight public data files, Redivis endeavors to make data-driven research accessible to all.

Who uses Redivis?

Research institutions provide a unified discovery and administrative layer for their organizations' data across Redivis.

Research organizations upload, catalogue, and grant access to datasets.

Researchers upload their own datasets, discover existing data, and execute flexible, high-performance analytic workflows in a collaborative and reproducible manner.

Redivis makes it easy for everyone to:

• Discover datasets

• Apply for access

• Query large datasets

• Analyze data

• Collaborate

• Host, brand, and distribute data within an organization

• Version & document datasets

• Integrate with existing data pipelines

Using this documentation

This documentation is broken up into Guides, Examples, and Reference sections.

We recommend that new users begin by following the relevant guides and examples, while the reference section can serve as a comprehensive overview of everything that Redivis can do.

Certain users may want to dive into the more technical API documentation for interfacing with Redivis via R, Python, Javascript, and the HTTP REST API.

Overview

In the bottom pane of the transform you will need to select the variables you'd like to keep in your output table. One or more variables must be kept in order to run the transform.

Usage

Click on a variable and use the center buttons to move it between the two lists. You can use command-/control- and shift-click to select multiple variables at once.

For larger variable lists, clicking the search icon will allow you to quickly filter the available variable options down based on search terms.

Distinct

You can select the Distinct box to drop any records that are an exact copy of another record across values in all variables in your output table.

This can be useful in situations where you create variables using analytic methods or if your data is in a particular format. Otherwise this might slow down your transform run time unnecessarily.

Source indicators

Hovering on any variable will show a tooltip with information about where the variable came from, it's type, and any assigned label.

You will also see to the left of variables in the list an icon showing the variable's source. For variables originating in the source table, the icon will show t0 and for variables from subsequently joined in tables the icon will show t1, t2, etc. For variables created in this transform it will show v.

Selection options

Some steps will add or remove variables as options for selection to keep in the discard table. Join and occasionally Merge will bring in variables from other tables. Create variables will make new variables. Aggregate will remove all variables that are not included in the collapse, but has the potential to add newly created aggregate variables.

When using a create variables step (or when creating aggregate variables in an aggregation or pivot step) you will need to select a method to specify how the variable will be created.

Every method is documented here with what information the Redivis interface needs, as well as a link to the underlying BigQuery documentation for more details.

Citing datasets

It's important to make sure that you fully cite any datasets used in your work. Each dataset page includes citation information in the provenance section in multiple format options.

For example the Global Historical Climatology Network dataset hosted by the Stanford Center for Population Health Sciences would be cited in the APA format as:

Stanford Center for Population Health Sciences. (2022). GHCN Daily Weather Data (Version 1.2) [Data set]. Redivis. https://doi.org/10.57761/h9ff-vy04

Citing workflows

Workflows across Redivis also include citation information in the provenance section in multiple format options.

Bibliography

Datasets and workflows on Redivis can be supplemented with related identifiers that situate a resource in its broader context. You can view the bibliography for a dataset or workflow from the corresponding provenance section.

Overview

Any type of file can be uploaded to a Redivis dataset, and we support previews for the most common file types. These data can be analyzed in notebooks within workflows.

Note that if you have tabular data we strongly recommend uploading it as a table (rather than a file) so you and your researchers can take advantage of our extensive toolkits for previewing and manipulating tabular data.

This guide assumes you have already started by Creating a dataset.

1. Locate the data you want to upload

You can upload data directly from your computer, or import from a linked account.

If importing, you'll want to get the relevant external account configured to your Redivis account before getting started.

The import tools allow for multiple uploads, so it's helpful to have them all in the same place.

2. Upload files

When you're ready click the Upload files button on the Files tab of the dataset page to get started and select the location your files will be coming from.

If you select "Computer" then you will get a browser window where you can choose the file, files, or folder you want to upload from. Choosing a different location will bring up the corresponding mechanism to select your files.

Once selected, you will need to choose a Folder to put them in. All files on a dataset live within one folder which have a corresponding index table to help researchers understand and navigate those files. You can change this later if needed. If this is your first time uploading files to this dataset you will need to create your first folder.

Click the "Upload" button to start the process. If you are uploading from your computer you'll need to wait on this screen until it's complete. Closing your window will end the upload process. If uploading from a linked source, you can close this window to allow this process to continue in the background.

3. Manage files and folders

You can click on the file name in the Files tab to preview the file and see its information.

You can also go to the Tables tab to view the index table for the folder you've uploaded files to. View the cells and hover or click on the file ID variable to see the preview here. This table can be used in Transforms within a Workflow to work with files on a large scale.

If you want to rename a folder you can do so on the Files tab by right clicking on the folder name in the right bar.

You can also create new folders and move files between two existing folders from this bar. When moving between folders, you can use a conditional statement to only move some files that match your conditions.

Next steps

Continue uploading your dataset

Great metadata makes your dataset useable. Complete your metadata, along with configuring access, creating a sample, and releasing this version.

Learn more in the Create & manage datasets guide.

Overview

A study is a group of users working together with a common goal. Studies allow users to apply for access to study requirements on datasets as a group. It also makes sharing workflows and user-created datasets easier.

Studies

You can create a study by clicking the + New button on the studies tab of your workspace. Any study you are a part of will appear in your workspace. Any edits you make will appear for all study collaborators.

Edit

You can add collaborators to your study. Collaborators can edit the study and will by default have edit access to all workflows created in this study.

When you create a study, you are assigned as the study's PI in the collaborators list. Organization administrators use this field to better track studies applying for their data.

If you add more users to a study you can change the PI assignment. Over on a user's name and click Make PI to transfer. Note that once you change the PI you will no longer have the ability to change the PI assignment.

You can also give a study a description, which will help administrators and other study members better understand its goals.

Workflow

Any workflow created in this study will share edit access with all other collaborators in this study by default.

Datasets

The list of datasets will be automatically populated as you use datasets in workflows in this study. You can all add datasets to it directly in order to make the access process easier later on.

Overview

Compute credits can be used to provision advanced compute environments for notebooks.

The default notebook configuration is a free resource on Redivis, and has access to 2 CPUs and 32GB working memory, alongside a 60GB SSD disk and gigabit networking. This is similar to most personal computers, and for many analyses should be plenty of compute power!

If you are working with larger tables, creating an ML model, or need to use a GPU you can choose to configure more advanced compute resources when setting up your notebook.

Advanced compute environments will cost a number of compute credits per hour to run depending on the machine type. This amount will be clear when configuring and starting the notebook. In order to start a notebook with an advanced compute configuration you must have enough compute credits to cover at least 15 minutes of running time.

Purchase compute credits

Credits cost $0.10 and can be purchased in increments of 100 ($10), 200 ($20), 500 ($50), or 1000 ($100). Click the Purchase credits button and choose the amount you'd like to purchase.

You can purchase credits immediately using a credit card or bank account, or through an invoice.

Credit card / Bank

When clicking Checkout you will be rerouted to a Stripe payment processing page. The card you enter here will be processed by Stripe and never seen by Redivis or stored in our databases.

Invoice

You can choose to generate an invoice for your credit purchase. Any information you enter in the custom fields section will appear on the invoice and might be required by the organization paying the invoice. Once generated the invoice will be downloaded as a PDF to your computer. You can also return to view the invoice and copy the link to the invoice to pay electronically. Once the invoice is paid the credits will appear in your account.

Auto-purchase compute credits

You can set up your account to purchase credits every time your account dips below 10 compute credits. This ensures that your notebooks will never halt mid-session due to a lack of compute credits. Here, you can select the amount of credits you'd like to purchase every time this condition is met and the card you would like to be charged for it. If you'd like to change the amount or cancel auto-purchase you can do so by returning to this screen.

Refund compute credits

Credits on Redivis never expire and can be refunded at any time. If you would like to refund the compute credits on your account please contact us to initiate the refund process.

Billing dashboard

Redivis uses Stripe for all payment processing. If you'd like to view or edit your card on file or view previous purchases you can do so in the Stripe billing dashboard, which is linked from this page.

Usage limits

User accounts are currently limited to 10GB of storage. You can see your current usage on the Settings page of your Workspace, under the Compute Credits and Billing tab.

If you'd like to increase that limit and host larger datasets, Contact us about creating an Organization!

Overview

All datasets that contain data will have at least one table (and up to 1,000), displayed on the Tables tab of the dataset page. Tables are the "container" for all data on Redivis, including for geospatial and unstructured data types.

Tables are created by data editors when they upload data, and users' access to these tables will be governed by the dataset's access configuration.

Tables can be explored from the dataset page, including the ability to view variables and summary statistics, cell contents, and run one-off queries against the table.

These tables are then further utilized from within a workflow.

See here for full documentation concerning tables and their functionality.

Creating tables

New tables can be created on the Tables tab of the dataset editor page by clicking on + New table. Tables can only be created on the unreleased next version of the dataset – if the dataset doesn't have a next version, you'll need to create the next version first.

Updating tables

Table metadata, such as the table name, description, and entity, can be edited at any time from the dataset editor page, even on released versions. The table's data can only be updated on an unreleased version; see the uploading data documentation for more information.

Deleting tables

Tables can only be deleted on the unreleased next version of the dataset – if the dataset doesn't have a next version, you'll need to create the next version first. Note that deleting a table on deletes that table on the version you're editing. It will still exist on any historic versions of the dataset.

Overview

Redivis is built on data principles to support data discovery and reusability at all levels. Data practices adhering to these principles emphasize data that is:

Findable

Data is findable when (1) data and metadata are assigned a globally unique and persistent identifier, (2) data are described with rich metadata, (3) metadata clearly and explicitly include the identifier of the data it describes, and data and (4) metadata and data are registered or indexed in a searchable resource.

At Redivis:

• can be issued for datasets, facilitating authoritative citation and attribution.

• All datasets have a section of documentation which auto-populates with administrator actions and allows for additional linking to other artifacts that were part of the dataset creation process.

• Redivis has comprehensive search tools that index all aspects of a including the , , variable names and variable documentation.

Accessible

Data is accessible when: (1) data and metadata are retrievable by their identifier using a standardized communications protocol, (2) the protocol is open, free, and universally implementable, (3) the protocol allows for an authentication and authorization procedure, where necessary, (4) metadata are accessible, even when the data are no longer available.

At Redivis:

• Data can be explored through any web browser.

• Public data and analyses can be explored without an .

• Researcher accounts to apply for data access or do analyses are always free.

• Researcher accounts can be linked to .

Interoperable

Data is interoperable when: (1) data and metadata use a formal, accessible, shared, and broadly applicable language for knowledge representation, (2) data and metadata use vocabularies that follow FAIR principles, (3) data and metadata include qualified references to other data and metadata.

At Redivis:

• Robust support interoperability with other tools.

• use common languages such as SQL, Python, R, Stata, and SAS.

• Data and metadata are available for in multiple common formats.

Reusable

Data is reusable when: (1) data and metadata are richly described with a plurality of accurate and relevant attributes, (2) data and metadata are released with a clear and accessible data usage license, (3) data and metadata are associated with detailed provenance, and (4) data and metadata meet domain-relevant community standards

At Redivis:

• Datasets are automatically .

• Many dataset, table, and variable metadata fields are automatically populated based on user action, with the ability to be manually adjusted.

• All datasets have a Usage tab with information on how it has been viewed and used across Redivis

• Redivis cloud-based analysis tools encourage data users to do analysis along the data rather than downloading it and breaking linkages.

• containing analyses are self-documenting and capture the full analysis pipeline.

• Anyone can a workflow they have access to in order to continue an analysis.

Overview

Studies are a way for one or more collaborators to work with data on the same conceptual topic. You can add multiple collaborators, datasets, and workflows to a study to organize your workflow. In addition, some restricted data requires you to submit access applications as a study via study requirements.

1. Create a new study

On the studies tab of your workspace you can create a new study and see all studies you are a part of. This study might represent a topic or group you are working with, maybe to investigate a similar cluster of research questions. You can give the study a name and description to reflect that purpose.

2. Add collaborators

If you are working with anyone else on this study you can add them as a collaborator. Anyone who is a collaborator on this study will be able to view and edit the study same as you can. Once added, this study will appear in their workspace.

PI

One person on the study can be designated as the PI. This designation does not have meaning within Redivis interfaces and is here to inform others in the group or administrators you are applying for access from of the group structure.

3. Add datasets

You can add any dataset to this study that you plan on working with, and remove it later if your topic shifts. This space is intended for you to gather resources that you are using in one place to make it easier to create workflows or manage data access. Datasets can be added to any number of studies.

4. Add workflows

You can create a new workflow in this study or move an existing one in. All workflows can be in at most one study, and you can see what study a workflow is in on the workflow's overview page.

Granting workflow access to study members

One of the options for sharing this workflow is to grant view or edit access to all members of the study it's in. This can be an easy way to give everyone in a group access to a workflow you are working on.

As with all workflows, even though someone might have access to the workflow they will need to have independent access to all the datasets it contains in order to view or query them.

5. Apply for data access via a study

If you're working with restricted data you might come across a dataset that has a study requirement as part of its access requirements. You can identify this by the study icon next to the requirement name.

Study requirements are filled out and submitted on behalf of an entire study, rather than one person. An approved study requirement will be valid for all collaborators on the study, unlike member requirements where each individual needs to complete the requirement on their own account.

To submit a study requirement you can navigate to a restricted dataset and open the access modal. You will need to select your study from the dropdown menu, and a submit button will appear. One member of your study will fill this out and submit it on behalf of your group. Once approved, all study collaborators will see an approved requirement in their access modal.

Data usage

When restricted data has a study requirement, the data administrator can approve your application submission for use of the data within that study. In order to query data in a workflow, that workflow will need to be in the approved study. If it is not, you will see a badge noting Limited access and you will not be allowed to run transforms or query this data in a notebook.

Next steps

Start analyzing your data

Once you've gained access and set up a study it's time to add your datasets to a workflow to transform and analyze them leveraging lightning fast tools from your browser.

Learn more in the guide.

Overview

Redivis combines powerful functionality to reshape and analyze data on platform, with an easy export and publishing flow, to ensure the results of your work can be displayed in the format of your choice and shared with your collaborators and research community.

Working to determine, first, the broader picture of the content you'd like to publish and where you'd like to publish it will help you, then, determine the desired shape and format of the assets to be exported and, finally, the specific sources of the data in your Redivis workspace.

1. Develop a publishing strategy

Ask yourself: What story are you trying to tell? Who/where is the audience? What component pieces are crucial to building the narrative? Your ideal package of assets may be very different if you're hoping to share progress on a workflow with a principal investigator, take a snapshot of a single variable distribution for a colleague, publish results in a journal, or build a custom dashboard to highlight multiple trends.

You may use a combination of tables, unstructured files, code snippets, graphs, and descriptive text to be published, so sketching out these component pieces in increasing detail will help define your end product.

2. Choose your desired formats

With an initial strategy in mind, learning about the different types of Redivis exports will help define your list of assets to generate.

Tabular data containing rows and columns can be in a variety of formats, accessed from many environments via our client libraries, or in a website.

Unstructured data files of any type can be in their original format, or accessed .

Notebooks containing code inputs and corresponding outputs can be as an .ipynb file or PDF or HTML, or (coming soon!) embedded in your site.

Learn more in the guide.

3. Identify the data sources

In a , you can generate output tables to capture the result of a set of data transformations, or use notebook to show a line-by-line data analysis flow and relevant figures.

As a , you can upload your own tabular data and unstructured files to create assets to use in your workflows or share with others.

Whether you're manipulating data in a workflow to showcase results or hosting your own dataset, you'll want to build a set of specific tables or notebooks you're trying to share. As you iteratively modify the shape and content of these assets – in a workflow to output new tables or to build analysis flows in Python or R – you'll fine-tune each piece of your final publication.

Next steps

Upload your own datasets

Augment your data analysis in Redivis by uploading your own datasets, with the option to share with your collaborators (or even the broader research community).

Learn more in the guide.

Build a custom dashboard

For full customizability, you can publish a static site that accesses Redivis data to power an interactive visual dashboard – a more permanent, web-based approach to highlighting results of your data analysis or generated data.

Learn more in the guide.

For more extensive full-workflow video walkthroughs, see our Events and press page:

Or read through our Example workflows:

Overview

Datasets are a core component of Redivis. Datasets contain various metadata and documentation, as well as one or more and/or containing data.

All datasets have their own persistent URL and are by either a user or an organization. Datasets can be added to to analyze and combine with other datasets across Redivis. Datasets are automatically versioned and you can always update the version you are viewing or working with.

Some components of a dataset may not be available to you until you are . In order to see the existence of a dataset, you must at least have .

You can to use in your workflows and share with colleagues, or create datasets within any organization that you administer.

Overview

In order to create a Redivis account, you must authenticate through your academic institution, through any Google account, or via passwordless login. Over 2,500 academic institutions from around the world are currently supported, with more being added regularly.

Redivis will request some basic information when you authenticate, such as your name, email, and any relevant affiliations; it will not have access to any other information through your Google or institutional account.

Adding emails / authentications to your account

It is strongly encouraged that each individual has only one Redivis account, and that your identity on Redivis map to your real-world identity. However, it is common that you might have multiple emails that you want to associate with your Redivis account: personal email, university email, visiting-scholar university email.

In order to add an email to your account, navigate to the , and click Add authentication in the "Authentications" tab. A new browser window will open requesting the relevant credentials, after which this new email will be associated with your account.

Logins and organization membership

When you join an organization, your membership in that organization will be associated with a particular login. In order to work with that organization's restricted data, you must have recently authenticated with the login associated with your membership. If you sign in to Redivis using a different login (for example, you signed in with your personal email, while your membership is associated with you academic institution's credentials), you'll be prompted to re-authenticate with the relevant login before you can work with restricted data.

If you lose access to the login originally associated with your membership — for example, if you change institutions — you can request that your membership be updated to associate it with a new login. An administrator of the organization will then need to approve this request before access through your membership will be restored.

Merging accounts

If you have more than one Redivis account, we strongly encourage you to merge them. In order to merge multiple accounts, you should perform the following steps:

1. Identify which account should be your "primary" Redivis account once the process is complete.

2. Log in to your non-primary account.

3. Transfer ownership of all datasets, workflows, and studies to your primary account. Alternatively, you can delete any datasets, workflows, and/or studies that are no longer relevant.

4. Delete the non-primary account by clicking "Delete account" within your . WARNING: This action cannot be undone.

5. Log in to your primary account, and add the authentication(s) that were previously associated with your non-primary account to your primary account.

Overview

Datasets provide a persistent, version-controlled store of data and their metadata. Datasets can be queried and analyzed within , and in most cases it will make sense to first perform initial analyses on Redivis before downloading data (particularly for larger datasets).

However, in certain situations you may want to download some or all of the dataset contents, or its metadata, for future reference and analysis on external systems.

Download metadata

Top-level metadata about the dataset can be downloaded by clicking on the Download metadata link in the . Metadata can be downloaded in the DataCite, Schema.org, and Redivis API schema specifications.

Download citations

From the , click on the Bibliography button to view a full bibliography of your dataset and its data sources. You can copy or download this citation information in APA, CFF, or BibTex formats.

Download and Export data

Any table within a dataset can be download or exported to a supported environment, or read into another environment through the Redivis API, pursuant to any applied by the dataset's administrators.

Extract content via the API

The (and and wrappers) provide numerous methods for interfacing with a workflow and its contents, and in many cases will be the most flexible mechanism to extract workflow metadata and data from external systems.

Overview

Your Redivis account establishes your identity on Redivis, and provides a home for various forms of data-driven investigation and collaboration.

Because researchers regularly move between institutions, and often want to be able to reference and access historic work, it is strongly encouraged that you:

• Only have a single Redivis account, much as you would only have a single Google Scholar account.

• (e.g., university credentials, personal google account) to your Redivis account, to ensure that you can always access it.

Communication and privacy

Redivis will never, ever distribute your personal information to a third party. We will send data access notifications to your contact email, though this can be disabled in your . You may opt-in to receive occasional product updates, though this is turned off by default.

When you apply for access to an organization's datasets, they will be able to see some information pertinent to your membership in their organization:

• Full name

• Contact email

• Email / institutional identifier

• Affiliation, if provided by your institution's login provider

Overview

To create a new account, click the Create account button at the top of any page. You can create an account using your academic institution's login credentials (e.g., [email protected]), logging in with any google account, or via passwordless authentication through any email. Your account can then be managed from .

By creating an account on Redivis, you can gain access to hundreds of restricted datasets and powerful querying tools. There is no cost to creating an account, and you may permanently delete your account at any time.

Your account tracks all of your workflows, queries, collaborations, and data approvals with different organizations. After creating an account, you will be able to apply for membership with various organizations on Redivis and request access to their restricted data.

Overview

In order to upload unstructured data to a you'll need to upload it as a on the Files tab in the dataset uploader.

Uploading files

From the Files tab of the , click the "Upload files" button to import one or many files. You can import files from a variety of , or perform uploads

Quotas & limits

Limits for upload file size and max files per dataset are .

Folder management

When preparing your upload you will need to select a destination folder. All files must be in one folder. These folders will have a corresponding which will allow researchers to more easily work with the files in bulk. If you are uploading many files, your folder structure will likely be important – for example, a different folder for different categories of imaging files.

You can create folders on the grey bar on the right side of the Files tab in a dataset. Folder names must be unique. You can click on the ⋮ menu next to any folder name to manage that folder.

To move files between folders you can right click on an individual file to change its folder. If you have multiple folders you can click the Move files button below files in the right bar to move files between folders.

Overview

Redivis workflows are high-performance, collaborative environments for analyzing data. In a workflow, you can combine datasets across Redivis, and build out your analysis in an iterative, reproducible manner. Workflows can contain:

• , which bring data from across Redivis into your workflow. Data sources can either be a dataset that you have access to, or another workflow, allowing you to build on top of others' analyses.

• , which join, filter, aggregate, and reshape tabular data. Transforms are particularly important when working with large tables, allowing you to query billions of records in seconds.

• , which execute Python, R, Stata, or SAS in a customizable, interactive compute environment.

• , which are created as outputs of transforms and notebooks. On any table, you can validate your results, build out further analyses analysis, create new Redivis datasets, or export the data to another environment.

Workflows are owned by either a user or , and can be with other users and organizations as you work together in real time.

This common method utilizes if-then-else logic, assigning a result when the corresponding condition evaluates to true, otherwise assigning the final ("else") result –>

Return type

dynamic (input-dependent)

Parameters

Overview

In addition to uploading data through the browser interface, you can leverage the and libraries, as well as the generic , to automate data ingest and data release pipelines. These libraries can be used for individual file uploads similar to the interface, as well as for streaming data ingest pipelines.

Basic examples are provided below. Consult the complete client library documentation (, ) for more details and additional examples.

Overview

Getting data into Redivis is the first and most important step in making it accessible for your research community. Redivis is designed to make it easy for you to securely ingest data, at scale. To begin the data upload process, first navigate to the dataset editor.

Redivis supports a wide variety of data types, sources, and upload methodologies. When beginning to upload data, you should first ask:

What type of data is it?

• Tabular data

• Geospatial data

• Unstructured data

Where is the data?

• A computer

• Google Drive

• Box

• Google Cloud

• AWS

• Another table on Redivis

• ...etc

Where is the metadata?

• Embedded in the file

• In another file

How do I want to perform the upload?

• Through the browser (default)

• Programmatically, via Python or the API

Overview

Datasets on Redivis are intended as a persistent store of information, and this persistence is critical to ensure the reproducibility of analyses using the datasets (and specific versions thereof).

In some cases, it may be necessary to archive a dataset, so as to prevent future usage and save on cloud costs. In other scenarios, you may need to delete a specific version (e.g., to reduce storage costs, or permanently retract inappropriately released data). Finally, in some situations it may be necessary to fully delete a dataset, for example, when a license to that dataset has expired.

Archival operations are always reversible, whereas deletion of versions and datasets becomes permanent after 7 days.

Dataset archival

When a dataset is marked as archived, its data can no longer be queried or modified (though its metadata can still be edited). All derivate tables in workflows will be marked as archived, which prevents reading or querying the data. This can help ensure that a dataset remains inactive and limit storage costs, though the dataset may be unarchived at any time.

To archive a dataset, either right-click on the dataset when viewing the list of your datasets, or click the Archive button on the dataset settings.

Version deletion

Any version of a dataset can be deleted, as long as it is not the currently released version. Deleting this version will delete all metadata and data associated with it.

This version will no longer be available in any workflows, and researchers will lose access to any data referencing that table in their workflows. In order to continue with the dataset, researchers will need to change their analyses to a non-deleted version.

If you delete an unreleased next version, this deletion is permanent and cannot be undone. If you delete a released version, there is a 7-day window within which the version can be undeleted, after which the deletion becomes permanent.

Dataset deletion

Once deleted, the dataset will no longer be discoverable, though it will still show up in users' workflows that reference the dataset, and bookmarked URLs and DOIs will still resolve to the dataset's landing page. You can also view a list of deleted datasets by navigating to your workspace or organization administrator panel, and filtering datasets by status: deleted.

To ensure future reproducibility, dataset metadata and documentation is preserved upon deletion. However, all data will be fully expunged, and the dataset will no longer by queryable.

The dataset's public access level will be persisted in its deleted state – meaning that if the dataset was previously visible, it will still be visible (but not discoverable) once deleted. Additionally, any users who explicitly had overview or metadata access to the dataset prior to deletion will have their access persisted upon deletion. These default access rules can be modified by navigating to the deleted dataset reconfiguring access accordingly.

Datasets can be undeleted for 7 days after deletion by navigating to the dataset settings and clicking the Undelete button. After 7 days, deletion becomes permanent and the dataset's data will be unrecoverable.

Overview

The ability to view, reproduce, and build upon other works is a core tenet of the scientific process that Redivis is built to uphold and facilitate. Redivis tools for data storage and analysis comply with federal mandates for open access and automatically generate artifacts documenting all steps in the data process, ensuring reproducibility.

Background

In August 2022, the United States Office of Science and Technology Policy (OSTP) released a memorandum on public access to federally funded research (known as “Nelson Memo”) which outlined new requirements affecting both faculty and students who conduct research using federal funding:

1. Make publications and their supporting data resulting from federally funded research publicly accessible without an embargo on their free and public release.

2. Enact transparent procedures that ensure scientific and research integrity is maintained in public access policies.

3. Ensure equitable delivery of federally funded research results and data.

1. Public accessibility

Redivis is built to make data sharing as easy and safe as possible, defaulting to fully accessible data, metadata, and documentation. However, some data must be restricted for privacy and to comply with data licensing agreements. In these cases Redivis follows an "as much as securely possible" sharing philosophy for sharing.

• Multiple levels of access allows data owners to grant access to documentation, metadata, samples, and data separately, so as much access as possible can be granted.

• When research is done using restricted data, analysis code is made available when possible.

• Viewing public data and analysis workflows on Redivis is completely public and does not require a Redivis account.

• If any piece of analysis or data is restricted, a user will need a Redivis account to apply for access. Redivis accounts are completely free with no restrictions.

2. Transparent procedures

Redivis systems are built to automatically capture and document any work done on datasets and analyses in standardized formats, while giving researchers and administrators the ability to supplement or override these when necessary.

• Datasets are automatically versioned. All changes to variables and rows are recorded and made available to administrators and data viewers.

• Redivis offers a no-code interface to build transform queries that compiles to SQL code. This code is available to the analyst and any viewers.

• Workflows are self-documenting. Every step taken in an analysis workflow is recorded sequentially in a visual format that is easy to follow.

• Workflows are version controlled. Every step is versioned and time-stamped, allowing users to revert to a previous iteration, or a viewer to understand how queries might have changed over time.

• Workflow can be forked, preserving relationships between workflows and datasets.

3. Equitable delivery

Redivis is free and accessible to any researcher, reviewer, or casual data browser. By taking multiple complex data storage and analysis systems and centralizing them with clear UI, it is clearer to understand full workflows without specific technical knowledge.

• Redivis uses transparent, open-source formats, allowing data, analyses, and workflows to be exported and used within other systems.

• make data and workflows available to researchers and administrators using their own systems.

• No-code interfaces allow researchers at any skill level to build and understand research steps.

• Common, open-source coding languages (SQL, Python, R) are a foundation of tools and supported in analysis workflows.

• There is no cost to researchers to store personal data, or run data queries.

Overview

One of the strengths of Redivis is that researchers can apply for, and gain access to, data on the same page as where they discover the dataset. Moreover, all access applications are tracked, both for administrator's reference and to provide transparency and timely notifications to researchers.

Responding to access requests in a timely manner creates a great environment for researchers and takes full advantage of these automatic systems.

1. Configure notifications

You can configure multiple email addresses to receive an email when a request for access is made to your organization, as well as the frequency you receive them. Especially if you are a smaller organization that isn't getting access requests every day, we very strongly recommend that you have at least one administrator receiving emails so you know when to respond to a request.

Learn more in the Settings reference section.

2. Approve membership requests

Users must be approved as members of your organization before any other access requests can be approved.

When a membership request is pending you will see an Alert (!) on the Members tab of your workspace. Click on any member with an alert to see their pending submission awaiting approval.

Pending members will have an Approve / Reject button as soon as you click on their page, alongside the authentication information they submitted when requesting to be a member. It is important to verify that this person has a genuine email that seems consistent with who they say they are.

3. Approve requirement submissions

As users ask for approval you will receive an alert (!) in the administrator panel where your attention is needed.

Member requirements and export restriction exceptions will appear on the Member tab.

Study requirements will appear on the Studies tab.

Direct access requests will appear on the Datasets tab.

When all of the pending requests are resolved, the alert will disappear.

For each request you will have the option to approve or reject the submission. You can always come in later to revoke the approval if needed. You might also see that the member has submitted updates to their existing approval. In that case the approval is still valid while updates are pending, unless it is otherwise revoked or expired.

Learn more in the Approving access reference section.

4. Leave comments for clarity

Both administrators and members have the option to leave a comment on any requirement submission. The act of leaving a comment will trigger an alert for administrators, or a notification for members.

Especially when you are rejecting or revoking a requirement submission, we recommend leaving a comment to let the member know what they did wrong, or what they still need to do to gain access.

5. Audit access

It's often a good idea to double check which researchers have access to your datasets.

In order to audit a particular member's access to your organization's datasets, double click on that member, and navigate to the Access overview tab. This will list all of your organization's datasets, and the member's corresponding access to each.

In order to audit all members that have access to a particular dataset, click the filter icon to the right of the member search bar. This will allow you to filter all members by their access to, and usage of, your organization's datasets.

Next steps

Expand your reach

If you are part of a larger institution, you can contact us about getting an institution-wide Redivis page to help users discover new organizations and datasets.

Overview

All activity surrounding your organization and it's data usage is logged and accessible to organization administrators. This can be helpful for year-end reports or any potential data security concerns!

You can easily organize, visualize, and download any of your organization's information by generating a report in your administrator panel.

1. Create a report

Navigate to the Reports tab of your administrator panel and click the + New report button.

In the future you can return to this page to see your existing reports and create duplicates to further edit.

2. Choose the entity of the report

Start by choosing the entity, or topic, of the report. Whatever you choose here will define what each entry in your report represents.

You can choose a Resource such as Members, Workflows, Studies, or Datasets which will allow you to get a sense of the named entities your organization owns or has interacted with.

Or you can choose a Usage event such as Queries, Notebooks, Exports (or all of these combined) which will allow you to see how your organization's data has been used.

3. Define constraints

From here, define the other criteria of your report:

Time frame

If you choose a usage event as your entity you will need to set a time frame to define which events will be included. (Resources don't have a time associated with them since they are not events.)

Aggregation

If you chose a usage event you can optionally choose to aggregate events based on a resource or period of time. Perhaps you want to see all queries grouped by member, or want to see all notebooks per day. You can also create custom aggregation groups by using member labels and dataset labels with colons. E.g. labeling members with school:medicine and school:business would allow you to aggregate a report by school.

Filter

You can additionally choose a filter that will restrict your report to only events or resources that meet your chosen criteria. For example, if you want to see all usage events that referenced a particular dataset.

Learn more in the Reports reference section.

4. Define the report's fields

Finally, you'll need to select what fields or topics are present in the report. Every field selected here will become a column in your resulting report table. Different fields are available based on what entity you have selected.

Some fields are selected by default that might be helpful but you can make your own selections. For example, maybe you have a report of data usage and you want to include the average number of tables referenced in each notebook session, or the max amount of compute used in queries per day.

5. View the report results

Click the Create button to generate this report! This will generate a table with entries that match all the criteria you've defined above. You can browse this table to understand the data and make sure it looks how you'd expect.

If your report contains any numeric data you can click the Visualize tab to see a visual representation of 1-2 of your numeric fields. The type of graph will be chosen by default based on the shape of your data. You can hover on this graphic to learn more about each entry.

5. Iterate and export

You can update this report by clicking the Edit button. Changing any criteria will refresh the table and visual with new contents based on your new criteria.

This report is now saved in your organization's reports and you can access it in the future.

Any time you edit the report or come back to it after more than one hour has elapsed, the table will be automatically regenerated. Based on the time frame you've selected this might drastically change the contents.

If you'd like to work with the data in a different environment you can download the report table as a csv by clicking to Download button.

Next steps

Expand your reach

If you are part of a larger institution, you can contact us about getting an institution-wide Redivis page to help users discover new organizations and datasets.

By default, you may upload data from your local computer, a public URL, or from another dataset or workflow on Redivis. However, Redivis supports numerous integrations for data ingest across common sources. You'll need to enable data sources in your account workspace settings in order to import data they contain.

When you the enable the data source, you'll be prompted to log into the corresponding account. Redivis will only ever read data from these sources when explicitly requested, and it will never modify or overwrite content.

Once configured, you'll see any added data sources appear as an option when uploading data.

Available data sources

Google Cloud Storage

You may import any object that you have read access to in GCS by specifying a bucket name and path to that object, in the form /my-bucket/path/to/file. You may import multiple objects at once by providing a prefix followed by wildcard characters, e.g.: /my-bucket/my-folder/* .

The following wildcard characters are supported:

• * : Match any number of characters within the current directory level. For example, /my-bucket/my-folder/d* matches my-folder/data.csv , but not my-folder/data/text.csv

• ** : Match any number of characters across directory boundaries. For example, my-folder/d** will match both examples provided above

• ? : Match a single character. For example, /my-bucket/da??.csv matches /my-bucket/data.csv

• [chars] : Match any of the specified characters once. For example, /my-bucket/[aeiou].csv matches any of the vowel characters followed by .csv

• [char range] : Match any of the range of characters once. For example, /my-bucket/[0-9].csv matches any number followed by .csv

Amazon S3

You may import any object that you have read access to in S3 by specifying a bucket name and path to that object, in the form /my-bucket/path/to/file. You may import multiple objects at once by providing a prefix followed by a wildcard character, following the same syntax and rules as outlined for Google Cloud Storage above.

Google Drive

You may import any file of valid format that you have stored within your Drive, including Google Sheets. Upon choosing as your import source, a modal will open that will allow you to browse and select files from your Google Drive.

Google BigQuery

You may import any table that you have read access to in BigQuery, including views, materialized views, and external tables. You must specify the table in the form project.dataset.table . To import multiple tables within a dataset, you may use wildcards. E.g., project.dataset.* or project.dataset.prefix* .

Box

You may import any file of valid format that you have stored within Box. Upon choosing as your import source, a modal will open that will allow you to browse and select files from your Box.

OneDrive

Coming soon. Please contact [email protected] if this integration would be helpful for your use case so that we can prioritize.

Redivis

You can import any table, which can be particularly helpful with ETL workloads where you want to import a cleaned version of your data (example). You can also import any uploaded files into your table, supporting workflows where tabular data is initially loaded as a file before being loaded into a table.

You must both have data access and the ability to export any table or file that you import.

Importing tables

You can reference any table on Redivis using the form user|organization.dataset|workflow.table. That is, specify its owner (a user or organization), its containing entity (a dataset or worfklow), as well as the table name, separated by periods.

Importing files

All files on Redivis belong to a file index table. To import a file, first specify the index table, followed by a forward slash (/), and then the file name. E.g.:

user_name.dataset_name.table_name/file_name.csv

To import multiple files at once, you can use wildcard characters, following the same pattern rules as specified for Google Cloud Storage above. E.g.:

user.dataset.table/prefix*.csv

Overview

For datasets with large tables, it is often a good idea to include a 1% sample of the data, supporting faster exploratory queries as new researchers work to understand your data. Moreover, if a sample is configured, you will have the ability to control access to that sample separately from the full dataset.

Sampling is applied independently to each version of a dataset. You may modify the sampling methodology on a version at any time — even after it's been released — though keep in mind that this may affect researchers that are currently working with the dataset sample. As a best practice, it's good to configure and validate your sample before releasing a new version.

To configure sampling on your dataset, click the Configure sample button on the Tables tab of a dataset page.

Random sample

The simplest form of sampling, this will create a corresponding sample for every table in the dataset (including file index tables). Every record / file will have a 1% chance of occurring in the sample.

As a general rule, you should only use random samples if:

• You have one table in your dataset, or

• Researchers won't be joining multiple tables in your dataset together

If this isn't the case, consider sampling on a specific variable. Otherwise, as researchers join different tables together, they will start getting samples of a sample, since there is no consistent cohort of records between tables.

Sampling on a variable

For situations when you want researchers to be able to join tables within your dataset, consider generating a sample on a variable that exists in at least some of the tables in your dataset. Every value for this variable will have a 1% chance of being in the output set.

Importantly, this sampling is deterministic. This guarantees that the same values that fall in the 1% sample for one table will also occur in the 1% sample for another table in the same dataset. In fact, these sampled values will be consistent across Redivis, allowing researchers to even merge samples across datasets.

When sampling on a variable, only tables with that variable will be sampled. This is useful for the case when some tables contain supplementary information to your primary cohort. For example, consider the case when your dataset has a "Patients" table, a "Hospitalizations" table, and a "Hospitals" table. We'd likely want to create a sample on the patient_id variable, which would create a 1% subset of patients and the corresponding hospitalizations for those patients. However, this wouldn't create a sample on the "Hospitals" table — which is what we want, given that the sample of patients is still distributed across a large number of hospitals.

If your dataset contains unstructured data files, you probably want to sample on either the file_name or file_id variables.

Overview

The Limit step reduces the table to a set number of rows.

Example starting data:

/*---------+--------*
 | student | score  |
 +---------+--------+
 | jane    | 83     |
 | neal    | 35     |
 | sam     | 74     |
 | pat     | 62     |
 *---------+--------*/

Example output data:

Limit to 2 rows

/*---------+--------*
 | student | score  |
 +---------+--------+
 | jane    | 83     |
 | neal    | 35     |
 *---------+--------*/

Step structure

• There will be one limit block where you will define your limit.

Field definitions

Example

Let's say we are doing initial exploratory work on data. We did a number of complex steps and want to quickly execute the query so we can iterate.

Starting data:

/*---------+-------+---------+------------*
 | test    | score | student | date       |
 +---------+-------+---------+------------+
 | quiz    | 83    | jane    | 2020-04-01 |
 | quiz    | 35    | pat     | 2020-04-01 |
 | quiz    | 89    | sam     | 2020-04-01 |
 | midterm | 74    | jane    | 2020-05-01 |
 | midterm | 62    | pat     | 2020-05-01 |
 | midterm | 93    | sam     | 2020-05-01 |
 | final   | 77    | jane    | 2020-06-01 |
 | final   | 59    | pat     | 2020-06-01 |
 | final   | 92    | sam     | 2020-06-01 |
 *---------+-------+---------+------------*/

Input fields:

Output data:

/*---------+-------+---------+------------*
 | test    | score | student | date       |
 +---------+-------+---------+------------+
 | quiz    | 89    | sam     | 2020-04-01 |
 | midterm | 62    | pat     | 2020-05-01 |
 | midterm | 93    | sam     | 2020-05-01 |
 | final   | 92    | sam     | 2020-06-01 |
 *---------+-------+---------+------------*/

Cumulative distribution

Return the relative rank of a row defined as NP/NR. NP is defined to be the number of rows that either precede or are peers with the current row. NR is the number of rows in the partition. –> learn more

CUME_DIST()

Return type

float

Parameters

Dense rank

Returns the ordinal (1-based) rank of each row within the ordered partition. All peer rows receive the same rank value, and the subsequent rank value is incremented by one. –> learn more

DENSE_RANK()

Return type

integer

Parameters

N-tile

Divides the rows into a set number of buckets based on row ordering and returns the 1-based bucket number that is assigned to each row. –> learn more

NTILE(@literal)

Return type

integer

Parameters

Percent rank

Return the percentile rank of a row defined as (RK-1)/(NR-1), where RK is the RANK of the row and NR is the number of rows in the partition. –> learn more

PERCENT_RANK()

Return type

float

Parameters

Rank

Returns the ordinal (1-based) rank of each row within the ordered partition. All peer rows receive the same rank value. The next row or set of peer rows receives a rank value which increments by the number of peers with the previous rank value –> learn more

RANK()

Return type

integer

Parameters

Row number

Returns the sequential row ordinal (1-based) of each row for each ordered partition. If the no order condition is specified then the result is non-deterministic. –> learn more

ROW_NUMBER()

Return type

integer

Parameters

Redivis has some built-in methods for helping you find and manage all of your organization's members, workflows, datasets, and access systems that you can use to ease your workflows.

In this example we will imagine that we need to get in touch with everyone working with a specific dataset.

1. Locate the dataset in the administrator panel

We can go to the Datasets tab of the panel and search or filter to locate our dataset.

2. Find relevant workflows

We can right click on this dataset and hover on View related to see all of the options relating to this dataset. In this case we want to see all workflows that have used this dataset.

Clicking this option navigates us to the Workflows tab, with a filter active for this dataset. This list is showing all workflows that contain this dataset.

3. See related members

Click the Select all checkbox to select all of these datasets. With these selected we can right click on any of them (or click the Actions button that appears in the top right) to View related activity.

In this menu we should select Members in workflows. This will navigate us to the Members tab filtered by all members who are owners or collaborators on the workflows we had selected.

4. Show emails

Now that we have narrowed it down to the members we are interested in, we can right click on the header (or the options menu in the top right of the list) to edit the columns shown. In this menu we can add the Contact email column.

The quickest way to get this information out of Redivis is to go back to that same menu and click Download CSV to begin downloading a CSV of the list we are looking at exactly as you've configured it (all columns, filters, and sorting will be preserved).

Next steps

We can use these similar steps to view:

• Queries run on a specific set of datasets

• Exports made from a workflow

• Requirements a member has filled out

And many more!

JSON query

Extracts a JSON value and converts it to a SQL JSON-formatted STRING. –> learn more

JSON_QUERY(@expression, @expression_2)

Return type

string

Parameters

JSON value

Extracts a JSON scalar value and converts it to a SQL JSON-formatted STRING, removing outermost quotes and un-escaping return values –> learn more

JSON_VALUE(@expression, @expression_2)

Return type

string

Parameters

Citing Redivis

Publications and other outputs should cite Redivis as research software, using the appropriate conventions for your publication medium.

• The Digital Object Identifier (DOI) for Redivis is:

• The canonical URL for Redivis is:

If referencing Redivis inline, the abbreviated citation can be used:

Redivis [https://doi.org/10.71778/V2DW-7A53]

The following BibTeX entry can be used:

These identifiers are also associated with Redivis, though do not need to be included in all citations:

• Re3data identifier:

• RRID:

Describing Redivis in grants and publications

Overview

Redivis is a secure, scalable, cloud-based data platform developed to meet the needs of academic research. Redivis was first developed in collaboration with the and is now deployed across a number of leading research institutions [1], supporting the distribution and analysis of large, sensitive datasets across multiple disciplines in alignment with FAIR data practices.

Users of Redivis are able to upload large-scale numeric, text, structured, and unstructured data directly through the browser or via APIs and other integrations. Built-in tools are available to curate and tag rich metadata, maximizing the shareability of datasets. The platform allows for robust search and exploration across multiple datasets and their metadata and variables.

Additionally, Redivis provides a rich toolset for analysis and exploration. Users can filter, merge, analyze and visualize billions of records in real-time, and can easily bring together disparate datasets to answer novel questions. They can leverage a massively-parallelized architecture to execute SQL queries (either composed as code or through a graphical user interface), in addition to customizable Jupyter notebooks running Python, R, Stata, and SAS. REST APIs give users and applications additional programmatic interfaces to the data, ensuring interoperability with other tools and ecosystems.

Technical infrastructure

The platform is built on Google Cloud Platform infrastructure using open-source software. The main application runs in containerized services orchestrated with Kubernetes. It integrates high-performance tools including Google BigQuery and its ANSI-SQL interface for large-scale tabular data processing and JupyterLab for interactive analytics. Researchers may provision environments preconfigured for R, Python, SAS, and Stata, with support for customizable environments. Compute capacity scales dynamically to meet workload demands, with configurations available up to 416 CPUs, 11.5 TB of RAM, and 16 NVIDIA A100 GPUs, facilitating complex statistical and machine learning workflows on terabyte-scale datasets.

Reproducibility

Redivis is designed such that reproducibility is an automatic byproduct of researchers' use of the platform. A novel version control system for datasets enables efficient data updates without duplication, supporting full reproducibility and cost-efficient storage management. All analytical activity — including code, queries, and derivative outputs — is tracked and recoverable, ensuring transparency and compliance with NIH data-sharing policies.

All datasets, workflows, and versions thereof can be assigned a unique Digital Object Identifier (DOI), allowing for researchers to persistently link to a fully-reproducible artifact of their research. Future investigators, assuming they have appropriate access to the underlying data, can then re-run these analyses and produce identical results, in turn modifying and building upon prior works.

Security

Redivis supports rigorous data governance with a tiered attribute-based access control (ABAC) system, customizable user agreements, egress restrictions, and an intuitive, searchable audit trail.

Redivis has been audited and approved for the use of FERPA, PII, PHI, and HIPAA data. The platform is SOC2 and NIST 800-171 (rev 3) compliant and undergoes regular audits and penetration testing. All data and metadata are stored in a multi-redundant, AES256 encrypted datastore. All connections to the platform are over an encrypted TLS 1.2 or greater protocol. User login is managed by single sign-on via and, allowing users and collaborators to use their institution credentials to authenticate. The platform supports HTTP strict transport security (HSTS) and is on the for all major browsers, preventing users from establishing an unencrypted connection.

Data administrators on Redivis have access to detailed, searchable audit logs, and reports can easily be generated for auditability and traceability. The platform allows for the restriction of data exports and downloads as well as the automatic expiration of dataset access. For highly sensitive data, multi-layer protection exists to prevent accidental data sharing, viewing, and downloading. Redivis also allows for the complete deletion of data, including the automatic and instantaneous deletion of all data derivatives, as needed.

—

[1] , , , , ,

Overview

Most data on Redivis is uploaded through an Organization, and that is the best place to find Datasets to work with. Some datasets will be public, while others will require certain steps before you can get full access to the data.

1. Find datasets

From an Organization's home page, click on the Datasets tab to browse all datasets and filter by their metadata.

You can use filters on the left bar to narrow down your results. All searches perform a full-text search across a dataset, its documentation, tables, variables, and rich metadata content.

If your organization is part of an institution (such as or ) you can also go to the institution page to search all datasets in your institution.

Not finding what you need? You can also to Redivis through your workspace to augment a research project you're working on and to share with other researchers. Or if you represent a research group, center, or institution, about setting up your own Organization page.

2. Get to know a dataset

Click on any dataset title to go to that Dataset page.

You can view the metadata (including an abstract, documentation, citation information, and a version history) and dig into the data directly on the Tables and/or Files tab.

Each table contains a data browser to view cells, generate summary statistics, and display metadata for this table and each variable. Each file can be previewed and downloaded. Both tables and files can be added to workflows for analysis.

Many datasets on Redivis are public, while others have requirements for certain levels of access enforced by the data owner.

On the right side of the dataset, you can see your current access level. If you have Metadata access you can see the variable information and summary statistics, but you'll need to gain Data access in order to view the data contents or work with this data.

Learn more in the guide.

3. Apply for access

On any restricted dataset you can click the Apply for access button in the top right of the page to see a list of steps required for you to gain different access levels to this dataset.

The first step is to become a member of this organization. When you go to apply you will be prompted to become a member of this organization using the credentials on your account.

Once you're a member of this organization, you might need to fill out some requirements that administrators have set up, or request access directly from them.

Requirements are forms that you will need to complete on this page and then submit. Once these are approved by your organization's administrators you will gain access to the dataset.

Learn more in the guide.

Next steps

Start working with your data

Once you have data you're interested in, bring it into a workflow to transform and analyze it leveraging lightning fast tools from your browser.

Learn more in the guide.

Upload your own datasets

Augment your data analysis in Redivis by uploading your own datasets, with the option to share with your collaborators (or even the broader research community).

Learn more in the guide.

Overview

Many datasets on Redivis are public, while others have requirements for certain levels of access enforced by the data owner. If the dataset is public, you can skip over this section, otherwise you'll need to gain access before you can fully utilize the data.

1. View access rules

On the right side of all restricted dataset pages is a box which displays your current access level. If this doesn't say Data access, then your next step will be to apply for access.

Click the Apply for access button on the top of the page to open the access modal. This modal allows you to manage your access to this particular dataset throughout the platform.

Access levels

Dataset access has five levels:

1. Overview: the ability to see a dataset and its documentation.

2. Metadata: the ability to view variable names and summary statistics.

3. Sample: the ability to view and query a dataset's 1% sample. This will only exist for datasets that have a sample configured.

4. Data: the ability to view and query a dataset's tables, and work with them in .

5. Edit: the ability to edit the dataset and release new versions.

Access levels are cumulative. For example, in order to gain data access you will need to have gained metadata access as well.

Usage rules

Usage rules are restrictions placed on the way you can use this data once you've gained access. For example, data exports may be restricted to certain export environments, and/or require administrator approval.

While no action is required when you first apply for access a dataset, it's helpful to understand these restrictions to better understand what limitations, if any, they might pose to your research.

2. Apply for access

The access modal for the dataset will include all steps designated by this dataset's administrators to gain access to work with data.

Membership

To apply for access to datasets hosted by a Redivis organization, you will first need to Join the organization.

To join an organization, you will need to provide information about your identity via an email authentication, usually the email associated with your Redivis account.

After submitting your membership application, it will either be automatically approved or will be marked as pending while an administrator reviews your submission. You'll receive a notification if your submission is approved or rejected.

Requirements

Some datasets have requirements as part of their access applications. Requirements are global to an organization and usually contain some type of form. Once all requirements for a particular level of access are approved, you will gain access to the relevant dataset(s).

In the access modal, click the Apply button on any requirement to fill out and submit the form.

When you submit a requirement, an administrator of the organization will be alerted and can review your submission for approval (in some cases, the requirement may be configured for auto-approval upon submission). You will receive a notification if your submission is rejected or approved, or in the future when it's about the expire. You can also leave a comment for an organization's administrators here, and they can reply to you.

There are two types of requirements - those you fill out on behalf of yourself (member requirements) and those you fill out on behalf of a (study requirements). If the dataset you are interested in has a study requirement, you'll need to create a study and select it in the access modal in order to submit the requirement. Additionally, any workflows that utilize this dataset will need to be a part of the approved study.

Direct access

Sometimes you don't need to fill out any requirements and need to simply Request access at a specific level. You can do so in this modal, and will receive a notification when you are granted access by the dataset owner.

3. Wait for data owner approval

For some membership and requirement submissions (and all direct access requests), you will need to wait for the data owner to respond to your submission. If the dataset is owned by a user, they will need to approve your request; if it is owned by an organization, any administrator at that organization can approve it.

The data owner will receive a notification as soon as the requirement is submitted, and you will receive a notification when it is approved, rejected, or for any requests for additional information. By default, you will receive an email as well as a notification within Redivis. You can customize this notification behavior on your .

4. Re-authenticate (if necessary)

If you are accessing restricted data hosted by an organization, you may have to re-authenticate after a period of inactivity anytime you perform an action governed by the "Data" access level.

After a period of inactivity (set between 15 minutes and 3 days by the organization administrator), a popup will ask you to re-authenticate anytime you view data, query tables in a transform, or export data from Redivis.

Next steps

Start working with your data

Add this dataset to a workflow to transform and analyze it leveraging lightning fast tools from your browser.

Learn more in the guide.

Overview

Redivis determines variable names and types during . Additionally, it will automatically parse certain metadata based on the uploaded file format:

• SAS (.sas7bdat): labels

• Stata (.dta): labels and value labels

• SPSS (.sav): labels and value labels

For other file types (e.g., csv), you will need to augment the metadata directly. To apply metadata in bulk, you can upload a file containing metadata information directly from your computer. This file can either be a CSV or JSON.

CSV metadata format

The CSV should be formatted without a header, with each row corresponding to a variable, with column 1 as the name, 2 as the label, 3 as the description. If the variable doesn't have a label or description, leave these columns empty.

For example:

JSON metadata format

When uploading a JSON file, specify the name, label, description, and valueLabels using the appropriately named attributes in the object corresponding to each variable. If the variable doesn't have a label, description, or value labels, you don't need to include these attributes.

For example:

Overview

Datasets are the core entity on Redivis, and finding the dataset(s) you want to work with is generally one of your first steps. All datasets are either uploaded by an organization or a user. If a user has uploaded a dataset they will need to share it with you directly, but you can browse datasets upload by any organization on their home page.

1. Find your organization

There are many ways to find your organization. If you know what you're looking for you can type it into the "Go to" bar on the top of the page. To browse all organizations you can go to the Organizations tab of your workspace where you can Find organization.

If your organization is part of an institution (such as or ) you can also go to the institution page to browse all organizations in that institution.

Note that some organizations may be hidden from public view, in which case you will need to contact someone at the organization to get the direct link.

2. Browse available datasets

On the organization home page you can see all of the organization's datasets that you have at least Overview access to. If you're looking for a dataset you expect to be here and you don't see it, you might need to contact the organization administrators to get access.

You can see the datasets that this organization has featured on the left section of the page, type a term into the search bar, or click the Datasets tab to browse everything. By default these datasets are sorted by popularity but you can reorder them however you'd like to find what you want.

You can also click the Workflows tab to see workflows featured by this organization which showcase how their data can be used. More on workflows later!

3. Search for data

Once on the dataset tab you can use multiple filters along with search terms to find what you'd like. All searches take a comprehensive look at information from the dataset documentation as well as variable metadata, including value labels.

You can also use the filters to find specific information you're looking for. If you know the name of the variable or year of the data you're interested in, narrow your search here.

4. Preview tables

Each dataset that appears here matches all search and filter configurations. You can click on the dataset tile to see more information, including a preview of all of the tables this dataset contains.

Click on any of these tables to dig into it further. As long as you have metadata access you can see the variable information and summary statistics.

Once you find a dataset you want to explore further, click on the title to go to the dataset page.

5. Explore the dataset page

This dataset page has all of the information you'll need to familiarize yourself with the data and get started working with it further. Add this dataset to your dataset library to easily find it later on by clicking the button alongside the dataset title.

Overview

This tab has a short summary of the dataset along with different sections filled out by the administrators such as the Methodology, Documentation, and Provenance information. This Provenance section will also have citation information and a DOI if one has been created. You can also see your current access level on the right side of the page.

Tables

This tab contains all of the dataset's tabular data. You can see each table alongside information about its contents. Click on each one to dig deeper into the data.

Files

This tab contains all of the dataset's unstructured data. You can preview each file by clicking on it.

Usage

This tab has information about how other people have used this dataset on Redivis. You can see the most popular variables across all tables to get a sense for which ones might be a good starting place to understanding the structure of the data.

Next steps

Apply for access to restricted data

If you don't have data access to this dataset you'll need to apply for access before you can work with it further

Learn more in the guide.

Start working with your data

Add this dataset to a workflow to transform and analyze it leveraging lightning fast tools from your browser.

Learn more in the guide.

Overview

Tables are brought into a workflow via its , and are then created as the output of various and in the workflow. Tables often serve to materialize intermediary results, sanity check an output, as well as final data derivative representing the culmination of your analysis pipeline.

Workflow tables follow the behavior and functionality of .

Usage in workflows

All table nodes have one upstream parent, either a data source, a transform, or a notebook.

On each table, you can create any number of transforms or notebooks, which then reference that table as their "source". Tables can also be joined into transforms and notebooks that do not have the table as their primary source.

Table node states

As you work in a workflow, nodes colors and symbols will change on the tree view to help you keep track of your work progress.

Overview

When working with non-public data on Redivis, it's often important to be able to authoritatively attest to their affiliation with a given institution or other entity. To this end, Redivis supports Single Sign-On (SSO) through most academic institutions, as well as the ability to establish identity through a Google account or validated email address.

Institution SSO via SAML

Redivis is a registered service provider within the US-based , which in turn is part of the , enabling secure, authoritative SSO across thousands of universities around the world, via the SAML 2.0 protocol. If you are a member of an academic institution (as well as certain other research enterprises), you can search for your institution by name and log in to Redivis through your institution's sign-in page.

Troubleshooting institution SSO

In most cases, logging in with your institution will "just work". However, due to inconsistencies in how certain standards are applied around the world, you may run into issues when logging in through your institution. These issues can often be resolved with a quick ticket with your IT support desk – we recommend that you direct them to this page and copy [email protected] so that we may provide further technical information if needed.

Some common issues are outlined below:

Your institution does not support the Redivis service provider

If, when choosing your institution to log in, you are immediately presented with an error page (before you can type in your password), this likely means that your institution needs to add Redivis to some sort of "service provider allowlist". As a registered service provider within InCommon / eduGAIN, most institutions will automatically accept login request from Redivis – but some require manual configuration. In this case, your IT desk will need to take a quick action to enable Redivis – it will likely be helpful to direct them to Redivis's SAML metadata, found here:

Redivis was unable to determine identity

This error will occur after you've logged in with your institution, upon being redirected back to Redivis. In this case, the authentication request completed successfully, but your institution didn't provide enough information for Redivis to know who you are (which is important in order for you to apply for restricted data, so that the data distributor can be confident of who they're granting access to!).

Redivis requires all institution identity providers to provide some minimal information about the individual, such as name, email, and a persistent identifier. These are codified as the "". If your institution uses OpenAthens for SSO, you can to learn more about releasing these attributes.

Other error messages

While uncommon, it's certainly possible that other errors might occur when logging in through your institutional credentials. If you do, please contact [email protected] and we'd be happy to help you troubleshoot.

SSO via Google

Redivis also supports the ability to sign in via any Google account. This can be a personal gmail account, or via your organization if it supports Google single sign-on. When you sign in with Google, your name, email, and an opaque persistent identifier will be shared with Redivis.

If your institution supports Google sign-on, but is also listed as a SAML identity provider (see above), the SAML SSO will be preferred. If you try logging in via Google, you will be redirected to your institution's login page.

Email sign-on

If your institution isn't listed and doesn't support SSO through Google (e.g., many @.gov emails), you can also sign in via any email address.

Redivis will send a unique code to this email every time you log in, making it such that the account owner continuously "proves" their ownership of the given email address.

Overview

Redivis workflows contain powerful tools to reshape and analyze data but if you prefer to export data into a different workflow you can easily do so. Redivis systems use open source tools and common formats to make the transition as easy as possible.

1. Check export restrictions

Some datasets on Redivis have export restrictions which prohibit or limit removing data from the system.

You can check this by going to the dataset page and clicking the top right button Manage access, or by right-clicking on a dataset in a workflow and selecting the View access option.

The bottom section of this access modal defines any export restrictions in place. It might not have any restrictions, might be completely restricted, or might have some limited options for export.

If there are limited options for export, this section will detail the available locations and any restrictions on using those (such as the size of the table being exported). These restrictions are enforced automatically when you try to take actions within the system.

If there are no options for export, you still have the option to work with your data in a Redivis workflow, where you can reshape and clean your data using transforms, and then analyze it in notebooks.

Learn more in the guide.

2. Prepare your data

If you would like to export the entire table, you can skip this step.

Otherwise, we recommend that you use transforms to cut your data down to a smaller size and reshape it into the table format you need for analysis before exporting it. Especially for large datasets, Redivis tools are created specifically for these purposes and work seamlessly in the browser with no additional setup.

Learn more in the guide.

3. Export data

You can open the Export modal to initiate an export from any table. You can do this on the dataset page by right clicking a table and selecting the Export menu option, or by right clicking on any table in a workflow and selecting the same option.

Here you can see all the options available for your table to export.

If your table has export restrictions set by the data owner, options will be disabled here. In the case where your table does not meet export requirements but the data owner allows exception requests, you will see the option to do so here.

Download

The first tab of this modal gives you different options for formats to download your data. Most common data formats are supported, including csv, json, avro, parquet, SAS, Stata, and SPSS. Select the format you'd like and click the Download button to start downloading the file.

Learn more in the reference section.

Programmatic reference

You can reference this file from your computer or another computational environment using the Redivis Python and R libraries. This modal gives specific information on how to reference this dataset, and you can reference our docs for other options.

Integrations: Google Looker Studio

This is a free Google dashboard visualization program you can directly link your table to. Use their point and click interface to build common visuals that will update as the underlying table updates.

Learn more in the reference section.

Integrations: Google Cloud Storage

You can export your table directly into a Google Cloud Storage bucket that you have access to.

Learn more in the reference section.

Integrations: Google BigQuery

You can export your table directly into a Google BigQuery project that you have access to.

Learn more in the reference section.

Next steps

Cite datasets in your publications

If the work you're doing leads to a publication, make sure to reference the dataset pages from datasets you've used for information from the data administrators on how to correctly cite it.

Query optimization

The Redivis transform connects to a highly performant, parallelized data store. Queries on terabytes of data can complete in seconds, often utilizing the resources of thousands of compute nodes. These are some best practices that can help you increase performance.

Limit output table size

Table writes are generally much slower than table reads — if your output table is exceptionally large, it may take the querying engine several minutes to materialize the output. Try restricting the number of rows returned by applying row filters to your transforms when possible, and be cognizant of joins that may substantially increase your row count. Avoid keeping variables that aren't needed.

When you are performing initial exploration, you may consider using a limit step in your transform to reduce the output table size.

Reduce the number of new variables

Each new variable adds to the computational complexity of the query; the new variable must be computed for every row in the table.

A common anti-pattern is to construct numerous boolean CASE new variables, and then use the result of these new variables in the row filter(s). If possible, it is far more efficient to inline the CASE logic within the row filters, or within fewer new variables, as this allows for improved efficiency in logical short-circuiting.

Optimize join patterns

When your query utilizes a join step, consider the order in which you are joining the data. The best practice is to place the largest table first, followed by the smallest, and then by decreasing size.

While the query optimizer can determine which table should be on which side of the join, it is still recommended to order your joined tables appropriately.

If all of your joins are INNER joins, the join order will have no impact on the final output. If your query leverages combinations of left / right / inner joins, the join order may affect your output; be careful in these cases.

Common errors

The Redivis transform will prevent you from initiating a run when any steps are invalid, and errors should be rare. However, some errors can only be detected as the query is run — in these cases the job will fail as soon as it encounters an error, logging the error message to the top of the transform.

If you come across an error message not on this list, please email [email protected] for further assistance.

Resources exceeded during query execution

This error occurs when a query utilizes too much memory, yet it is often easily resolvable and due to unintended behavior within the transform. This error is caused by a certain component of the query not being parallelizable, which often occurs when combining and / or ordering many distinct values. We recommend investigating the following culprits:

1. Investigate any order clauses in your transform — either at the bottom of the transform or in a partitioned query. Often, attempts to order on hundreds of millions or billions of distinct values will fail. Note that ordering rows at the bottom of the transform does not affect your output, and is only useful in preparing your data for export.

2. Confirm that none of your aggregation methods are creating massive cells. For example, using the String aggregate method on an exceptionally large partition can collapse and concatenate many values into one record — if the cell becomes too big, this error will be thrown.

Cast / type conversion errors

When converting between variable types, all values must be appropriately formatted for conversion to the new type. For example, the value "1,000" is not a valid integer and will throw an error when being converted from a string to an integer.

There are several options for getting around cast errors:

1. Choose the "If invalid for type, set to null" option in your retype blocks. Note that this will set all invalid values to NULL, potentially causing unintended side effects. Use with caution.

2. Filter out all records that have incompatible values.

3. Create a new variable, using the Case method to convert any invalid values to something that can be appropriately cast.

Maximum table size

If an output table is more than 1TB, it cannot exceed the size of the sum of all source tables, + 10%. Very large output tables that substiantially exceed their inputs are typically the result of a misconfigured join that generates a cross-product between a one-to-many or many-to-many relationhip between multiple tables. If you encounter this error, try to apply filter and aggregation steps first, and also validate that your join conditions are appropriately specific.

Too many tables, views and user-defined functions for query: Max: 1000

This error may occur when running queries on tables belonging to an unreleased version, particularly when these tables are made up of hundreds of independent uploads. Under the hood, these unreleased tables are represented as a logical view that stitches the various uploads together into a single table. If your query references multiple unreleased tables, with each approaching the 500 per-table upload limit, it's possible to exceed the total allowed number of tables referenced by a query.

To work around this issue, you can create a transform that simply selects all variables from the unreleased table, materializing the result in your workflow. This output table will now only count as a single table in your query, avoiding this error.

This error will also no longer be an issue once the version is released, as the table is materialized shortly after a version is released.

Other errors

If you come across any other errors or issues while using the transform please contact us directly at [email protected]

Overview

Redivis is a platform that allows researchers to seamlessly discover, access, and analyze data. This brief guide will walk you through the basics to get up and running and provide a launching point for exploring other resources in this documentation.

1. Create your account

Many datasets on Redivis are public, and you can browse them without creating an account. However, you'll need an account in order to analyze data, as well as to apply for access to restricted datasets.

Click the Create account button in the top right of any page to sign up. You can use your academic institution's login credentials or any Google account.

Once you create an account, you'll be navigated to your Workspace, which is your private area for creating workflows to work with data, uploading your own datasets, and managing your Redivis account.

2. Join your organization

The main entity on Redivis is a dataset. Most datasets on Redivis are uploaded by organizations.

You might already know the organization that you’ll be working with, or you can browse organizations by clicking the Find organizations button on the Organizations tab your Workspace. You might know your organization's URL already, or see it listed in the Recent section of the Dashboard if you were already viewing that page.

Once you join an organization, you'll be able to apply for any of their restricted datasets, and also see a link on the left bar of your workspace to quickly get back to the organization's home page.

3. Find datasets

The best place to find datasets on Redivis is to search on an organization's home page. Here you can click on the Datasets tab to browse all datasets and filter by their metadata. All searches perform a full-text search across a dataset, its documentation, tables, variables, and rich metadata content.

If your organization is part of an institution (such as Stanford or Columbia) you can also go to the institution page to search all datasets in your institution.

Click on any dataset title to go to that Dataset page where you can view the data and metadata it contains. The data for this dataset will be available on the Tables tab, which you can explore further.

You will also probably come across restricted datasets. For these you will need to click the Apply for access button on the top right of the Dataset page and complete the requirements to gain approval from the dataset's administrators.

Learn more in the Discover & access data guide.

4. Analyze data

Once you've found a dataset that you want to work with, you can add it to a Workflow. Workflows are the fundamental analysis interface on Redivis, where you can query, merge, reshape, and analyze any dataset that you have access to — all from within your web browser.

Add a dataset to a workflow by clicking the Add to workflow button on the top right of the dataset page.

In a workflow we can create a Transform by selecting any table and clicking the +Transform button, which allows us to combine and reshape our data into a final output table that best serves our analysis. These transforms use a powerful SQL engine under the hood, allowing us to query incredibly large tables - even billions of records - in seconds.

After creating output tables for analysis, we can create a computational Notebook in R, Python, Stata, or SAS to further analyze our data and develop our final figures. Select any table and click the +Notebook button to get started. The notebook will initialize and pull in the table you've selected (or a sample of the table if it is a large table).

We can also export and query this table from external environments, allowing us to use whatever analytical tools best suit our research question by clicking the Export table button on the right side of any table. Here we can download the table in a number of common formats, interface with the it programmatically via the API, or export the table to supported integrations.

Learn more in the Analyze data in a workflow guide.

Next steps

Upload your own datasets

Augment your data analysis in Redivis by uploading your own datasets, with the option to share with your collaborators (or even the broader research community).

Learn more in the Create & manage datasets guide.

Administer your organization

Organizations allow for research groups and centers to securely distribute data to their research community. Organization administrators can create datasets, manage access, review logs, and create customized reports of their data utilization.

Contact an existing administrator to add you to their organization, or contact us to set up a new organization. Learn more in the Administer an organization guide.

Overview

Notebooks on Redivis offer a performant and highly flexible environment for doing data analysis. This includes the ability to run state-of-the-art machine learning (ML) models, including the ability to train new models, fine-tune various existing models, and use these models to perform inference and generate novel outputs.

This guide is focused on a number of common use cases when running ML workloads on Redivis. Here we generally focus on using the Hugging Face + PyTorch ecosystem in Python, though these examples are broadly applicable to other ML libraries and languages.

For a detailed example of using Redivis to fine tune a large-language model, see the complementary example:

1. Create a notebook with appropriate computation capacity

Training ML models and running inference can require a substantial amount of compute capacity, depending on various factors such as your model and dataset size, usage parameters, and performance goals.

The default, free notebook on Redivis offers 2CPUs and 32GB of RAM. While this may work for initial exploration, running practical machine learning workflows typically requires the availability of a GPU. When creating your notebook, you can choose a custom compute configuration to match your needs.

Redivis offers a number of custom compute configurations, mapping to the various machine types available to Google Cloud. We recommend starting with a more modest GPU for initial exploration, and then upgrading as needed when computational or performance bottlenecks are reached. For this example, we'll use the NVIDIA L4 GPU, which provides reasonable performance at a reasonable cost.

2. Define dependencies

The Redivis python notebook is based off the jupyter-pytorch notebook image, with PyTorch, CUDA bindings, and various common data science libraries pre-installed. However, if you require additional dependencies for your work, you can specify them under the "dependencies" section of your notebook.

2a. [Optional]: Pre-load external models when internet is disabled

If your notebook references export-restricted data, for security reasons, internet will be disabled while the notebook is running. This can present a challenge for some common approaches to ML in python, such as downloading a model or dataset from Hugging Face. For example, we might reference a model as follows:

from sentence_transformers import SentenceTransformer
model = SentenceTransformer("all-MiniLM-L6-v2")

This code typically downloads the model from Hugging Face and caches it to our local disk. However, if internet is disabled, this command will hang and ultimately fail. Instead, we need to download the model during notebook startup, before the internet is disabled, as part of the post_install.sh script under the notebook's dependencies:

python -c '
from huggingface_hub import snapshot_download
snapshot_download(repo_id="sentence-transformers/all-MiniLM-L6-v2")
'

This will download the model weights and other files to the default Hugging Face cache directory, ~/.cache/huggingface/hub.

Now, within our notebook, we can load the cached model. Make sure to set local_files_only=True, so that Hugging Face doesn't try to connect to the internet to check for a newer version of the model.

from sentence_transformers import SentenceTransformer
model = SentenceTransformer("all-MiniLM-L6-v2", local_files_only=True)

3. Load a model

3a. Load a model from Redivis

Machine learning models can be stored directly within a Redivis dataset as unstructured files. For example, this dataset contains the various files that make up the bert-base-cased model on Hugging Face. We can then download the model to our notebook's local filesystem:

import redivis
table = redivis.organization("demo").dataset("huggingface_models").table("bert_base_cased")
table.download_files("/scratch/bert-base-cased")

And then reference this as a local model. E.g.:

from transformers import AutoTokenizer, AutoModelForSequenceClassification

model = AutoModelForSequenceClassification.from_pretrained("/scratch/bert-base-cased", num_labels=5)
tokenizer = AutoTokenizer.from_pretrained("/scratch/bert-base-cased")

When using models stored on Redivis, we don't have to worry about whether our notebook has internet access, nor do we need to rely on the future availability of that particular model on Hugging Face.

from huggingface_hub import snapshot_download
import redivis

# Download the model files from Hugging Face
snapshot_download(repo_id="google-bert/bert-base-cased")

# Specify an existing dataset and table on Redivis. 
# Consult the python docs for how to programmatically create datasets (apidocs.redivis.com)
table = redivis.organization("demo").dataset("huggingface_models").table("bert_base_cased")

# Add the downloaded model files to the table
table.add_files(directory='/home/root/.cache/huggingface/hub/models--google-bert--bert-base-cased/snapshots/cd5ef92a9fb2f889e972770a36d4ed042daf221e')

3b. Load a model from an external source

If your notebook has internet access, you can also use any other models that may be available on the internet. For example, we can load the same bert-base-cased model directly from Hugging Face:

from transformers import AutoTokenizer, AutoModelForSequenceClassification

model = AutoModelForSequenceClassification.from_pretrained("google-bert/bert-base-cased", num_labels=5)
tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-cased")

# Do cool things!

Note that if you are working in a notebook with disabled internet, this approach won't work, and you'll need to use the methods mentioned in either 2a or 3a above.

4. Load data

As a final step, you'll likely want to load data into your notebook to either fine-tune the model, perform inference, or otherwise experiment. There are thousands of datasets on Redivis, and you can upload your own data as well. You can learn more about loading data into a notebook in the python notebooks documentation. As a quick example:

import redivis

# Load tabular data as a pandas data frame, or a number of other formats (arrow, polars, iterator, etc)
table = redivis.organization("demo").dataset("ghcn_daily_weather_data").table("stations")
df = table.to_pandas_dataframe()

# Download unstructured data as files to your local disk
files_table = redivis.organization("demo").dataset("chest_x_ray_8").table("images")
files_table.download_files("/scratch/xray_images")

Of course, assuming your notebook has access to the external internet, you can also call various APIs to load external data sources.

from datasets import load_dataset

# load a dataset from Hugging Face
hf_dataset = load_dataset("Yelp/yelp_review_full") 

Next steps

At this point, you have all the tools at your disposal to perform cutting edge ML research. But of course, what you do next is totally up to you. We do recommend further familiarizing yourself with the examples and detailed documentation to take full advantage of the capabilities of Redivis notebooks:

• Example workflow, Fine tuning a Large Language Model

• Python notebook reference

• R notebook reference

This guide demonstrates using a Redivis workflow to import an existing LLM and then use relevant data to fine tune it and run it on another similar set of data we are interested in.

Workflow objective

Here, we want to fine-tune a pre-trained "foundational" LLM so that it can be used to score reviews. We will leverage an existing dataset that contains a collection of Yelp reviews and their scores to perform the fine-tuning, and then apply this classification model to other reviews (from Reddit) that do not contain an accompanying score. The goal here is to demonstrate how Redivis can be used to leverage, modify, and ultimately apply state-of-the-art LLMs to novel data.

1. Choose and explore data

For this workflow we'll need our initial data to train the model on (in this case Yelp reviews) and the data we want to apply the model to (Reddit posts). These data are already on Redivis, split across two datasets uploaded to the Redivis Demo organization: Yelp Reviews (Hugging Face) and Reddit.

Yelp reviews

To get started we want to understand this dataset and what information is in each table. We can look at the dataset page to learn more about it, including its overview information, metadata, and variable summary statistics. Since this dataset is public we can also look directly at the data to confirm it has the information we need.

It looks like there are two tables, one with reviews for testing a model and another with reviews for training a model. Clicking on each table in this interface shows that they both have two variables (label and text) and that the Train table has 650,000 records while the Test table has 50,000 records.

This data seems to be formatted exactly how we'll want to use it so we don't need to do additional cleaning .

Reddit

This dataset has two tables with over 150 million Reddit posts and subreddit information, split across two tables. We can look more closely at the 33 variables in the Reddit posts, including univariate statistics.

For this workflow, we just want to look at reviews from one specific subreddit which reviews mouse traps: MouseReview. If we click on the Subreddit variable name, we can see a searchable frequency table with all this variable's values. If we search MouseReview we can see that this dataset contains 26,801 posts.

To move forward with this workflow we'll want to train a model on the Yelp dataset, and filter and clean the Reddit table make it more usable with our model. In order to clean or transform data and do our analysis we'll need to create a workflow.

2. Identify a base model to fine-tune

We want to leverage an existing model that understands language and can generally be used for language classification. There are many open-source models that might meet our needs here; in this example, we'll use Google's BERT-base-cased model.

This model is hosted on Hugging Face, so we could load it directly into our notebook at runtime. However, if our notebook uses restricted data, it might not have access to the external internet, in which case we'll need to load the model into a dataset on Redivis.

The Redivis dataset for this model can be found here. You can also learn more about loading ML models into Redivis datasets in our accompanying guide.

3. Create a workflow

At the top of any dataset page, we can click the Analyze in workflow button to get started working with this data.

You can add this dataset to an existing workflow you already have access to, or create a new workflow to start from scratch.

Add the additional datasets by clicking the + Add data button in the top left corner of the workflow and searching for the dataset by name.

4. Create a notebook and load model + data

Once we've added all our datasets to the workflow, we can get started. To begin, we'll create a python notebook based on the Yelp reviews training data, by selecting that table and clicking the + Notebook button.

To enable GPU acceleration, before starting the notebook, we'll choose a custom compute configuration with an NVIDIA-L4 GPU, which costs about $0.75 per hour to run (we could use the default, free notebook for this analysis, but it would take substantially longer to execute).

We'll also need to install a few additional dependencies to perform training and inference via Hugging Face python packages:

With that, we can start our notebook! The full annotated notebook is embedded below, and also viewable on Redivis.

The general steps here are as follows:

1. Load the training and test data from the Yelp reviews dataset

2. Load the pretrained BERT model

3. Train this base model on the Yelp data to create a fine-tuned model that can classify text reviews from a score of 0-4.

5. Prepare the Reddit data for inference

The Yelp data was ready to go as-is, with a simple text field for the review and integer value for the score. For the Reddit data, we just need to run a quick filter to choose posts from the appropriate sub-reddit.

We will use a transform to clean the data, as transforms are best suited for reshaping data at scale. Even though we might be more comfortable with Python or R, this dataset table is 83GB and it will be much easier and faster to filter it in a transform rather than a notebook.

Create a transform

Click on the Posts table in the Reddit dataset and press the + Transform button. This is the interface we will use to build our query.

Add a Filter step. Conceptually we want to keep records that are part of the subreddit we are interested in, and are not empty, deleted, or removed.

The final step in a transform is selecting which variables we would like to populate the resulting output table. In this case we just need the variables title and selftext.

With everything in place we will run this transform to create a new table, by pressing the Run button in the top right corner.

7. Use the fine-tuned model to classify Reddit reviews

Finally, we can apply our fine-tuned model to the subset of Reddit comments that we want to analyze. Ultimately, we produce a single output table from the notebook, containing the reddit post and associate score generated by our model.

Next steps

Workflows are iterative and at any point you can go back and change our source data, our transform configuration or notebooks and rerun them. Perhaps we want to look at other subreddits, or run the model on a larger sample of the Yelp data.

You can also fork this workflow to work on a similar analysis, or export any table in this workflow to analyze elsewhere.

We do recommend further familiarizing yourself with the examples and detailed documentation to take full advantage of the capabilities of Redivis notebooks:

• Guide to running ML Workloads on Redivis

• Python notebook reference

• R notebook reference

Overview

Datasets are a core component of Redivis. They are a versioned collection of tables containing data, alongside rich documentation and metadata.

Datasets can be hosted by organizations or individual users, and every dataset has its own Dataset page. Datasets can be shared with other users on Redivis according to their access configuration.

1. Create a dataset

Administrators can create datasets for their organization from the Datasets tab of their organization's Administrator panel. These datasets can be seen and managed by any administrator in the organization. When released they will be visible on the organization’s home page to anyone who as overview access tot he dataset.

Alternatively, anyone with a Redivis account can create a dataset on the Datasets tab of their Workspace. These datasets are by default only visible to their owner, and have simplified options to support sharing with your collaborators.

When you first create a dataset, it will be unpublished and only visible to other editors. This means you can edit the dataset and validate everything before releasing it. You can also validate and reconfigure its access rules before release.

2. Import data

At the core of every dataset is the data it contains, so we recommend starting here.

All data in a dataset is stored in tables. You can create a new table on the Tables tab of your dataset and start importing data. Redivis can upload data from your computer or another location you’ve linked, such as Box, Google Drive, AWS, and Google Cloud.

Once your data is finished importing you can validate that this table looks as you expect it to.

You can create more tables here if this dataset has multiple separate tables.

However, if your data is split across multiple files that all follow the same structure (such as a different file for each state, or each year of data but with generally the same variables) you will want to import all of these files to the same table, where they will be automatically appended together.

Learn more in the Upload tabular data as tables guide.

3. Populate metadata

Metadata is essential to helping your researchers find and utilize your dataset. While some metadata will be generated automatically, such as variable summary statistics and counts, other metadata will require additional input.

Dataset metadata

On the Overview tab of the dataset there are multiple suggested sections to help break down information. You can fill out any of these that apply, such as the methodology, tags, contact information, etc.

Redivis will automatically generate citation and provenance information based on what we know about the dataset, but you can update this information with anything more specific.

If you have additional information that you want to include that doesn't fit one of these headers, you can create a custom section. Custom sections can also be set to be visible only to certain access levels if you have sensitive information.

Table metadata

You should also populate the metadata on each table. Tables can have a description, as well as an entity field that defines what each row in the table represents. You can also define the temporal and geographic range on the table, when relevant.

Variable metadata

Each variable within a table has its own metadata. The variable name and type will be pre-determined from your data, but you should add a short label and longer description to each variable to help researchers understand what that variable measures.

Additionally, some variables will contain coded values, in which case you should provide value labels that represent the human-readable term for each code.

Learn more in the Create and populate a dataset guide.

4. Release

Once you are ready to make your dataset available to others, you'll need to release it. You can click the Review and publish or Review and release button in the top right of the page.

You should also confirm your access settings you set up when creating this dataset.

Once this dataset is released, it will become visible and available to anyone who would qualify for access.

Next steps

Edit data

You can use tools right on Redivis to create new versions of your dataset.

Learn more in the Edit data in a dataset guide.

Administer your organization

Organizations allow for groups, centers, and institutions to more easily work with data by providing administrators with tools to effectively version and distribute data from a central location.

Contact us to set up a new organization, and learn more in the Administer an organization guide.

Overview

Redivis datasets are a great place to host data for interrogating in a workflow, but you can also edit the underlying data.

Perhaps you found an issue with the source, or want to restructure it, before making it available to others. Redivis has all the tools you'll need to do this in a versioned, step-by-step process with transparency into the changes you've made.

1. Upload the raw data

If you haven't yet, upload the data you want to work with to a Redivis dataset and release the version. A personal dataset or one belonging to an organization works just as well.

2. Add this dataset to a workflow

Create a new workflow and add this dataset to it.

If you want to share the data transformation process with others for transparency, you can make this workflow public in the share modal. (Due to the way Redivis works, still only people with access to the underlying data will be able to see the data in the workflow even though the workflow is public.)

For this guide you can follow along in this workflow Demo tables edits.

3. Transform data

Select the table you want to make changes to and create a new transform.

Use this transform to edit this table. Some common actions include:

Example edit 1: Rename and retype a variable

Create new steps to rename and retype any variables you'd like to update.

Example edit 2: Recoding the values of a variable

Create a new variable with the same name. "Keep" this variable and "Discard" the original variable. Select the method as "Case (if/else)" and create the conditions you want to recode to.

Example edit 3: Add a new variable

This example shows a new variable with today's data as the "date_uploaded" but you could create any variable you want from the data in the table. Perhaps a aggregation that would be helpful to see with this data? Or the sum of multiple other variables?

You can transform this table however you want, using the Graphical interface or SQL code. Make sure to move all variables in this table from the "Discard" section to the "Keep" section except for any variables that have been replaced and will be left behind (in this case the store_and_fwd_flag from the source table).

Validate that this new table looks correct by looking at the output table below this transform!

Make changes to any tables in this dataset in this same wokflow.

4. Re-upload the finished tables

Go back to the original dataset, and create a new version.

Open a table that you made changes to in your workflow, and click to "Import data."

Choose your merge strategy as "Replace" since you want to replace the existing table with the new one you created.

Select the data source as "Redivis" and type in the information from the workflow you made your changes. The sequence here will be:

1. Your user name

2. The name of the workflow (underscores replace spaces)

3. The table name (underscores replace spaces)

For example: username.testworkflow.table_name

Once it's done uploading you can validate that the data looks like it's supposed to.

Now you can close this table and repeat this process for any other tables in this dataset you've edited.

Once this is finished, you can add the link to the worfklow into this dataset's documentation or release notes along with a note on the changes. Remember that if you made the workflow public, anyone who has data access to this dataset can view the changes you made.

Now release this version! This new version of the data contains the edited data. Anyone using this data in a workflow will see that this dataset has a new version next time they open a workflow. You can see the release notes and the updated table of our Demo tables updates live.

Next steps

Start working with your data

Once your dataset is released, bring it into a workflow to transform and analyze it leveraging lightning fast tools from your browser.

Learn more in the Analyze data in a workflow guide.

Overview

Notebooks provide a flexible computation environment for data analysis, connecting Redivis tables to the scientific compute stack in Python, R, Stata, and SAS. Redivis notebooks are built on top of Jupyter notebooks and the .ipynb notebook format.

From within a notebook, you will be able to query and analyze any table within your workflow, and (optionally) generate an output table from the notebook. Because data referenced in a notebook never leaves Redivis, you can securely analyze data that would otherwise be bound by export restrictions.

Notebook node states

As you work in a workflow, nodes colors and symbols will change on the tree view to help you keep track of your work progress.

Overview

The Rename step changes the name of an existing variable.

Example starting data:

/*---------+--------*
 | student | score  |
 +---------+--------+
 | jane    | 83     |
 | neal    | 35     |
 | sam     | 74     |
 | pat     | 62     |
 *---------+--------*/

Example output data:

Rename score to final_score

/*---------+--------------*
 | student | final_score  |
 +---------+--------------+
 | jane    | 83           |
 | neal    | 35           |
 | sam     | 74           |
 | pat     | 62           |
 *---------+--------------*/

Step structure

• There will be at least one rename block where you will define a variable and a new name.

Input field definitions

Example

We can change the name of a variable to give more meaning to the data.

Starting data:

/*---------+-------+---------+------------*
 | test    | score | student | date       |
 +---------+-------+---------+------------+
 | quiz    | 83    | jane    | 2020-04-01 |
 | quiz    | 35    | pat     | 2020-04-01 |
 | quiz    | 89    | sam     | 2020-04-01 |
 | midterm | 74    | jane    | 2020-05-01 |
 | midterm | 62    | pat     | 2020-05-01 |
 | midterm | 100   | sam     | 2020-05-01 |
 | final   | 77    | jane    | 2020-06-01 |
 | final   | 59    | pat     | 2020-06-01 |
 | final   | 100   | sam     | 2020-06-01 |
 *---------+-------+---------+------------*/

Input fields:

• Source variable: The variable we want to rename is date so we select that here.

• New name: date_taken would be a better name for this variable, so we enter it here.

Output data:

/*---------+-------+---------+------------*
 | test    | score | student | date_taken |
 +---------+-------+---------+------------+
 | quiz    | 83    | jane    | 2020-04-01 |
 | quiz    | 35    | pat     | 2020-04-01 |
 | quiz    | 89    | sam     | 2020-04-01 |
 | midterm | 74    | jane    | 2020-05-01 |
 | midterm | 62    | pat     | 2020-05-01 |
 | midterm | 100   | sam     | 2020-05-01 |
 | final   | 77    | jane    | 2020-06-01 |
 | final   | 59    | pat     | 2020-06-01 |
 | final   | 100   | sam     | 2020-06-01 |
 *---------+-------+---------+------------*/

Overview

A transform node in a Redivis workflow provides a powerful querying tool to reshape one or more tables into a new output table.

You will build your transform by choosing steps one at a time based on the actions you want to take on your data. At any point you can run the transform to execute each step sequentially and view the resulting table.

The most commonly used transforms steps can:

• Aggregate a table

• Create variables based on existing data

• Filter out rows you no longer need

• Join additional tables

The transform interface translates your interface inputs to SQL code that you can view and edit. You can also intermix SQL code you've written between interface steps, or build an entire transform of your own SQL code if you wish.

Transform node states

As you work in a workflow, nodes colors and symbols will change on the tree view to help you keep track of your work progress.

Overview

Redivis offers extensive tools for previewing tabular data and transforming it in workflows, but the data needs to be uploaded correctly in a tabular format for researchers to utilize these tools.

This guide assumes you have already started by Creating a dataset.

1. Locate the data you want to upload

You can upload data directly from your computer, or import from a linked account.

If importing, you'll want to get the relevant external account configured to your Redivis account before getting started.

The import tools allow for multiple uploads, so no need to combine files together before importing them, but it's helpful to have them all in the same place.

2. Create tables

On your newly created dataset, the first step to uploading data is to create one or more tables that the data will be uploaded to.

The data files you currently have may or may not be how you want to store them on this dataset, so it's important to think about your data's structure before getting started.

For example, if you have multiple files that all follow the same schema, we strongly recommend uploading them as one table (for example, if you have a separate table for each year, or a separate table for each state, but the structure of each is the same). In the example of one file per state, this would allow researchers to query across all states, skipping the first step of doing up to 50 joins. Additionally, you generally shouldn't split out tables for performance reasons — even when querying billions of records, Redivis will execute in seconds.

When uploading files to tables, remember that every row in a table should represent the same "thing", or entity; we wouldn't want to combine county-level and state-level observations in one table.

If you haven't already, we very strongly recommend experimenting with the reshaping and analytic tools in a workflow which researchers will use to work with your dataset. Knowing how they will work with it might inform how you structure it during this setup process, and can save time for everyone. You can even add your unreleased dataset to a workflow for testing — click on "View dataset page" from the dataset overview, and then add the unreleased dataset to your workflow.

When you're ready click the Create new table button on the Tables tab of the dataset page and name your table to get started.

3. Upload tabular file(s) to create a table

To get started uploading, choose the data source. By default this is your computer, but you can choose any option from the dropdown menu.

Next, choose the file(s) or enter the paths of the file(s) you want to import.

If you select multiple files here, they will be automatically appended in this single table on upload based on common variable names. If a variable is missing in some of the files, that's ok, it will just be recorded as null for all records in that file.

For a full list of supported file types, as well as advanced functionality (such as wildcard imports) and error handling techniques, consult the Uploading data reference.

Once your files are selected, click the Import button. If the files are coming from your computer, you might need to wait until they are finished uploading to the browser before they can be imported into Redivis.

Learn more in the Uploading data reference section.

4. Verify uploads

As you upload files, you will see an overview of any files' progress and can click to view each file's data and additional information.

Once all uploads have completed, you can inspect the table (representing the concatenation of all of your uploads). Make sure to check the summary statistics and other analytical information to validate that the data are as you expected.

If you have more files to upload you can click the Manage imports button on the right side of the table at any time (up until releasing this version of the dataset).

Next steps

Continue uploading your dataset

Great metadata makes your dataset useable. Complete your metadata, along with configuring access, creating a sample, and releasing this version.

Learn more in the Create & manage datasets guide.

Overview

Redivis supports direct SQL queries for advanced functionality and for users who prefer to work in a programmatic interface. You can add one or more SQL query steps to any transform.

This step accepts SQL code and will execute in the order it has been placed within the rest of the steps.

Query syntax

SQL Dialect

Redivis supports the BigQuery Standard SQL Syntax and features, with certain limitations.

Referencing the source table

All queries must reference the SQL step's source, represented as _source_ in the query. If the SQL step is the first step in the transform, the source represents the transform's source table. Otherwise, the source represents the output of the previous step in the transform.

For example:

SELECT * 
FROM _source_
ORDER BY mean_drg_cost
LIMIT 1000

Joining tables

You can also reference any other table in the current workflow from your SQL step, following the same reference rules as those in the Redivis API. After typing the backtick character (`), available tables will be presented as autocomplete options.

For example:

SELECT t0.id, t0.name, t1.latitude, t1.longitude
FROM _source_ AS t0
INNER JOIN `demo.ghcn_daily_weather_data:7br5:v1_1.stations:g2q3` AS t1
    ON t0.id = t1.id

Referencing upstream variables

You can mix and match SQL steps with other step types within your transform. If you choose to do so, this will introduce some additional language semantics, since transforms allow variables with the same name to co-exist at any given step, delineated in the UI with the t1, t2, ... and v prefixes. The v prefix denotes any newly created variable, while the tN prefix denotes variables that have been joined from other tables.

In order to reference these variables, you must include the prefix in your SQL code before the variable name. The only exception to this rule is for variables prefixed by t0 — these should be referenced normally, allowing you to ignore these rules if you aren't mixing the SQL interface with other step types.

For example:

SELECT 
-- These variables are from the transform's source table, denoted by `t0` in the UI
    id, name, 
-- These variables were joined upstream, denoted by `t1` in the UI
    t1_latitude, t1_longitude, 
-- This variable was created upstream, denoted by `v` in the UI
    v_extracted_year
FROM _source_

Referencing value lists

You may also use value lists in your query — they can be referenced as a SQL parameter, prefixed with the @ symbol. Note that all value lists are explicitly typed, and you will need to cast the value list if there is a type mismatch.

For example:

SELECT provider_id 
FROM _source_
WHERE provider_state IN UNNEST(@`states:8a2h`) 
    -- If a value list is in the incorrect type, use the following code to convert it
    OR provider_fips IN UNNEST(
        (
            SELECT ARRAY_AGG(CAST(val AS STRING))
            FROM UNNEST(@`fips_codes:134a`)val
        )
    )
ORDER BY mean_drg_cost
LIMIT 1000

-- @states = ['CA', 'TX', etc...]
-- @fips_codes = [6071, 2999, 3242]

Limitations

• DDL / DML queries are not supported

• All queries must return fields of a valid Redivis type. The following BigQuery types are not supported in the result set, though they may be used internally within the query:

  • TIMESTAMP

  • NUMERIC

  • BIGNUMERIC

  • BYTES

  • INTERVAL

  • JSON

  • ARRAY

  • STRUCT

• BigQuery ML syntax is not supported

• BigQuery scripting language is not supported

• Statements may only reference tables that exist in your workflow

• Unnamed variables in the result are not supported

• Duplicate variable names in the result are not supported

Errors

If your query is invalid, you will see an invalid icon and the error message displayed above your code. Transforms can't be run while any step is invalid.

Notebooks on Redivis provide a highly flexible computational environment. Notebooks can be used for anything from quick visualizations to training sophisticated ML models on a large corpus of data.

Understanding the compute resources available, and when to modify which parameters, can help you take full and efficient advantage of the high-performance computing resources on Redivis.

Default (free) notebooks

The default notebook configuration on Redivis is always free, and provides a performant environment for working with most datasets. The computational resources in the default notebook are comparable to a typical personal computer, though likely with substantially better network performance.

The default free notebook configuration offers:

• 2 vCPUs (Intel Ice Lake or Cascade Lake)

• 32GB RAM

• 100GB SSD:

  • IOPS: 170,000 read | 90,000 write

  • Throughput: 660MB/s read | 350MB/s write

• 16Gbps networking

• No GPU (see custom environments below)

• 6 hr max duration

• 30min idle timeout (no code is being written or executed)

Custom compute configurations

For scenarios where you need additional computational resources, you can choose a custom compute configuration for your notebook. This enables you to specify CPU, memory, GPU, and hard disk resources, while also giving you control over the notebook's max duration and idle timeout.

In order to customize the compute configuration for your notebook, click the Edit compute configuration button in the notebook start modal or toolbar.

Custom machine types

Redivis makes available nearly every machine type available on Google Cloud. These machines can scale from small servers all the way to massively powerful VMs with thousands of cores, terabytes of memory, and dozens of state-of-the-art GPUs.

These machine types are classified by four high-level compute platforms: General purpose, memory optimized, compute optimized, and GPU. Choose the platform, and machine type therein, that is mose appropriate for your workload.

Custom machine costs

All custom machines have an associated hourly cost (charged by the second). This cost is determined by the then-current price for that machine configuration on Google Cloud.

In order to run a custom machine, you must first purchase compute credits, and have enough credits to run the notebook for at least 15 minutes. If you run low on credits and don't have credit auto-purchase configured, you will receive various alerts as your credits run low, and ultimately the notebook will shut down when you are out of credits.

Maximizing notebook performance

All notebooks on Redivis use either Python, R, Stata, or SAS. While Redivis notebooks are highly performant and scalable, the coding paradigms in these languages can introduce bottlenecks when working with very large tabular data. If you are running into issues with performance we suggest:

• Use transforms to clean and reduce the size of your data before analyzing them further in a notebook. When possible, this will often be the most performant and cost-efficient approach.

• Adjust your programming model to load data lazily or on-disk to exceeding memory limits. See our suggestions for working with larger tables in Python, R, Stata, and SAS.

• Adjust the compute resources in your notebook. This may help to resolve these bottlenecks depending on what is causing them!

Some quick rules of thumb:

• < 1GB: probably doesn't matter, use what suits you!

• 1-10GB: probably fine for a notebook, though a transform might be faster.

• 10-100GB: maybe doable in a notebook, but you'll want to make sure to apply the right programming methodologies. Try to pre-cut your data if you can.

• >100GB: You should probably cut the data first in a transform, unless if you really know what you're doing.

Overview

A data source node contains data across Redivis that you want to work with and have access to. These are usually but can also be other workflows. These nodes display overview information about the dataset or workflow it represents, and a list of the tables it contains.

You can click on any table to view its contents, or click "Transform" to build a transform on it.

Adding data source nodes to a workflow

Adding a data source to a workflow will make a copy of that dataset or workflow in the form of a circular node at the top of your workflow tree. You can add data to the workflow by:

• Click the + Add data button in the top left of the toolbar within a workflow.

• Click the Analyze in workflow button on a dataset page.

• Click the Fork button in the toolbar of any workflow.

Restrictions

No workflow can contain two copies of the data source or the same version of the same dataset. You can add a different version of the dataset to your workflow by right clicking on the dataset name in the + Add data modal.

Datasets as a data source

are the most common Data source you will add to your workflow. They contain the data you want to work with in the original state curated by the data owner. Usually datasets will contain 1 or more tables you can choose to transform or analyze in a notebook.

Dataset samples

Some large datasets have 1% which are useful for quickly testing querying strategies before running transforms against the full dataset.

If a 1% sample is available for a dataset, it will automatically be added to your workflow by default instead of the full sample. Samples are indicated by the dark circle icon to the top left of a dataset node in the left panel and in the list of the dataset's tables.

All sampled tables in the same dataset will be sampled on the same variable with the same group of values (so joining two tables in the same dataset with 1% samples will still result in a 1% sample).

​To switch to the full sample, click "Sample" button in the top right of the menu bar when you have a dataset selected.

Your downstream transforms and tables will become stale, since an upstream change has been made. You can run these nodes individually to update their contents, or use the run all functionality by clicking on the workflow's name in the top menu bar.

Dataset versions

When a new of a dataset is released by an administrator, the corresponding dataset node on your workflow tree will become purple. To upgrade the dataset's version, click the "Version" button in the top right of the menu bar when you have a dataset selected.

You can view version diffs and select whichever version you want to use here.

After updating, your downstream transforms and tables will become stale. You can run these nodes individually to update their contents, or use the run all functionality by clicking on the workflow's name in the top menu bar.

Workflows as a data source

can be added to another workflow in order to build off existing analysis. You might want to continue an analytical pipeline that you've built elsewhere, or elaborate on someone else's analysis. You will have access to all tables this workflow contains.

Data source node states

As you work in a workflow, nodes colors and symbols will change on the tree view to help you keep track of your work progress.

Overview

Organizations are where administrators centrally manage datasets and who can access them. All organization administration happens from the Administrator panel which administrators can access from the Organization home page.

If you're interested in using Redivis for your research group, center, or institution, to get an organization set up.

1. Add other administrators

You can add any Redivis account to be an administrator of your organization in the Administrator panel from the Settings tab.

Note that all administrators will have full access to the organization and its datasets, including the ability to modify access rules, approve access requests, and add or remove other administrators, including you.

2. Build your organization's presence

To help brand your organization and best communicate the organization’s purpose to others, you'll want to fully populate and brand the Organization home page.

In the Administrator panel on the Settings tab / Public profile section you can customize:

• The organization's full name

• An organization description which will appear at the top of your about section, and in any institution searches

• A custom brand color

• The organization's logo

• The data portal's cover photo

• A header image that will appear in the top left of users' windows when on your data portal

• Header links to other resources, such as organization-specific documentation, events, and other resources.

• Rich text and images that will display in the main content section of the data portal. A place to provide high level information about your organization and how researchers should user your data portal.

3. Create datasets

You'll need to create datasets to upload and distribute data to your researchers. You can create datasets from the Datasets tab of your Administrator panel.

Here are some pieces to consider before getting started:

Dataset structure

It's worth thinking through your broader data management strategy. Assess the data that you currently have — what format are the files in? Where are they located? How are they organized?

Consider what datasets you'll want to create. Datasets on Redivis are made up of one or more semantically related tables, and each table can contain data from one or more uploaded files. Moreover, all permissions happen at the dataset level — if you grant a user access to a dataset, they will have access to all of its tables.

Citations and DOIs

If you’d like to issue DOIs for your organization’s datasets, you can configure your organization to issue DOIs in the Advanced section Settings tab. DOIs will make it easier for researchers to cite your data when they publish, and allow you to track publications using your data as well as their broader impact.

Learn more in the guide.

4. Build access systems

Before you can configure access to a dataset, you'll want to consider and build out the relevant access systems for your organization as a whole.

Direct access & Member access

If you have a smaller organization where you know everyone working with your data, you might want to consider directly granting access to datasets to specific members, or granting access to everyone who is a member.

This can be set up on any dataset individually or you can create a Permission group on the Permission group tab to save the entire configuration of access levels and assign it to multiple datasets.

Requirement-based access

In many cases, however, you’ll want to develop more process-driven access control systems and gather additional information from your members To do so, you can set up requirements that must be completed by members or their studies, and approved by an administrator, before gaining access.

You can build requirements on the Requirements tab of the administrator panel, and save groups of them together as Permission groups on the Permission groups tab which you can assign to multiple datasets.

Export restrictions

Finally, you may want to control how your data is exported from Redivis, if at all. While members with data access will always be able to work with data in Redivis workflow, you can limit their ability to export data. For example, you may choose to allow exports only to certain environments, or only upon administrator approval, or not at all.

Learn more in the guide.

5. Manage members and studies

As members and studies apply to access data, you may need to take action to approve (or reject) their access requests. You can find all of your pending access requests wherever you see an alert on the administrator panel.

Members

As users join or organization and apply to work with data, they will appear in the Members tab of your administrator panel. You can also invite anyone to become a member by adding them to the members list.

Studies

Studies allow for researchers to organize their work around a conceptual study, or research project, and for you to grant access in the context of that study. Any studies that are working with your organization's data will appear in the Studies tab of the administrator panel automatically, and you can approve any requirements for studies here.

Learn more in the guide.

6. Gain insight into usage

Redivis provides tools to help you keep an eye on your organization and how its data is being utilized.

You can filter, sort, and change the columns for any list in your administrator panel, and download the corresponding report for a snapshot of current information. You can also dig into the logs and generate analytical reports for more information.

Next steps

Expand your reach

If you are part of a larger institution, you can about getting an institution-wide Redivis page to help users discover new organizations and datasets.

Overview

Geospatial data on Redivis behaves quite similarly to : each feature is ingested as a single row within a table, alongside any metadata for this feature. This approach mirrors tools like PostGIS, R spatial features, and geopandas, allowing you to query and join your geospatial data, at scale.

Supported file types

Redivis supports importing geospatial data from several common GIS formats: geojson, shp, shp.zip,kml, and parquet. Parquet files with geospatial metadata (often referred to as GeoParquet) are the most performant and robust option, though as a newer standard, these files are less common. For other geospatial file types, Redivis first converts the file to a geojson representation (using the relevant ), and then imports the geojson into a table.

Each feature will be imported as one row, with the geometry column containing the WKT representation for that feature. Additional feature properties will be mapped to variables in your table, with any nested properties flattened using the . separator. Note that Redivis only supports 2-dimensional, unprojected () geometries. Other projections might cause the import to fail, and any extra dimensions will be stripped during ingest. If you are uploading a .shp.zip that contains projection information, the geometries will automatically be reprojected as part of the import process.

Geography data in text-delimited files

In addition to uploading geospatial data using one of the formats listed above, you can also import geographic data encoded within a text-delimited file (e.g., a csv). In this case, the geographic data should be encoded as strings using the . This is also the same format used when exporting geography variables as a CSV. WKT in CSVs will be auto-detected during data ingest.

Quotas & limits

Limits for upload file size, max variables, and other parameters are .

Overview

All datasets on Redivis are automatically versioned. Any change to a dataset's data in or will require a new version. A version is a locked copy of the data, supporting future reproducibility and confidence in the persistence of researchers' data workflows.

Changes to documentation and metadata do not create a new version, though different versions do have independent documentation and metadata. For example, if a new version contains a new table, or new data, you will likely want to document this information separately from the previous version. However, if you only want to enrich the existing metadata on the current version, you can do so without creating a new version.

If you are a dataset editor and have new data, find a mistake to correct, or would otherwise like to modify the existing content of a dataset, you can and a new version.

Semantic version tags

To help researchers better understand if the differences across versions, Redivis uses semantic versioning, of the form v[major].[minor] The first version of every dataset is v1.0. For subsequent versions, the tag will augment automatically depending on the changes being released.

• Major update: Existing code may not run.

  • Triggered when variables in the new version are renamed, deleted, or retyped. Also occurs if any tables from the previous version were deleted.

• Minor update: Existing code will generally run.

  • Changes are limited to adding / removing records, recoding variables, adding variables, and adding tables.

Version history

On any dataset page, you can view the current version tag next to the dataset title, and click on this tag to switch to view a full version history and switch to a different version of this dataset.

Within a workflow, you can change the version of a dataset by selecting the and clicking the Version button at top right. If there is a new version available, the dataset node will be highlighted to indicate that you might want to upgrade.

Creating a new version

When it's time to update a dataset's data, you'll want to create a new version. To do this, navigate to the dataset editor and click Create next version.

Before this version is released, it will be tagged as next. Only dataset editors will be able to see the next version on the dataset page and use it in their workflows.

Version storage

All versions of a dataset contribute to that dataset's total size, which in turn will count towards your or (depending on whether the dataset is owned by you or an organization).

This total size will be displayed in the dataset editor, alongside the size for the current version. For datasets with one version, this total size may be slightly larger than the current version, as Redivis stores certain metadata behind the scenes to support future versioning.

As new versions are created, Redivis will efficiently compute a row-level difference between the versions — only additions and updates to existing data will contribute to the dataset's total storage size, preventing data that is consistent across versions from being double-counted.

Adding or removing columns (or changing column types) won’t affect row uniqueness, as the underlying storage represents all values as strings. Only the storage size of the new column would be added.

Version unrelease

If the most recent version of a dataset has been released in the last 7 days, and there is no Next version already created, you'll have the option to unrelease it.

This will revert the dataset to the exact state it was in before the version was released. If anyone who is not a dataset editor has this version in a workflow, they will lose access to the data, though can revert to a previous version if it exists.

Unreleased and published datasets

When a dataset is first created it will be marked as unreleased.

As you work on the initial version of the dataset you can see the changes you make to the data in the version history modal. in this unreleased state you can add it to a workflow and analyze the data it contains. Note that the only people who can see it in the workflow are dataset editors. If you make changes to the data or delete it, your workflow will change instantly to reflect those changes.

Once you are ready to make your data available to non-editors, you will need to publish it. This will simultaneously release your version as well as make your dataset discoverable. As soon as it is published it will be accessible to all who meet its access rules.

If this is an organization dataset, it can be unpublished at any time. This will "unlist" the dataset so that it will not be listed on your organization home page and even users who meet the access requirements will not be able to see it. Anywhere it is used in a workflow it will instantly be made unavailable.

Unpublishing might be used if you need to temporarily halt usage of the dataset but don't want to disrupt all of the access rules.

Working with previous versions

When analyzing data in a workflow you can change any dataset to a previous version to instantly change every table to the corresponding table in that version. To do this, click on the dataset and click the Version button in the toolbar. Select the version you'd like to analyze and confirm. You can always switch versions at any point.

Normally you can only have one copy of dataset in a workflow but it's possible to add a second version if you'd like to compare them. In any workflow, click the + Add data button and locate the dataset. Right click on the dataset and choose to add another version to the workflow.

Overview

A transform is a series of steps detailing a data transformation that are executed in sequential order when the transform is run. These steps and a final variable selection create an output table that can be inspected and further transformed.

Steps

The majority of building a transform is choosing and completing your data transformation steps. Steps represent one specific action (e.g. filter rows, join tables) and are completed in the Redivis transform interface. Every step is a fundamentally a piece of SQL code that can be examined and could be reproduced by running in any environment with the same data.

When running the transform, steps will be executed in order, building off of what directly came before them.

Steps have additional options in the ⋮ menu to help in the building process:

• Collapse / expand

• Annotate

• Disable / enable

• Reorder (this might change the outcome)

Output variables

The final task in constructing a transform is to select which variables you want to have in your output table. You will use the variable selector in the bottom pane of the transform to choose which to keep.

Running transforms

Transforms are iterative and are designed to easily check your progress as you work. As long as a transform is valid, you can press the Run button to generate your output table. Click on this output table node in the workflow tree to see your results once the run is complete.

Invalid state

The transform interface will prevent you from running a transform that is invalid. This will be displayed with an invalid icon on the step or the part of a step where the issue is. Usually when you have an invalid alert it means some crucial information is missing and you can follow the invalid icons to find the locations you'll need to update.

History and revert

Every time you run the transform, a snapshot of the transform at that point in time is saved. You can click on History to view the log of these runs, and click on any snapshot to revert the transform to that point in time.

Your transform will then be in the Edited state and no changes will be made to the output table. Once you run it, this configuration will become a new snapshot at the top of the transform history list.

Checking the output

Output tables don't just exist to store data, you can use the variable statistics generated in the table node to make sure your transform did what you expected it to do.

Common things to check might be total number of rows and variables in your output table, distinct and null counts of a key variable, or even running a query on the Query tab of the table to filter by a particular value.

If you find the table isn't what you want, you can go back to the transform to make changes and rerun it. Transforms were created to be iterative!

Additional tools for working with transforms

Name and rename

By default transforms are named with a number based on the order they were created. We suggest renaming your transforms with descriptive names so you and your collaborators can quickly find them later on. You can rename a transform by clicking on the title of the node, or on the Rename option in the transform ⋮ More menu.

If you have not specifically renamed the connected output table, renaming a transform will also rename this output table.

Change the source table

All transform nodes have one source table. You can join in multiple tables, but the source table designation will affect how the joins execute. You can change the source table from the ⋮ More menu in the transform. Note that if the new source table has different variables your transform might become invalid until you update it.

Split and combine transforms

All transforms can be split at the step level into two different transforms by clicking Split in any step's menu. Additionally, two transforms can be combined into one by right clicking on a table to Remove it.

You might want to split a transform above a tricky step to see what the output table would look like at that point in the process. This can be a key tool in troubleshooting any issues and understanding what might be going wrong.

After splitting a transform to check an output table, the next logical step might be to combine these two transforms back into one again. Or perhaps you have a string of transforms which you no longer need the output tables for and want to reduce the size of your workflow.

Copy, paste and insert

As you go, you might find yourself wanting to take certain data cleaning actions multiple times or move them around in your process. You can right click on any transform in the workflow tree to see options for copying it or deleting it. Right click on any table node to paste the copied transform. You can also insert transforms above other transforms.

Steps and parts of steps can also be copied and pasted within the same transform or across multiple transforms.

View SQL code

All transforms generate SQL, which will executed the transform is run. To view the code for a particular step, click on View SQL in the step menu. You can also see the SQL for the entire transform from within the ⋮ More menu of the transform.

You can convert this code into a SQL query step and run it to achieve the same outcome, or edit the code directly.

SQL Concepts

Redivis uses SQL as the basis for data transformation for many reasons. It is designed for working quickly and efficiently with large tables, and is easy to learn concepts and basic executions, and it is highly reproducible.

While Python, R, Stata, and SAS are commonly used in data analysis, they are not suited to working with large tables. Executing queries on millions of records can take hours and in many cases fail. We have provided a point and click interface for using transforms to make using a perhaps unfamiliar coding language easier.

Fundamentals

• Tables are data structures containing rows and variables

• Variables represent fields in the table that all rows contain values for

• Rows represent an individual observation in table with values in across all variables

• A SQL query will always operate on data from one or more tables and output a single data table.

Building a query

When you are using the interface or SQL code, it is best to start the process by figuring out what you would like your output table to look like. What are the variables? What do you want each row of the table to represent?

Once you figure out what you want your output to look like, you can work backwards and formulate the steps of your query to build the shape you'd like.

Overview

One of the main tasks of organization administrators is to manage who has access to datasets. Redivis has tools to support any level of complexity in data systems for your restricted data and makes it easy for organization administrators to define, manage, and audit access rules across these various use cases.

1. Consider your organization's access needs

Do you mostly have public datasets? Will you have a small group of researchers with access to pretty much every dataset? Will you need to gather information from researchers before they use your datasets? Do you have high risk data that can't ever leave the system? Or some combination of the above?

Redivis supports access systems for all of these configurations – for organizations of varying sizes – but you'll want to use different tools to achieve these very different goals.

For example, if you have few datasets, you might find that permission groups for bulk managing access rules aren't particularly helpful. Or if you have a small group of researchers who you all know closely, you might not need to set up process-driven access rules through requirements, and can instead grant access to certain researchers on an individual basis.

We'll walk through all the tools available to organizations for managing access, but you'll likely want to pick and choose from what's available to meet your organization's specific needs.

2. Understand access levels

All interactions with data on Redivis require the user to have the appropriate access level to the data for a given action. Ideally your data would be as permissive as possible to allow for the greatest exploration from researchers before they need to start applying for access.

Dataset access has five levels:

1. Overview: the ability to see a dataset and its documentation.

2. Metadata: the ability to view variable names and univariate summary statistics, but not to retrieve any identifiable information or multivariate relationships from the data.

3. Sample: the ability to view and query a dataset's 1% sample. This will only exist for datasets that have a sample configured.

4. Data: the ability to view and query a dataset's tables, and work with them in workflows.

5. Edit: the ability to edit the dataset and release new versions.

Access levels are cumulative. For example, in order to gain data access you will need to have gained metadata access as well.

We strongly recommend making your dataset's metadata as open as possible. This will reveal variable names and aggregate summary statistics, but will not allow researchers to view, query, or export the raw data in any way.

Being able to see metadata greatly improves researchers' discovery experience, and allows them to better assess a dataset's utility upfront, and even reduce your administrative workload. If researchers can understand a dataset before applying for access, they'll be submitting fewer access applications to datasets that are ultimately a dead end.

Learn more in the Access levels reference section.

3. Understand available access tools

Membership

Anyone wanting to apply for access to restricted datasets in your organization must first be a member. You can configure whether memberships are restricted to certain identity providers (such as your institutional login), and whether they are approved automatically or require administrator review. You also have the option to configure access to datasets to "All members."

Direct access

Permission granted directly to a researcher to instantly gain access to a dataset at a specific level. Researchers can also request access to datasets with this configuration.

Example usage: a dataset that will only be shared with a small number of people who are already known to administrators

Member requirements

A form for members to fill out. This can be set to require approval from an administrator or be automatically approved. It can also have an expiration date. These are global to your organization and when assigned to multiple datasets a user will only fill it out one time.

Example usage: a demographic form gathering researcher personal information, or a data use agreement signed PDF

Study requirements

Similar to requirements, but instead of each user needing to fill them out individually, only one requirement needs to be completed for the entire study, which can include multiple researchers. A single researcher may also have multiple studies. Each study working with the dataset will need to fill out its own study requirement, and any queries or exports of the data will be tied to that study.

Example usage: a funding proposal for a research project

Data export restrictions

A rule defining that a dataset can only be exported to a specific export environment, as configured on the Administrator panel Settings tab.

Example usage: limiting exports to a specific server environment

Learn more in the Configuring access reference section.

4. Create requirements

If you want to work with requirements, you'll want to get started making them and planning out how they will work across datasets.

Perhaps you want one requirement for all members to fill out about their field of research, which is necessary to gain access to any of your datasets, but another 4 requirements with different data use agreements that will apply only to their specific datasets.

To get started, go to the Requirements tab of the administrator panel and click the New requirement button. You will need to start by selecting if this will be a Member requirement or a Study requirement.

You can use the form builder to collect different information, including standard form responses, file uploads, and e-signatures.

Learn more in the Requirements reference section.

5. Create permission groups

A permission group is an access configuration that can be assigned to multiple datasets and managed centrally.

You don't need to use permission groups, and it might not make sense to do so if each of your datasets has a different access configuration and you aren't using requirements. But if you have any overlap between datasets and want to enforce consistency, or want to use requirements you'll want to make one.

To get started, go to the Permission groups tab of the administrator panel and click the New permission group button.

This interface requires you to set an access paradigm for each access level of the dataset.

Perhaps you will set the overview access level to be public, the metadata access level to be available to all members, and data access level to be direct access (meaning you will have to directly grant access to users, or respond to their requests for access).

Or perhaps you will set the overview access level to public, and assign multiple requirements to the metadata and data access levels (meaning that anyone who is approved for all of the requirements will automatically gain that access level).

For any case you can assign data export restrictions here and choose whether you want to manage access to the dataset's sample (if it exists) differently than the full data.

Learn more in the Permission groups reference section.

6. Assign access permissions to datasets

Finally you'll need to apply these access permissions to actual data!

Open any of your datasets and click the Configure access button on the top of the page. This configuration setup will look very similar to configuring the permission group.

You can either create a custom configuration here, or you can assign this dataset to one of the Permission groups by clicking the dropdown menu in the top right corner of this modal.

This is also where you will manage any direct access requests for the dataset.

7. Verify your setup

As an administrator of this organization, you will have access to all datasets no matter what your access configuration is or what requirements you have filled out.

We recommend checking that your access system works as you expect by either looking at a dataset while you are logged out or in your browser's incognito mode, or by making a second Redivis account using a different email address that is not linked to your administrator account.

Next steps

Grant access to data

You have some shiny access systems, but they won't work if you don't approve user requests for access.

Learn more in the Grant access to data guide.

Expand your reach

If you are part of a larger institution, you can contact us about getting an institution-wide Redivis page to help users discover new organizations and datasets.

Overview

The workspace is the home of your content on Redivis. It is created when you make a Redivis account and accessible by clicking the My workspace button in the right of the header bar from anywhere on the site. Your workspace is only visible to you and is where you will manage your work on Redivis.

Your workspace dashboard has links and information to help ground yourself in the work you are doing on Redivis. Beyond the dashboard, you will see pages listed on the left bar with different aspects of your account.

Workflows

Any workflows that you've created, or that have been shared with you, will show up here. You can create a workflow by clicking the + New workflow button on this page in order to start working with data. Clicking on a workflow here will open up the workflow interface.

Datasets

Any datasets that you've uploaded or can edit will appear here. You can create a dataset by clicking the + New dataset button on this page in order to start uploading and sharing your data. Clicking on a dataset in this list will open the dataset editor which will allow you to work on the dataset.

Organizations

This page lists all organizations that you're a member of. If you have applied for access to any restricted data, a copy of your submissions will be listed under the organization you applied to. You can search for new organizations to join by clicking the + Find new organization button.

Studies

Studies allow users to more easily work in groups. You can create a study by clicking the + New study button on the studies tab of your workspace. Any study you are a part of will appear in your workspace. Any edits you make will appear for all study collaborators.

Dataset library

This section contains any dataset that you have bookmarked. You can manually bookmark a dataset by clicking on the bookmark button next the dataset title on any dataset page. Redivis will also automatically bookmark datasets that you add to prjoects, or where you have applied for access. You can remove these at any time by right clicking on a dataset in this list and selecting the Remove from library option.

Logs

All activity you take when working with data on Redivis is logged. You can visit the Logs page of your account to view the details of your Query, Notebook, and Export actions.

Settings

Public profile

You can specify the following information, public to all users and organizations on Redivis:

• Name: This is your full name displayed across Redivis. This will appear when collaborators search for you to share their workflows, and the name administrators will see when interacting with any access steps you've submitted. It does not need to be unique.

• Username: A unique handle for your account.

• Profile picture: A picture visible next to your name throughout the platform.

• ORCID iD: Link your ORCID ID so organizations can see research you are working on. See orcid.org for more information on setting up an account.

• Disciplines: Areas of interest and study that best represent your work and research.

Changing your username

If you would like to change your username, you may do so at any time. Any URLs or API references to your old username will automatically redirect. You may change your username up to 10 times.

Authentications

You can use any email listed here to log in to your account. If you have more than one email address, the one marked as 'Primary' will be the one to receive any emails. You can authenticate with additional institutional or Google emails by clicking Add authentication button. If the email you add is associated with an institution or organization, the authenticated information we receive from them will be listed here. A single authentication will be used by organizations to verify your identity when you apply for membership.

Data sources

To allow you to import your data from wherever it resides, Redivis supports multiple integrations with outside services.

Redivis will only ever read data from these sources; it will never modify or overwrite content. Additionally, Redivis will only access and transfer content when you specify a location in one of those sources as part of the data upload process.

API tokens

You can create and manage tokens here for working with the Redivis API. See the API documentation for more information.

Compute credits and billing

You can use compute credits to purchase advanced compute environments for notebook nodes within workflows.

Secrets

Secrets allow for you to define simple key/value pairs that are securely stored and later be referenced from within a Redivis notebook.

Common use cases for secrets include storing external API tokens and other credentials for systems outside of Redivis.

Secrets are only accessible to you from within a Redivis notebook.

Learn more about working with secrets in notebooks >

Advanced

This tab allows configuration of the following:

• Contact email: Specify a contact email to receive email notifications for any update to your data access applications. If you apply to access data hosted by an organization, that organization will be able to see your contact email. By default, this is the email you used initially to create your Redivis account.

• Communications: Allows you to configure email notifications for any update to your data access applications, as well as occasional product update emails regarding changes to Redivis. Note that, if you opt to receive email notifications, you will still see a comprehensive list of notifications in the web platform.

• Security: Shows all current sessions of Redivis across devices and browsers, and provides the ability to log out of any of these sessions.

• Delete account: Allows permanent deletion of your account, including all workflows you have created.

Stata license

In order to enable Stata notebooks for your account, you can provide your Stata license information here (alternatively, you'll be able to use Stata if one of your organizations provides a license). Specifically, you'll need the license "code", "authorization", and "serial number", which should all have been provided as part of purchasing Stata.

SAS license

In order to enable SAS notebooks for your account, you can provide your license information here (alternatively, you'll be able to use SAS if one of your organizations provides a license). Because of how SAS handles cloud deployments, the steps to enable SAS are a bit more complicated than for Stata. Specifically, you'll need to complete the following steps:

1. Run SAS Deployment Wizard to install SAS Studio on a supported Linux 64-bit operating system. During the installation, change the default location for the SAS Studio installation to /usr/local/SASHome.

2. For details about how to install SAS Studio, refer to the installation and configuration instructions provided by the product.

3. Create a TAR file that includes the SASHome directory:

   Copy

   tar -czvf SASHome.tar.gz /usr/local/SASHome

Once this tar file has been created, please reach out to [email protected] to share the file so that we can enable SAS for your account.

E.g.: I want the date of the first and last hospital admission for every patient in my table

Let's take an example table which contains all hospital visits in one facility over many years. Each record contains information about what happened in the visit, including a patient id to identify who was present. An individual patient might be represented in one or many records in this table since each record represents a separate hospital visit.

Let's say that for our research we want to find the first and last encounter — or more formally, the min and max encounter date for each patient.

Computing a min or max value is an aggregate operation that scans all records and chooses the highest or lowest value. To compute the min or max value in a variable we will want to use an analytic new variable method. We can do this either by aggregating our table (which would drop records and/or variables) or by using a partition to calculate a new variable without changing the rest of the data.

In this example we will use a partition to create a new variable since we want the rest of the data to remain unchanged.

Variable concepts

Conceptually we will define our aggregation variable as the date of the encounter since that is where we want to look for the min and max values.

We will define our partition as the patient id since we want to find the min or max for each patient. This will limit the min or max value scan to all values of our aggregation variable across each unique value of our partitioned variable (patient id).

You can see more examples of using partitions with analytic methods on the Create variables step page.

Defining the new variable

For this particular case, we want to find the MIN and MAX of the encounter_date of each patient; in other words, partitioned on patient_id . This will create two new variables, min_encounter_date and max_encounter_date for each patient.

When we run this query, note that no records will be dropped; rather, these new variables will be have consistent values for any given patient_id .

Now, let's say we only want to keep records which contain each patient's first and last encounter. We can do this easily be creating a new row filter:

We can also generalize this approach to find the Nth encounter for each patient — take a look at the RANK, DENSE_RANK, and ROW_NUMBER methods.

And if we want to be more specific on our partition (e.g., for the first / last encounter in a given calendar year), we can always apply additional variables (year ) when defining our partition.

Overview

The Order step sorts the table based on one or more variables.

Example starting data:

/*---------+--------*
 | student | score  |
 +---------+--------+
 | jane    | 83     |
 | neal    | 35     |
 | sam     | 74     |
 | pat     | 62     |
 *---------+--------*/

Example output data:

Order by score descending

/*---------+--------*
 | student | score  |
 +---------+--------+
 | jane    | 83     |
 | sam     | 74     |
 | pat     | 62     |
 | neal    | 35     |
 *---------+--------*/

Step structure