Redivis is a platform for data distribution and analysis that prioritizes open access and FAIR data practices. We believe that supporting open science principles drives better research outcomes, collaboration, and scientific impact.

Redivis is creating tools not just for data distribution and analysis, but also to help drive forward standards for transparent research that is accessible to all.

Reproducibility

Every Redivis dataset has a versioned history that is automatically generated while the data owner uploads and edits data contents. This history is fully exposed to end users and the dataset can be accessed at any version. Citations are generated for every version of a dataset which include the ORCID iDs for all creators. Data owners can opt to create Digital Object Identifier (DOI) for all versions of their dataset, which is available at a persistent URL along with the required metadata describing the content at that point in time.

As users analyze data in a Redivis workflow each step they take is recorded in a visual tree that can be navigated sequentially. Every step has a full history which can be viewed at any point in time.

Availability

Datasets in any format can be deposited into Redivis and made available immediately to the world via open access licenses at no cost to the end user.

All Redivis datasets and analysis workflows are available to anyone on the internet without a Redivis account unless the owner decides to add restrictions. In this case someone wishing to view the work would need to create a free Redivis account to request access from the owner.

Longevity

The stable technical infrastructure and dedicated Redivis team is funded by a coalition of academic institutions paying license fees for large scale data distribution. This diverse group of funders ensures the long-term availability of the datasets.

Overview

Retaining data is vital to establishing long-lasting data linkages and making research available. Redivis' data retention policy is aligned with the goal of keeping all data available and accessible whenever possible.

Owner-initiated deletion

Data owners always have the ability to remove their data from Redivis or make it inaccessible if they choose to do so.

Redivis also has policies in place in case of an accidental deletion:

• All data is fully backed up, with point-in-time recovery, over a 7 day rolling window. Deletions and modifications to data is permanent after 7 days.

• All metadata is backed up with point-in-time recovery over a 7 day rolling window, with daily backups of metadata extending to 1 year.

Redivis-initiated deletion

Data storage that exceeds the free tier must be paid for by the data owner or other sponsor. Upon payment failure, every effort will be made to contact the data owner and re-establish payment.

If unsuccessful, data may be destroyed after 30 days of non-payment, though all metadata, and permanent landing pages will be persisted.

Overview

The ability to view, reproduce, and build upon other works is a core tenet of the scientific process that Redivis is built to uphold and facilitate. Redivis tools for data storage and analysis comply with federal mandates for open access and automatically generate artifacts documenting all steps in the data process, ensuring reproducibility.

Background

In August 2022, the United States Office of Science and Technology Policy (OSTP) released a memorandum on public access to federally funded research (known as “Nelson Memo”) which outlined new requirements affecting both faculty and students who conduct research using federal funding:

1. Make publications and their supporting data resulting from federally funded research publicly accessible without an embargo on their free and public release.

2. Enact transparent procedures that ensure scientific and research integrity is maintained in public access policies.

3. Ensure equitable delivery of federally funded research results and data.

1. Public accessibility

Redivis is built to make data sharing as easy and safe as possible, defaulting to fully accessible data, metadata, and documentation. However, some data must be restricted for privacy and to comply with data licensing agreements. In these cases Redivis follows an "as much as securely possible" sharing philosophy for sharing.

• Multiple levels of access allows data owners to grant access to documentation, metadata, samples, and data separately, so as much access as possible can be granted.

• When research is done using restricted data, analysis code is made available when possible.

• Viewing public data and analysis workflows on Redivis is completely public and does not require a Redivis account.

• If any piece of analysis or data is restricted, a user will need a Redivis account to apply for access. Redivis accounts are completely free with no restrictions.

2. Transparent procedures

Redivis systems are built to automatically capture and document any work done on datasets and analyses in standardized formats, while giving researchers and administrators the ability to supplement or override these when necessary.

• Datasets are automatically versioned. All changes to variables and rows are recorded and made available to administrators and data viewers.

• Redivis offers a no-code interface to build transform queries that compiles to SQL code. This code is available to the analyst and any viewers.

• Workflows are self-documenting. Every step taken in an analysis workflow is recorded sequentially in a visual format that is easy to follow.

• Workflows are version controlled. Every step is versioned and time-stamped, allowing users to revert to a previous iteration, or a viewer to understand how queries might have changed over time.

• Workflow can be forked, preserving relationships between workflows and datasets.

3. Equitable delivery

Redivis is free and accessible to any researcher, reviewer, or casual data browser. By taking multiple complex data storage and analysis systems and centralizing them with clear UI, it is clearer to understand full workflows without specific technical knowledge.

• Redivis uses transparent, open-source formats, allowing data, analyses, and workflows to be exported and used within other systems.

• No-code interfaces allow researchers at any skill level to build and understand research steps.

• Common, open-source coding languages (SQL, Python, R) are a foundation of tools and supported in analysis workflows.

• There is no cost to researchers to store personal data, or run data queries.

Overview

Redivis is designed to support data-driven research throughout the research lifecycle, and can serve as a permanent repository for your data, analytical workflows, and data derivatives.

In May 2022, the Subcommittee on Open Science (SOS) of the United States Office of Science and Technology Policy (OSTP) released a document outlining the "desirable characteristics of data repositories".

These characteristics are intended to help agencies direct Federally funded researchers toward repositories that enable management and sharing of research data consistent with the principles of FAIR data practices. Various agencies have adopted these guidelines, including the NIH.

Redivis is specifically designed to meet these desirable characteristics of a data repository, outlined below:

Desirable Characteristics for All Data Repositories

Unique Persistent Identifiers

Long-Term Sustainability

Metadata

Curation and Quality Assurance

Free and Easy Access

Broad and Measured Reuse

Clear Use Guidance

Security and Integrity

Confidentiality

Common Format

Provenance

Retention Policy

Additional Considerations for Human Data

Fidelity to Consent

Restricted Use Compliant

Privacy

Plan for Breach

Download Control

Violations

Request Review

Detailed information

Unique Persistent Identifiers

Assigns datasets a citable, unique persistent identifier, such as a digital object identifier (DOI) or accession number, to support data discovery, reporting, and research assessment. The identifier points to a persistent landing page that remains accessible even if the dataset is de-accessioned or no longer available.

Data can be uploaded to datasets within an organization, where every version of that dataset is assigned a DOI through the organization's DOI-issuing credentials. DOIs will always resolve to the URL of the dataset page. In the case when a dataset is restricted or deleted, base metadata will remain available.

Long-Term Sustainability

Has a plan for long-term management of data, including maintaining integrity, authenticity, and availability of datasets; building on a stable technical infrastructure and funding plans; and having contingency plans to ensure data are available and maintained during and after unforeseen events.

Redivis uses highly-available and redundant Google Cloud infrastructure to ensure data is stored securely and to the highest technical standards. Redivis maintains a formal disaster recovery and business continuity plan that is regularly exercised to ensure our ability to maintain availability and data durability during unforeseen events.

Redivis undergoes annual security audits and penetration testing by an external firm, and utilizes a formalized software development and review process to maintain the soundness of its technical infrastructure.

Funding for Redivis is provided by recurring annual subscriptions from its member academic institutions. This model provides consistent annual revenue to support the ongoing maintenance of the platform. Redivis is an employee-owned company without any external investors with an equity stake, allowing us solely focus on the needs of our customers, our employees, and our mission of improving accessibility in the research data science.

Metadata

Ensures datasets are accompanied by metadata to enable discovery, reuse, and citation of datasets, using schema that are appropriate to, and ideally widely used across, the community(ies) the repository serves. Domain-specific repositories would generally have more detailed metadata than generalist repositories.

All Redivis datasets contain extensive metadata and documentation. Some metadata fields (including summary statistics for all variables) are automatically generated. Some fields are optional depending on the editor's insight and preference. Every dataset has space for short and long-form text, supporting files, and links, alongside variable labels and descriptions.

As a generalist repository, the metadata schema is intentionally broad and flexible, but specific groups on Redivis can choose to enforce more specific metadata standards within their datasets.

Curation and Quality Assurance

Provides, or has a mechanism for others to provide, expert curation and quality assurance to improve the accuracy and integrity of datasets and metadata.

All datasets on Redivis are owned either by an Organization (curated by any administrator) or an individual User. Additional users may be added as editors to a dataset, so as to provide further curation and quality assurance.

It is ultimately up to the editors of a dataset to provide curation, though Redivis is designed to support this process as much as possible. Redivis automatically computes checksums and runs fixity checks on all uploaded files, and computes univariate summary statistics of all variables to aid in the quality assurance process. Metadata completeness is also reported to editors, encouraging them to provide as much information as possible.

Free and Easy Access

Provides broad, equitable, and maximally open access to datasets and their metadata free of charge in a timely manner after submission, consistent with legal and ethical limits required to maintain privacy and confidentiality, Tribal sovereignty, and protection of other sensitive data.

Datasets and workflows can be explored on Redivis without any requirement to have an account or log in. If someone would need to apply to access restricted data, or analyze and download data they will need to make an account to do so. All individual accounts are complete free and require no specific affiliation. Data access restrictions are set and maintained by the data owner.

Broad and Measured Reuse

Makes datasets and their metadata available with broadest possible terms of reuse; and provides the ability to measure attribution, citation, and reuse of data (i.e., through assignment of adequate metadata and unique PIDs).

The data owner can choose to publish any dataset publicly or set appropriate access restrictions based on the sensitivity of the data. Redivis imposes no additional limits on the availability and reuse of data. Any dataset that has a DOI can be cited and tracked in publications, and any reuse on Redivis is displayed on the dataset's usage tab.

Clear Use Guidance

Provides accompanying documentation describing terms of dataset access and use (e.g., particular licenses, need for approval by a data use committee).

Dataset owners can describe any usage agreements in their access requirements and have space to document any additional usage rules. Redivis imposes no additional limits on the availability and use of data.

Security and Integrity

Has documented measures in place to meet generally accepted criteria for preventing unauthorized access to, modification of, or release of data, with levels of security that are appropriate to the sensitivity of data.

Redivis is SOC2 certified and prioritizes technical security. There are multiple layers of administrative controls to make it clear what actions data owners are taking, and all actions whether administrator or researcher are automatically logged for review.

Redivis is well-designed to handle workflows around sharing of sensitive and high-risk data. It provides technical mechanisms to support the reuse of sensitive data when allowed, while enabling the enforcement of appropriate guardrails and access controls defined by data administrators.

Confidentiality

Has documented capabilities for ensuring that administrative, technical, and physical safeguards are employed to comply with applicable confidentiality, risk management, and continuous monitoring requirements for sensitive data.

Redivis is SOC2 certified and prioritizes technical security. All data is encrypted in transit and at rest, and stored on Google Cloud infrastructure that maintains robust technical and physical access controls.

In addition to an annual security audit, Redivis also undergoes annual penetration testing by an outside firm to further ensure the soundness of its security posture.

Common Format

Allows datasets and metadata downloaded, accessed, or exported from the repository to be in widely used, preferably non-proprietary, formats consistent with those used in the community(ies) the repository serves.

Data and metadata on Redivis can be imported and exported in multiple common formats. Data analysis is performed in common, generally open-source programming languages (SAS and Stata being available exceptions). Redivis does not introduce any of its own proprietary formats or programming languages.

Provenance

Has mechanisms in place to record the origin, chain of custody, and any modifications to submitted datasets and metadata.

All datasets contain Provenance documentation, which is automatically populated based on administrator actions. This information can be further edited or supplemented with additional related identifiers.

All modifications to a dataset or tracked in the adminstrative logs.

Retention Policy

Provides documentation on policies for data retention within the repository.

Redivis publishes a data retention policy. Data is always owned by the user or organization who uploads the dataset, and they have control over a dataset's presence and availability. The dataset owner may apply additional policies towards data retention.

Fidelity to Consent

Uses documented procedures to restrict dataset access and use to those that are consistent with participant consent and changes in consent.

Redivis provides extensive options to limit access on sensitive or restricted data, including allowing access on different levels (e.g. metadata, sample, data). Access is granted and revoked on Redivis instantly, allowing for immediate changes in access based on changing circumstances.

Access rules on Redivis are defined and enforced by the dataset owner.

Restricted Use Compliant

Uses documented procedures to communicate and enforce data use restrictions, such as preventing reidentification or redistribution to unauthorized users.

Redivis has built in data restrictions available to administrators. Data download or export can be restricted completely or limited only to administrator approval in order to prevent redistribution.

Data administrators can also communicate and collect formal acknowledgement of other use restrictions through access requirements. Moreover, data administrators can easily audit the use of restricted data in the audit logs to further check for and limit any non-compliance or other misuse.

Privacy

Implements and provides documentation of measures (for example, tiered access, credentialing of data users, security safeguards against potential breaches) to protect human subjects’ data from inappropriate access.

Redivis has a built-in tiered access system that allows administrators to restrict access to sensitive data. These controls apply to data derivatives as well, where any data output inherits the access rules of the source dataset(s) used to create that output. These controls on derivative data allow researchers to share analyses with colleagues without worrying about accidentally leaking sensitive information, since all collaborators will need to comply with the access rules in order to view those outputs.

Additional technical safeguards ensure data privacy. Data users can establish their identity through their institutional identity provider, and Redivis undergoes regular security audits to ensure the soundness of its security posture.

Plan for Breach

Has security measures that include a response plan for detected data breaches.

Redivis has detailed internal security protocols and documented security breach plans which are regularly exercised by technical personnel via tabletop exercises. All systems are continuously monitored for potential breaches, with immediate alert pathways and clear escalation protocols to respond to any breach.

Download Control

Controls and audits access to and download of datasets (if download is permitted).

Redivis has built in data restrictions available to administrators. Data download or export can be restricted completely or limited only to specific external systems / upon administrator approval. All data downloads are logged for subsequent audit and review. Downloads are only available to authenticated users who have access to the underlying data.

Violations

Has procedures for addressing violations of terms-of-use by users and data mismanagement by the repository.

Violations of the terms-of-use may lead to account suspension or revocation, as outlined in the terms. Additional technical controls are in place to prevent abuse or misuse of Redivis's systems.

As a matter of policy, Redivis aims to be as permissive as possible, recognizing that often misuse is the result of accidental behavior or misunderstanding. These controls are designed to protect the system for all users, and are not intended to ever be punitive towards good-faith actors.

Request Review

Makes use of an established and transparent process for reviewing data access requests.

Administrators can define access requirements on any restricted dataset. These access requirements are transparent to all users, and researchers must apply and be approved for a given set of requirements in order to gain access. Requirements also have space for both administrators and applicants to leave comments specifically in the context of the data application.

Overview

Findable

Data is findable when (1) data and metadata are assigned a globally unique and persistent identifier, (2) data are described with rich metadata, (3) metadata clearly and explicitly include the identifier of the data it describes, and data and (4) metadata and data are registered or indexed in a searchable resource.

At Redivis:

Accessible

Data is accessible when: (1) data and metadata are retrievable by their identifier using a standardized communications protocol, (2) the protocol is open, free, and universally implementable, (3) the protocol allows for an authentication and authorization procedure, where necessary, (4) metadata are accessible, even when the data are no longer available.

At Redivis:

• Data can be explored through any web browser.

• Researcher accounts to apply for data access or do analyses are always free.

Interoperable

Data is interoperable when: (1) data and metadata use a formal, accessible, shared, and broadly applicable language for knowledge representation, (2) data and metadata use vocabularies that follow FAIR principles, (3) data and metadata include qualified references to other data and metadata.

At Redivis:

Reusable

Data is reusable when: (1) data and metadata are richly described with a plurality of accurate and relevant attributes, (2) data and metadata are released with a clear and accessible data usage license, (3) data and metadata are associated with detailed provenance, and (4) data and metadata meet domain-relevant community standards

At Redivis:

• Many dataset, table, and variable metadata fields are automatically populated based on user action, with the ability to be manually adjusted.

• All datasets have a Usage tab with information on how it has been viewed and used across Redivis

• Redivis cloud-based analysis tools encourage data users to do analysis along the data rather than downloading it and breaking linkages.

What is Redivis?

Who uses Redivis?

Organization administrators upload, catalogue, and grant access to datasets.

Researchers discover, query., and analyze data.

Redivis makes it easy for everyone to:

Using this documentation

This documentation is broken up into Guides, Examples, and Reference sections.

We recommend that new users begin by following the relevant guides and examples, while the reference section can serve as a comprehensive overview of everything that Redivis can do.

Overview

Redivis is a platform that allows researchers to seamlessly discover, access, and analyze data. This brief guide will walk you through the basics to get up and running and provide a launching point for exploring other resources in this documentation.

1. Create your account

Many datasets on Redivis are public, and you can browse them without creating an account. However, you'll need an account in order to analyze data, as well as to apply for access to restricted datasets.

Click the Create account button in the top right of any page to sign up. You can use your academic institution's login credentials or any Google account.

Once you create an account, you'll be navigated to your Workspace, which is your private area for creating workflows to work with data, uploading your own datasets, and managing your Redivis account.

2. Join your organization

The main entity on Redivis is a dataset. Most datasets on Redivis are uploaded by organizations.

You might already know the organization that you’ll be working with, or you can browse organizations by clicking the Find organizations button on the Organizations tab your Workspace. You might know your organization's URL already, or see it listed in the Recent section of the Dashboard if you were already viewing that page.

Once you join an organization, you'll be able to apply for any of their restricted datasets, and also see a link on the left bar of your workspace to quickly get back to the organization's home page.

3. Find datasets

The best place to find datasets on Redivis is to search on an organization's home page. Here you can click on the Datasets tab to browse all datasets and filter by their metadata. All searches perform a full-text search across a dataset, its documentation, tables, variables, and rich metadata content.

Click on any dataset title to go to that Dataset page where you can view the data and metadata it contains. The data for this dataset will be available on the Tables tab, which you can explore further.

You will also probably come across restricted datasets. For these you will need to click the Apply for access button on the top right of the Dataset page and complete the requirements to gain approval from the dataset's administrators.

4. Analyze data

Once you've found a dataset that you want to work with, you can add it to a Workflow. Workflows are the fundamental analysis interface on Redivis, where you can query, merge, reshape, and analyze any dataset that you have access to — all from within your web browser.

Add a dataset to a workflow by clicking the Add to workflow button on the top right of the dataset page.

In a workflow we can create a Transform by selecting any table and clicking the +Transform button, which allows us to combine and reshape our data into a final output table that best serves our analysis. These transforms use a powerful SQL engine under the hood, allowing us to query incredibly large tables - even billions of records - in seconds.

After creating output tables for analysis, we can create a computational Notebook in R, Python, Stata, or SAS to further analyze our data and develop our final figures. Select any table and click the +Notebook button to get started. The notebook will initialize and pull in the table you've selected (or a sample of the table if it is a large table).

We can also export and query this table from external environments, allowing us to use whatever analytical tools best suit our research question by clicking the Export table button on the right side of any table. Here we can download the table in a number of common formats, interface with the it programmatically via the API, or export the table to supported integrations.

Next steps

Upload your own datasets

Augment your data analysis in Redivis by uploading your own datasets, with the option to share with your collaborators (or even the broader research community).

Administer your organization

Organizations allow for research groups and centers to securely distribute data to their research community. Organization administrators can create datasets, manage access, review logs, and create customized reports of their data utilization.

Overview

Most data on Redivis is uploaded through an Organization, and that is the best place to find Datasets to work with. Some datasets will be public, while others will require certain steps before you can get full access to the data.

1. Find datasets

From an Organization's home page, click on the Datasets tab to browse all datasets and filter by their metadata.

You can use filters on the left bar to narrow down your results. All searches perform a full-text search across a dataset, its documentation, tables, variables, and rich metadata content.

2. Get to know a dataset

Click on any dataset title to go to that Dataset page.

You can view the metadata (including an abstract, documentation, citation information, and a version history) and dig into the data directly on the Tables and/or Files tab.

Each table contains a data browser to view cells, generate summary statistics, and display metadata for this table and each variable. Each file can be previewed and downloaded. Both tables and files can be added to workflows for analysis.

Many datasets on Redivis are public, while others have requirements for certain levels of access enforced by the data owner.

On the right side of the dataset, you can see your current access level. If you have Metadata access you can see the variable information and summary statistics, but you'll need to gain Data access in order to view the data contents or work with this data.

3. Apply for access

On any restricted dataset you can click the Apply for access button in the top right of the page to see a list of steps required for you to gain different access levels to this dataset.

The first step is to become a member of this organization. When you go to apply you will be prompted to become a member of this organization using the credentials on your account.

Once you're a member of this organization, you might need to fill out some requirements that administrators have set up, or request access directly from them.

Requirements are forms that you will need to complete on this page and then submit. Once these are approved by your organization's administrators you will gain access to the dataset.

Next steps

Start working with your data

Once you have data you're interested in, bring it into a workflow to transform and analyze it leveraging lightning fast tools from your browser.

Upload your own datasets

Augment your data analysis in Redivis by uploading your own datasets, with the option to share with your collaborators (or even the broader research community).

Overview

Datasets are the core entity on Redivis, and finding the dataset(s) you want to work with is generally one of your first steps. All datasets are either uploaded by an organization or a user. If a user has uploaded a dataset they will need to share it with you directly, but you can browse datasets upload by any organization on their home page.

1. Find your organization

There are many ways to find your organization. If you know what you're looking for you can type it into the "Go to" bar on the top of the page. To browse all organizations you can go to the Organizations tab of your workspace where you can Find organization.

Note that some organizations may be hidden from public view, in which case you will need to contact someone at the organization to get the direct link.

2. Browse available datasets

On the organization home page you can see all of the organization's datasets that you have at least Overview access to. If you're looking for a dataset you expect to be here and you don't see it, you might need to contact the organization administrators to get access.

You can see the datasets that this organization has featured on the left section of the page, type a term into the search bar, or click the Datasets tab to browse everything. By default these datasets are sorted by popularity but you can reorder them however you'd like to find what you want.

You can also click the Workflows tab to see workflows featured by this organization which showcase how their data can be used. More on workflows later!

3. Search for data

Once on the dataset tab you can use multiple filters along with search terms to find what you'd like. All searches take a comprehensive look at information from the dataset documentation as well as variable metadata, including value labels.

You can also use the filters to find specific information you're looking for. If you know the name of the variable or year of the data you're interested in, narrow your search here.

4. Preview tables

Each dataset that appears here matches all search and filter configurations. You can click on the dataset tile to see more information, including a preview of all of the tables this dataset contains.

Click on any of these tables to dig into it further. As long as you have metadata access you can see the variable information and summary statistics.

Once you find a dataset you want to explore further, click on the title to go to the dataset page.

5. Explore the dataset page

This dataset page has all of the information you'll need to familiarize yourself with the data and get started working with it further. Add this dataset to your dataset library to easily find it later on by clicking the button alongside the dataset title.

Overview

This tab has a short summary of the dataset along with different sections filled out by the administrators such as the Methodology, Documentation, and Provenance information. This Provenance section will also have citation information and a DOI if one has been created. You can also see your current access level on the right side of the page.

Tables

This tab contains all of the dataset's tabular data. You can see each table alongside information about its contents. Click on each one to dig deeper into the data.

Files

This tab contains all of the dataset's unstructured data. You can preview each file by clicking on it.

Usage

This tab has information about how other people have used this dataset on Redivis. You can see the most popular variables across all tables to get a sense for which ones might be a good starting place to understanding the structure of the data.

Next steps

Apply for access to restricted data

If you don't have data access to this dataset you'll need to apply for access before you can work with it further

Start working with your data

Add this dataset to a workflow to transform and analyze it leveraging lightning fast tools from your browser.

We've started building a library of common workflow types and analytic tasks that might be useful when you're getting started working with data.

Please let us know if there are other actions or concepts you would like us to provide examples for by reaching out or emailing contact@redivis.com

Overview

Many datasets on Redivis are public, while others have requirements for certain levels of access enforced by the data owner. If the dataset is public, you can skip over this section, otherwise you'll need to gain access before you can fully utilize the data.

1. View access rules

On the right side of all restricted dataset pages is a box which displays your current access level. If this doesn't say Data access, then your next step will be to apply for access.

Click the Apply for access button on the top of the page to open the access modal. This modal allows you to manage your access to this particular dataset throughout the platform.

Access levels

Dataset access has five levels:

1. Overview: the ability to see a dataset and its documentation.

2. Metadata: the ability to view variable names and summary statistics.

3. Sample: the ability to view and query a dataset's 1% sample. This will only exist for datasets that have a sample configured.

4. Data: the ability to view and query a dataset's tables, and work with them in workflows.

5. Edit: the ability to edit the dataset and release new versions.

Access levels are cumulative. For example, in order to gain data access you will need to have gained metadata access as well.

Usage rules

Usage rules are restrictions placed on the way you can use this data once you've gained access. For example, data exports may be restricted to certain export environments, and/or require administrator approval.

While no action is required when you first apply for access a dataset, it's helpful to understand these restrictions to better understand what limitations, if any, they might pose to your research.

2. Apply for access

The access modal for the dataset will include all steps designated by this dataset's administrators to gain access to work with data.

Membership

To apply for access to datasets hosted by a Redivis organization, you will first need to Join the organization.

To join an organization, you will need to provide information about your identity via an email authentication, usually the email associated with your Redivis account.

After submitting your membership application, it will either be automatically approved or will be marked as pending while an administrator reviews your submission. You'll receive a notification if your submission is approved or rejected.

Requirements

Some datasets have requirements as part of their access applications. Requirements are global to an organization and usually contain some type of form. Once all requirements for a particular level of access are approved, you will gain access to the relevant dataset(s).

In the access modal, click the Apply button on any requirement to fill out and submit the form.

When you submit a requirement, an administrator of the organization will be alerted and can review your submission for approval (in some cases, the requirement may be configured for auto-approval upon submission). You will receive a notification if your submission is rejected or approved, or in the future when it's about the expire. You can also leave a comment for an organization's administrators here, and they can reply to you.

There are two types of requirements - those you fill out on behalf of yourself (member requirements) and those you fill out on behalf of a study (study requirements). If the dataset you are interested in has a study requirement, you'll need to create a study and select it in the access modal in order to submit the requirement. Additionally, any workflows that utilize this dataset will need to be a part of the approved study.

Direct access

Sometimes you don't need to fill out any requirements and need to simply Request access at a specific level. You can do so in this modal, and will receive a notification when you are granted access by the dataset owner.

3. Wait for data owner approval

For some membership and requirement submissions (and all direct access requests), you will need to wait for the data owner to respond to your submission. If the dataset is owned by a user, they will need to approve your request; if it is owned by an organization, any administrator at that organization can approve it.

The data owner will receive a notification as soon as the requirement is submitted, and you will receive a notification when it is approved, rejected, or for any requests for additional information. By default, you will receive an email as well as a notification within Redivis. You can customize this notification behavior on your workspace settings.

4. Re-authenticate (if necessary)

If you are accessing restricted data hosted by an organization, you may have to re-authenticate after a period of inactivity anytime you perform an action governed by the "Data" access level.

After a period of inactivity (set between 15 minutes and 3 days by the organization administrator), a popup will ask you to re-authenticate anytime you view data, query tables in a transform, or export data from Redivis.

Next steps

Start working with your data

Add this dataset to a workflow to transform and analyze it leveraging lightning fast tools from your browser.

Learn more in the Work with data in a workflow guide.

Citing datasets

It's important to make sure that you fully cite any datasets used in your work. Each dataset page includes citation information in the provenance section in multiple format options.

For example the Global Historical Climatology Network dataset hosted by the Stanford Center for Population Health Sciences would be cited in the APA format as:

Stanford Center for Population Health Sciences. (2022). GHCN Daily Weather Data (Version 1.2) [Data set]. Redivis. https://doi.org/10.57761/h9ff-vy04

Citing workflows

Workflows across Redivis also include citation information in the provenance section in multiple format options.

Bibliography

Datasets and workflows on Redivis can be supplemented with related identifiers that situate a resource in its broader context. You can view the bibliography for a dataset or workflow from the corresponding provenance section.

Citing Redivis

To cite Redivis in publications use:

Redivis (RRID:SCR_023111). https://redivis.com

If referencing Redivis inline, the abbreviated citation is also acceptable:

Redivis (RRID:SCR_023111)

To cite Redivis, you can use the following BibTeX:

@misc{https://doi.org/10.25495/7gxk-rd71,
  doi = {10.25495/7GXK-RD71},
  url = {https://redivis.com/},
  author = {Redivis},
  keywords = {FOS: Dataset, Publication},
  language = {en},
  title = {Redivis},
  publisher = {Redivis},
  year = {2024}
}

You can also refer to the Redivis Re3data identifier

Overview

Studies are a way for one or more collaborators to work with data on the same conceptual topic. You can add multiple collaborators, datasets, and workflows to a study to organize your workflow. In addition, some restricted data requires you to submit access applications as a study via study requirements.

1. Create a new study

On the studies tab of your workspace you can create a new study and see all studies you are a part of. This study might represent a topic or group you are working with, maybe to investigate a similar cluster of research questions. You can give the study a name and description to reflect that purpose.

2. Add collaborators

If you are working with anyone else on this study you can add them as a collaborator. Anyone who is a collaborator on this study will be able to view and edit the study same as you can. Once added, this study will appear in their workspace.

PI

One person on the study can be designated as the PI. This designation does not have meaning within Redivis interfaces and is here to inform others in the group or administrators you are applying for access from of the group structure.

3. Add datasets

You can add any dataset to this study that you plan on working with, and remove it later if your topic shifts. This space is intended for you to gather resources that you are using in one place to make it easier to create workflows or manage data access. Datasets can be added to any number of studies.

4. Add workflows

You can create a new workflow in this study or move an existing one in. All workflows can be in at most one study, and you can see what study a workflow is in on the workflow's overview page.

Granting workflow access to study members

One of the options for sharing this workflow is to grant view or edit access to all members of the study it's in. This can be an easy way to give everyone in a group access to a workflow you are working on.

As with all workflows, even though someone might have access to the workflow they will need to have independent access to all the datasets it contains in order to view or query them.

5. Apply for data access via a study

If you're working with restricted data you might come across a dataset that has a study requirement as part of its access requirements. You can identify this by the study icon next to the requirement name.

Study requirements are filled out and submitted on behalf of an entire study, rather than one person. An approved study requirement will be valid for all collaborators on the study, unlike member requirements where each individual needs to complete the requirement on their own account.

To submit a study requirement you can navigate to a restricted dataset and open the access modal. You will need to select your study from the dropdown menu, and a submit button will appear. One member of your study will fill this out and submit it on behalf of your group. Once approved, all study collaborators will see an approved requirement in their access modal.

Data usage

When restricted data has a study requirement, the data administrator can approve your application submission for use of the data within that study. In order to query data in a workflow, that workflow will need to be in the approved study. If it is not, you will see a badge noting Limited access and you will not be allowed to run transforms or query this data in a notebook.

Next steps

Start analyzing your data

Once you've gained access and set up a study it's time to add your datasets to a workflow to transform and analyze them leveraging lightning fast tools from your browser.

Overview

Workflows are where you work with data on Redivis. In a workflow you can query, merge, reshape, and analyze any data that you have access to, all from within your web browser.

In a workflow, you can construct reproducible data transformations and analyses, and share and collaborate with your peers in real time.

1. Create a workflow

Add a dataset to a new or existing workflow from any Dataset page where you have "Data access" by clicking the Analyze in workflow button.

Within workflows you can navigate between entities on the left side of the screen, and inspect them further on the right panel. You can inspect your dataset further by clicking on any table to see its cells and summary statistics.

To add more data to this workflow you can click the Add data button in the top left of the workflow toolbar. It is also possible to add other linked workflows to this workflow. This is useful as you develop more complex analyses that you want to segment into discrete pieces of work that you can link together.

You can find this workflow later by going back to your workspace.

2. Transform data

Transforming tables is a crucial step in working with data on Redivis. Conceptually, transforms execute a query on source table(s), whose results are materialized in a new output table. In most cases you'll want to use transforms to reshape your data to contain the information you're interested in, before analyzing that table in a notebook or exporting it for further use.

To create a Transform, select a table in this dataset and click the +Transform button. You can get started here building a query through the point and click interface or writing SQL code by adding a SQL step.

For all transforms you will need to select which variables you want to keep in your output table. The rest of the steps are up to you. Some common operations you can get started with include:

Once you've built your query, execute it by clicking the Run button in the top right of the transform. This will create a new output table where you can inspect the output of your query by clicking on the table beneath the transform in the map and making sure it contains the data we would expect.

From here you can create a new transform from this output table to continue reshaping your data, or go back to your original transform to make changes and rerun it.

As you become more familiar with transforms, you can start doing more advanced work such as geospatial joins, complex aggregations, and statistical analyses.

3. Analyze data in a notebook

Once you have a table you're ready to analyze, you can select any table and click the + Notebook button to create a notebook that references this table.

The default notebook configuration is free, and provides access to 2 CPUs and 32GB working memory, alongside a 60GB (SSD) disk and gigabit network. The computational powerful of these default notebooks are comparable to most personal computers, and will be more than enough for many analyses.

From here it’s all up to you in how you want to analyze and visualize your data. Once you’ve finalized your notebook, you can easily export it in different formats to share your findings!

4. Share and collaborate

You can share your in-progress work or finished results with collaborators by sharing this workflow.

Researchers can work side by side in this workflow in real-time. Leave comments to communicate, and see a visual cue for what each person is working on. You can even collaborate within a running notebook at the same time.

If any of the data in your workflow is restricted, your collaborator must also have access to those datasets in order to view their derivatives within your workflow.

Next steps

Share and collaborate

Redivis workflows are built for collaboration and include real-time visuals to see where collaborators with edit access are working in the workflow, and a comments interface to discuss changes asynchronously.

Export data

Browse our example workflows

Redivis workflows excel at working with large tables, whether it's filtering and joining, complex aggregation and date manipulation, or visualization and analysis.

Overview

Transforming tables in a workflow is a crucial step when working with data on Redivis. Conceptually, transforms execute a query on a source table and results are materialized in a new output table. They are optimized to run on billions of records in seconds, and create a transparent, reproducible record of data transformation steps you've taken in your workflow.

In most cases you'll want to use transforms to create an output table containing all the information you're interested in before analyzing that table in a notebook or exporting it for further use.

1. Create a transform

Once you've created a workflow and added a dataset to it that you'd like to work with, get started by creating a Transform. You can do this by clicking on any dataset or table.

To build this transform, you will add steps which each take an action to shape the data in the output table. You can choose from many named steps which include a point and click interface that compile to SQL code, or you can add a SQL query step to write code directly. You can always view the code that your step is generating and switch to code if you'd like to edit it.

While SQL might not be a familiar language, it is optimized for data cleaning procedures and allows transforms to execute extremely quickly. It also allows you to write your data transformations in a declarative, reproducible manner.

2. Join additional tables

The first thing you'll want to do is decide if you have all the information you need in this table or if you'd like to join in an additional table or tables. You can reference any other table from either 1) this dataset, 2) another dataset, or 3) an output table from a different transform in this workflow. Any table you want to reference needs to be in this workflow, so you can click the Add dataset button to add a new dataset to the workflow if it's not here already.

To join a table, add a Join step and select the table you'd like to join. You'll then need to select what type of join it will be, and build your join condition. In most cases your join condition will be linking two variables of the same name and type together (e.g. join all records where id = id).

Learn more in the Joins reference section.

3. Create new variables

You might want to generate new columns in this table by creating a new variable. You can do so by adding a Create variables step. Start by giving this variable a name and then choosing what method you want to use to create it.

Some methods are restricted to the type of variable you are working with. For example there are options to add or subtract years from a date variable, or concatenate string variables.

One of the most common new variable methods is Case (if/else). This allows you to set up a statement that looks at the content of each record and evaluates it based on conditions you've set up to generate the value. For example you can say that in the new variable you're creating, if the amount in column A is greater than 1000, the value of this new variable will be set to "high" and if not, then it will be set to "low."

You can create any number of new variables, and they will execute sequentially, allowing you to reference variable's you've created in other variables and subsequent sections.

Learn more in the Create variables reference section.

4. Filter records

You'll probably want to reduce the number of records in this table to exclude any that aren't relevant to the final output table you're creating. This will allow you to execute transforms quickly and get a better understanding of the number of relevant records you have.

To filter records, add a Filter step to your transform and start building the conditions that records will need to meet in order to stay in your output table. These statements can be nested and allow you to reference any variables or record values.

If you find yourself working with a standard list of values you're using in multiple places, this might also be a place to save time and enhance reproducibility by creating and referencing a value list.

Learn more in the Filters reference section.

5. Aggregate data

Depending on the structure of your data you might want to aggregate the data to collapse multiple records down to one, while preserving some information about what was dropped. Conceptually this might look like aggregating a table of charges from one record per charge into one record per person including a variable for the total charge per person.

To get started, add an Aggregate step and select the variables that you will to aggregate on. These will be the variables that exist in your output data after the aggregation is finished. All records that are an exact match in these selected variables will be dropped.

You can also capture information about records being dropped by creating a new aggregate variable. For example, maybe you are aggregating a table with multiple test scores per person down to a table with just one record per person. You can create an aggregate variable with the average test score, or the count of the number of tests each person took.

Learn more in the Aggegate reference section.

5. Select variables

Finally, before running your transform you'll always need to select which variables you want to keep in your output table. Perhaps you referenced a variable in this transform to create a new one, and now you don't need it anymore. Cutting variables means faster execution, and it is easy to add any variables back later and re-run the transform if you realize you need it, so in general try to keep this list as short as possible.

To propagate variables forward into your output table, they need to be in the right-hand box at the top of the transform labeled Keep. You can select any variable, or set of variables and click the > arrow button to move them over.

Learn more in the Variable selection reference section.

6. Run transform and sanity check output

When you're ready to execute the transform click the Run button in the top right of the toolbar.

If this button is disabled, it might be because the transform is invalid for some reason. Hovering on this button will give you more information, or you can look for the alert symbol (!) to see where you'll need to fix the transform in order to make it valid to run.

Successfully running a transform will create a new output table. Click on this table to see your output and verify that the transform executed as you would expect. Does it have the approximate number of records you expect to see? If you click on a variable you can see summary statistics which can help you verify key variable concepts, such as the distinct number of values.

Every time you run a transform you will want to sanity check your output.

7. Make changes and re-run

If there are any issues with your output table, or if you decide to go in a different direction, or add another step, it is easy to go back to your transform and start making changes.

You'll notice that this transform is now yellow and so is its output table. These edited and stale state indications help you keep track of work you're doing in your workflow, and you can easily revert to the previous successfully executed transform state at any point.

Run this transform again to see changes in the output table. This guide describes all the steps you can take in a single transform but perhaps you want to do one step at a time and run it in between each step to sanity check your output. This system is designed for iteration so make as many changes as you want to experiment and build the output you want.

8. Create another transform

From here you can continue to create transforms on your output table, tables from the dataset you added, or add any other dataset on Redivis. You might want to fit as many steps as you can into one transform, or make a long chain to more easily track your work and communicate it to others.

As you build more transforms you'll see that sometimes actions you take create stale chains of transforms. You can easily make upstream changes in your workflow (such as upgrading a dataset from the sample to the full dataset, or updating to a new version) and then run all transforms in your workflow with one click (a Run all option is available in the Map button menu).

Next steps

Work with data in a notebook

You can use notebooks in a workflow to analyze data using Python, R, Stata, or SAS. These notebooks run in the browser with no additional configuration and seamless sharing with collaborators.

Learn more in the Work with data in notebooks guide.

Export data

If you'd like to export data to a different system, you can download it in various file formats, reference it programmatically in Python / R, or visualize in tools such as Google Data Studio.

Learn more in the Export to other environments guide.

Upload your own datasets

Augment your data analysis in Redivis by uploading your own datasets, with the option to share with your collaborators (or even the broader research community).

Learn more in the Create and populate a dataset guide.

Share and collaborate

Redivis workflows are built for collaboration and include real-time visuals to see where collaborators with edit access are in the workflow, and a comments interface to discuss changes asynchronously.

Share your workflow to work with collaborators in real time, and make it public so that others can fork off of and build upon your work.

This guide demonstrates using a Redivis workflow to gather key variables from different tables, clean them, and and consolidate them into a single table for analysis in Google Looker, all using point and click interfaces.

Workflow objective

We'll use Gini coefficient data (a common measure of income distribution) from the American Community Survey (ACS) to build a map that shows the range of income inequality by US county in 2018.

1. Explore data and start a workflow

We have already uploaded all relevant data for this workflow into a single dataset in the Redivis Demo organization called Example Workflow Dataset which contains contains three tables:

• ACS Gini coefficient table

• ACS table that measures population by county

• National Weather Service dataset that maps a US county to lat/long coordinates.

Add this dataset to your workflow and click on the dataset node to see all of the available tables. Select Gini Index by County and can look at the data format by clicking Cells.

2. Clean data

We want to map the Gini estimate and population by US county in our final data visualization. To map data points by county, we will need a variable that represents the latitude, longitude coordinates of that county. We also want the state, county name, and five-digit county code.

Therefore, we want our final dataset to include six distinct variables:‌

• Gini Estimate

• State

• County Name

• County Code

• Latitude, Longitude

• Population

We will get Lat/Long and Population by joining our table with other tables later on in the workflow. For now, we can create a transform to start reshaping our data.

Rename: Gini Estimate

We already have this variable in our dataset: B19083_001E. We will create a Rename step to rename this to gini_estimate so we can keep better track of it.

New variables: County and State Names

We want to break up our NAME variable into two separate columns, county and state. So we will add the Create variables step and use the Regexp extract method.

This method allows users to create a new variable with characters that match a specific pattern of the old variable. For example, creating a new variable for county, we can select all characters in NAME preceding the comma. We can add a new block and do a similar process on the NAME variable to get the State.

New variable: County Code

The last five digits in the GEO_ID variable represent the county code. We create another new variable block and name our new variable County_code. Select the Substring method and select all values starting at index 10 up to a max length of five.

Variable selection

Finally, we choose which variables to keep or discard in our transform. We’ll keep all variables except for GEO_ID, NAME (county and state combined), and B19083_001M (the margin of error for the Gini estimate).

Click the run button in the upper right-hand corner to run this transform.

3. Sanity check the output table

Click on the newly generated output table to investigate the output of the transform. We can see that this table has the same number of records as our initial table and has the four variables we selected. If we click on the County and State variables we can see in the frequency tables that they look like we wanted them to.

Since this table looks like we expect we can move on to the next step! Otherwise we'd need to go back to the initial transform to change our inputs.

4. ‌Join geographic information

We now have a table with the Gini estimate, state, county name, and county code, but we need the latitude and longitude information in order to map each county and we also want to account for the size of each county with a population variable. To do this, we'll perform two joins.

We could continue to work in our initial transform but we are choosing to create new transforms for each join to keep steps separated conceptually and provide output tables to reference along the way.

Matching county to latitude, longitude data

First, we will match each county code to its lat/long coordinates by joining with the "County to Lat/Long Coordinates" table.

Create a new transform and add a Join step.

A Left Join returns the complete source table (Gini Index by County) and all matching records from the right table (County to Lat/Long). We set the County_code variable in the source table equal to the County_code variable in the right table, which matches a set of lat/long coordinates to each county code (if there is no coordinate, it will return Null).

Some county codes, however, map to more than one set of latitude, longitude coordinates so we must create two new variables in this transform: unique_lat and unique_long.

For unique_lat, we will partition on all kept variables except LAT and take the average of the LAT variable. This says that for that each county, compute the average of all possible latitudes and store that average in the variable unique_lat.

We will select all variables from the source table as well as unique_lat and unique_long in our output table. Finally, we select "distinct" in our variable selector so that we will drop all records that are unique.

Running this transform outputs the following table, where each county now corresponds to a unique set of latitude and longitude variables.

Matching county to population data

We now have all the desired variables in our dataset except for population per county, which we will need when we create the final visualization.‌

Create a new transform and add a Join step.

Using the same steps from the previous join, we will perform a second join to match each county to its total population by incorporating the Population by County table (also in the original dataset). We select a Left Join and set the County_code variable in the source table (Gini Index by County) equal to the County_code variable in the right table (Population by County).

Selecting all variables to remain in our output table and running this transform outputs the following table.

5. Final data cleaning

We will be using Google Looker Studio to visualize this data so we will need to update our data to make it work smoothly with their specified format for geographic data. Using their reference sheet, we can see that we will need our latitude and longitude information in the format:

Comma separated latitude and longitude decimal values (e.g., "51.5074,-0.1278" specifies to London, England)

We will need to combine our two separate values and add a comma between them.

We can do that using the Concat method for creating a new variable, but that method only accepts string inputs so we'll first need to retyping the unique_lat and unique_long variables from float to string using the Retype step.

Then we use the Create new variables step to make a new variable named lattitude_longitude and use the Concat method to concatenate the unique_lat string, a string with a comma, and the unique_long string.

We can now discard unique_lat and unique_long and keep the combined variable, latitude_longitude.

Running this transform yields the final table. From here, we can edit variable metadata and/or download our new dataset in the format that we'd like.

6. Export data to Looker Studio

The next step would normally be to create a notebook in this workflow and use Python, R, or Stata to analyze this table. However if we want to create a quick visualization and aren't familiar with any of those coding languages we have easy options to export data to other systems such as Google Looker Studio.

For this example we will link this table directly to Google Looker Studio by clicking Export table on this table.

We could download our final table and the re-upload it into Looker Studio, but we choose to link the table through the Redivis connector so that if we come back to this original workflow and make changes they will be reflected in the visual that we're about to make there.

Follow the prompts of the connector to log in and authorize access, then we'll need to indicate the table we'd like to use. For that we'll need:

• The owner of the workflow's ID (shown in the top menu bar of the workflow)

• The name of the workflow (shown in the middle of the black workflow toolbar)

• The name of the table (shown on the title of the table node)

When the table is being imported we will have the option to change the type for variables. We'll need to change the type of our lattitude_longitude variable from string to the Lattitude, Longitude geography type.

Then click Create report to get started!

7. Build a visualization

For this example we are going to build a bubble map that shows the size of population against the intensity of gini disparity in certain regions.

Get started by adding a chart and selecting the bubble map type. We will need to select Redivis as the data source for this map.

Then we will need to define which of our variables map to which parts of the visual. We will also want to change how they are aggregated, from SUM to AVG.

• Location: lattitude_longitude

• Tooltip: County

• Size: (AVG) population

• Color metric: (SUM) gini_estimate

The resulting map is automatically interactive. Users can hover over each bubble and view the county name, population, and Gini coefficient. The bubble size is determined by the county population and the bubble color is determined by the Gini value. View the interactive report:

You can continue to create a variety of visualizations in this report, including scatter plots, Google maps, and stacked bar charts.

Refer to the Looker Studio Help pages for additional specific guidance!

Next steps

Perhaps we see something in this workflow we want to tweak, or we want to go back and change some of our data decisions. Workflows are iterative and at any point you can go back and change our source data, our transform configuration or notebooks and rerun them.

Notebooks can also create output tables which allow you to sanity check the work we did in the notebook or perhaps create a table to use in another notebook or transform. You can also fork this workflow to work on a similar analysis, or export any table in this workflow for work elsewhere.

This guide demonstrates using a Redivis workflow to gather key variables from different tables, clean them, and and consolidate them into a single table for analysis in a notebook.

Workflow objective

We want to take weather data collected around the world and use it to understand how precipitation trends in the US have changed over time.

1. Explore data

All the weather data we need is contained in the Demo organization dataset GHCN Daily Weather Data.

To get started we want to understand this dataset and what information is in each table. We can look at the dataset page to learn more about it, including its overview information, metadata, and variable summary statistics. Since this dataset is public we can also look directly at the data to confirm it has the information we need. Some tables jump out as ones we will want to work with:

Daily observations

This table has nearly 3 billion records and seems to be the table in the dataset with the most information so we will start here. There is a variable that looks like it will be very helpful for our goal: element

If we click on the variable name, we can see a searchable frequency table with all this variable's values. One value that jumps out is PRCP which we can see on the value label represents the precipitation. Paired with this is the variable value which contains a numeric value recording the magnitude of the element. We can learn even more about this by clicking on the data dictionary in the dataset's overview page to see that this value is measured in either mm or inches.

We can see that this table doesn't contain any information about location though, so we'll need to find that somewhere else.

Stations

This table contains latitude and longitude variables as well as a linking variable id which will allow us to link in precipitation information from the Daily observations table, since it has the same id variable.

Since there are two tables we want information from we know that we will need to do a join as part of our data reshaping.

2. Create a workflow

At the top of this dataset page we can click the Analyze in workflow button to get started working with this data.

You can add this dataset to an existing workflow you already have access to, or create a new workflow to start from scratch.

3. Clean and reshape data

We will use transforms to clean the data, as they are best suited for reshaping data and will quickly output new tables we can continue to work with. Even though we might be more comfortable with Python or R, the table we want to work with is 132GB and we would not be able to work with it in a notebook without specialized equipment.

Filter for precipitation

If we click on the element variable we can see that there are many types of elements recorded in this table. Since we only want information about precipitation (PRCP), we will start by filtering out any records with a different value in the element variable.

Start by creating a new transform on the Daily observations table, and add a Filter step. Configure the filter to select for records where element = PRCP.

Compute annual precipitation

Looking at the data, it's clear there are many observations of precipitation at each location over time. We know we want to look at annual trends so we can start by aggregating this data to only have one value per year per location.

Conceptually, we will need to aggregate on the year of each record, but if we look at the date variable in this table it also contains the month and day. For aggregation we will need a field which contains only the year which we can collapse on.

In this same transform we will add a Create variables step and make a new variable named year using the date extract method, which will pull the year out of the date value.

Then we will add an Aggregate step to aggregate on the values based on our new year variable. The first thing we will do is select the variables to aggregate on, or select which variables that we want all records with duplicate values to be grouped together. Since we want information on precipitation per station per year we should choose to aggregate on the id variable (station) and year variable. When executed this will group all records with the same combination of values in id and year, drop all other variables from the table, and then drop duplicate records down to just one record per unique combination in those two variables.

But the most important step here is that we want to gather information about the duplicate records that were dropped! If there were 400 records of precipitation in a year for a particular station, we want to know what those records all add up to. To do this we will Create a new aggregate variable within this aggregation step named annual_precip. We want to aggregate the information in the value column, since that contains the numeric amount of the corresponding element variable. Since we want the total amount across all dropped records we will use the Sum method.

Select variables and run

The final step in a transform is selecting which variables we would like to populate the resulting output table. Since we did an aggregation step, this list becomes limited to only the variables that we aggregated on. Since we want both of these variables in our output we make sure both are selected and visible in the right side list of the footer.

With everything in place we will run this transform to create a new table, by pressing the Run button in the top right corner!

4. Sanity check the output table

Now that we have created a new table, we can inspect it to make sure our steps accomplished what we expected them to.

Table characteristics

Click on the output table below the transform to view it. We can see that it contains the two variables we expected it to, based on our variable selection. The table has almost 3 million records, down from the almost 3 billion in our original table, which makes sense given our aggregation step.

Variable characteristics

We can also inspect each variable further. If we click on the year variable we can see that all values are a four digit year value and we can see from the min and max values in the summary statistics that the years range from 1781 - 2021 which make sense. Looking at the frequency table we can see that we have a high frequency of each year represented in the records which is what we expect since there are multiple ids or locations per year.

If we click on the annual precipitation variable we can see a max value of 792,402 (mm or in) which seems very high but possible for a given year. The minimum value is -148,917 which doesn't seem right. When we look at the data dictionary it doesn't indicate that there should be negative values.

We can investigate our data further by clicking on the Query tab of the table and choosing "Sort on variable" from the templates menu, and sort on our annual_precip variable by ascending (ASC). It looks like there are a relatively few number of stations with negative values.

If we were doing a rigorous analysis of this data we might dig deeper into why these value exist in this dataset, or decide to exclude these outliers. However since this is an example we can decide to leave them in and see how it affects our outputs. We can easily go back and exclude them from the workflow at a later point.

5. Join geographic information

Now that we have a table we are satisfied with that contains annual precipitation information, we want to join in the latitude and longitude information about each station so we can make a geographical visual as part of our analysis.

This geographic information is included in the Stations table, so we will need to do a join which connects records from two tables.

While we could do this transformation step in our previous transform and rerun it, let's create a new transform to make this join so that we will have our previous output table saved to compare the output to.

In the new transform add a Join step. Set the joined table to the Stations table from our original dataset. The join type will depend on what variables we want to keep based on how the matching process goes. Since we only want to keep records that have both information from our source table (annual precipitation) AND information from our joined table (station lattitude and longitude) we set the join type to Inner join.

Since our identifier in both table is the variable id, set id from the source table (t0) as the left side of the join condition and set id from the joined table (t1) as the right side of the join condition. This means that for every record in our source table, look for a corresponding value in the Stations table and join the values from additional columns in that table.

There are a number of additional variables in the stations table but we are only interested in the latitude and longitude variables. In our variable selector we move all variables from our source table (year, id, annual_precip) as well as latitude and longitude.

With everything set up, run this transform and sanity check the output table! We started with a 132GB table and have narrowed it down to 134MB containing only the data we want in the format we will need.

6. Analyze outputs in a notebook

Next we want to generate an interactive visual to understand annual precipitation events in different locations over time. To do this we will create a Notebook node on our output table.

For this workflow we will use Python but you can also use R, Stata, or SAS if you'd like. When you create the notebook for the first time it will start up. Notebooks must be running to execute code.

Reference data

Newly created notebooks come with standard code to import the Redivis library and reference the source table in a pandas dataframe within the notebook.

You can use this pandas code or replace it to use the dataframe of your choice. To use the standard code, click inside this cell and press the run button, or the Shift + Enter keys.

Select data from the United States

Now we will create a new cell to organize our work by pressing the + button in the top bar.

In this cell we want to limit our records to only ones whose latitude and longitude values fall in a specific range of the continental United States.

us_df = df[
    (df.latitude >= 24.396308)
    & (df.latitude <= 49.384358)
    & (df.longitude >= -124.848974)
    & (df.longitude <= -66.885444)

Compute average change

We want to see how our annual precipitation variable has changed at stations in the United states so for each station we will group by the id variable and compute an average for annual_precip.

# Compute avg precip for each station between 1980-2010
df_1980_2010_average = us_df[(us_df.year > 1980) & (us_df.year < 2010)].drop(columns=['year'])
df_1980_2010_average = df_1980_2010_average.groupby(['id']).mean()
df_1980_2010_average.rename(columns={"annual_precip": "_1980_2010_avg"}, inplace=True)

# Compute avg precip for each station between 1990-2020
df_1990_2020_average = us_df[(us_df.year > 1990) & (us_df.year < 2020)].drop(columns=['year'])
df_1990_2020_average = df_1990_2020_average.groupby(['id']).mean()
df_1990_2020_average.rename(columns={"annual_precip": "_1990_2020_avg"}, inplace=True)

diff_df = df_1980_2010_average.join(df_1990_2020_average["_1990_2020_avg"])
diff_df['deviation'] = (diff_df._1990_2020_avg - diff_df._1980_2010_avg) / diff_df._1980_2010_avg * 100
diff_df = diff_df[(diff_df.deviation < 25) & (diff_df.deviation > -25)]

diff_df = diff_df.dropna()
diff_df

Plot the results

Now that we have the information calculated, we can plot it! We'll use Plotly to do the heavy lifting, and point it at our relevant variables.

import plotly.figure_factory as ff

import numpy as np
import matplotlib.pyplot as plt
import plotly.express as px
px.set_mapbox_access_token("pk.eyJ1IjoiaW1hdGhld3MiLCJhIjoiY2thdnl2cGVsMGtldTJ6cGl3c2tvM2NweSJ9.TXtG4gARAf4bUbnPVxk6uA")

fig = ff.create_hexbin_mapbox(
    data_frame=diff_df, lat="latitude", lon="longitude",
    color="deviation",
    agg_func=np.mean,
    title="% Change precipitation, 1981-2010 vs 1991-2020",
    range_color=[-15,15],
    nx_hexagon=50, opacity=0.4, labels={"color": "Percent change"}, color_continuous_scale="Icefire_r",
)

fig.update_layout(margin=dict(b=0, t=0, l=0, r=0))
fig.show("nteract")

Now we've created an interactive figure showing change in precipitation by location in the United states.

Next steps

Perhaps we see something in this plot we want to investigate further, or we want to go back and change some of our assumptions. Workflows are iterative and at any point you can go back and change our source data, our transform configuration or notebooks and rerun them.

Notebooks can also create output tables which allow you to sanity check the work we did in the notebook or perhaps create a table to use in another notebook or transform. You can also fork this workflow to work on a similar analysis, or export any table in this workflow for work elsewhere.

This guide demonstrates using a Redivis workflow to train an ML model on a set of images stored in a Redivis dataset.

Workflow objective

This is an example workflow demonstrating image classification via Convolutional Neural Networks. It imports an example dataset containing several thousand test and training images of cats and dogs, with which we can train and evaluate our model.

This workflow is heavily adapted from its initial publication at: https://gsurma.medium.com/image-classifier-cats-vs-dogs-with-convolutional-neural-networks-cnns-and-google-colabs-4e9af21ae7a8

1. Explore data

We can go to this dataset to browse it's tables to understand the structure of the data it contains.

We see three tables here, and all of them are file index tables. That means that each table contains an index of the files (unstructured data) this dataset contains, sorted by the folder the administrator uploaded them into. We can click on the Files tab of the dataset to see each file individually, and click on it to see a preview.

This dataset has three groupings of files:

• Training images (we will use these to build the model)

• Test images (images not included in the training set that we can verify the model with)

• Example file types (unrelated to this workflow)

If we click on the Tables tab, and click on the training images table, we can see high level information about this set of files. We can see that there are 25,000 files, and when we click the Cells tab, all of the file names we can see end in .jpg. We can hover on these to see a preview of the image, and we can click on the file_id variable to see a preview of the image with more information.

2. Create a workflow

At the top of this dataset page we can click the Analyze in workflow button to get started working with this data.

You can add this dataset to an existing workflow you already have access to, or create a new workflow to start from scratch.

3. Define a training set of images

We will use transforms to clean the data, as they are best suited for reshaping data and will quickly output new tables we can continue to work with.

Define training set

We need to start by defining the training set, which conceptually means the set of images we know are cats and know are dogs to train the model on. Information about whether an image is a cat or dog is in the file name, so we need to pull it out into a new variable we can more easily sort on.

We want to include only some of our training set images into the training set we use to train the model, since we want to leave some aside to validate the model. So here we want to include exactly 5000 cat images and 5000 dog images. So we we will create a new variable rank and filter on it so that we only keep the first 5000 images of each type.

The final step in the transform is deciding which variables we want in our output table. We will keep our new boolean variable is_cat to use later, along with the file_id and file_name

With everything in place we can run this transform by clicking the Run button in the top right corner.

4. Sanity check the output table

Now that we have created a new table, we can inspect it to make sure our steps accomplished what we expected them to.

Click on the output table below the transform to view it. We can see that it contains 10,000 records which is exactly what we expected

We can also inspect each variable further. If we click on the is_cat variable we can see that there are 5000 true values and 5000 false values, which shows that our filtering was successful. We can also validate that the method we used to determine if an image is a cat or a dog worked by clicking on the Cells tab. Here we can see that records marked True have "cat" in their file name, and when we hover on the file_ID value to see a preview, the image clearly contains a cat.

Since this table looks like we expect we can move on to the next step! Otherwise we'd need to go back to the initial transform to change our inputs.

5. Define validation set of images

We need to create a set of image files separate from our training set where we know if the image contains a cat or dog. This will be used to validate the model training.

Create a new transform and take all the same steps as we did in the previous transform, but we will change the filter to keep images ranked 5001-7500, rather than 1-5000.

We will keep the same variables as we did in our training model, and then run this transform.

When we run this transform and inspect the output table we see what we expect here as well. There are 5000 total files and we can validate a few of them visually on the Cells tab.

6. Training the model in a notebook

When you create the notebook for the first time it will start up. Notebooks must be running to execute code.

Install packages

Redivis notebooks come with many common packages preinstalled, and you can install additional packages by clicking the Dependencies button and importing libraries in the code.

Since this notebook contains only public data we can install packages at any time, but for restricted data notebooks do not have internet access and packages can only be installed when they are stopped.

Load training and validation sets

Newly created notebooks come with standard code to import the Redivis library and reference the source table in a pandas dataframe within the notebook. For this example we will remove this sample code to import data according to our library's parameters.

Define model parameters

This is where we will heavily rely on our selected libraries to build the model.

Model training

This is where we will train the model we just built using the image data we cleaned.

Evaluate model results

Now we will use the validation set to see how well our model works

Next steps

This guide demonstrates using a Redivis workflow to import an existing LLM and then use relevant data to fine tune it and run it on another similar set of data we are interested in.

Workflow objective

Here, we want to fine-tune a pre-trained "foundational" LLM so that it can be used to score reviews. We will leverage an existing dataset that contains a collection of Yelp reviews and their scores to perform the fine-tuning, and then apply this classification model to other reviews (from Reddit) that do not contain an accompanying score. The goal here is to demonstrate how Redivis can be used to leverage, modify, and ultimately apply state-of-the-art LLMs to novel data.

1. Choose and explore data

For this workflow we'll need our initial data to train the model on (in this case Yelp reviews) and the data we want to apply the model to (Reddit posts). These data are already on Redivis, split across two datasets uploaded to the Redivis Demo organization: Yelp Reviews (Hugging Face) and Reddit.

Yelp reviews

To get started we want to understand this dataset and what information is in each table. We can look at the dataset page to learn more about it, including its overview information, metadata, and variable summary statistics. Since this dataset is public we can also look directly at the data to confirm it has the information we need.

It looks like there are two tables, one with reviews for testing a model and another with reviews for training a model. Clicking on each table in this interface shows that they both have two variables (label and text) and that the Train table has 650,000 records while the Test table has 50,000 records.

This data seems to be formatted exactly how we'll want to use it so we don't need to do additional cleaning .

Reddit

This dataset has two tables with over 150 million Reddit posts and subreddit information, split across two tables. We can look more closely at the 33 variables in the Reddit posts, including univariate statistics.

For this workflow, we just want to look at reviews from one specific subreddit which reviews mouse traps: MouseReview. If we click on the Subreddit variable name, we can see a searchable frequency table with all this variable's values. If we search MouseReview we can see that this dataset contains 26,801 posts.

To move forward with this workflow we'll want to train a model on the Yelp dataset, and filter and clean the Reddit table make it more usable with our model. In order to clean or transform data and do our analysis we'll need to create a workflow.

2. Identify a base model to fine-tune

We want to leverage an existing model that understands language and can generally be used for language classification. There are many open-source models that might meet our needs here; in this example, we'll use Google's BERT-base-cased model.

This model is hosted on Hugging Face, so we could load it directly into our notebook at runtime. However, if our notebook uses restricted data, it might not have access to the external internet, in which case we'll need to load the model into a dataset on Redivis.

The Redivis dataset for this model can be found here. You can also learn more about loading ML models into Redivis datasets in our accompanying guide.

3. Create a workflow

At the top of any dataset page, we can click the Analyze in workflow button to get started working with this data.

You can add this dataset to an existing workflow you already have access to, or create a new workflow to start from scratch.

Add the additional datasets by clicking the + Add data button in the top left corner of the workflow and searching for the dataset by name.

4. Create a notebook and load model + data

Once we've added all our datasets to the workflow, we can get started. To begin, we'll create a python notebook based on the Yelp reviews training data, by selecting that table and clicking the + Notebook button.

To enable GPU acceleration, before starting the notebook, we'll choose a custom compute configuration with an NVIDIA-L4 GPU, which costs about $0.75 per hour to run (we could use the default, free notebook for this analysis, but it would take substantially longer to execute).

We'll also need to install a few additional dependencies to perform training and inference via Hugging Face python packages:

With that, we can start our notebook! The full annotated notebook is embedded below, and also viewable on Redivis.

The general steps here are as follows:

1. Load the training and test data from the Yelp reviews dataset

2. Load the pretrained BERT model

3. Train this base model on the Yelp data to create a fine-tuned model that can classify text reviews from a score of 0-4.

5. Prepare the Reddit data for inference

The Yelp data was ready to go as-is, with a simple text field for the review and integer value for the score. For the Reddit data, we just need to run a quick filter to choose posts from the appropriate sub-reddit.

We will use a transform to clean the data, as transforms are best suited for reshaping data at scale. Even though we might be more comfortable with Python or R, this dataset table is 83GB and it will be much easier and faster to filter it in a transform rather than a notebook.

Create a transform

Click on the Posts table in the Reddit dataset and press the + Transform button. This is the interface we will use to build our query.

Add a Filter step. Conceptually we want to keep records that are part of the subreddit we are interested in, and are not empty, deleted, or removed.

The final step in a transform is selecting which variables we would like to populate the resulting output table. In this case we just need the variables title and selftext.

With everything in place we will run this transform to create a new table, by pressing the Run button in the top right corner.

7. Use the fine-tuned model to classify Reddit reviews

Finally, we can apply our fine-tuned model to the subset of Reddit comments that we want to analyze. Ultimately, we produce a single output table from the notebook, containing the reddit post and associate score generated by our model.

Next steps

Workflows are iterative and at any point you can go back and change our source data, our transform configuration or notebooks and rerun them. Perhaps we want to look at other subreddits, or run the model on a larger sample of the Yelp data.

You can also fork this workflow to work on a similar analysis, or export any table in this workflow to analyze elsewhere.

We do recommend further familiarizing yourself with the examples and detailed documentation to take full advantage of the capabilities of Redivis notebooks:

• Guide to running ML Workloads on Redivis

• Python notebook reference

• R notebook reference

Overview

Redivis workflows contain powerful tools to reshape and analyze data but if you prefer to export data into a different workflow you can easily do so. Redivis systems use open source tools and common formats to make the transition as easy as possible.

1. Check export restrictions

Some datasets on Redivis have export restrictions which prohibit or limit removing data from the system.

You can check this by going to the dataset page and clicking the top right button Manage access, or by right-clicking on a dataset in a workflow and selecting the View access option.

The bottom section of this access modal defines any export restrictions in place. It might not have any restrictions, might be completely restricted, or might have some limited options for export.

If there are limited options for export, this section will detail the available locations and any restrictions on using those (such as the size of the table being exported). These restrictions are enforced automatically when you try to take actions within the system.

If there are no options for export, you still have the option to work with your data in a Redivis workflow, where you can reshape and clean your data using transforms, and then analyze it in notebooks.

Learn more in the Work with data in a workflow guide.

2. Prepare your data

If you would like to export the entire table, you can skip this step.

Otherwise, we recommend that you use transforms to cut your data down to a smaller size and reshape it into the table format you need for analysis before exporting it. Especially for large datasets, Redivis tools are created specifically for these purposes and work seamlessly in the browser with no additional setup.

Learn more in the Reshape tables in transform guide.

3. Export data

You can open the Export modal to initiate an export from any table. You can do this on the dataset page by right clicking a table and selecting the Export menu option, or by right clicking on any table in a workflow and selecting the same option.

Here you can see all the options available for your table to export.

If your table has export restrictions set by the data owner, options will be disabled here. In the case where your table does not meet export requirements but the data owner allows exception requests, you will see the option to do so here.

Download

The first tab of this modal gives you different options for formats to download your data. Most common data formats are supported, including csv, json, avro, parquet, SAS, Stata, and SPSS. Select the format you'd like and click the Download button to start downloading the file.

Learn more in the Downloads reference section.

Programmatic reference

You can reference this file from your computer or another computational environment using the Redivis Python and R libraries. This modal gives specific information on how to reference this dataset, and you can reference our docs for other options.

Redivis-python library

Redivis-r library

Integrations: Google Looker Studio

This is a free Google dashboard visualization program you can directly link your table to. Use their point and click interface to build common visuals that will update as the underlying table updates.

Learn more in the Google Looker Studio reference section.

Integrations: Google Cloud Storage

You can export your table directly into a Google Cloud Storage bucket that you have access to.

Learn more in the Google Cloud Storage reference section.

Integrations: Google BigQuery

You can export your table directly into a Google BigQuery project that you have access to.

Learn more in the Google BigQuery reference section.

Next steps

Cite datasets in your publications

If the work you're doing leads to a publication, make sure to reference the dataset pages from datasets you've used for information from the data administrators on how to correctly cite it.

Overview

Redivis combines powerful functionality to reshape and analyze data on platform, with an easy export and publishing flow, to ensure the results of your work can be displayed in the format of your choice and shared with your collaborators and research community.

Working to determine, first, the broader picture of the content you'd like to publish and where you'd like to publish it will help you, then, determine the desired shape and format of the assets to be exported and, finally, the specific sources of the data in your Redivis workspace.

1. Develop a publishing strategy

Ask yourself: What story are you trying to tell? Who/where is the audience? What component pieces are crucial to building the narrative? Your ideal package of assets may be very different if you're hoping to share progress on a workflow with a principal investigator, take a snapshot of a single variable distribution for a colleague, publish results in a journal, or build a custom dashboard to highlight multiple trends.

You may use a combination of tables, unstructured files, code snippets, graphs, and descriptive text to be published, so sketching out these component pieces in increasing detail will help define your end product.

2. Choose your desired formats

With an initial strategy in mind, learning about the different types of Redivis exports will help define your list of assets to generate.

Tabular data containing rows and columns can be exported in a variety of formats, accessed programmatically from many environments via our client libraries, or embedded in a website.

Unstructured data files of any type can be previewed and downloaded in their original format, or accessed programatically.

Notebooks containing code inputs and corresponding outputs can be exported as an .ipynb file or PDF or HTML, or (coming soon!) embedded in your site.

Learn more in the Export to other environments guide.

3. Identify the data sources

In a workflow, you can generate output tables to capture the result of a set of data transformations, or use notebook to show a line-by-line data analysis flow and relevant figures.

As a dataset creator, you can upload your own tabular data and unstructured files to create assets to use in your workflows or share with others.

Whether you're manipulating data in a workflow to showcase results or hosting your own dataset, you'll want to build a set of specific tables or notebooks you're trying to share. As you iteratively modify the shape and content of these assets – transform your data in a workflow to output new tables or use notebooks to build analysis flows in Python or R – you'll fine-tune each piece of your final publication.

Next steps

Upload your own datasets

Augment your data analysis in Redivis by uploading your own datasets, with the option to share with your collaborators (or even the broader research community).

Learn more in the Create & manage datasets guide.

Build a custom dashboard

For full customizability, you can publish a static site that accesses Redivis data to power an interactive visual dashboard – a more permanent, web-based approach to highlighting results of your data analysis or generated data.

Learn more in the Build your own site guide.

We've started building a library of common actions that you might take when administering an organization.

Please let us know if there are other actions or concepts you would like us to provide examples for by reaching out or emailing contact@redivis.com

Overview

Any type of file can be uploaded to a Redivis dataset, and we support previews for the most common file types. These data can be analyzed in notebooks within workflows.

Note that if you have tabular data we strongly recommend uploading it as a table (rather than a file) so you and your researchers can take advantage of our extensive toolkits for previewing and manipulating tabular data.

This guide assumes you have already started by Creating a dataset.

1. Locate the data you want to upload

You can upload data directly from your computer, or import from a linked account.

If importing, you'll want to get the relevant external account configured to your Redivis account before getting started.

The import tools allow for multiple uploads, so it's helpful to have them all in the same place.

2. Upload files

When you're ready click the Upload files button on the Files tab of the dataset page to get started and select the location your files will be coming from.

If you select "Computer" then you will get a browser window where you can choose the file, files, or folder you want to upload from. Choosing a different location will bring up the corresponding mechanism to select your files.

Once selected, you will need to choose a Folder to put them in. All files on a dataset live within one folder which have a corresponding index table to help researchers understand and navigate those files. You can change this later if needed. If this is your first time uploading files to this dataset you will need to create your first folder.

Click the "Upload" button to start the process. If you are uploading from your computer you'll need to wait on this screen until it's complete. Closing your window will end the upload process. If uploading from a linked source, you can close this window to allow this process to continue in the background.

3. Manage files and folders

You can click on the file name in the Files tab to preview the file and see its information.

You can also go to the Tables tab to view the index table for the folder you've uploaded files to. View the cells and hover or click on the file ID variable to see the preview here. This table can be used in Transforms within a Workflow to work with files on a large scale.

If you want to rename a folder you can do so on the Files tab by right clicking on the folder name in the right bar.

You can also create new folders and move files between two existing folders from this bar. When moving between folders, you can use a conditional statement to only move some files that match your conditions.

Next steps

Continue uploading your dataset

Great metadata makes your dataset useable. Complete your metadata, along with configuring access, creating a sample, and releasing this version.

Learn more in the Create & manage datasets guide.

Overview

It's easy to use of Redivis data anywhere on the web, either by interfacing programmatically via our client libraries, visualizing via one of our export integrations, or embedding a table directly in another webpage.

To present Redivis data in a fully customizable way, you may want to build your own dashboard-style site. The following example shows a static site built with Observable Framework, deployed via Github Pages, that connects to Redivis data using our client libraries.

1. Identify your Redivis data

Identify the data sources on Redivis that will provide the content to build your site. You may be interested in presenting facets of a dataset already hosted on Redivis, highlighting tables you uploaded yourself, or showing outputs generated in one of your workflows. In our example, we'll use weather station locations contained in the GHCN Daily Weather dataset, and outputs from a public Redivis workflow that uses the same source data, to visualize locations and aggregate precipitation measurements from around the world.

To reference a Redivis table, choose the "Export table" option and navigate to the "Programmatic" tab (e.g., the GHCN Daily Weather "Stations" table), and see the code snippet for unique identifiers for the owner, dataset/workflow, and table. More details about referencing Redivis resources here.

2. Build your Observable project

With chosen data content in mind, the next step is to choose a set of tools to build our custom dashboard and deploy it to the web. Of the numerous choices, Observable Framework provides an approachable option with support for working with data in many programming languages (not just javascript), powerful visualization libraries built in, and excellent documentation. To start with a copy of our example dashboard, you can clone our public Github repo and follow the README.md instructions to develop a similar site in a local environment. To create a brand new project, and to reference additional details and development strategies, see the Observable Getting started guide.

3. Load data with a client library of choice

You'll need to generate a Redivis API access token with appropriate permissions. See Authorization in our API documentation to create your own token to read data from Redivis.

To successfully authorize data fetching functionality in your development environment, be sure to export a REDIVIS_API_TOKEN variable to your path, with the following terminal command: export REDIVIS_API_TOKEN = 'MY_API_TOKEN'

Writing a "data loader"

In Observable, a data loader is a file that accesses data and writes it to a standard output channel for consumption by renderers, to be ultimately displayed on your site.

In our example, we'll illustrate use of both javascript and python to pull data from Redivis into our dashboard, using the Redivis API via js and python client libraries.

Using the redivis-js client library

In a web environment, the redivis-js client library allows for fetching and manipulating data via simple javascript functions.

In our example, we first install the library by specifying the latest version in our package.json and running npm install in our development environment to install or update all specified packages.

Then, in the Observable data loader file redivis-js-precipitation.json.js , we authorize access, and then listVariables and listRows from our specified precipitation table.

Using the redivis-python client library

In a python environment, the redivis-python client library allows for fetching and manipulating data via simple python functions.

In our example, we'll create a requirements.txt file that specifies the latest version of the client library, both for development and future deployment. Running the following command in your virtual python environment will install pip, and then any package specified in the requirements.txt file.

python3 -m pip install --upgrade pip
pip install -r requirements.txt

Then, in the Observable data loader file redivis-python-geodata.json.py , we create a pandas dataframe from our specified stations table, and output the records.

4. Visualize data

With data loaded and parsed into JSON format, it can now be manipulated and presented as you see fit. Among a wealth of modern web visualization tools is Observable Plot, which provides a polished selection of simple visualizations and corresponding functions to group and filter data, and pairs nicely with our development framework.

In our example, we use several Observable Plot functions to display our data on a map – using Plot.plot() with Plot.geo() to show a globe – and in a hex-binned histogram – using Plot.dot() and Plot.hexgrid() for data aggregation. We also add a range input to allow viewers to rotate the globe visualization.

Then, we use the built-in display command to print the JSON-shaped payloads, to give the viewer a quick look at the raw data.

5. Deploy and share your site

With our dashboard finalized, you can deploy your project anywhere on the web. Observable provides a quick command (npm run build) to generate static files to be hosted on a server of your choice, as well as a deployment service if you'd like to host with Observable.

Deploying to Observable

You can run npm run deploy in the command line, and follow the prompts to deploy your project to an address on Observable. Further configuration and permissioning is available through your Observable account.

Deploying to Github Pages

Github Pages provides a hosting service that's easy to integrate with your Github repository and any Github Actions needed to deploy your project.

In our example, we deployed our project to Github Pages in a few simple steps.

First, we need to specify a REDIVIS_API_TOKEN to support Redivis client library authorization in the live deployment. We used "action secret" within the Settings of the Github repo, which sets an environment variable REDIVIS_API_TOKEN to our appropriately scoped token, which is completely private while our repository remains public.

Next, we write a Github Action to build and deploy the project, which will run anytime we push to the main branch of our repository. The action is specified via the deploy.yaml file, and relies on a set of open-source actions and our own custom code to do the following steps:

• Checkout the code

• Set up a Node.js environment, with a specific node version (20.6)

• Install our python dependencies (via pip,specified in requirements.txt)

• Build the static files for the site

• Deploy the files (in the generated dist folder) to Github Pages via a specific branch (gh-pages)

After adding this file to your Github repository, any push to the main branch will trigger a deployment following the steps above, the status of which can be viewed under the Actions tab of the repository.

Finally, Redivis has hosted a number of different team projects at a custom endpoint specified in our organization's Github Pages settings, which you can see at labs.redivis.com.

Overview

Redivis datasets are a great place to host data for interrogating in a workflow, but you can also edit the underlying data.

Perhaps you found an issue with the source, or want to restructure it, before making it available to others. Redivis has all the tools you'll need to do this in a versioned, step-by-step process with transparency into the changes you've made.

1. Upload the raw data

If you haven't yet, upload the data you want to work with to a Redivis dataset and release the version. A personal dataset or one belonging to an organization works just as well.

2. Add this dataset to a workflow

Create a new workflow and add this dataset to it.

If you want to share the data transformation process with others for transparency, you can make this workflow public in the share modal. (Due to the way Redivis works, still only people with access to the underlying data will be able to see the data in the workflow even though the workflow is public.)

For this guide you can follow along in this workflow Demo tables edits.

3. Transform data

Select the table you want to make changes to and create a new transform.

Use this transform to edit this table. Some common actions include:

Example edit 1: Rename and retype a variable

Create new steps to rename and retype any variables you'd like to update.

Example edit 2: Recoding the values of a variable

Create a new variable with the same name. "Keep" this variable and "Discard" the original variable. Select the method as "Case (if/else)" and create the conditions you want to recode to.

Example edit 3: Add a new variable

This example shows a new variable with today's data as the "date_uploaded" but you could create any variable you want from the data in the table. Perhaps a aggregation that would be helpful to see with this data? Or the sum of multiple other variables?

You can transform this table however you want, using the Graphical interface or SQL code. Make sure to move all variables in this table from the "Discard" section to the "Keep" section except for any variables that have been replaced and will be left behind (in this case the store_and_fwd_flag from the source table).

Validate that this new table looks correct by looking at the output table below this transform!

Make changes to any tables in this dataset in this same wokflow.

4. Re-upload the finished tables

Go back to the original dataset, and create a new version.

Open a table that you made changes to in your workflow, and click to "Import data."

Choose your merge strategy as "Replace" since you want to replace the existing table with the new one you created.

Select the data source as "Redivis" and type in the information from the workflow you made your changes. The sequence here will be:

1. Your user name

2. The name of the workflow (underscores replace spaces)

3. The table name (underscores replace spaces)

For example: username.testworkflow.table_name

Once it's done uploading you can validate that the data looks like it's supposed to.

Now you can close this table and repeat this process for any other tables in this dataset you've edited.

Once this is finished, you can add the link to the worfklow into this dataset's documentation or release notes along with a note on the changes. Remember that if you made the workflow public, anyone who has data access to this dataset can view the changes you made.

Now release this version! This new version of the data contains the edited data. Anyone using this data in a workflow will see that this dataset has a new version next time they open a workflow. You can see the release notes and the updated table of our Demo tables updates live.

Next steps

Start working with your data

Once your dataset is released, bring it into a workflow to transform and analyze it leveraging lightning fast tools from your browser.

Learn more in the Analyze data in a workflow guide.

Many insurance claims datasets on Redivis contain information about enrollment, detailing the periods of time when an individual represented in a dataset was covered by an insurance plan. If you intend to characterize patients based on their insurance coverage (or lack thereof) during certain key events (procedures, diagnoses, etc), it's often important to identify periods of continuous enrollment for each individual – and capture each continuous enrollment period for each patient in a single row.‌

These claims datasets describe enrollment information in multiple discrete rows per patient, each corresponding to patient per month. However, an overall continuous enrollment period may be broken up across rows into months or other non-uniform chunks, so we'll employ the following process to combine multiple sequential rows into a single row with one start date and one end date, describing one continuous period.‌

In this example, we will process the MarketScan Enrollment Detail table to create a table in which each row describes a single period of continuous enrollment. We show an artificial snapshot of the dataset below, where patient 1 has multiple periods of continuous enrollment due to some gaps in coverage, and patient 2 has a single period of continuous enrollment.

We want to create a final table with 3 rows for patient 1 to account for gaps in enrollment in March and May of 2012, and 1 row for patient 2. Our desired output has a row for each continuous period per patient, shown below:

The variable names in this example are not actual MarketScan variable names. But, with appropriate data access, you can see the real variables used in the first transform of the Redivis example workflow by hovering over the (renamed) variables patient_id, enrollment_start_date, and enrollment_end_date in the Keep section.‌

(1) Add start and end of adjacent periods to each row

Throughout this example, we'll create variables that partition the dataset by patient identifier (here, patient_id) to ensure that each patient is processed individually. But, to account for the fact that a single patient may have many periods of continuous enrollment, each spanning many months, we need to identify the correct start (from enrollment_start_date) and end (from enrollment_end_date) of a continuous period out of multiple rows and capture them in a single row.‌

First, we create a partition variable lag_end_date using the lag method, which will order each row by enrollment_start_date in a given patient_id partition and copy the previous row's enrollment_end_date value into the each row. We also create lead_start_date using lead, to copy the following row's enrollment_start_date value into each row.‌

These methods generate values which tell us how close the preceding and following enrollment periods are with respect the each row's enrollment period.

(2) Select rows with start and end of continuous periods

In a second, downstream transform we'll create new variables, which will use the above lead and lag values to identify which rows correspond to the beginning and end of a continuous enrollment period.‌

First, we'll compare enrollment_start_date and lag_end_date to find the difference (in days) of the start of each period and the end of the previous period. and compare enrollment_end_date and lead_start_date to find the end of the period.‌

We see via diff_lag_end_enrollment_start (created using the date diff method) which rows describe an enrollment period directly following the previous period, and which rows describe an enrollment period with a larger gap since the previous period. We also create diff_enrollment_end_lead_start to identify gaps between an enrollment period and the next period.

Next, we'll encode booleans from our difference variables to simplify further filtering. We'll identify rows corresponding to the start of a continuous period as those with a diff_lag_end_enrollment_start value of either NULL (the row is the first period in the partition) or greater than 1 (the row comes after a gap in enrollment). And we identify rows corresponding to the end of a continuous period as those with a diff_enrollment_end_lead_start value of either NULL (the row is the last period in the partition) or greater than 1 (the row comes before a gap in enrollment).‌

We see via boolean variables is_start_continuous_period and is_end_continuous_period (created the case method) if a given row corresponds to the start of a continuous period, the end of a continuous period, or both.

Then, to capture only the start and end of a continuous period, we'll use a filter to keep only rows which are true for either is_start_continuous_period or is_end_continuous_period.‌

This leaves us with either 1 or 2 rows corresponding to a continuous enrollment period. If a continuous period spans multiple rows (months, in this case), we'll have 2 rows (a start row and an end row). But if a period only spans one row, we'll have both start and end captured by that 1 row. In our example, the row containing the middle (neither start nor end) of the 2012-06-01 to 2012-08-31 enrollment period for patient 1 was dropped. We can also ignore our intermediate lead..., lag..., and diff... variables, since our final processing step will only consider a row's is_start_continuous_period and is_end_continuous_period values.

(3) Add end date of following enrollment periods to each row and collapse

Finally, we want to collapse our table to ensure 1 row per continuous enrollment period per patient. Since we have only rows corresponding to start and end of continuous periods, we create another partition variable lead_end_date (in the same transform is fine, since this step will happen after the previous filter) which copies the enrollment_end_date value of the following row on to each row.‌

We can also use the partition row filter to keep only rows with is_start_continuous_period as true, since our lead_end_date has copied over the end date of the continuous enrollment period, contained in each row's following row.

We end up with a table where each and every row contains both the start date of the continuous enrollment period and the end date of that continuous enrollment period.

A final processing step captures the correct end date of a continuous period. We now have only rows whose enrollment_start_date value contains the start of a continuous period, but these rows fall into two categories:‌

• First, we have the rows that do not also correspond to the end of an enrollment period (where the is_end_continuous_period value is false). For these, we want to look at lead_end_date, the end date of the next row, which represents the final date of the continuous period, since we filtered out all the intermediate rows above.

• Second, we have which also correspond to the end of an enrollment period (where the is_end_continuous_period value is true) – in this example, if the continuous period was only 1 month. For these, the row defines a period (in this example, 1 month) that also contains the end date, so we just get the enrollment_end_date from that same row. Note that the lead_end_date value is incorrect in this case, since the next row contains the start of the next continuous period, or NULL if the period falls at the end of a partition.

We capture the above logic in a new variable enrollment_end_date_continuous, created in an additional transform, since our previous operation involved a partition.

We end up with a final table below, containing the patient identifier, and the start date (renamed to enrollment_start_date_continuous for consistency, and end date of each continuous enrollment period.

Overview

Datasets are a core component of Redivis. They are a versioned collection of tables containing data, alongside rich documentation and metadata.

Datasets can be hosted by organizations or individual users, and every dataset has its own Dataset page. Datasets can be shared with other users on Redivis according to their access configuration.

1. Create a dataset

Administrators can create datasets for their organization from the Datasets tab of their organization's Administrator panel. These datasets can be seen and managed by any administrator in the organization. When released they will be visible on the organization’s home page to anyone who as overview access tot he dataset.

Alternatively, anyone with a Redivis account can create a dataset on the Datasets tab of their Workspace. These datasets are by default only visible to their owner, and have simplified options to support sharing with your collaborators.

When you first create a dataset, it will be unpublished and only visible to other editors. This means you can edit the dataset and validate everything before releasing it. You can also validate and reconfigure its access rules before release.

2. Import data

At the core of every dataset is the data it contains, so we recommend starting here.

All data in a dataset is stored in tables. You can create a new table on the Tables tab of your dataset and start importing data. Redivis can upload data from your computer or another location you’ve linked, such as Box, Google Drive, AWS, and Google Cloud.

Once your data is finished importing you can validate that this table looks as you expect it to.

You can create more tables here if this dataset has multiple separate tables.

However, if your data is split across multiple files that all follow the same structure (such as a different file for each state, or each year of data but with generally the same variables) you will want to import all of these files to the same table, where they will be automatically appended together.

Learn more in the Upload tabular data as tables guide.

3. Populate metadata

Metadata is essential to helping your researchers find and utilize your dataset. While some metadata will be generated automatically, such as variable summary statistics and counts, other metadata will require additional input.

Dataset metadata

On the Overview tab of the dataset there are multiple suggested sections to help break down information. You can fill out any of these that apply, such as the methodology, tags, contact information, etc.

Redivis will automatically generate citation and provenance information based on what we know about the dataset, but you can update this information with anything more specific.

If you have additional information that you want to include that doesn't fit one of these headers, you can create a custom section. Custom sections can also be set to be visible only to certain access levels if you have sensitive information.

Table metadata

You should also populate the metadata on each table. Tables can have a description, as well as an entity field that defines what each row in the table represents. You can also define the temporal and geographic range on the table, when relevant.

Variable metadata

Each variable within a table has its own metadata. The variable name and type will be pre-determined from your data, but you should add a short label and longer description to each variable to help researchers understand what that variable measures.

Additionally, some variables will contain coded values, in which case you should provide value labels that represent the human-readable term for each code.

Learn more in the Create and populate a dataset guide.

4. Release

Once you are ready to make your dataset available to others, you'll need to release it. You can click the Review and publish or Review and release button in the top right of the page.

You should also confirm your access settings you set up when creating this dataset.

Once this dataset is released, it will become visible and available to anyone who would qualify for access.

Next steps

Edit data

You can use tools right on Redivis to create new versions of your dataset.

Learn more in the Edit data in a dataset guide.

Administer your organization

Organizations allow for groups, centers, and institutions to more easily work with data by providing administrators with tools to effectively version and distribute data from a central location.

Contact us to set up a new organization, and learn more in the Administer an organization guide.

Overview

Datasets are a core component of Redivis. Consisting of documentation, metadata, and tables, datasets allow you to store, version, and distribute a wide variety of data.

Anyone with a Redivis account can create a dataset in their workspace, and organization administrators can upload one to an organization via the administrator panel.

1. Create the dataset

You can create a new dataset by navigating to the Datasets tab of your workspace or administrator panel and clicking the New dataset button.

All datasets must have a name (unique to datasets for the user / organization).

You can set up your dataset in whatever order you'd like, but we recommend following the order below when getting started.

2. Upload data

This data might be in a tabular format (.csv, .tsv, .sas etc.) or more rarely in unstructured data such as images and text files.

Tabular data

All tabular data is associated with a table and each dataset can have one or more tables. While you may release a dataset without any tables, this will be of limited use to other researchers, as Redivis provides numerous tools for understanding, querying, and generally working with tabular data.

If you haven't already worked with data in a workflow, we strongly recommend exploring that before creating dataset so you can understand how researchers will work with your data.

When you're ready to upload data, we have broken that out into a separate guide.

Learn more in the Upload tabular data as tables guide.

Unstructured data

For unstructured data, go to the Files tab where you can upload files from your computer or another location your data is stored via an integration. You can put these into folders and create index tables to better keep track of.

Note that any files uploaded here can't be transformed in the workflow tool or queried across Redivis (which require the table format).

Make sure any files you upload here contain this dataset's data. Any files with information about the data (such as data dictionaries or usage guides) should be uploaded as documentation on the Overview tab.

Learn more in the Upload unstructured data as files guide.

3. Edit metadata

It's easy to feel "done" after uploading your data, but documentation and metadata are essential to the usability of your dataset. Moreover, rich metadata will improve the discoverability of your dataset by providing more information and terms to the Redivis search engine.

Metadata can always be updated after your dataset has been released. While good metadata are essential, it can be a time consuming and iterative process, so you might prefer to provide some basic content initially, and then improve it over time.

Dataset metadata

On the overview tab of the dataset, you can provide an abstract, detailed documentation blocks, supporting files and links, and subject tags for the dataset.

The abstract should be a brief overview of the dataset, while the rest of the documentation can be as thorough as you'd like. Each documentation block has a header for suggested content, and any you don't fill out won't be shown on the dataset page. These blocks contain rich text editor complete with embedded images. Most of this information will be visible to anyone with overview access, though you can also create custom documentation sections that require a higher level of access.

Make sure to audit your data's provenance information to give attribution to whoever is working on the data. If this dataset is part of an organization you can configure a DataCite account to issue a DOI for each dataset. Note that if your organization is configured to issue DOIs, then one will automatically be issued for this dataset when you first publish it.

Table metadata

To help users understand what each table represents, you should update the description, entity, and temporal range for each table in the dataset. The entity should define what each row in a table represents: is it a person? an event? a charge? The temporal range can be tied to a specific variable (using the min/max of that variable), or defined explicitly.

Variable metadata

The tables in your dataset are made up of named variables, though rarely is this name enough to understand what the variable measures. On any table, click "Edit variable metadata" to order to populate the variable metadata.

On each variable, Redivis supports a label, description, and value labels. The label is the most essential item, think of it as a more human-readable variable name. The description should contain more detailed information, everything from caveats and notes to collection methodology. Value labels are only applicable when the variable is encoded with keys (often integers or short strings) that map to the actual value — for example, a survey might be encoded as 0: "No" 1: "Yes" 2: "Don't know" 3: "Declined to answer".

Editing variable metadata can be a tedious process, but Redivis does support the ability to import metadata from a file, and will also automatically extract metadata if it's present in the uploaded data files (e.g., Stata or SAS upload types).

Learn more in the Documentation reference section.

4. Create a sample

If your dataset is particularly large, or if you want to control access to a sample of the data separately from the whole dataset, you should configure sampling on your dataset. This will allow researchers to work with a 1% sample of the data during initial exploration, and allow you to grant access to the sample independently of the full dataset.

To update the dataset's sample configuration, click on any table, and then click "Configure sample". When configuring the sample, you can generate a random sample for each table, or sample on a particular variable that is common across tables. If researchers will be joining tables across your dataset, it is highly recommended that you sample on that common join variable so that researchers can contain a consistent 1% sample as they work with your data.

Learn more in the Dataset sampling reference section.

5. Configure access

Before releasing your dataset, it is important to define who can access the dataset and what the procedures are for applying and gaining access. Click the Configure access button on the top of the page to set up the access configuration.

Datasets owned by organizations have more options for access than datasets owned by users.

Access levels

Dataset access has five levels:

1. Overview: the ability to see a dataset and its documentation.

2. Metadata: the ability to view variable names and summary statistics.

3. Sample: the ability to view and query a dataset's 1% sample. This will only exist for datasets that have a sample configured.

4. Data: the ability to view and query a dataset's tables, and work with them in workflows.

5. Edit: the ability to edit the dataset and release new versions.

Access levels are cumulative. For example, in order to gain data access you will need to have gained metadata access as well.

Usage rules

Even with data access, you may want to limit what other users can do with your dataset. Currently, you can configure export restrictions that limit:

• The download location (e.g., to prevent researchers from downloading to their personal computer)

• The download size, in bytes and/or rows

• Enforce admin approval before any export

Editors

You may also add additional dataset editors to help upload data and provide metadata content. These editors will be able to create and release new versions, and will have full access to the underlying data, though they cannot add other users, modify the access configuration, or bypass the dataset usage rules.

If the dataset is hosted by an organization, all administrators of the organization will be able to the edit the dataset as well as its access configuration.

Permission groups

If the dataset is hosted by an organization, you will have additional options for configuring access to the dataset. The dataset can be assigned to a permission group to help standardize access procedures, and this permission group can contain requirements that help data managers fulfill contractual requirements and gather relevant information about the research being done on the dataset.

Learn more in the Configure access systems guide.

5. Release the dataset

Congratulations! Your dataset is ready to be released and utilized by the research community. But first, it is highly recommended that you validate and audit your dataset beforehand. Take a look at the number of rows, variables, and uploads in each table. Validate some of the variable summary statistics against what you expect. And to be truly thorough, add the dataset to a workflow and run some queries as if you were a researcher. Catching a mistake now will prevent headaches down the line if researchers uncover unexpected discrepancies in the data.

When you're confident that you're ready to go, click the "Release" button on the top of the page. If the button is disabled, hover over it to understand what issues are currently preventing you from releasing.

After clicking the button, you'll be presented with a final checklist of tasks. When you click the Release version button, the dataset will be immediately released and available to all users with access.

This dataset is now also considered Published. If you need to pause all activity and access to this dataset, you can return to this page in the future and Unpublish it temporarily.

6. Make updates as new versions

Once a dataset is released, you can return to it to make changes at any time. Changes to datasets are tracked in Redivis as versions. Anyone with access to a dataset can view and work with any of its versions.

How to work with versions when updating a dataset:

• Any edits to the data content in tables will need to be released as a new version.

• Edits to the dataset information, table information, or variable metadata can be made on the current version (or historic versions) and will be live as soon as it's saved.

• Edits to the dataset name and access configuration will always affect all versions.

Creating the next version

All data within a dataset is encapsulated in discrete, immutable versions. Every part of the dataset except for the name and access settings are versioned. All tables in a dataset are versioned together.

After releasing the first version of the dataset, you can choose to create a new version at any time by clicking the button in the top right "Create next version". This version will be created as vNext, and you may toggle between this and historic versions at any time.

Subsequent versions always build on the previous version of the dataset, and changes made in the next version will have no affect on previous versions. Alongside modifications to the dataset's metadata, you may create, update, or delete any of the previous version's tables.

Replacing vs appending data

When uploading data to a previous table, you can choose whether you want to append these new uploads to your existing data, or replace the entire table with the new data.

Version storage costs

Redivis computes row-level diffs for each version, efficiently storing the complete version history in one master table. This allows you to regularly release new versions and maintain a robust version history without ballooning storage costs.

Learn more in the Usage and limits for users, and Billing for organizations reference sections.

Next steps

Start working with your data

Once your dataset is released, bring it into a workflow to transform and analyze it leveraging lightning fast tools from your browser.

Learn more in the Analyze data in a workflow guide.

Overview

Organizations are where administrators centrally manage datasets and who can access them. All organization administration happens from the Administrator panel which administrators can access from the Organization home page.

If you're interested in using Redivis for your research group, center, or institution, contact us to get an organization set up.

1. Add other administrators

You can add any Redivis account to be an administrator of your organization in the Administrator panel from the Settings tab.

Note that all administrators will have full access to the organization and its datasets, including the ability to modify access rules, approve access requests, and add or remove other administrators, including you.

2. Build your organization's presence

To help brand your organization and best communicate the organization’s purpose to others, you'll want to fully populate and brand the Organization home page.

In the Administrator panel on the Settings tab / Public profile section you can customize:

• The organization's full name

• An organization description which will appear at the top of your about section, and in any institution searches

• A custom brand color

• The organization's logo

• The data portal's cover photo

• A header image that will appear in the top left of users' windows when on your data portal

• Header links to other resources, such as organization-specific documentation, events, and other resources.

• Rich text and images that will display in the main content section of the data portal. A place to provide high level information about your organization and how researchers should user your data portal.

3. Create datasets

You'll need to create datasets to upload and distribute data to your researchers. You can create datasets from the Datasets tab of your Administrator panel.

Here are some pieces to consider before getting started:

Dataset structure

It's worth thinking through your broader data management strategy. Assess the data that you currently have — what format are the files in? Where are they located? How are they organized?

Consider what datasets you'll want to create. Datasets on Redivis are made up of one or more semantically related tables, and each table can contain data from one or more uploaded files. Moreover, all permissions happen at the dataset level — if you grant a user access to a dataset, they will have access to all of its tables.

Citations and DOIs

If you’d like to issue DOIs for your organization’s datasets, you can configure your organization to issue DOIs in the Advanced section Settings tab. DOIs will make it easier for researchers to cite your data when they publish, and allow you to track publications using your data as well as their broader impact.

Learn more in the Create & manage datasets guide.

4. Build access systems

Before you can configure access to a dataset, you'll want to consider and build out the relevant access systems for your organization as a whole.

Direct access & Member access

If you have a smaller organization where you know everyone working with your data, you might want to consider directly granting access to datasets to specific members, or granting access to everyone who is a member.

This can be set up on any dataset individually or you can create a Permission group on the Permission group tab to save the entire configuration of access levels and assign it to multiple datasets.

Requirement-based access

In many cases, however, you’ll want to develop more process-driven access control systems and gather additional information from your members To do so, you can set up requirements that must be completed by members or their studies, and approved by an administrator, before gaining access.

You can build requirements on the Requirements tab of the administrator panel, and save groups of them together as Permission groups on the Permission groups tab which you can assign to multiple datasets.

Export restrictions

Finally, you may want to control how your data is exported from Redivis, if at all. While members with data access will always be able to work with data in Redivis workflow, you can limit their ability to export data. For example, you may choose to allow exports only to certain environments, or only upon administrator approval, or not at all.

Learn more in the Configure access systems guide.

5. Manage members and studies

As members and studies apply to access data, you may need to take action to approve (or reject) their access requests. You can find all of your pending access requests wherever you see an alert on the administrator panel.

Members

As users join or organization and apply to work with data, they will appear in the Members tab of your administrator panel. You can also invite anyone to become a member by adding them to the members list.

Studies

Studies allow for researchers to organize their work around a conceptual study, or research project, and for you to grant access in the context of that study. Any studies that are working with your organization's data will appear in the Studies tab of the administrator panel automatically, and you can approve any requirements for studies here.

Learn more in the Grant access to data guide.

6. Gain insight into usage

Redivis provides tools to help you keep an eye on your organization and how its data is being utilized.

You can filter, sort, and change the columns for any list in your administrator panel, and download the corresponding report for a snapshot of current information. You can also dig into the logs and generate analytical reports for more information.

Next steps

Expand your reach

If you are part of a larger institution, you can contact us about getting an institution-wide Redivis page to help users discover new organizations and datasets.

Overview

One of the strengths of Redivis is that researchers can apply for, and gain access to, data on the same page as where they discover the dataset. Moreover, all access applications are tracked, both for administrator's reference and to provide transparency and timely notifications to researchers.

Responding to access requests in a timely manner creates a great environment for researchers and takes full advantage of these automatic systems.

1. Configure notifications

You can configure multiple email addresses to receive an email when a request for access is made to your organization, as well as the frequency you receive them. Especially if you are a smaller organization that isn't getting access requests every day, we very strongly recommend that you have at least one administrator receiving emails so you know when to respond to a request.

Learn more in the Settings reference section.

2. Approve membership requests

Users must be approved as members of your organization before any other access requests can be approved.

When a membership request is pending you will see an Alert (!) on the Members tab of your workspace. Click on any member with an alert to see their pending submission awaiting approval.

Pending members will have an Approve / Reject button as soon as you click on their page, alongside the authentication information they submitted when requesting to be a member. It is important to verify that this person has a genuine email that seems consistent with who they say they are.

3. Approve requirement submissions

As users ask for approval you will receive an alert (!) in the administrator panel where your attention is needed.

Member requirements and export restriction exceptions will appear on the Member tab.

Study requirements will appear on the Studies tab.

Direct access requests will appear on the Datasets tab.

When all of the pending requests are resolved, the alert will disappear.

For each request you will have the option to approve or reject the submission. You can always come in later to revoke the approval if needed. You might also see that the member has submitted updates to their existing approval. In that case the approval is still valid while updates are pending, unless it is otherwise revoked or expired.

Learn more in the Approving access reference section.

4. Leave comments for clarity

Both administrators and members have the option to leave a comment on any requirement submission. The act of leaving a comment will trigger an alert for administrators, or a notification for members.

Especially when you are rejecting or revoking a requirement submission, we recommend leaving a comment to let the member know what they did wrong, or what they still need to do to gain access.

5. Audit access

It's often a good idea to double check which researchers have access to your datasets.

In order to audit a particular member's access to your organization's datasets, double click on that member, and navigate to the Access overview tab. This will list all of your organization's datasets, and the member's corresponding access to each.

In order to audit all members that have access to a particular dataset, click the filter icon to the right of the member search bar. This will allow you to filter all members by their access to, and usage of, your organization's datasets.

Next steps

Expand your reach

If you are part of a larger institution, you can contact us about getting an institution-wide Redivis page to help users discover new organizations and datasets.

Overview

All activity surrounding your organization and it's data usage is logged and accessible to organization administrators. This can be helpful for year-end reports or any potential data security concerns!

You can easily organize, visualize, and download any of your organization's information by generating a report in your administrator panel.

1. Create a report

Navigate to the Reports tab of your administrator panel and click the + New report button.

In the future you can return to this page to see your existing reports and create duplicates to further edit.

2. Choose the entity of the report

Start by choosing the entity, or topic, of the report. Whatever you choose here will define what each entry in your report represents.

You can choose a Resource such as Members, Workflows, Studies, or Datasets which will allow you to get a sense of the named entities your organization owns or has interacted with.

Or you can choose a Usage event such as Queries, Notebooks, Exports (or all of these combined) which will allow you to see how your organization's data has been used.

3. Define constraints

From here, define the other criteria of your report:

Time frame

If you choose a usage event as your entity you will need to set a time frame to define which events will be included. (Resources don't have a time associated with them since they are not events.)

Aggregation

If you chose a usage event you can optionally choose to aggregate events based on a resource or period of time. Perhaps you want to see all queries grouped by member, or want to see all notebooks per day. You can also create custom aggregation groups by using member labels and dataset labels with colons. E.g. labeling members with school:medicine and school:business would allow you to aggregate a report by school.

Filter

You can additionally choose a filter that will restrict your report to only events or resources that meet your chosen criteria. For example, if you want to see all usage events that referenced a particular dataset.

Learn more in the Reports reference section.

4. Define the report's fields

Finally, you'll need to select what fields or topics are present in the report. Every field selected here will become a column in your resulting report table. Different fields are available based on what entity you have selected.

Some fields are selected by default that might be helpful but you can make your own selections. For example, maybe you have a report of data usage and you want to include the average number of tables referenced in each notebook session, or the max amount of compute used in queries per day.

5. View the report results

Click the Create button to generate this report! This will generate a table with entries that match all the criteria you've defined above. You can browse this table to understand the data and make sure it looks how you'd expect.

If your report contains any numeric data you can click the Visualize tab to see a visual representation of 1-2 of your numeric fields. The type of graph will be chosen by default based on the shape of your data. You can hover on this graphic to learn more about each entry.

5. Iterate and export

You can update this report by clicking the Edit button. Changing any criteria will refresh the table and visual with new contents based on your new criteria.

This report is now saved in your organization's reports and you can access it in the future.

Any time you edit the report or come back to it after more than one hour has elapsed, the table will be automatically regenerated. Based on the time frame you've selected this might drastically change the contents.

If you'd like to work with the data in a different environment you can download the report table as a csv by clicking to Download button.

Next steps

Expand your reach

If you are part of a larger institution, you can contact us about getting an institution-wide Redivis page to help users discover new organizations and datasets.

Overview

One of the main tasks of organization administrators is to manage who has access to datasets. Redivis has tools to support any level of complexity in data systems for your restricted data and makes it easy for organization administrators to define, manage, and audit access rules across these various use cases.

1. Consider your organization's access needs

Do you mostly have public datasets? Will you have a small group of researchers with access to pretty much every dataset? Will you need to gather information from researchers before they use your datasets? Do you have high risk data that can't ever leave the system? Or some combination of the above?

Redivis supports access systems for all of these configurations – for organizations of varying sizes – but you'll want to use different tools to achieve these very different goals.

For example, if you have few datasets, you might find that permission groups for bulk managing access rules aren't particularly helpful. Or if you have a small group of researchers who you all know closely, you might not need to set up process-driven access rules through requirements, and can instead grant access to certain researchers on an individual basis.

We'll walk through all the tools available to organizations for managing access, but you'll likely want to pick and choose from what's available to meet your organization's specific needs.

2. Understand access levels

All interactions with data on Redivis require the user to have the appropriate access level to the data for a given action. Ideally your data would be as permissive as possible to allow for the greatest exploration from researchers before they need to start applying for access.

Dataset access has five levels:

1. Overview: the ability to see a dataset and its documentation.

2. Metadata: the ability to view variable names and univariate summary statistics, but not to retrieve any identifiable information or multivariate relationships from the data.

3. Sample: the ability to view and query a dataset's 1% sample. This will only exist for datasets that have a sample configured.

4. Data: the ability to view and query a dataset's tables, and work with them in workflows.

5. Edit: the ability to edit the dataset and release new versions.

Access levels are cumulative. For example, in order to gain data access you will need to have gained metadata access as well.

We strongly recommend making your dataset's metadata as open as possible. This will reveal variable names and aggregate summary statistics, but will not allow researchers to view, query, or export the raw data in any way.

Being able to see metadata greatly improves researchers' discovery experience, and allows them to better assess a dataset's utility upfront, and even reduce your administrative workload. If researchers can understand a dataset before applying for access, they'll be submitting fewer access applications to datasets that are ultimately a dead end.

Learn more in the Access levels reference section.

3. Understand available access tools

Membership

Anyone wanting to apply for access to restricted datasets in your organization must first be a member. You can configure whether memberships are restricted to certain identity providers (such as your institutional login), and whether they are approved automatically or require administrator review. You also have the option to configure access to datasets to "All members."

Direct access

Permission granted directly to a researcher to instantly gain access to a dataset at a specific level. Researchers can also request access to datasets with this configuration.

Example usage: a dataset that will only be shared with a small number of people who are already known to administrators

Member requirements

A form for members to fill out. This can be set to require approval from an administrator or be automatically approved. It can also have an expiration date. These are global to your organization and when assigned to multiple datasets a user will only fill it out one time.

Example usage: a demographic form gathering researcher personal information, or a data use agreement signed PDF

Study requirements

Similar to requirements, but instead of each user needing to fill them out individually, only one requirement needs to be completed for the entire study, which can include multiple researchers. A single researcher may also have multiple studies. Each study working with the dataset will need to fill out its own study requirement, and any queries or exports of the data will be tied to that study.

Example usage: a funding proposal for a research project

Data export restrictions

A rule defining that a dataset can only be exported to a specific export environment, as configured on the Administrator panel Settings tab.

Example usage: limiting exports to a specific server environment

Learn more in the Configuring access reference section.

4. Create requirements

If you want to work with requirements, you'll want to get started making them and planning out how they will work across datasets.

Perhaps you want one requirement for all members to fill out about their field of research, which is necessary to gain access to any of your datasets, but another 4 requirements with different data use agreements that will apply only to their specific datasets.

To get started, go to the Requirements tab of the administrator panel and click the New requirement button. You will need to start by selecting if this will be a Member requirement or a Study requirement.

You can use the form builder to collect different information, including standard form responses, file uploads, and e-signatures.

Learn more in the Requirements reference section.

5. Create permission groups

A permission group is an access configuration that can be assigned to multiple datasets and managed centrally.

You don't need to use permission groups, and it might not make sense to do so if each of your datasets has a different access configuration and you aren't using requirements. But if you have any overlap between datasets and want to enforce consistency, or want to use requirements you'll want to make one.

To get started, go to the Permission groups tab of the administrator panel and click the New permission group button.

This interface requires you to set an access paradigm for each access level of the dataset.

Perhaps you will set the overview access level to be public, the metadata access level to be available to all members, and data access level to be direct access (meaning you will have to directly grant access to users, or respond to their requests for access).

Or perhaps you will set the overview access level to public, and assign multiple requirements to the metadata and data access levels (meaning that anyone who is approved for all of the requirements will automatically gain that access level).

For any case you can assign data export restrictions here and choose whether you want to manage access to the dataset's sample (if it exists) differently than the full data.

Learn more in the Permission groups reference section.

6. Assign access permissions to datasets

Finally you'll need to apply these access permissions to actual data!

Open any of your datasets and click the Configure access button on the top of the page. This configuration setup will look very similar to configuring the permission group.

You can either create a custom configuration here, or you can assign this dataset to one of the Permission groups by clicking the dropdown menu in the top right corner of this modal.

This is also where you will manage any direct access requests for the dataset.

7. Verify your setup

As an administrator of this organization, you will have access to all datasets no matter what your access configuration is or what requirements you have filled out.

We recommend checking that your access system works as you expect by either looking at a dataset while you are logged out or in your browser's incognito mode, or by making a second Redivis account using a different email address that is not linked to your administrator account.

Next steps

Grant access to data

You have some shiny access systems, but they won't work if you don't approve user requests for access.

Learn more in the Grant access to data guide.

Expand your reach

If you are part of a larger institution, you can contact us about getting an institution-wide Redivis page to help users discover new organizations and datasets.

Overview

Redivis notebooks a performant, flexible environment for analysis that allow you to analyze and visualize data in workflows in Python, R, Stata, or SAS. With the notebook computation happening on Redivis, you don't need to configure an environment on a local machine or server, or export data from Redivis. This makes for easy iteration and collaboration, not to mention ensuring better security and data throughput.

1. Create a notebook

Once you have a table that you're ready to analyze, you can create a notebook by clicking the + Notebook button at any time. You'll need to name it and choose a kernel (Python, R, Stata, or SAS).

Notebooks can only reference tables within their workflow, so we recommend keeping all related work together in the same workflow.

2. Define dependencies

3. Compute resources

The default notebook configuration is free, and provides access to 2 CPUs and 32GB working memory, alongside a 60GB (SSD) disk and gigabit network. The computational powerful of these default notebooks are comparable to most personal computers, and will be more than enough for many analyses.

Clicking Edit compute configuration button in the start modal or the toolbar will allow you to choose from different preconfigured machine types. The notebook will then default to this compute configuration each time it starts up.

4. Start the notebook

Notebook nodes need to be started in order to edit or execute cells. When first clicking on a notebook node, you will see a read-only view of its contents (including cell outputs). Click the Start notebook button in the toolbar to connect this notebook to compute resources.

When you create a notebook for the first time it will start automatically.

5. Load data

To do meaningful work in your notebook, you'll want to bring in the tabular and/or unstructured data that exists in your workflow into your notebook.

Referencing tables

Notebooks come pre-populated with templated code that pulls in data from the notebook's source table. You will need to run this cell to pull the data into the notebook, and you can see that it worked because this code will print a preview of the loaded data.

Referencing files

Any files with unstructured data stored in Redivis tables can be referenced by their globally unique file_id. You can also reference these file_id's in any derivative tables, allowing you to query and download specific subsets of files.

When working with large files, you'll want to consider saving the files to disk and/or working with the streaming interfaces to reduce memory overhead and improve performance.

6. Analyze data

At this point, you have all the tools you need to work with your data in your chosen language. The Python, R, Stata, and SAS ecosystems contain myriad tools and libraries for performing sophisticated data analysis and visualization.

7. Create an output table

Notebooks can produce an output table, which you can sanity check and further analyze in your workflow by including in other notebooks or exporting to other systems.

Next steps

Share and collaborate

All Redivis notebooks support real-time collaboration, allowing multiple editors to edit and run cells in a running notebook. When another editor is active in a notebook, you will see a colored cursor associated with them (much like a Google Doc).

Cite datasets in your publications

Overview

Your Redivis account establishes your identity on Redivis, and provides a home for various forms of data-driven investigation and collaboration.

Because researchers regularly move between institutions, and often want to be able to reference and access historic work, it is strongly encouraged that you:

• Only have a single Redivis account, much as you would only have a single Google Scholar account.

Communication and privacy

When you apply for access to an organization's datasets, they will be able to see some information pertinent to your membership in their organization:

• Full name

• Contact email

• Email / institutional identifier

• Affiliation, if provided by your institution's login provider

For more extensive full-workflow video walkthroughs, see our Events and press page:

Or read through our Example workflows:

Overview

By creating an account on Redivis, you can gain access to hundreds of restricted datasets and powerful querying tools. There is no cost to creating an account, and you may permanently delete your account at any time.

Your account tracks all of your workflows, queries, collaborations, and data approvals with different organizations. After creating an account, you will be able to apply for membership with various organizations on Redivis and request access to their restricted data.

E.g.: I want the date of the first and last hospital admission for every patient in my table

Let's take an example table which contains all hospital visits in one facility over many years. Each record contains information about what happened in the visit, including a patient id to identify who was present. An individual patient might be represented in one or many records in this table since each record represents a separate hospital visit.

Let's say that for our research we want to find the first and last encounter — or more formally, the min and max encounter date for each patient.

In this example we will use a partition to create a new variable since we want the rest of the data to remain unchanged.

Variable concepts

Conceptually we will define our aggregation variable as the date of the encounter since that is where we want to look for the min and max values.

We will define our partition as the patient id since we want to find the min or max for each patient. This will limit the min or max value scan to all values of our aggregation variable across each unique value of our partitioned variable (patient id).

Defining the new variable

For this particular case, we want to find the MIN and MAX of the encounter_date of each patient; in other words, partitioned on patient_id . This will create two new variables, min_encounter_date and max_encounter_date for each patient.

When we run this query, note that no records will be dropped; rather, these new variables will be have consistent values for any given patient_id .

Now, let's say we only want to keep records which contain each patient's first and last encounter. We can do this easily be creating a new row filter:

We can also generalize this approach to find the Nth encounter for each patient — take a look at the RANK, DENSE_RANK, and ROW_NUMBER methods.

And if we want to be more specific on our partition (e.g., for the first / last encounter in a given calendar year), we can always apply additional variables (year ) when defining our partition.

Redivis has some built-in methods for helping you find and manage all of your organization's members, workflows, datasets, and access systems that you can use to ease your workflows.

In this example we will imagine that we need to get in touch with everyone working with a specific dataset.

1. Locate the dataset in the administrator panel

We can go to the Datasets tab of the panel and search or filter to locate our dataset.

2. Find relevant workflows

We can right click on this dataset and hover on View related to see all of the options relating to this dataset. In this case we want to see all workflows that have used this dataset.

Clicking this option navigates us to the Workflows tab, with a filter active for this dataset. This list is showing all workflows that contain this dataset.

3. See related members

Click the Select all checkbox to select all of these datasets. With these selected we can right click on any of them (or click the Actions button that appears in the top right) to View related activity.

In this menu we should select Members in workflows. This will navigate us to the Members tab filtered by all members who are owners or collaborators on the workflows we had selected.

4. Show emails

Now that we have narrowed it down to the members we are interested in, we can right click on the header (or the options menu in the top right of the list) to edit the columns shown. In this menu we can add the Contact email column.

The quickest way to get this information out of Redivis is to go back to that same menu and click Download CSV to begin downloading a CSV of the list we are looking at exactly as you've configured it (all columns, filters, and sorting will be preserved).

Next steps

We can use these similar steps to view:

• Queries run on a specific set of datasets

• Exports made from a workflow

• Requirements a member has filled out

And many more!

Overview

Notebooks on Redivis offer a performant and highly flexible environment for doing data analysis. This includes the ability to run state-of-the-art machine learning (ML) models, including the ability to train new models, fine-tune various existing models, and use these models to perform inference and generate novel outputs.

This guide is focused on a number of common use cases when running ML workloads on Redivis. Here we generally focus on using the Hugging Face + PyTorch ecosystem in Python, though these examples are broadly applicable to other ML libraries and languages.

For a detailed example of using Redivis to fine tune a large-language model, see the complementary example:

1. Create a notebook with appropriate computation capacity

Training ML models and running inference can require a substantial amount of compute capacity, depending on various factors such as your model and dataset size, usage parameters, and performance goals.

The default, free notebook on Redivis offers 2CPUs and 32GB of RAM. While this may work for initial exploration, running practical machine learning workflows typically requires the availability of a GPU. When creating your notebook, you can choose a custom compute configuration to match your needs.

Redivis offers a number of custom compute configurations, mapping to the various machine types available to Google Cloud. We recommend starting with a more modest GPU for initial exploration, and then upgrading as needed when computational or performance bottlenecks are reached. For this example, we'll use the NVIDIA L4 GPU, which provides reasonable performance at a reasonable cost.

2. Define dependencies

The Redivis python notebook is based off the jupyter-pytorch notebook image, with PyTorch, CUDA bindings, and various common data science libraries pre-installed. However, if you require additional dependencies for your work, you can specify them under the "dependencies" section of your notebook.

2a. [Optional]: Pre-load external models when internet is disabled

If your notebook references export-restricted data, for security reasons, internet will be disabled while the notebook is running. This can present a challenge for some common approaches to ML in python, such as downloading a model or dataset from Hugging Face. For example, we might reference a model as follows:

This code typically downloads the model from Hugging Face and caches it to our local disk. However, if internet is disabled, this command will hang and ultimately fail. Instead, we need to download the model during notebook startup, before the internet is disabled, as part of the post_install.sh script under the notebook's dependencies:

This will download the model weights and other files to the default Hugging Face cache directory, ~/.cache/huggingface/hub.

Now, within our notebook, we can load the cached model. Make sure to set local_files_only=True, so that Hugging Face doesn't try to connect to the internet to check for a newer version of the model.

3. Load a model

3a. Load a model from Redivis

Machine learning models can be stored directly within a Redivis dataset as unstructured files. For example, this dataset contains the various files that make up the bert-base-cased model on Hugging Face. We can then download the model to our notebook's local filesystem:

And then reference this as a local model. E.g.:

When using models stored on Redivis, we don't have to worry about whether our notebook has internet access, nor do we need to rely on the future availability of that particular model on Hugging Face.

3b. Load a model from an external source

If your notebook has internet access, you can also use any other models that may be available on the internet. For example, we can load the same bert-base-cased model directly from Hugging Face:

Note that if you are working in a notebook with disabled internet, this approach won't work, and you'll need to use the methods mentioned in either 2a or 3a above.

4. Load data

As a final step, you'll likely want to load data into your notebook to either fine-tune the model, perform inference, or otherwise experiment. There are thousands of datasets on Redivis, and you can upload your own data as well. You can learn more about loading data into a notebook in the python notebooks documentation. As a quick example:

Of course, assuming your notebook has access to the external internet, you can also call various APIs to load external data sources.

Next steps

At this point, you have all the tools at your disposal to perform cutting edge ML research. But of course, what you do next is totally up to you. We do recommend further familiarizing yourself with the examples and detailed documentation to take full advantage of the capabilities of Redivis notebooks: