Overview

Retaining data is vital to establishing long-lasting data linkages and making research available. Redivis' data retention policy is aligned with the goal of keeping all data available and accessible whenever possible.

Owner-initiated deletion

Data owners always have the ability to remove their data from Redivis or make it inaccessible if they choose to do so.

Redivis also has policies in place in case of an accidental deletion:

• All data is fully backed up, with point-in-time recovery, over a 7 day rolling window. Deletions and modifications to data is permanent after 7 days.

• All metadata is backed up with point-in-time recovery over a 7 day rolling window, with daily backups of metadata extending to 1 year.

Redivis-initiated deletion

Data storage that exceeds the free tier must be paid for by the data owner or other sponsor. Upon payment failure, every effort will be made to contact the data owner and re-establish payment.

If unsuccessful, data may be destroyed after 30 days of non-payment, though all metadata, and permanent landing pages will be persisted.

Overview

The ability to view, reproduce, and build upon other works is a core tenet of the scientific process that Redivis is built to uphold and facilitate. Redivis tools for data storage and analysis comply with federal mandates for open access and automatically generate artifacts documenting all steps in the data process, ensuring reproducibility.

Background

1. Make publications and their supporting data resulting from federally funded research publicly accessible without an embargo on their free and public release.

2. Enact transparent procedures that ensure scientific and research integrity is maintained in public access policies.

3. Ensure equitable delivery of federally funded research results and data.

1. Public accessibility

Redivis is built to make data sharing as easy and safe as possible, defaulting to fully accessible data, metadata, and documentation. However, some data must be restricted for privacy and to comply with data licensing agreements. In these cases Redivis follows an "as much as securely possible" sharing philosophy for sharing.

2. Transparent procedures

Redivis systems are built to automatically capture and document any work done on datasets and analyses in standardized formats, while giving researchers and administrators the ability to supplement or override these when necessary.

3. Equitable delivery

Redivis is free and accessible to any researcher, reviewer, or casual data browser. By taking multiple complex data storage and analysis systems and centralizing them with clear UI, it is clearer to understand full workflows without specific technical knowledge.

• Redivis uses transparent, open-source formats, allowing data, analyses, and workflows to be exported and used within other systems.

• No-code interfaces allow researchers at any skill level to build and understand research steps.

Overview

Findable

Data is findable when (1) data and metadata are assigned a globally unique and persistent identifier, (2) data are described with rich metadata, (3) metadata clearly and explicitly include the identifier of the data it describes, and data and (4) metadata and data are registered or indexed in a searchable resource.

At Redivis:

Accessible

Data is accessible when: (1) data and metadata are retrievable by their identifier using a standardized communications protocol, (2) the protocol is open, free, and universally implementable, (3) the protocol allows for an authentication and authorization procedure, where necessary, (4) metadata are accessible, even when the data are no longer available.

At Redivis:

• Data can be explored through any web browser.

• Researcher accounts to apply for data access or do analyses are always free.

Interoperable

Data is interoperable when: (1) data and metadata use a formal, accessible, shared, and broadly applicable language for knowledge representation, (2) data and metadata use vocabularies that follow FAIR principles, (3) data and metadata include qualified references to other data and metadata.

At Redivis:

Reusable

Data is reusable when: (1) data and metadata are richly described with a plurality of accurate and relevant attributes, (2) data and metadata are released with a clear and accessible data usage license, (3) data and metadata are associated with detailed provenance, and (4) data and metadata meet domain-relevant community standards

At Redivis:

• Many dataset, table, and variable metadata fields are automatically populated based on user action, with the ability to be manually adjusted.

• All datasets have a Usage tab with information on how it has been viewed and used across Redivis

• Redivis cloud-based analysis tools encourage data users to do analysis along the data rather than downloading it and breaking linkages.

What is Redivis?

Who uses Redivis?

Organization administrators upload, catalogue, and grant access to datasets.

Researchers discover, query., and analyze data.

Redivis makes it easy for everyone to:

Using this documentation

This documentation is broken up into Guides, Examples, and Reference sections.

We recommend that new users begin by following the relevant guides and examples, while the reference section can serve as a comprehensive overview of everything that Redivis can do.

Overview

Most data on Redivis is uploaded through an Organization, and that is the best place to find Datasets to work with. Some datasets will be public, while others will require certain steps before you can get full access to the data.

1. Find datasets

From an Organization's home page, click on the Datasets tab to browse all datasets and filter by their metadata.

You can use filters on the left bar to narrow down your results. All searches perform a full-text search across a dataset, its documentation, tables, variables, and rich metadata content.

2. Get to know a dataset

Click on any dataset title to go to that Dataset page.

You can view the metadata (including an abstract, documentation, citation information, and a version history) and dig into the data directly on the Tables and/or Files tab.

Each table contains a data browser to view cells, generate summary statistics, and display metadata for this table and each variable. Each file can be previewed and downloaded. Both tables and files can be added to workflows for analysis.

Many datasets on Redivis are public, while others have requirements for certain levels of access enforced by the data owner.

On the right side of the dataset, you can see your current access level. If you have Metadata access you can see the variable information and summary statistics, but you'll need to gain Data access in order to view the data contents or work with this data.

3. Apply for access

On any restricted dataset you can click the Apply for access button in the top right of the page to see a list of steps required for you to gain different access levels to this dataset.

The first step is to become a member of this organization. When you go to apply you will be prompted to become a member of this organization using the credentials on your account.

Once you're a member of this organization, you might need to fill out some requirements that administrators have set up, or request access directly from them.

Requirements are forms that you will need to complete on this page and then submit. Once these are approved by your organization's administrators you will gain access to the dataset.

Next steps

Start working with your data

Once you have data you're interested in, bring it into a workflow to transform and analyze it leveraging lightning fast tools from your browser.

Upload your own datasets

Augment your data analysis in Redivis by uploading your own datasets, with the option to share with your collaborators (or even the broader research community).

Overview

Redivis is a platform that allows researchers to seamlessly discover, access, and analyze data. This brief guide will walk you through the basics to get up and running and provide a launching point for exploring other resources in this documentation.

1. Create your account

Many datasets on Redivis are public, and you can browse them without creating an account. However, you'll need an account in order to analyze data, as well as to apply for access to restricted datasets.

Click the Create account button in the top right of any page to sign up. You can use your academic institution's login credentials or any Google account.

Once you create an account, you'll be navigated to your Workspace, which is your private area for creating workflows to work with data, uploading your own datasets, and managing your Redivis account.

2. Join your organization

The main entity on Redivis is a dataset. Most datasets on Redivis are uploaded by organizations.

You might already know the organization that you’ll be working with, or you can browse organizations by clicking the Find organizations button on the Organizations tab your Workspace. You might know your organization's URL already, or see it listed in the Recent section of the Dashboard if you were already viewing that page.

Once you join an organization, you'll be able to apply for any of their restricted datasets, and also see a link on the left bar of your workspace to quickly get back to the organization's home page.

3. Find datasets

The best place to find datasets on Redivis is to search on an organization's home page. Here you can click on the Datasets tab to browse all datasets and filter by their metadata. All searches perform a full-text search across a dataset, its documentation, tables, variables, and rich metadata content.

Click on any dataset title to go to that Dataset page where you can view the data and metadata it contains. The data for this dataset will be available on the Tables tab, which you can explore further.

You will also probably come across restricted datasets. For these you will need to click the Apply for access button on the top right of the Dataset page and complete the requirements to gain approval from the dataset's administrators.

4. Analyze data

Once you've found a dataset that you want to work with, you can add it to a Workflow. Workflows are the fundamental analysis interface on Redivis, where you can query, merge, reshape, and analyze any dataset that you have access to — all from within your web browser.

Add a dataset to a workflow by clicking the Add to workflow button on the top right of the dataset page.

In a workflow we can create a Transform by selecting any table and clicking the +Transform button, which allows us to combine and reshape our data into a final output table that best serves our analysis. These transforms use a powerful SQL engine under the hood, allowing us to query incredibly large tables - even billions of records - in seconds.

After creating output tables for analysis, we can create a computational Notebook in R, Python, Stata, or SAS to further analyze our data and develop our final figures. Select any table and click the +Notebook button to get started. The notebook will initialize and pull in the table you've selected (or a sample of the table if it is a large table).

We can also export and query this table from external environments, allowing us to use whatever analytical tools best suit our research question by clicking the Export table button on the right side of any table. Here we can download the table in a number of common formats, interface with the it programmatically via the API, or export the table to supported integrations.

Next steps

Upload your own datasets

Augment your data analysis in Redivis by uploading your own datasets, with the option to share with your collaborators (or even the broader research community).

Administer your organization

Organizations allow for research groups and centers to securely distribute data to their research community. Organization administrators can create datasets, manage access, review logs, and create customized reports of their data utilization.

Overview

Studies are a way for one or more collaborators to work with data on the same conceptual topic. You can add multiple collaborators, datasets, and workflows to a study to organize your workflow. In addition, some restricted data requires you to submit access applications as a study via study requirements.

1. Create a new study

On the studies tab of your workspace you can create a new study and see all studies you are a part of. This study might represent a topic or group you are working with, maybe to investigate a similar cluster of research questions. You can give the study a name and description to reflect that purpose.

2. Add collaborators

If you are working with anyone else on this study you can add them as a collaborator. Anyone who is a collaborator on this study will be able to view and edit the study same as you can. Once added, this study will appear in their workspace.

PI

One person on the study can be designated as the PI. This designation does not have meaning within Redivis interfaces and is here to inform others in the group or administrators you are applying for access from of the group structure.

3. Add datasets

You can add any dataset to this study that you plan on working with, and remove it later if your topic shifts. This space is intended for you to gather resources that you are using in one place to make it easier to create workflows or manage data access. Datasets can be added to any number of studies.

4. Add workflows

You can create a new workflow in this study or move an existing one in. All workflows can be in at most one study, and you can see what study a workflow is in on the workflow's overview page.

Granting workflow access to study members

One of the options for sharing this workflow is to grant view or edit access to all members of the study it's in. This can be an easy way to give everyone in a group access to a workflow you are working on.

As with all workflows, even though someone might have access to the workflow they will need to have independent access to all the datasets it contains in order to view or query them.

5. Apply for data access via a study

If you're working with restricted data you might come across a dataset that has a study requirement as part of its access requirements. You can identify this by the study icon next to the requirement name.

Study requirements are filled out and submitted on behalf of an entire study, rather than one person. An approved study requirement will be valid for all collaborators on the study, unlike member requirements where each individual needs to complete the requirement on their own account.

To submit a study requirement you can navigate to a restricted dataset and open the access modal. You will need to select your study from the dropdown menu, and a submit button will appear. One member of your study will fill this out and submit it on behalf of your group. Once approved, all study collaborators will see an approved requirement in their access modal.

Data usage

When restricted data has a study requirement, the data administrator can approve your application submission for use of the data within that study. In order to query data in a workflow, that workflow will need to be in the approved study. If it is not, you will see a badge noting Limited access and you will not be allowed to run transforms or query this data in a notebook.

Next steps

Start analyzing your data

Once you've gained access and set up a study it's time to add your datasets to a workflow to transform and analyze them leveraging lightning fast tools from your browser.

Overview

Many datasets on Redivis are public, while others have requirements for certain levels of access enforced by the data owner. If the dataset is public, you can skip over this section, otherwise you'll need to gain access before you can fully utilize the data.

1. View access rules

On the right side of all restricted dataset pages is a box which displays your current access level. If this doesn't say Data access, then your next step will be to apply for access.

Click the Apply for access button on the top of the page to open the access modal. This modal allows you to manage your access to this particular dataset throughout the platform.

Access levels

Dataset access has five levels:

1. Overview: the ability to see a dataset and its documentation.

2. Metadata: the ability to view variable names and summary statistics.

3. Sample: the ability to view and query a dataset's 1% sample. This will only exist for datasets that have a sample configured.

5. Edit: the ability to edit the dataset and release new versions.

Access levels are cumulative. For example, in order to gain data access you will need to have gained metadata access as well.

Usage rules

Usage rules are restrictions placed on the way you can use this data once you've gained access. For example, data exports may be restricted to certain export environments, and/or require administrator approval.

While no action is required when you first apply for access a dataset, it's helpful to understand these restrictions to better understand what limitations, if any, they might pose to your research.

2. Apply for access

The access modal for the dataset will include all steps designated by this dataset's administrators to gain access to work with data.

Membership

To apply for access to datasets hosted by a Redivis organization, you will first need to Join the organization.

To join an organization, you will need to provide information about your identity via an email authentication, usually the email associated with your Redivis account.

After submitting your membership application, it will either be automatically approved or will be marked as pending while an administrator reviews your submission. You'll receive a notification if your submission is approved or rejected.

Requirements

Some datasets have requirements as part of their access applications. Requirements are global to an organization and usually contain some type of form. Once all requirements for a particular level of access are approved, you will gain access to the relevant dataset(s).

In the access modal, click the Apply button on any requirement to fill out and submit the form.

When you submit a requirement, an administrator of the organization will be alerted and can review your submission for approval (in some cases, the requirement may be configured for auto-approval upon submission). You will receive a notification if your submission is rejected or approved, or in the future when it's about the expire. You can also leave a comment for an organization's administrators here, and they can reply to you.

Direct access

Sometimes you don't need to fill out any requirements and need to simply Request access at a specific level. You can do so in this modal, and will receive a notification when you are granted access by the dataset owner.

3. Wait for data owner approval

For some membership and requirement submissions (and all direct access requests), you will need to wait for the data owner to respond to your submission. If the dataset is owned by a user, they will need to approve your request; if it is owned by an organization, any administrator at that organization can approve it.

4. Re-authenticate (if necessary)

If you are accessing restricted data hosted by an organization, you may have to re-authenticate after a period of inactivity anytime you perform an action governed by the "Data" access level.

After a period of inactivity (set between 15 minutes and 3 days by the organization administrator), a popup will ask you to re-authenticate anytime you view data, query tables in a transform, or export data from Redivis.

Next steps

Start working with your data

Add this dataset to a workflow to transform and analyze it leveraging lightning fast tools from your browser.

Overview

Transforming tables in a workflow is a crucial step when working with data on Redivis. Conceptually, transforms execute a query on a source table and results are materialized in a new output table. They are optimized to run on billions of records in seconds, and create a transparent, reproducible record of data transformation steps you've taken in your workflow.

In most cases you'll want to use transforms to create an output table containing all the information you're interested in before analyzing that table in a notebook or exporting it for further use.

1. Create a transform

Once you've created a workflow and added a dataset to it that you'd like to work with, get started by creating a Transform. You can do this by clicking on any dataset or table.

To build this transform, you will add steps which each take an action to shape the data in the output table. You can choose from many named steps which include a point and click interface that compile to SQL code, or you can add a SQL query step to write code directly. You can always view the code that your step is generating and switch to code if you'd like to edit it.

While SQL might not be a familiar language, it is optimized for data cleaning procedures and allows transforms to execute extremely quickly. It also allows you to write your data transformations in a declarative, reproducible manner.

2. Join additional tables

The first thing you'll want to do is decide if you have all the information you need in this table or if you'd like to join in an additional table or tables. You can reference any other table from either 1) this dataset, 2) another dataset, or 3) an output table from a different transform in this workflow. Any table you want to reference needs to be in this workflow, so you can click the Add dataset button to add a new dataset to the workflow if it's not here already.

To join a table, add a Join step and select the table you'd like to join. You'll then need to select what type of join it will be, and build your join condition. In most cases your join condition will be linking two variables of the same name and type together (e.g. join all records where id = id).

3. Create new variables

You might want to generate new columns in this table by creating a new variable. You can do so by adding a Create variables step. Start by giving this variable a name and then choosing what method you want to use to create it.

Some methods are restricted to the type of variable you are working with. For example there are options to add or subtract years from a date variable, or concatenate string variables.

One of the most common new variable methods is Case (if/else). This allows you to set up a statement that looks at the content of each record and evaluates it based on conditions you've set up to generate the value. For example you can say that in the new variable you're creating, if the amount in column A is greater than 1000, the value of this new variable will be set to "high" and if not, then it will be set to "low."

You can create any number of new variables, and they will execute sequentially, allowing you to reference variable's you've created in other variables and subsequent sections.

4. Filter records

You'll probably want to reduce the number of records in this table to exclude any that aren't relevant to the final output table you're creating. This will allow you to execute transforms quickly and get a better understanding of the number of relevant records you have.

To filter records, add a Filter step to your transform and start building the conditions that records will need to meet in order to stay in your output table. These statements can be nested and allow you to reference any variables or record values.

5. Aggregate data

Depending on the structure of your data you might want to aggregate the data to collapse multiple records down to one, while preserving some information about what was dropped. Conceptually this might look like aggregating a table of charges from one record per charge into one record per person including a variable for the total charge per person.

To get started, add an Aggregate step and select the variables that you will to aggregate on. These will be the variables that exist in your output data after the aggregation is finished. All records that are an exact match in these selected variables will be dropped.

You can also capture information about records being dropped by creating a new aggregate variable. For example, maybe you are aggregating a table with multiple test scores per person down to a table with just one record per person. You can create an aggregate variable with the average test score, or the count of the number of tests each person took.

5. Select variables

Finally, before running your transform you'll always need to select which variables you want to keep in your output table. Perhaps you referenced a variable in this transform to create a new one, and now you don't need it anymore. Cutting variables means faster execution, and it is easy to add any variables back later and re-run the transform if you realize you need it, so in general try to keep this list as short as possible.

To propagate variables forward into your output table, they need to be in the right-hand box at the top of the transform labeled Keep. You can select any variable, or set of variables and click the > arrow button to move them over.

6. Run transform and sanity check output

When you're ready to execute the transform click the Run button in the top right of the toolbar.

If this button is disabled, it might be because the transform is invalid for some reason. Hovering on this button will give you more information, or you can look for the alert symbol (!) to see where you'll need to fix the transform in order to make it valid to run.

Successfully running a transform will create a new output table. Click on this table to see your output and verify that the transform executed as you would expect. Does it have the approximate number of records you expect to see? If you click on a variable you can see summary statistics which can help you verify key variable concepts, such as the distinct number of values.

Every time you run a transform you will want to sanity check your output.

7. Make changes and re-run

If there are any issues with your output table, or if you decide to go in a different direction, or add another step, it is easy to go back to your transform and start making changes.

Run this transform again to see changes in the output table. This guide describes all the steps you can take in a single transform but perhaps you want to do one step at a time and run it in between each step to sanity check your output. This system is designed for iteration so make as many changes as you want to experiment and build the output you want.

8. Create another transform

From here you can continue to create transforms on your output table, tables from the dataset you added, or add any other dataset on Redivis. You might want to fit as many steps as you can into one transform, or make a long chain to more easily track your work and communicate it to others.

As you build more transforms you'll see that sometimes actions you take create stale chains of transforms. You can easily make upstream changes in your workflow (such as upgrading a dataset from the sample to the full dataset, or updating to a new version) and then run all transforms in your workflow with one click (a Run all option is available in the Map button menu).

Next steps

Work with data in a notebook

You can use notebooks in a workflow to analyze data using Python, R, Stata, or SAS. These notebooks run in the browser with no additional configuration and seamless sharing with collaborators.

Export data

Upload your own datasets

Augment your data analysis in Redivis by uploading your own datasets, with the option to share with your collaborators (or even the broader research community).

Share and collaborate

Redivis workflows are built for collaboration and include real-time visuals to see where collaborators with edit access are in the workflow, and a comments interface to discuss changes asynchronously.

Overview

Redivis notebooks a performant, flexible environment for analysis that allow you to analyze and visualize data in workflows in Python, R, Stata, or SAS. With the notebook computation happening on Redivis, you don't need to configure an environment on a local machine or server, or export data from Redivis. This makes for easy iteration and collaboration, not to mention ensuring better security and data throughput.

1. Create a notebook

Once you have a table that you're ready to analyze, you can create a notebook by clicking the + Notebook button at any time. You'll need to name it and choose a kernel (Python, R, Stata, or SAS).

Notebooks can only reference tables within their workflow, so we recommend keeping all related work together in the same workflow.

2. Define dependencies

3. Compute resources

The default notebook configuration is free, and provides access to 2 CPUs and 32GB working memory, alongside a 60GB (SSD) disk and gigabit network. The computational powerful of these default notebooks are comparable to most personal computers, and will be more than enough for many analyses.

Clicking Edit compute configuration button in the start modal or the toolbar will allow you to choose from different preconfigured machine types. The notebook will then default to this compute configuration each time it starts up.

4. Start the notebook

Notebook nodes need to be started in order to edit or execute cells. When first clicking on a notebook node, you will see a read-only view of its contents (including cell outputs). Click the Start notebook button in the toolbar to connect this notebook to compute resources.

When you create a notebook for the first time it will start automatically.

5. Load data

To do meaningful work in your notebook, you'll want to bring in the tabular and/or unstructured data that exists in your workflow into your notebook.

Referencing tables

Notebooks come pre-populated with templated code that pulls in data from the notebook's source table. You will need to run this cell to pull the data into the notebook, and you can see that it worked because this code will print a preview of the loaded data.

Referencing files

Any files with unstructured data stored in Redivis tables can be referenced by their globally unique file_id. You can also reference these file_id's in any derivative tables, allowing you to query and download specific subsets of files.

When working with large files, you'll want to consider saving the files to disk and/or working with the streaming interfaces to reduce memory overhead and improve performance.

6. Analyze data

At this point, you have all the tools you need to work with your data in your chosen language. The Python, R, Stata, and SAS ecosystems contain myriad tools and libraries for performing sophisticated data analysis and visualization.

7. Create an output table

Notebooks can produce an output table, which you can sanity check and further analyze in your workflow by including in other notebooks or exporting to other systems.

Next steps

Share and collaborate

All Redivis notebooks support real-time collaboration, allowing multiple editors to edit and run cells in a running notebook. When another editor is active in a notebook, you will see a colored cursor associated with them (much like a Google Doc).

Cite datasets in your publications

Redivis is creating tools not just for data distribution and analysis, but also to help drive forward standards for transparent research that is accessible to all.

Reproducibility

Availability

Datasets in any format can be deposited into Redivis and made available immediately to the world via open access licenses at no cost to the end user.

All Redivis datasets and analysis workflows are available to anyone on the internet without a Redivis account unless the owner decides to add restrictions. In this case someone wishing to view the work would need to create a free Redivis account to request access from the owner.

Longevity

The stable technical infrastructure and dedicated Redivis team is funded by a coalition of academic institutions paying license fees for large scale data distribution. This diverse group of funders ensures the long-term availability of the datasets.

We've started building a library of common workflow types and analytic tasks that might be useful when you're getting started working with data.

Please let us know if there are other actions or concepts you would like us to provide examples for by reaching out or emailing contact@redivis.com

This guide demonstrates using a Redivis workflow to gather key variables from different tables, clean them, and and consolidate them into a single table for analysis in Google Looker, all using point and click interfaces.

Workflow objective

We'll use Gini coefficient data (a common measure of income distribution) from the American Community Survey (ACS) to build a map that shows the range of income inequality by US county in 2018.

1. Explore data and start a workflow

• ACS Gini coefficient table

• ACS table that measures population by county

• National Weather Service dataset that maps a US county to lat/long coordinates.

Add this dataset to your workflow and click on the dataset node to see all of the available tables. Select Gini Index by County and can look at the data format by clicking Cells.

2. Clean data

We want to map the Gini estimate and population by US county in our final data visualization. To map data points by county, we will need a variable that represents the latitude, longitude coordinates of that county. We also want the state, county name, and five-digit county code.

Therefore, we want our final dataset to include six distinct variables:‌

• Gini Estimate

• State

• County Name

• County Code

• Latitude, Longitude

• Population

We will get Lat/Long and Population by joining our table with other tables later on in the workflow. For now, we can create a transform to start reshaping our data.

Rename: Gini Estimate

New variables: County and State Names

This method allows users to create a new variable with characters that match a specific pattern of the old variable. For example, creating a new variable for county, we can select all characters in NAME preceding the comma. We can add a new block and do a similar process on the NAME variable to get the State.

New variable: County Code

The last five digits in the GEO_ID variable represent the county code. We create another new variable block and name our new variable County_code. Select the Substring method and select all values starting at index 10 up to a max length of five.

Variable selection

Finally, we choose which variables to keep or discard in our transform. We’ll keep all variables except for GEO_ID, NAME (county and state combined), and B19083_001M (the margin of error for the Gini estimate).

Click the run button in the upper right-hand corner to run this transform.

3. Sanity check the output table

Click on the newly generated output table to investigate the output of the transform. We can see that this table has the same number of records as our initial table and has the four variables we selected. If we click on the County and State variables we can see in the frequency tables that they look like we wanted them to.

Since this table looks like we expect we can move on to the next step! Otherwise we'd need to go back to the initial transform to change our inputs.

4. ‌Join geographic information

We now have a table with the Gini estimate, state, county name, and county code, but we need the latitude and longitude information in order to map each county and we also want to account for the size of each county with a population variable. To do this, we'll perform two joins.

We could continue to work in our initial transform but we are choosing to create new transforms for each join to keep steps separated conceptually and provide output tables to reference along the way.

Matching county to latitude, longitude data

First, we will match each county code to its lat/long coordinates by joining with the "County to Lat/Long Coordinates" table.

A Left Join returns the complete source table (Gini Index by County) and all matching records from the right table (County to Lat/Long). We set the County_code variable in the source table equal to the County_code variable in the right table, which matches a set of lat/long coordinates to each county code (if there is no coordinate, it will return Null).

Some county codes, however, map to more than one set of latitude, longitude coordinates so we must create two new variables in this transform: unique_lat and unique_long.

For unique_lat, we will partition on all kept variables except LAT and take the average of the LAT variable. This says that for that each county, compute the average of all possible latitudes and store that average in the variable unique_lat.

We will select all variables from the source table as well as unique_lat and unique_long in our output table. Finally, we select "distinct" in our variable selector so that we will drop all records that are unique.

Running this transform outputs the following table, where each county now corresponds to a unique set of latitude and longitude variables.

Matching county to population data

We now have all the desired variables in our dataset except for population per county, which we will need when we create the final visualization.‌

Using the same steps from the previous join, we will perform a second join to match each county to its total population by incorporating the Population by County table (also in the original dataset). We select a Left Join and set the County_code variable in the source table (Gini Index by County) equal to the County_code variable in the right table (Population by County).

Selecting all variables to remain in our output table and running this transform outputs the following table.

5. Final data cleaning

We will be using Google Looker Studio to visualize this data so we will need to update our data to make it work smoothly with their specified format for geographic data. Using their reference sheet, we can see that we will need our latitude and longitude information in the format:

Comma separated latitude and longitude decimal values (e.g., "51.5074,-0.1278" specifies to London, England)

We will need to combine our two separate values and add a comma between them.

We can do that using the Concat method for creating a new variable, but that method only accepts string inputs so we'll first need to retyping the unique_lat and unique_long variables from float to string using the Retype step.

Then we use the Create new variables step to make a new variable named lattitude_longitude and use the Concat method to concatenate the unique_lat string, a string with a comma, and the unique_long string.

We can now discard unique_lat and unique_long and keep the combined variable, latitude_longitude.

Running this transform yields the final table. From here, we can edit variable metadata and/or download our new dataset in the format that we'd like.

6. Export data to Looker Studio

The next step would normally be to create a notebook in this workflow and use Python, R, or Stata to analyze this table. However if we want to create a quick visualization and aren't familiar with any of those coding languages we have easy options to export data to other systems such as Google Looker Studio.

For this example we will link this table directly to Google Looker Studio by clicking Export table on this table.

We could download our final table and the re-upload it into Looker Studio, but we choose to link the table through the Redivis connector so that if we come back to this original workflow and make changes they will be reflected in the visual that we're about to make there.

Follow the prompts of the connector to log in and authorize access, then we'll need to indicate the table we'd like to use. For that we'll need:

• The owner of the workflow's ID (shown in the top menu bar of the workflow)

• The name of the workflow (shown in the middle of the black workflow toolbar)

• The name of the table (shown on the title of the table node)

When the table is being imported we will have the option to change the type for variables. We'll need to change the type of our lattitude_longitude variable from string to the Lattitude, Longitude geography type.

Then click Create report to get started!

7. Build a visualization

For this example we are going to build a bubble map that shows the size of population against the intensity of gini disparity in certain regions.

Get started by adding a chart and selecting the bubble map type. We will need to select Redivis as the data source for this map.

Then we will need to define which of our variables map to which parts of the visual. We will also want to change how they are aggregated, from SUM to AVG.

• Location: lattitude_longitude

• Tooltip: County

• Size: (AVG) population

• Color metric: (SUM) gini_estimate

The resulting map is automatically interactive. Users can hover over each bubble and view the county name, population, and Gini coefficient. The bubble size is determined by the county population and the bubble color is determined by the Gini value. View the interactive report:

You can continue to create a variety of visualizations in this report, including scatter plots, Google maps, and stacked bar charts.

Next steps

This guide demonstrates using a Redivis workflow to gather key variables from different tables, clean them, and and consolidate them into a single table for analysis in a notebook.

Workflow objective

We want to take weather data collected around the world and use it to understand how precipitation trends in the US have changed over time.

1. Explore data

To get started we want to understand this dataset and what information is in each table. We can look at the dataset page to learn more about it, including its overview information, metadata, and variable summary statistics. Since this dataset is public we can also look directly at the data to confirm it has the information we need. Some tables jump out as ones we will want to work with:

Daily observations

This table has nearly 3 billion records and seems to be the table in the dataset with the most information so we will start here. There is a variable that looks like it will be very helpful for our goal: element

If we click on the variable name, we can see a searchable frequency table with all this variable's values. One value that jumps out is PRCP which we can see on the value label represents the precipitation. Paired with this is the variable value which contains a numeric value recording the magnitude of the element. We can learn even more about this by clicking on the data dictionary in the dataset's overview page to see that this value is measured in either mm or inches.

We can see that this table doesn't contain any information about location though, so we'll need to find that somewhere else.

Stations

This table contains latitude and longitude variables as well as a linking variable id which will allow us to link in precipitation information from the Daily observations table, since it has the same id variable.

Since there are two tables we want information from we know that we will need to do a join as part of our data reshaping.

2. Create a workflow

At the top of this dataset page we can click the Analyze in workflow button to get started working with this data.

You can add this dataset to an existing workflow you already have access to, or create a new workflow to start from scratch.

3. Clean and reshape data

We will use transforms to clean the data, as they are best suited for reshaping data and will quickly output new tables we can continue to work with. Even though we might be more comfortable with Python or R, the table we want to work with is 132GB and we would not be able to work with it in a notebook without specialized equipment.

Filter for precipitation

If we click on the element variable we can see that there are many types of elements recorded in this table. Since we only want information about precipitation (PRCP), we will start by filtering out any records with a different value in the element variable.

Compute annual precipitation

Looking at the data, it's clear there are many observations of precipitation at each location over time. We know we want to look at annual trends so we can start by aggregating this data to only have one value per year per location.

Conceptually, we will need to aggregate on the year of each record, but if we look at the date variable in this table it also contains the month and day. For aggregation we will need a field which contains only the year which we can collapse on.

But the most important step here is that we want to gather information about the duplicate records that were dropped! If there were 400 records of precipitation in a year for a particular station, we want to know what those records all add up to. To do this we will Create a new aggregate variable within this aggregation step named annual_precip. We want to aggregate the information in the value column, since that contains the numeric amount of the corresponding element variable. Since we want the total amount across all dropped records we will use the Sum method.

Select variables and run

The final step in a transform is selecting which variables we would like to populate the resulting output table. Since we did an aggregation step, this list becomes limited to only the variables that we aggregated on. Since we want both of these variables in our output we make sure both are selected and visible in the right side list of the footer.

With everything in place we will run this transform to create a new table, by pressing the Run button in the top right corner!

4. Sanity check the output table

Now that we have created a new table, we can inspect it to make sure our steps accomplished what we expected them to.

Table characteristics

Click on the output table below the transform to view it. We can see that it contains the two variables we expected it to, based on our variable selection. The table has almost 3 million records, down from the almost 3 billion in our original table, which makes sense given our aggregation step.

Variable characteristics

We can also inspect each variable further. If we click on the year variable we can see that all values are a four digit year value and we can see from the min and max values in the summary statistics that the years range from 1781 - 2021 which make sense. Looking at the frequency table we can see that we have a high frequency of each year represented in the records which is what we expect since there are multiple ids or locations per year.

If we click on the annual precipitation variable we can see a max value of 792,402 (mm or in) which seems very high but possible for a given year. The minimum value is -148,917 which doesn't seem right. When we look at the data dictionary it doesn't indicate that there should be negative values.

We can investigate our data further by clicking on the Query tab of the table and choosing "Sort on variable" from the templates menu, and sort on our annual_precip variable by ascending (ASC). It looks like there are a relatively few number of stations with negative values.

If we were doing a rigorous analysis of this data we might dig deeper into why these value exist in this dataset, or decide to exclude these outliers. However since this is an example we can decide to leave them in and see how it affects our outputs. We can easily go back and exclude them from the workflow at a later point.

5. Join geographic information

Now that we have a table we are satisfied with that contains annual precipitation information, we want to join in the latitude and longitude information about each station so we can make a geographical visual as part of our analysis.

This geographic information is included in the Stations table, so we will need to do a join which connects records from two tables.

While we could do this transformation step in our previous transform and rerun it, let's create a new transform to make this join so that we will have our previous output table saved to compare the output to.

Since our identifier in both table is the variable id, set id from the source table (t0) as the left side of the join condition and set id from the joined table (t1) as the right side of the join condition. This means that for every record in our source table, look for a corresponding value in the Stations table and join the values from additional columns in that table.

There are a number of additional variables in the stations table but we are only interested in the latitude and longitude variables. In our variable selector we move all variables from our source table (year, id, annual_precip) as well as latitude and longitude.

With everything set up, run this transform and sanity check the output table! We started with a 132GB table and have narrowed it down to 134MB containing only the data we want in the format we will need.

6. Analyze outputs in a notebook

For this workflow we will use Python but you can also use R, Stata, or SAS if you'd like. When you create the notebook for the first time it will start up. Notebooks must be running to execute code.

Reference data

Newly created notebooks come with standard code to import the Redivis library and reference the source table in a pandas dataframe within the notebook.

Select data from the United States

Now we will create a new cell to organize our work by pressing the + button in the top bar.

In this cell we want to limit our records to only ones whose latitude and longitude values fall in a specific range of the continental United States.

us_df = df[
    (df.latitude >= 24.396308)
    & (df.latitude <= 49.384358)
    & (df.longitude >= -124.848974)
    & (df.longitude <= -66.885444)

Compute average change

We want to see how our annual precipitation variable has changed at stations in the United states so for each station we will group by the id variable and compute an average for annual_precip.

# Compute avg precip for each station between 1980-2010
df_1980_2010_average = us_df[(us_df.year > 1980) & (us_df.year < 2010)].drop(columns=['year'])
df_1980_2010_average = df_1980_2010_average.groupby(['id']).mean()
df_1980_2010_average.rename(columns={"annual_precip": "_1980_2010_avg"}, inplace=True)

# Compute avg precip for each station between 1990-2020
df_1990_2020_average = us_df[(us_df.year > 1990) & (us_df.year < 2020)].drop(columns=['year'])
df_1990_2020_average = df_1990_2020_average.groupby(['id']).mean()
df_1990_2020_average.rename(columns={"annual_precip": "_1990_2020_avg"}, inplace=True)

diff_df = df_1980_2010_average.join(df_1990_2020_average["_1990_2020_avg"])
diff_df['deviation'] = (diff_df._1990_2020_avg - diff_df._1980_2010_avg) / diff_df._1980_2010_avg * 100
diff_df = diff_df[(diff_df.deviation < 25) & (diff_df.deviation > -25)]

diff_df = diff_df.dropna()
diff_df

Plot the results

import plotly.figure_factory as ff

import numpy as np
import matplotlib.pyplot as plt
import plotly.express as px
px.set_mapbox_access_token("pk.eyJ1IjoiaW1hdGhld3MiLCJhIjoiY2thdnl2cGVsMGtldTJ6cGl3c2tvM2NweSJ9.TXtG4gARAf4bUbnPVxk6uA")

fig = ff.create_hexbin_mapbox(
    data_frame=diff_df, lat="latitude", lon="longitude",
    color="deviation",
    agg_func=np.mean,
    title="% Change precipitation, 1981-2010 vs 1991-2020",
    range_color=[-15,15],
    nx_hexagon=50, opacity=0.4, labels={"color": "Percent change"}, color_continuous_scale="Icefire_r",
)

fig.update_layout(margin=dict(b=0, t=0, l=0, r=0))
fig.show("nteract")

Now we've created an interactive figure showing change in precipitation by location in the United states.

Next steps

Citing datasets

It's important to make sure that you fully cite any datasets used in your work. Each dataset page includes citation information in the provenance section in multiple format options.

Stanford Center for Population Health Sciences. (2022). GHCN Daily Weather Data (Version 1.2) [Data set]. Redivis. https://doi.org/10.57761/h9ff-vy04

Citing workflows

Workflows across Redivis also include citation information in the provenance section in multiple format options.

Bibliography

Datasets and workflows on Redivis can be supplemented with related identifiers that situate a resource in its broader context. You can view the bibliography for a dataset or workflow from the corresponding provenance section.

Citing Redivis

To cite Redivis in publications use:

Redivis (RRID:SCR_023111). https://redivis.com

If referencing Redivis inline, the abbreviated citation is also acceptable:

Redivis (RRID:SCR_023111)

To cite Redivis, you can use the following BibTeX:

@misc{https://doi.org/10.25495/7gxk-rd71,
  doi = {10.25495/7GXK-RD71},
  url = {https://redivis.com/},
  author = {Redivis},
  keywords = {FOS: Dataset, Publication},
  language = {en},
  title = {Redivis},
  publisher = {Redivis},
  year = {2024}
}

You can also refer to the Redivis Re3data identifier

Overview

Notebooks on Redivis offer a performant and highly flexible environment for doing data analysis. This includes the ability to run state-of-the-art machine learning (ML) models, including the ability to train new models, fine-tune various existing models, and use these models to perform inference and generate novel outputs.

This guide is focused on a number of common use cases when running ML workloads on Redivis. Here we generally focus on using the Hugging Face + PyTorch ecosystem in Python, though these examples are broadly applicable to other ML libraries and languages.

For a detailed example of using Redivis to fine tune a large-language model, see the complementary example:

1. Create a notebook with appropriate computation capacity

Training ML models and running inference can require a substantial amount of compute capacity, depending on various factors such as your model and dataset size, usage parameters, and performance goals.

The default, free notebook on Redivis offers 2CPUs and 32GB of RAM. While this may work for initial exploration, running practical machine learning workflows typically requires the availability of a GPU. When creating your notebook, you can choose a custom compute configuration to match your needs.

Redivis offers a number of custom compute configurations, mapping to the various machine types available to Google Cloud. We recommend starting with a more modest GPU for initial exploration, and then upgrading as needed when computational or performance bottlenecks are reached. For this example, we'll use the NVIDIA L4 GPU, which provides reasonable performance at a reasonable cost.

2. Define dependencies

The Redivis python notebook is based off the jupyter-pytorch notebook image, with PyTorch, CUDA bindings, and various common data science libraries pre-installed. However, if you require additional dependencies for your work, you can specify them under the "dependencies" section of your notebook.

2a. [Optional]: Pre-load external models when internet is disabled

If your notebook references export-restricted data, for security reasons, internet will be disabled while the notebook is running. This can present a challenge for some common approaches to ML in python, such as downloading a model or dataset from Hugging Face. For example, we might reference a model as follows:

from sentence_transformers import SentenceTransformer
model = SentenceTransformer("all-MiniLM-L6-v2")

This code typically downloads the model from Hugging Face and caches it to our local disk. However, if internet is disabled, this command will hang and ultimately fail. Instead, we need to download the model during notebook startup, before the internet is disabled, as part of the post_install.sh script under the notebook's dependencies:

python -c '
from huggingface_hub import snapshot_download
snapshot_download(repo_id="sentence-transformers/all-MiniLM-L6-v2")
'

This will download the model weights and other files to the default Hugging Face cache directory, ~/.cache/huggingface/hub.

Now, within our notebook, we can load the cached model. Make sure to set local_files_only=True, so that Hugging Face doesn't try to connect to the internet to check for a newer version of the model.

from sentence_transformers import SentenceTransformer
model = SentenceTransformer("all-MiniLM-L6-v2", local_files_only=True)

3. Load a model

3a. Load a model from Redivis

Machine learning models can be stored directly within a Redivis dataset as unstructured files. For example, this dataset contains the various files that make up the bert-base-cased model on Hugging Face. We can then download the model to our notebook's local filesystem:

import redivis
table = redivis.organization("demo").dataset("huggingface_models").table("bert_base_cased")
table.download_files("/scratch/bert-base-cased")

And then reference this as a local model. E.g.:

from transformers import AutoTokenizer, AutoModelForSequenceClassification

model = AutoModelForSequenceClassification.from_pretrained("/scratch/bert-base-cased", num_labels=5)
tokenizer = AutoTokenizer.from_pretrained("/scratch/bert-base-cased")

When using models stored on Redivis, we don't have to worry about whether our notebook has internet access, nor do we need to rely on the future availability of that particular model on Hugging Face.

from huggingface_hub import snapshot_download
import redivis

# Download the model files from Hugging Face
snapshot_download(repo_id="google-bert/bert-base-cased")

# Specify an existing dataset and table on Redivis. 
# Consult the python docs for how to programmatically create datasets (apidocs.redivis.com)
table = redivis.organization("demo").dataset("huggingface_models").table("bert_base_cased")

# Add the downloaded model files to the table
table.add_files(directory='/home/root/.cache/huggingface/hub/models--google-bert--bert-base-cased/snapshots/cd5ef92a9fb2f889e972770a36d4ed042daf221e')

3b. Load a model from an external source

If your notebook has internet access, you can also use any other models that may be available on the internet. For example, we can load the same bert-base-cased model directly from Hugging Face:

from transformers import AutoTokenizer, AutoModelForSequenceClassification

model = AutoModelForSequenceClassification.from_pretrained("google-bert/bert-base-cased", num_labels=5)
tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-cased")

# Do cool things!

Note that if you are working in a notebook with disabled internet, this approach won't work, and you'll need to use the methods mentioned in either 2a or 3a above.

4. Load data

As a final step, you'll likely want to load data into your notebook to either fine-tune the model, perform inference, or otherwise experiment. There are thousands of datasets on Redivis, and you can upload your own data as well. You can learn more about loading data into a notebook in the python notebooks documentation. As a quick example:

import redivis

# Load tabular data as a pandas data frame, or a number of other formats (arrow, polars, iterator, etc)
table = redivis.organization("demo").dataset("ghcn_daily_weather_data").table("stations")
df = table.to_pandas_dataframe()

# Download unstructured data as files to your local disk
files_table = redivis.organization("demo").dataset("chest_x_ray_8").table("images")
files_table.download_files("/scratch/xray_images")

Of course, assuming your notebook has access to the external internet, you can also call various APIs to load external data sources.

from datasets import load_dataset

# load a dataset from Hugging Face
hf_dataset = load_dataset("Yelp/yelp_review_full") 

Next steps

At this point, you have all the tools at your disposal to perform cutting edge ML research. But of course, what you do next is totally up to you. We do recommend further familiarizing yourself with the examples and detailed documentation to take full advantage of the capabilities of Redivis notebooks:

Overview

Workflows are where you work with data on Redivis. In a workflow you can query, merge, reshape, and analyze any data that you have access to, all from within your web browser.

In a workflow, you can construct reproducible data transformations and analyses, and share and collaborate with your peers in real time.

1. Create a workflow

Add a dataset to a new or existing workflow from any Dataset page where you have "Data access" by clicking the Analyze in workflow button.

Within workflows you can navigate between entities on the left side of the screen, and inspect them further on the right panel. You can inspect your dataset further by clicking on any table to see its cells and summary statistics.

To add more data to this workflow you can click the Add data button in the top left of the workflow toolbar. It is also possible to add other linked workflows to this workflow. This is useful as you develop more complex analyses that you want to segment into discrete pieces of work that you can link together.

You can find this workflow later by going back to your workspace.

2. Transform data

Transforming tables is a crucial step in working with data on Redivis. Conceptually, transforms execute a query on source table(s), whose results are materialized in a new output table. In most cases you'll want to use transforms to reshape your data to contain the information you're interested in, before analyzing that table in a notebook or exporting it for further use.

To create a Transform, select a table in this dataset and click the +Transform button. You can get started here building a query through the point and click interface or writing SQL code by adding a SQL step.

For all transforms you will need to select which variables you want to keep in your output table. The rest of the steps are up to you. Some common operations you can get started with include:

Once you've built your query, execute it by clicking the Run button in the top right of the transform. This will create a new output table where you can inspect the output of your query by clicking on the table beneath the transform in the map and making sure it contains the data we would expect.

From here you can create a new transform from this output table to continue reshaping your data, or go back to your original transform to make changes and rerun it.

As you become more familiar with transforms, you can start doing more advanced work such as geospatial joins, complex aggregations, and statistical analyses.

3. Analyze data in a notebook

Once you have a table you're ready to analyze, you can select any table and click the + Notebook button to create a notebook that references this table.

The default notebook configuration is free, and provides access to 2 CPUs and 32GB working memory, alongside a 60GB (SSD) disk and gigabit network. The computational powerful of these default notebooks are comparable to most personal computers, and will be more than enough for many analyses.

From here it’s all up to you in how you want to analyze and visualize your data. Once you’ve finalized your notebook, you can easily export it in different formats to share your findings!

4. Share and collaborate

You can share your in-progress work or finished results with collaborators by sharing this workflow.

Researchers can work side by side in this workflow in real-time. Leave comments to communicate, and see a visual cue for what each person is working on. You can even collaborate within a running notebook at the same time.

If any of the data in your workflow is restricted, your collaborator must also have access to those datasets in order to view their derivatives within your workflow.

Next steps

Share and collaborate

Redivis workflows are built for collaboration and include real-time visuals to see where collaborators with edit access are working in the workflow, and a comments interface to discuss changes asynchronously.

Export data

Browse our example workflows

Redivis workflows excel at working with large tables, whether it's filtering and joining, complex aggregation and date manipulation, or visualization and analysis.

Overview

Redivis combines powerful functionality to reshape and analyze data on platform, with an easy export and publishing flow, to ensure the results of your work can be displayed in the format of your choice and shared with your collaborators and research community.

Working to determine, first, the broader picture of the content you'd like to publish and where you'd like to publish it will help you, then, determine the desired shape and format of the assets to be exported and, finally, the specific sources of the data in your Redivis workspace.

1. Develop a publishing strategy

Ask yourself: What story are you trying to tell? Who/where is the audience? What component pieces are crucial to building the narrative? Your ideal package of assets may be very different if you're hoping to share progress on a workflow with a principal investigator, take a snapshot of a single variable distribution for a colleague, publish results in a journal, or build a custom dashboard to highlight multiple trends.

You may use a combination of tables, unstructured files, code snippets, graphs, and descriptive text to be published, so sketching out these component pieces in increasing detail will help define your end product.

2. Choose your desired formats

With an initial strategy in mind, learning about the different types of Redivis exports will help define your list of assets to generate.

3. Identify the data sources

Next steps

Upload your own datasets

Augment your data analysis in Redivis by uploading your own datasets, with the option to share with your collaborators (or even the broader research community).

Build a custom dashboard

For full customizability, you can publish a static site that accesses Redivis data to power an interactive visual dashboard – a more permanent, web-based approach to highlighting results of your data analysis or generated data.

Overview

Datasets are the core entity on Redivis, and finding the dataset(s) you want to work with is generally one of your first steps. All datasets are either uploaded by an organization or a user. If a user has uploaded a dataset they will need to share it with you directly, but you can browse datasets upload by any organization on their home page.

1. Find your organization

There are many ways to find your organization. If you know what you're looking for you can type it into the "Go to" bar on the top of the page. To browse all organizations you can go to the Organizations tab of your workspace where you can Find organization.

Note that some organizations may be hidden from public view, in which case you will need to contact someone at the organization to get the direct link.

2. Browse available datasets

On the organization home page you can see all of the organization's datasets that you have at least Overview access to. If you're looking for a dataset you expect to be here and you don't see it, you might need to contact the organization administrators to get access.

You can see the datasets that this organization has featured on the left section of the page, type a term into the search bar, or click the Datasets tab to browse everything. By default these datasets are sorted by popularity but you can reorder them however you'd like to find what you want.

You can also click the Workflows tab to see workflows featured by this organization which showcase how their data can be used. More on workflows later!

3. Search for data

Once on the dataset tab you can use multiple filters along with search terms to find what you'd like. All searches take a comprehensive look at information from the dataset documentation as well as variable metadata, including value labels.

You can also use the filters to find specific information you're looking for. If you know the name of the variable or year of the data you're interested in, narrow your search here.

4. Preview tables

Each dataset that appears here matches all search and filter configurations. You can click on the dataset tile to see more information, including a preview of all of the tables this dataset contains.

Click on any of these tables to dig into it further. As long as you have metadata access you can see the variable information and summary statistics.

Once you find a dataset you want to explore further, click on the title to go to the dataset page.

5. Explore the dataset page

This dataset page has all of the information you'll need to familiarize yourself with the data and get started working with it further. Add this dataset to your dataset library to easily find it later on by clicking the button alongside the dataset title.

Overview

This tab has a short summary of the dataset along with different sections filled out by the administrators such as the Methodology, Documentation, and Provenance information. This Provenance section will also have citation information and a DOI if one has been created. You can also see your current access level on the right side of the page.

Tables

This tab contains all of the dataset's tabular data. You can see each table alongside information about its contents. Click on each one to dig deeper into the data.

Files

This tab contains all of the dataset's unstructured data. You can preview each file by clicking on it.

Usage

This tab has information about how other people have used this dataset on Redivis. You can see the most popular variables across all tables to get a sense for which ones might be a good starting place to understanding the structure of the data.

Next steps

Apply for access to restricted data

If you don't have data access to this dataset you'll need to apply for access before you can work with it further

Start working with your data

Add this dataset to a workflow to transform and analyze it leveraging lightning fast tools from your browser.

Overview

Redivis is designed to support data-driven research throughout the research lifecycle, and can serve as a permanent repository for your data, analytical workflows, and data derivatives.

Redivis is specifically designed to meet these desirable characteristics of a data repository, outlined below:

Desirable Characteristics for All Data Repositories

Additional Considerations for Human Data

Detailed information

Unique Persistent Identifiers

Assigns datasets a citable, unique persistent identifier, such as a digital object identifier (DOI) or accession number, to support data discovery, reporting, and research assessment. The identifier points to a persistent landing page that remains accessible even if the dataset is de-accessioned or no longer available.

Long-Term Sustainability

Has a plan for long-term management of data, including maintaining integrity, authenticity, and availability of datasets; building on a stable technical infrastructure and funding plans; and having contingency plans to ensure data are available and maintained during and after unforeseen events.

Redivis undergoes annual security audits and penetration testing by an external firm, and utilizes a formalized software development and review process to maintain the soundness of its technical infrastructure.

Funding for Redivis is provided by recurring annual subscriptions from its member academic institutions. This model provides consistent annual revenue to support the ongoing maintenance of the platform. Redivis is an employee-owned company without any external investors with an equity stake, allowing us solely focus on the needs of our customers, our employees, and our mission of improving accessibility in the research data science.

Metadata

Ensures datasets are accompanied by metadata to enable discovery, reuse, and citation of datasets, using schema that are appropriate to, and ideally widely used across, the community(ies) the repository serves. Domain-specific repositories would generally have more detailed metadata than generalist repositories.

As a generalist repository, the metadata schema is intentionally broad and flexible, but specific groups on Redivis can choose to enforce more specific metadata standards within their datasets.

Curation and Quality Assurance

Provides, or has a mechanism for others to provide, expert curation and quality assurance to improve the accuracy and integrity of datasets and metadata.

It is ultimately up to the editors of a dataset to provide curation, though Redivis is designed to support this process as much as possible. Redivis automatically computes checksums and runs fixity checks on all uploaded files, and computes univariate summary statistics of all variables to aid in the quality assurance process. Metadata completeness is also reported to editors, encouraging them to provide as much information as possible.

Free and Easy Access

Provides broad, equitable, and maximally open access to datasets and their metadata free of charge in a timely manner after submission, consistent with legal and ethical limits required to maintain privacy and confidentiality, Tribal sovereignty, and protection of other sensitive data.

Broad and Measured Reuse

Makes datasets and their metadata available with broadest possible terms of reuse; and provides the ability to measure attribution, citation, and reuse of data (i.e., through assignment of adequate metadata and unique PIDs).

Clear Use Guidance

Provides accompanying documentation describing terms of dataset access and use (e.g., particular licenses, need for approval by a data use committee).

Security and Integrity

Has documented measures in place to meet generally accepted criteria for preventing unauthorized access to, modification of, or release of data, with levels of security that are appropriate to the sensitivity of data.

Redivis is well-designed to handle workflows around sharing of sensitive and high-risk data. It provides technical mechanisms to support the reuse of sensitive data when allowed, while enabling the enforcement of appropriate guardrails and access controls defined by data administrators.

Confidentiality

Has documented capabilities for ensuring that administrative, technical, and physical safeguards are employed to comply with applicable confidentiality, risk management, and continuous monitoring requirements for sensitive data.

In addition to an annual security audit, Redivis also undergoes annual penetration testing by an outside firm to further ensure the soundness of its security posture.

Common Format

Allows datasets and metadata downloaded, accessed, or exported from the repository to be in widely used, preferably non-proprietary, formats consistent with those used in the community(ies) the repository serves.

Provenance

Has mechanisms in place to record the origin, chain of custody, and any modifications to submitted datasets and metadata.

Retention Policy

Provides documentation on policies for data retention within the repository.

Fidelity to Consent

Uses documented procedures to restrict dataset access and use to those that are consistent with participant consent and changes in consent.

Access rules on Redivis are defined and enforced by the dataset owner.

Restricted Use Compliant

Uses documented procedures to communicate and enforce data use restrictions, such as preventing reidentification or redistribution to unauthorized users.

Privacy

Implements and provides documentation of measures (for example, tiered access, credentialing of data users, security safeguards against potential breaches) to protect human subjects’ data from inappropriate access.

Additional technical safeguards ensure data privacy. Data users can establish their identity through their institutional identity provider, and Redivis undergoes regular security audits to ensure the soundness of its security posture.

Plan for Breach

Has security measures that include a response plan for detected data breaches.

Redivis has detailed internal security protocols and documented security breach plans which are regularly exercised by technical personnel via tabletop exercises. All systems are continuously monitored for potential breaches, with immediate alert pathways and clear escalation protocols to respond to any breach.

Download Control

Controls and audits access to and download of datasets (if download is permitted).

Violations

Has procedures for addressing violations of terms-of-use by users and data mismanagement by the repository.

As a matter of policy, Redivis aims to be as permissive as possible, recognizing that often misuse is the result of accidental behavior or misunderstanding. These controls are designed to protect the system for all users, and are not intended to ever be punitive towards good-faith actors.

Request Review

Makes use of an established and transparent process for reviewing data access requests.

This guide demonstrates using a Redivis workflow to import an existing LLM and then use relevant data to fine tune it and run it on another similar set of data we are interested in.

Workflow objective

Here, we want to fine-tune a pre-trained "foundational" LLM so that it can be used to score reviews. We will leverage an existing dataset that contains a collection of Yelp reviews and their scores to perform the fine-tuning, and then apply this classification model to other reviews (from Reddit) that do not contain an accompanying score. The goal here is to demonstrate how Redivis can be used to leverage, modify, and ultimately apply state-of-the-art LLMs to novel data.

1. Choose and explore data

Yelp reviews

To get started we want to understand this dataset and what information is in each table. We can look at the dataset page to learn more about it, including its overview information, metadata, and variable summary statistics. Since this dataset is public we can also look directly at the data to confirm it has the information we need.

It looks like there are two tables, one with reviews for testing a model and another with reviews for training a model. Clicking on each table in this interface shows that they both have two variables (label and text) and that the Train table has 650,000 records while the Test table has 50,000 records.

This data seems to be formatted exactly how we'll want to use it so we don't need to do additional cleaning .

Reddit

For this workflow, we just want to look at reviews from one specific subreddit which reviews mouse traps: MouseReview. If we click on the Subreddit variable name, we can see a searchable frequency table with all this variable's values. If we search MouseReview we can see that this dataset contains 26,801 posts.

To move forward with this workflow we'll want to train a model on the Yelp dataset, and filter and clean the Reddit table make it more usable with our model. In order to clean or transform data and do our analysis we'll need to create a workflow.

2. Identify a base model to fine-tune

This model is hosted on Hugging Face, so we could load it directly into our notebook at runtime. However, if our notebook uses restricted data, it might not have access to the external internet, in which case we'll need to load the model into a dataset on Redivis.

3. Create a workflow

At the top of any dataset page, we can click the Analyze in workflow button to get started working with this data.

You can add this dataset to an existing workflow you already have access to, or create a new workflow to start from scratch.

Add the additional datasets by clicking the + Add data button in the top left corner of the workflow and searching for the dataset by name.

4. Create a notebook and load model + data

Once we've added all our datasets to the workflow, we can get started. To begin, we'll create a python notebook based on the Yelp reviews training data, by selecting that table and clicking the + Notebook button.

We'll also need to install a few additional dependencies to perform training and inference via Hugging Face python packages:

The general steps here are as follows:

1. Load the training and test data from the Yelp reviews dataset

2. Load the pretrained BERT model

3. Train this base model on the Yelp data to create a fine-tuned model that can classify text reviews from a score of 0-4.

5. Prepare the Reddit data for inference

The Yelp data was ready to go as-is, with a simple text field for the review and integer value for the score. For the Reddit data, we just need to run a quick filter to choose posts from the appropriate sub-reddit.

Create a transform

Click on the Posts table in the Reddit dataset and press the + Transform button. This is the interface we will use to build our query.

The final step in a transform is selecting which variables we would like to populate the resulting output table. In this case we just need the variables title and selftext.

With everything in place we will run this transform to create a new table, by pressing the Run button in the top right corner.

7. Use the fine-tuned model to classify Reddit reviews

Finally, we can apply our fine-tuned model to the subset of Reddit comments that we want to analyze. Ultimately, we produce a single output table from the notebook, containing the reddit post and associate score generated by our model.

Next steps

We do recommend further familiarizing yourself with the examples and detailed documentation to take full advantage of the capabilities of Redivis notebooks:

Overview

1. Identify your Redivis data

2. Build your Observable project

3. Load data with a client library of choice

To successfully authorize data fetching functionality in your development environment, be sure to export a REDIVIS_API_TOKEN variable to your path, with the following terminal command: export REDIVIS_API_TOKEN = 'MY_API_TOKEN'

Writing a "data loader"

Using the redivis-js client library

Using the redivis-python client library

python3 -m pip install --upgrade pip
pip install -r requirements.txt

4. Visualize data

5. Deploy and share your site

With our dashboard finalized, you can deploy your project anywhere on the web. Observable provides a quick command (npm run build) to generate static files to be hosted on a server of your choice, as well as a deployment service if you'd like to host with Observable.

Deploying to Observable

Deploying to Github Pages

Github Pages provides a hosting service that's easy to integrate with your Github repository and any Github Actions needed to deploy your project.

In our example, we deployed our project to Github Pages in a few simple steps.

First, we need to specify a REDIVIS_API_TOKEN to support Redivis client library authorization in the live deployment. We used "action secret" within the Settings of the Github repo, which sets an environment variable REDIVIS_API_TOKEN to our appropriately scoped token, which is completely private while our repository remains public.

• Checkout the code

• Set up a Node.js environment, with a specific node version (20.6)

• Install our python dependencies (via pip,specified in requirements.txt)

• Build the static files for the site

• Deploy the files (in the generated dist folder) to Github Pages via a specific branch (gh-pages)

This guide demonstrates using a Redivis workflow to train an ML model on a set of images stored in a Redivis dataset.

Workflow objective

This is an example workflow demonstrating image classification via Convolutional Neural Networks. It imports an example dataset containing several thousand test and training images of cats and dogs, with which we can train and evaluate our model.

This workflow is heavily adapted from its initial publication at: https://gsurma.medium.com/image-classifier-cats-vs-dogs-with-convolutional-neural-networks-cnns-and-google-colabs-4e9af21ae7a8

1. Explore data

We can go to this dataset to browse it's tables to understand the structure of the data it contains.

We see three tables here, and all of them are file index tables. That means that each table contains an index of the files (unstructured data) this dataset contains, sorted by the folder the administrator uploaded them into. We can click on the Files tab of the dataset to see each file individually, and click on it to see a preview.

This dataset has three groupings of files:

• Training images (we will use these to build the model)

• Test images (images not included in the training set that we can verify the model with)

• Example file types (unrelated to this workflow)

If we click on the Tables tab, and click on the training images table, we can see high level information about this set of files. We can see that there are 25,000 files, and when we click the Cells tab, all of the file names we can see end in .jpg. We can hover on these to see a preview of the image, and we can click on the file_id variable to see a preview of the image with more information.

2. Create a workflow

At the top of this dataset page we can click the Analyze in workflow button to get started working with this data.

You can add this dataset to an existing workflow you already have access to, or create a new workflow to start from scratch.

3. Define a training set of images

We will use transforms to clean the data, as they are best suited for reshaping data and will quickly output new tables we can continue to work with.

Define training set

We need to start by defining the training set, which conceptually means the set of images we know are cats and know are dogs to train the model on. Information about whether an image is a cat or dog is in the file name, so we need to pull it out into a new variable we can more easily sort on.

We want to include only some of our training set images into the training set we use to train the model, since we want to leave some aside to validate the model. So here we want to include exactly 5000 cat images and 5000 dog images. So we we will create a new variable rank and filter on it so that we only keep the first 5000 images of each type.

The final step in the transform is deciding which variables we want in our output table. We will keep our new boolean variable is_cat to use later, along with the file_id and file_name

With everything in place we can run this transform by clicking the Run button in the top right corner.

4. Sanity check the output table

Now that we have created a new table, we can inspect it to make sure our steps accomplished what we expected them to.

Click on the output table below the transform to view it. We can see that it contains 10,000 records which is exactly what we expected

We can also inspect each variable further. If we click on the is_cat variable we can see that there are 5000 true values and 5000 false values, which shows that our filtering was successful. We can also validate that the method we used to determine if an image is a cat or a dog worked by clicking on the Cells tab. Here we can see that records marked True have "cat" in their file name, and when we hover on the file_ID value to see a preview, the image clearly contains a cat.

Since this table looks like we expect we can move on to the next step! Otherwise we'd need to go back to the initial transform to change our inputs.

5. Define validation set of images

We need to create a set of image files separate from our training set where we know if the image contains a cat or dog. This will be used to validate the model training.

Create a new transform and take all the same steps as we did in the previous transform, but we will change the filter to keep images ranked 5001-7500, rather than 1-5000.

We will keep the same variables as we did in our training model, and then run this transform.

When we run this transform and inspect the output table we see what we expect here as well. There are 5000 total files and we can validate a few of them visually on the Cells tab.

6. Training the model in a notebook

When you create the notebook for the first time it will start up. Notebooks must be running to execute code.

Install packages

Redivis notebooks come with many common packages preinstalled, and you can install additional packages by clicking the Dependencies button and importing libraries in the code.

Since this notebook contains only public data we can install packages at any time, but for restricted data notebooks do not have internet access and packages can only be installed when they are stopped.

import keras
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
from keras.preprocessing.image import ImageDataGenerator
from keras.models import Sequential, Model
from tensorflow.keras.optimizers import RMSprop
from keras.layers import Activation, Dropout, Flatten, Dense, GlobalMaxPooling2D, Conv2D, MaxPooling2D
from keras.callbacks import CSVLogger
from livelossplot.keras import PlotLossesCallback
import efficientnet.keras as efn
import redivis
import os

Load training and validation sets

Newly created notebooks come with standard code to import the Redivis library and reference the source table in a pandas dataframe within the notebook. For this example we will remove this sample code to import data according to our library's parameters.

TRAINING_LOGS_FILE = "training_logs.csv"
MODEL_SUMMARY_FILE = "model_summary.txt"
MODEL_FILE = "cats_vs_dogs.h5"

# Data
path = f"{os.getcwd()}/cats_and_dogs/"
training_data_dir = path + "training/"
validation_data_dir = path + "validation/" 
test_data_dir = path + "test/" 

Define model parameters

This is where we will heavily rely on our selected libraries to build the model.

# Hyperparams
IMAGE_SIZE = 200
IMAGE_WIDTH, IMAGE_HEIGHT = IMAGE_SIZE, IMAGE_SIZE
EPOCHS = 20
BATCH_SIZE = 32
TEST_SIZE = 30

input_shape = (IMAGE_WIDTH, IMAGE_HEIGHT, 3)

# CNN Model 5 (https://towardsdatascience.com/image-classifier-cats-vs-dogs-with-convolutional-neural-networks-cnns-and-google-colabs-4e9af21ae7a8)
model = Sequential()

model.add(Conv2D(32, 3, 3, padding='same', input_shape=input_shape, activation='relu'))
model.add(Conv2D(32, 3, 3, padding='same', activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2), padding='same'))

model.add(Conv2D(64, 3, 3, padding='same', activation='relu'))
model.add(Conv2D(64, 3, 3, padding='same', activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2), padding='same'))

model.add(Conv2D(128, 3, 3, padding='same', activation='relu'))
model.add(Conv2D(128, 3, 3, padding='same', activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2), padding='same'))

model.add(Conv2D(256, 3, 3, padding='same', activation='relu'))
model.add(Conv2D(256, 3, 3, padding='same', activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2), padding='same'))

model.add(Flatten())
model.add(Dense(256, activation='relu'))
model.add(Dropout(0.5))

model.add(Dense(256, activation='relu'))
model.add(Dropout(0.5))

model.add(Dense(1))
model.add(Activation('sigmoid'))

model.compile(loss='binary_crossentropy',
            optimizer=RMSprop(learning_rate=0.0001),
            metrics=['accuracy'])

with open(MODEL_SUMMARY_FILE,"w") as fh:
    model.summary(print_fn=lambda line: fh.write(line + "\n"))

# Data augmentation
training_data_generator = ImageDataGenerator(
    rescale=1./255,
    shear_range=0.1,
    zoom_range=0.1,
    horizontal_flip=True)
validation_data_generator = ImageDataGenerator(rescale=1./255)
test_data_generator = ImageDataGenerator(rescale=1./255)

# Data preparation
training_generator = training_data_generator.flow_from_directory(
    training_data_dir,
    target_size=(IMAGE_WIDTH, IMAGE_HEIGHT),
    batch_size=BATCH_SIZE,
    class_mode="binary")
validation_generator = validation_data_generator.flow_from_directory(
    validation_data_dir,
    target_size=(IMAGE_WIDTH, IMAGE_HEIGHT),
    batch_size=BATCH_SIZE,
    class_mode="binary")
test_generator = test_data_generator.flow_from_directory(
    test_data_dir,
    target_size=(IMAGE_WIDTH, IMAGE_HEIGHT),
    batch_size=1,
    class_mode="binary", 
    shuffle=False)

Model training

This is where we will train the model we just built using the image data we cleaned.

# Training
model.fit_generator(
    training_generator,
    steps_per_epoch=len(training_generator.filenames) // BATCH_SIZE,
    epochs=EPOCHS,
    validation_data=validation_generator,
    validation_steps=len(validation_generator.filenames) // BATCH_SIZE,
    callbacks=[PlotLossesCallback(), CSVLogger(TRAINING_LOGS_FILE,
                                            append=False,
                                            separator=";")], 
    verbose=1)
model.save_weights(MODEL_FILE)

Evaluate model results

Now we will use the validation set to see how well our model works

# Testing
probabilities = model.predict_generator(test_generator, TEST_SIZE)
for index, probability in enumerate(probabilities):
    image_path = test_data_dir + "/" +test_generator.filenames[index]
    img = mpimg.imread(image_path)
    plt.imshow(img)
    if probability > 0.5:
        plt.title("%.2f" % (probability[0]*100) + "% dog")
    else:
        plt.title("%.2f" % ((1-probability[0])*100) + "% cat")
    plt.show()

Next steps

Overview

Redivis workflows contain powerful tools to reshape and analyze data but if you prefer to export data into a different workflow you can easily do so. Redivis systems use open source tools and common formats to make the transition as easy as possible.

1. Check export restrictions

Some datasets on Redivis have export restrictions which prohibit or limit removing data from the system.

You can check this by going to the dataset page and clicking the top right button Manage access, or by right-clicking on a dataset in a workflow and selecting the View access option.

The bottom section of this access modal defines any export restrictions in place. It might not have any restrictions, might be completely restricted, or might have some limited options for export.

If there are limited options for export, this section will detail the available locations and any restrictions on using those (such as the size of the table being exported). These restrictions are enforced automatically when you try to take actions within the system.

If there are no options for export, you still have the option to work with your data in a Redivis workflow, where you can reshape and clean your data using transforms, and then analyze it in notebooks.

2. Prepare your data

If you would like to export the entire table, you can skip this step.

Otherwise, we recommend that you use transforms to cut your data down to a smaller size and reshape it into the table format you need for analysis before exporting it. Especially for large datasets, Redivis tools are created specifically for these purposes and work seamlessly in the browser with no additional setup.

3. Export data

You can open the Export modal to initiate an export from any table. You can do this on the dataset page by right clicking a table and selecting the Export menu option, or by right clicking on any table in a workflow and selecting the same option.

Here you can see all the options available for your table to export.

If your table has export restrictions set by the data owner, options will be disabled here. In the case where your table does not meet export requirements but the data owner allows exception requests, you will see the option to do so here.

Download

The first tab of this modal gives you different options for formats to download your data. Most common data formats are supported, including csv, json, avro, parquet, SAS, Stata, and SPSS. Select the format you'd like and click the Download button to start downloading the file.

Programmatic reference

You can reference this file from your computer or another computational environment using the Redivis Python and R libraries. This modal gives specific information on how to reference this dataset, and you can reference our docs for other options.

Integrations: Google Looker Studio

This is a free Google dashboard visualization program you can directly link your table to. Use their point and click interface to build common visuals that will update as the underlying table updates.

Integrations: Google Cloud Storage

You can export your table directly into a Google Cloud Storage bucket that you have access to.

Integrations: Google BigQuery

You can export your table directly into a Google BigQuery project that you have access to.

Next steps

Cite datasets in your publications

If the work you're doing leads to a publication, make sure to reference the dataset pages from datasets you've used for information from the data administrators on how to correctly cite it.

Overview

Datasets are a core component of Redivis. Consisting of documentation, metadata, and tables, datasets allow you to store, version, and distribute a wide variety of data.

Anyone with a Redivis account can create a dataset in their workspace, and organization administrators can upload one to an organization via the administrator panel.

1. Create the dataset

All datasets must have a name (unique to datasets for the user / organization).

You can set up your dataset in whatever order you'd like, but we recommend following the order below when getting started.

2. Upload data

This data might be in a tabular format (.csv, .tsv, .sas etc.) or more rarely in unstructured data such as images and text files.

Tabular data

If you haven't already worked with data in a workflow, we strongly recommend exploring that before creating dataset so you can understand how researchers will work with your data.

When you're ready to upload data, we have broken that out into a separate guide.

Unstructured data

For unstructured data, go to the Files tab where you can upload files from your computer or another location your data is stored via an integration. You can put these into folders and create index tables to better keep track of.

Note that any files uploaded here can't be transformed in the workflow tool or queried across Redivis (which require the table format).

Make sure any files you upload here contain this dataset's data. Any files with information about the data (such as data dictionaries or usage guides) should be uploaded as documentation on the Overview tab.

3. Edit metadata

It's easy to feel "done" after uploading your data, but documentation and metadata are essential to the usability of your dataset. Moreover, rich metadata will improve the discoverability of your dataset by providing more information and terms to the Redivis search engine.

Metadata can always be updated after your dataset has been released. While good metadata are essential, it can be a time consuming and iterative process, so you might prefer to provide some basic content initially, and then improve it over time.

Dataset metadata

On the overview tab of the dataset, you can provide an abstract, detailed documentation blocks, supporting files and links, and subject tags for the dataset.

Table metadata

Variable metadata

4. Create a sample

If your dataset is particularly large, or if you want to control access to a sample of the data separately from the whole dataset, you should configure sampling on your dataset. This will allow researchers to work with a 1% sample of the data during initial exploration, and allow you to grant access to the sample independently of the full dataset.

To update the dataset's sample configuration, click on any table, and then click "Configure sample". When configuring the sample, you can generate a random sample for each table, or sample on a particular variable that is common across tables. If researchers will be joining tables across your dataset, it is highly recommended that you sample on that common join variable so that researchers can contain a consistent 1% sample as they work with your data.

5. Configure access

Before releasing your dataset, it is important to define who can access the dataset and what the procedures are for applying and gaining access. Click the Configure access button on the top of the page to set up the access configuration.

Datasets owned by organizations have more options for access than datasets owned by users.

Access levels

Dataset access has five levels:

1. Overview: the ability to see a dataset and its documentation.

2. Metadata: the ability to view variable names and summary statistics.

3. Sample: the ability to view and query a dataset's 1% sample. This will only exist for datasets that have a sample configured.

4. Data: the ability to view and query a dataset's tables, and work with them in workflows.

5. Edit: the ability to edit the dataset and release new versions.

Access levels are cumulative. For example, in order to gain data access you will need to have gained metadata access as well.

Usage rules

Even with data access, you may want to limit what other users can do with your dataset. Currently, you can configure export restrictions that limit:

• The download location (e.g., to prevent researchers from downloading to their personal computer)

• The download size, in bytes and/or rows

• Enforce admin approval before any export

Editors

You may also add additional dataset editors to help upload data and provide metadata content. These editors will be able to create and release new versions, and will have full access to the underlying data, though they cannot add other users, modify the access configuration, or bypass the dataset usage rules.

If the dataset is hosted by an organization, all administrators of the organization will be able to the edit the dataset as well as its access configuration.

Permission groups

5. Release the dataset

Congratulations! Your dataset is ready to be released and utilized by the research community. But first, it is highly recommended that you validate and audit your dataset beforehand. Take a look at the number of rows, variables, and uploads in each table. Validate some of the variable summary statistics against what you expect. And to be truly thorough, add the dataset to a workflow and run some queries as if you were a researcher. Catching a mistake now will prevent headaches down the line if researchers uncover unexpected discrepancies in the data.

When you're confident that you're ready to go, click the "Release" button on the top of the page. If the button is disabled, hover over it to understand what issues are currently preventing you from releasing.

After clicking the button, you'll be presented with a final checklist of tasks. When you click the Release version button, the dataset will be immediately released and available to all users with access.

This dataset is now also considered Published. If you need to pause all activity and access to this dataset, you can return to this page in the future and Unpublish it temporarily.

6. Make updates as new versions

How to work with versions when updating a dataset:

• Any edits to the data content in tables will need to be released as a new version.

• Edits to the dataset information, table information, or variable metadata can be made on the current version (or historic versions) and will be live as soon as it's saved.

• Edits to the dataset name and access configuration will always affect all versions.

Creating the next version

All data within a dataset is encapsulated in discrete, immutable versions. Every part of the dataset except for the name and access settings are versioned. All tables in a dataset are versioned together.

After releasing the first version of the dataset, you can choose to create a new version at any time by clicking the button in the top right "Create next version". This version will be created as vNext, and you may toggle between this and historic versions at any time.

Subsequent versions always build on the previous version of the dataset, and changes made in the next version will have no affect on previous versions. Alongside modifications to the dataset's metadata, you may create, update, or delete any of the previous version's tables.

Replacing vs appending data

When uploading data to a previous table, you can choose whether you want to append these new uploads to your existing data, or replace the entire table with the new data.

Version storage costs

Redivis computes row-level diffs for each version, efficiently storing the complete version history in one master table. This allows you to regularly release new versions and maintain a robust version history without ballooning storage costs.

Next steps

Start working with your data

Once your dataset is released, bring it into a workflow to transform and analyze it leveraging lightning fast tools from your browser.

Overview

Any type of file can be uploaded to a Redivis dataset, and we support previews for the most common file types. These data can be analyzed in notebooks within workflows.

Note that if you have tabular data we strongly recommend uploading it as a table (rather than a file) so you and your researchers can take advantage of our extensive toolkits for previewing and manipulating tabular data.

1. Locate the data you want to upload

If importing, you'll want to get the relevant external account configured to your Redivis account before getting started.

The import tools allow for multiple uploads, so it's helpful to have them all in the same place.

2. Upload files

When you're ready click the Upload files button on the Files tab of the dataset page to get started and select the location your files will be coming from.

If you select "Computer" then you will get a browser window where you can choose the file, files, or folder you want to upload from. Choosing a different location will bring up the corresponding mechanism to select your files.

Click the "Upload" button to start the process. If you are uploading from your computer you'll need to wait on this screen until it's complete. Closing your window will end the upload process. If uploading from a linked source, you can close this window to allow this process to continue in the background.

3. Manage files and folders

You can click on the file name in the Files tab to preview the file and see its information.

You can also go to the Tables tab to view the index table for the folder you've uploaded files to. View the cells and hover or click on the file ID variable to see the preview here. This table can be used in Transforms within a Workflow to work with files on a large scale.

If you want to rename a folder you can do so on the Files tab by right clicking on the folder name in the right bar.

You can also create new folders and move files between two existing folders from this bar. When moving between folders, you can use a conditional statement to only move some files that match your conditions.

Next steps

Continue uploading your dataset

Great metadata makes your dataset useable. Complete your metadata, along with configuring access, creating a sample, and releasing this version.

Overview

Redivis offers extensive tools for previewing tabular data and transforming it in workflows, but the data needs to be uploaded correctly in a tabular format for researchers to utilize these tools.

1. Locate the data you want to upload

If importing, you'll want to get the relevant external account configured to your Redivis account before getting started.

The import tools allow for multiple uploads, so no need to combine files together before importing them, but it's helpful to have them all in the same place.

2. Create tables

On your newly created dataset, the first step to uploading data is to create one or more tables that the data will be uploaded to.

The data files you currently have may or may not be how you want to store them on this dataset, so it's important to think about your data's structure before getting started.

For example, if you have multiple files that all follow the same schema, we strongly recommend uploading them as one table (for example, if you have a separate table for each year, or a separate table for each state, but the structure of each is the same). In the example of one file per state, this would allow researchers to query across all states, skipping the first step of doing up to 50 joins. Additionally, you generally shouldn't split out tables for performance reasons — even when querying billions of records, Redivis will execute in seconds.

When you're ready click the Create new table button on the Tables tab of the dataset page and name your table to get started.

3. Upload tabular file(s) to create a table

To get started uploading, choose the data source. By default this is your computer, but you can choose any option from the dropdown menu.

Next, choose the file(s) or enter the paths of the file(s) you want to import.

If you select multiple files here, they will be automatically appended in this single table on upload based on common variable names. If a variable is missing in some of the files, that's ok, it will just be recorded as null for all records in that file.

Once your files are selected, click the Import button. If the files are coming from your computer, you might need to wait until they are finished uploading to the browser before they can be imported into Redivis.

4. Verify uploads

As you upload files, you will see an overview of any files' progress and can click to view each file's data and additional information.

Once all uploads have completed, you can inspect the table (representing the concatenation of all of your uploads). Make sure to check the summary statistics and other analytical information to validate that the data are as you expected.

If you have more files to upload you can click the Manage imports button on the right side of the table at any time (up until releasing this version of the dataset).

Next steps

Continue uploading your dataset

Great metadata makes your dataset useable. Complete your metadata, along with configuring access, creating a sample, and releasing this version.

Overview

Organizations are where administrators centrally manage datasets and who can access them. All organization administration happens from the Administrator panel which administrators can access from the Organization home page.

1. Add other administrators

You can add any Redivis account to be an administrator of your organization in the Administrator panel from the Settings tab.

Note that all administrators will have full access to the organization and its datasets, including the ability to modify access rules, approve access requests, and add or remove other administrators, including you.

2. Build your organization's presence

To help brand your organization and best communicate the organization’s purpose to others, you'll want to fully populate and brand the Organization home page.

In the Administrator panel on the Settings tab / Public profile section you can customize:

• The organization's full name

• An organization description which will appear at the top of your about section, and in any institution searches

• A custom brand color

• The organization's logo

• The data portal's cover photo

• A header image that will appear in the top left of users' windows when on your data portal

• Header links to other resources, such as organization-specific documentation, events, and other resources.

• Rich text and images that will display in the main content section of the data portal. A place to provide high level information about your organization and how researchers should user your data portal.

3. Create datasets

You'll need to create datasets to upload and distribute data to your researchers. You can create datasets from the Datasets tab of your Administrator panel.

Here are some pieces to consider before getting started:

Dataset structure

It's worth thinking through your broader data management strategy. Assess the data that you currently have — what format are the files in? Where are they located? How are they organized?

Consider what datasets you'll want to create. Datasets on Redivis are made up of one or more semantically related tables, and each table can contain data from one or more uploaded files. Moreover, all permissions happen at the dataset level — if you grant a user access to a dataset, they will have access to all of its tables.

Citations and DOIs

If you’d like to issue DOIs for your organization’s datasets, you can configure your organization to issue DOIs in the Advanced section Settings tab. DOIs will make it easier for researchers to cite your data when they publish, and allow you to track publications using your data as well as their broader impact.

4. Build access systems

Before you can configure access to a dataset, you'll want to consider and build out the relevant access systems for your organization as a whole.

Direct access & Member access

If you have a smaller organization where you know everyone working with your data, you might want to consider directly granting access to datasets to specific members, or granting access to everyone who is a member.

This can be set up on any dataset individually or you can create a Permission group on the Permission group tab to save the entire configuration of access levels and assign it to multiple datasets.

Requirement-based access

In many cases, however, you’ll want to develop more process-driven access control systems and gather additional information from your members To do so, you can set up requirements that must be completed by members or their studies, and approved by an administrator, before gaining access.

You can build requirements on the Requirements tab of the administrator panel, and save groups of them together as Permission groups on the Permission groups tab which you can assign to multiple datasets.

Export restrictions

Finally, you may want to control how your data is exported from Redivis, if at all. While members with data access will always be able to work with data in Redivis workflow, you can limit their ability to export data. For example, you may choose to allow exports only to certain environments, or only upon administrator approval, or not at all.

5. Manage members and studies

As members and studies apply to access data, you may need to take action to approve (or reject) their access requests. You can find all of your pending access requests wherever you see an alert on the administrator panel.

Members

As users join or organization and apply to work with data, they will appear in the Members tab of your administrator panel. You can also invite anyone to become a member by adding them to the members list.

Studies

Studies allow for researchers to organize their work around a conceptual study, or research project, and for you to grant access in the context of that study. Any studies that are working with your organization's data will appear in the Studies tab of the administrator panel automatically, and you can approve any requirements for studies here.

6. Gain insight into usage

Redivis provides tools to help you keep an eye on your organization and how its data is being utilized.

You can filter, sort, and change the columns for any list in your administrator panel, and download the corresponding report for a snapshot of current information. You can also dig into the logs and generate analytical reports for more information.

Next steps

Expand your reach

Many insurance claims datasets on Redivis contain information about enrollment, detailing the periods of time when an individual represented in a dataset was covered by an insurance plan. If you intend to characterize patients based on their insurance coverage (or lack thereof) during certain key events (procedures, diagnoses, etc), it's often important to identify periods of continuous enrollment for each individual – and capture each continuous enrollment period for each patient in a single row.‌

These claims datasets describe enrollment information in multiple discrete rows per patient, each corresponding to patient per month. However, an overall continuous enrollment period may be broken up across rows into months or other non-uniform chunks, so we'll employ the following process to combine multiple sequential rows into a single row with one start date and one end date, describing one continuous period.‌

We want to create a final table with 3 rows for patient 1 to account for gaps in enrollment in March and May of 2012, and 1 row for patient 2. Our desired output has a row for each continuous period per patient, shown below:

The variable names in this example are not actual MarketScan variable names. But, with appropriate data access, you can see the real variables used in the first transform of the Redivis example workflow by hovering over the (renamed) variables patient_id, enrollment_start_date, and enrollment_end_date in the Keep section.‌

(1) Add start and end of adjacent periods to each row

Throughout this example, we'll create variables that partition the dataset by patient identifier (here, patient_id) to ensure that each patient is processed individually. But, to account for the fact that a single patient may have many periods of continuous enrollment, each spanning many months, we need to identify the correct start (from enrollment_start_date) and end (from enrollment_end_date) of a continuous period out of multiple rows and capture them in a single row.‌

First, we create a partition variable lag_end_date using the lag method, which will order each row by enrollment_start_date in a given patient_id partition and copy the previous row's enrollment_end_date value into the each row. We also create lead_start_date using lead, to copy the following row's enrollment_start_date value into each row.‌

These methods generate values which tell us how close the preceding and following enrollment periods are with respect the each row's enrollment period.

(2) Select rows with start and end of continuous periods

In a second, downstream transform we'll create new variables, which will use the above lead and lag values to identify which rows correspond to the beginning and end of a continuous enrollment period.‌

First, we'll compare enrollment_start_date and lag_end_date to find the difference (in days) of the start of each period and the end of the previous period. and compare enrollment_end_date and lead_start_date to find the end of the period.‌

We see via diff_lag_end_enrollment_start (created using the date diff method) which rows describe an enrollment period directly following the previous period, and which rows describe an enrollment period with a larger gap since the previous period. We also create diff_enrollment_end_lead_start to identify gaps between an enrollment period and the next period.

Next, we'll encode booleans from our difference variables to simplify further filtering. We'll identify rows corresponding to the start of a continuous period as those with a diff_lag_end_enrollment_start value of either NULL (the row is the first period in the partition) or greater than 1 (the row comes after a gap in enrollment). And we identify rows corresponding to the end of a continuous period as those with a diff_enrollment_end_lead_start value of either NULL (the row is the last period in the partition) or greater than 1 (the row comes before a gap in enrollment).‌

This leaves us with either 1 or 2 rows corresponding to a continuous enrollment period. If a continuous period spans multiple rows (months, in this case), we'll have 2 rows (a start row and an end row). But if a period only spans one row, we'll have both start and end captured by that 1 row. In our example, the row containing the middle (neither start nor end) of the 2012-06-01 to 2012-08-31 enrollment period for patient 1 was dropped. We can also ignore our intermediate lead..., lag..., and diff... variables, since our final processing step will only consider a row's is_start_continuous_period and is_end_continuous_period values.

(3) Add end date of following enrollment periods to each row and collapse

Finally, we want to collapse our table to ensure 1 row per continuous enrollment period per patient. Since we have only rows corresponding to start and end of continuous periods, we create another partition variable lead_end_date (in the same transform is fine, since this step will happen after the previous filter) which copies the enrollment_end_date value of the following row on to each row.‌

We can also use the partition row filter to keep only rows with is_start_continuous_period as true, since our lead_end_date has copied over the end date of the continuous enrollment period, contained in each row's following row.

We end up with a table where each and every row contains both the start date of the continuous enrollment period and the end date of that continuous enrollment period.

A final processing step captures the correct end date of a continuous period. We now have only rows whose enrollment_start_date value contains the start of a continuous period, but these rows fall into two categories:‌

• First, we have the rows that do not also correspond to the end of an enrollment period (where the is_end_continuous_period value is false). For these, we want to look at lead_end_date, the end date of the next row, which represents the final date of the continuous period, since we filtered out all the intermediate rows above.

• Second, we have which also correspond to the end of an enrollment period (where the is_end_continuous_period value is true) – in this example, if the continuous period was only 1 month. For these, the row defines a period (in this example, 1 month) that also contains the end date, so we just get the enrollment_end_date from that same row. Note that the lead_end_date value is incorrect in this case, since the next row contains the start of the next continuous period, or NULL if the period falls at the end of a partition.

We capture the above logic in a new variable enrollment_end_date_continuous, created in an additional transform, since our previous operation involved a partition.

We end up with a final table below, containing the patient identifier, and the start date (renamed to enrollment_start_date_continuous for consistency, and end date of each continuous enrollment period.

Overview

One of the main tasks of organization administrators is to manage who has access to datasets. Redivis has tools to support any level of complexity in data systems for your restricted data and makes it easy for organization administrators to define, manage, and audit access rules across these various use cases.

1. Consider your organization's access needs

Do you mostly have public datasets? Will you have a small group of researchers with access to pretty much every dataset? Will you need to gather information from researchers before they use your datasets? Do you have high risk data that can't ever leave the system? Or some combination of the above?

Redivis supports access systems for all of these configurations – for organizations of varying sizes – but you'll want to use different tools to achieve these very different goals.

For example, if you have few datasets, you might find that permission groups for bulk managing access rules aren't particularly helpful. Or if you have a small group of researchers who you all know closely, you might not need to set up process-driven access rules through requirements, and can instead grant access to certain researchers on an individual basis.

We'll walk through all the tools available to organizations for managing access, but you'll likely want to pick and choose from what's available to meet your organization's specific needs.

2. Understand access levels

Dataset access has five levels:

1. Overview: the ability to see a dataset and its documentation.

2. Metadata: the ability to view variable names and univariate summary statistics, but not to retrieve any identifiable information or multivariate relationships from the data.

3. Sample: the ability to view and query a dataset's 1% sample. This will only exist for datasets that have a sample configured.

4. Data: the ability to view and query a dataset's tables, and work with them in workflows.

5. Edit: the ability to edit the dataset and release new versions.

Access levels are cumulative. For example, in order to gain data access you will need to have gained metadata access as well.

We strongly recommend making your dataset's metadata as open as possible. This will reveal variable names and aggregate summary statistics, but will not allow researchers to view, query, or export the raw data in any way.

Being able to see metadata greatly improves researchers' discovery experience, and allows them to better assess a dataset's utility upfront, and even reduce your administrative workload. If researchers can understand a dataset before applying for access, they'll be submitting fewer access applications to datasets that are ultimately a dead end.

3. Understand available access tools

Membership

Anyone wanting to apply for access to restricted datasets in your organization must first be a member. You can configure whether memberships are restricted to certain identity providers (such as your institutional login), and whether they are approved automatically or require administrator review. You also have the option to configure access to datasets to "All members."

Direct access

Permission granted directly to a researcher to instantly gain access to a dataset at a specific level. Researchers can also request access to datasets with this configuration.

Example usage: a dataset that will only be shared with a small number of people who are already known to administrators

Member requirements

A form for members to fill out. This can be set to require approval from an administrator or be automatically approved. It can also have an expiration date. These are global to your organization and when assigned to multiple datasets a user will only fill it out one time.

Example usage: a demographic form gathering researcher personal information, or a data use agreement signed PDF

Study requirements

Similar to requirements, but instead of each user needing to fill them out individually, only one requirement needs to be completed for the entire study, which can include multiple researchers. A single researcher may also have multiple studies. Each study working with the dataset will need to fill out its own study requirement, and any queries or exports of the data will be tied to that study.

Example usage: a funding proposal for a research project

Data export restrictions

A rule defining that a dataset can only be exported to a specific export environment, as configured on the Administrator panel Settings tab.

Example usage: limiting exports to a specific server environment

4. Create requirements

If you want to work with requirements, you'll want to get started making them and planning out how they will work across datasets.

Perhaps you want one requirement for all members to fill out about their field of research, which is necessary to gain access to any of your datasets, but another 4 requirements with different data use agreements that will apply only to their specific datasets.

To get started, go to the Requirements tab of the administrator panel and click the New requirement button. You will need to start by selecting if this will be a Member requirement or a Study requirement.

You can use the form builder to collect different information, including standard form responses, file uploads, and e-signatures.

5. Create permission groups

A permission group is an access configuration that can be assigned to multiple datasets and managed centrally.

You don't need to use permission groups, and it might not make sense to do so if each of your datasets has a different access configuration and you aren't using requirements. But if you have any overlap between datasets and want to enforce consistency, or want to use requirements you'll want to make one.

To get started, go to the Permission groups tab of the administrator panel and click the New permission group button.

This interface requires you to set an access paradigm for each access level of the dataset.

Perhaps you will set the overview access level to be public, the metadata access level to be available to all members, and data access level to be direct access (meaning you will have to directly grant access to users, or respond to their requests for access).

Or perhaps you will set the overview access level to public, and assign multiple requirements to the metadata and data access levels (meaning that anyone who is approved for all of the requirements will automatically gain that access level).

For any case you can assign data export restrictions here and choose whether you want to manage access to the dataset's sample (if it exists) differently than the full data.

6. Assign access permissions to datasets

Finally you'll need to apply these access permissions to actual data!

Open any of your datasets and click the Configure access button on the top of the page. This configuration setup will look very similar to configuring the permission group.

You can either create a custom configuration here, or you can assign this dataset to one of the Permission groups by clicking the dropdown menu in the top right corner of this modal.

This is also where you will manage any direct access requests for the dataset.

7. Verify your setup

As an administrator of this organization, you will have access to all datasets no matter what your access configuration is or what requirements you have filled out.

We recommend checking that your access system works as you expect by either looking at a dataset while you are logged out or in your browser's incognito mode, or by making a second Redivis account using a different email address that is not linked to your administrator account.

Next steps

Grant access to data

You have some shiny access systems, but they won't work if you don't approve user requests for access.

Expand your reach

Overview

Redivis datasets are a great place to host data for interrogating in a workflow, but you can also edit the underlying data.

Perhaps you found an issue with the source, or want to restructure it, before making it available to others. Redivis has all the tools you'll need to do this in a versioned, step-by-step process with transparency into the changes you've made.

1. Upload the raw data

2. Add this dataset to a workflow

Create a new workflow and add this dataset to it.

If you want to share the data transformation process with others for transparency, you can make this workflow public in the share modal. (Due to the way Redivis works, still only people with access to the underlying data will be able to see the data in the workflow even though the workflow is public.)

3. Transform data

Select the table you want to make changes to and create a new transform.

Example edit 1: Rename and retype a variable

Example edit 2: Recoding the values of a variable

Create a new variable with the same name. "Keep" this variable and "Discard" the original variable. Select the method as "Case (if/else)" and create the conditions you want to recode to.

Example edit 3: Add a new variable

This example shows a new variable with today's data as the "date_uploaded" but you could create any variable you want from the data in the table. Perhaps a aggregation that would be helpful to see with this data? Or the sum of multiple other variables?

You can transform this table however you want, using the Graphical interface or SQL code. Make sure to move all variables in this table from the "Discard" section to the "Keep" section except for any variables that have been replaced and will be left behind (in this case the store_and_fwd_flag from the source table).

Validate that this new table looks correct by looking at the output table below this transform!

Make changes to any tables in this dataset in this same wokflow.

4. Re-upload the finished tables

Go back to the original dataset, and create a new version.

Open a table that you made changes to in your workflow, and click to "Import data."

Choose your merge strategy as "Replace" since you want to replace the existing table with the new one you created.

Select the data source as "Redivis" and type in the information from the workflow you made your changes. The sequence here will be:

1. Your user name

2. The name of the workflow (underscores replace spaces)

3. The table name (underscores replace spaces)

For example: username.testworkflow.table_name

Once it's done uploading you can validate that the data looks like it's supposed to.

Now you can close this table and repeat this process for any other tables in this dataset you've edited.

Once this is finished, you can add the link to the worfklow into this dataset's documentation or release notes along with a note on the changes. Remember that if you made the workflow public, anyone who has data access to this dataset can view the changes you made.

Next steps

Start working with your data

Once your dataset is released, bring it into a workflow to transform and analyze it leveraging lightning fast tools from your browser.

Overview

One of the strengths of Redivis is that researchers can apply for, and gain access to, data on the same page as where they discover the dataset. Moreover, all access applications are tracked, both for administrator's reference and to provide transparency and timely notifications to researchers.

Responding to access requests in a timely manner creates a great environment for researchers and takes full advantage of these automatic systems.

1. Configure notifications

You can configure multiple email addresses to receive an email when a request for access is made to your organization, as well as the frequency you receive them. Especially if you are a smaller organization that isn't getting access requests every day, we very strongly recommend that you have at least one administrator receiving emails so you know when to respond to a request.

2. Approve membership requests

Users must be approved as members of your organization before any other access requests can be approved.

When a membership request is pending you will see an Alert (!) on the Members tab of your workspace. Click on any member with an alert to see their pending submission awaiting approval.

Pending members will have an Approve / Reject button as soon as you click on their page, alongside the authentication information they submitted when requesting to be a member. It is important to verify that this person has a genuine email that seems consistent with who they say they are.

3. Approve requirement submissions

As users ask for approval you will receive an alert (!) in the administrator panel where your attention is needed.

Member requirements and export restriction exceptions will appear on the Member tab.

Study requirements will appear on the Studies tab.

Direct access requests will appear on the Datasets tab.

When all of the pending requests are resolved, the alert will disappear.

For each request you will have the option to approve or reject the submission. You can always come in later to revoke the approval if needed. You might also see that the member has submitted updates to their existing approval. In that case the approval is still valid while updates are pending, unless it is otherwise revoked or expired.

4. Leave comments for clarity

Both administrators and members have the option to leave a comment on any requirement submission. The act of leaving a comment will trigger an alert for administrators, or a notification for members.

Especially when you are rejecting or revoking a requirement submission, we recommend leaving a comment to let the member know what they did wrong, or what they still need to do to gain access.

5. Audit access

It's often a good idea to double check which researchers have access to your datasets.

In order to audit a particular member's access to your organization's datasets, double click on that member, and navigate to the Access overview tab. This will list all of your organization's datasets, and the member's corresponding access to each.

In order to audit all members that have access to a particular dataset, click the filter icon to the right of the member search bar. This will allow you to filter all members by their access to, and usage of, your organization's datasets.

Next steps

Expand your reach

E.g.: I want the date of the first and last hospital admission for every patient in my table

Let's take an example table which contains all hospital visits in one facility over many years. Each record contains information about what happened in the visit, including a patient id to identify who was present. An individual patient might be represented in one or many records in this table since each record represents a separate hospital visit.

Let's say that for our research we want to find the first and last encounter — or more formally, the min and max encounter date for each patient.

In this example we will use a partition to create a new variable since we want the rest of the data to remain unchanged.

Variable concepts

Conceptually we will define our aggregation variable as the date of the encounter since that is where we want to look for the min and max values.

We will define our partition as the patient id since we want to find the min or max for each patient. This will limit the min or max value scan to all values of our aggregation variable across each unique value of our partitioned variable (patient id).

Defining the new variable

For this particular case, we want to find the MIN and MAX of the encounter_date of each patient; in other words, partitioned on patient_id . This will create two new variables, min_encounter_date and max_encounter_date for each patient.

When we run this query, note that no records will be dropped; rather, these new variables will be have consistent values for any given patient_id .

Now, let's say we only want to keep records which contain each patient's first and last encounter. We can do this easily be creating a new row filter:

We can also generalize this approach to find the Nth encounter for each patient — take a look at the RANK, DENSE_RANK, and ROW_NUMBER methods.

And if we want to be more specific on our partition (e.g., for the first / last encounter in a given calendar year), we can always apply additional variables (year ) when defining our partition.