Switch data sources

When creating a workflow, you may want to start with a data source that is convenient for development and later switch to the data you ultimately want to analyze. For example, you might build an analysis using a small sample, switch to a new version of a dataset, or develop using synthetic data before running your workflow on restricted data.

The steps for switching data sources depend in how the data is represented in your workflow.

1. Open your workflow

Open the workflow containing the data you want to change and navigate to the data source node in the workflow tree. From here, you can choose how you'd like to change the data used in your workflow.

2. Switch the data source

Depending on what you are trying to accomplish, there are several ways you can switch the data source:

Switch to a different version

Changing versions can be useful when a dataset has been updated and you want to rerun your existing analysis using the newest version, or when you want to reproduce an analysis using a specific version. Redivis will indicate when a new version is released.

To run your workflow using a different version of the same dataset, select the version number next to the dataset title.

Alternatively, you may select Version in the top resource bar.

You can navigate between different versions of the dataset and review the changes to each version. You'll want to check that the version you are switching to contains the tables and variables your workflow expects. If the schema has changed, you may need to update your workflow before running it.

After making necessary updates, rerun the workflow to apply the changes and materialize the updated results.

Switch from a sample to the full dataset

When you create a workflow with data that has a sample configured, Redivis will use the sample by default. If you have been developing your workflow using a sample, you can switch to the full dataset when you're ready to run you analysis on all of the available data.

In addition, access to a sample does not necessarily grant access to the full dataset. A dataset owner may configure different access rules for sample and the full dataset. If you do not have access to the full dataset, you'll need to apply for access before you can switch to it.

To switch from a sample to the full dataset, select the 1% sample next to the dataset version, beside the dataset title.

Alternatively, you may select Sampling in the top resource bar.

Here, you can move between the sample and full dataset.

For exceptionally large datasets, you may want to develop your workflow using a sample, even if you have access to the full dataset. Redivis is designed to support fast computation even on large datasets, but working with a sample can further improve computation speed and allow you to iterate more quickly while developing your analysis. Switch to the full dataset once your workflow is ready for the final analysis.

Replace with a different data source

You can also replace a data source with an entirely different dataset. This can be useful when you develop a workflow using one dataset and want to run it on another dataset with a compatible structure.

For example, you might:

• Work with high risk data: Develop and test a workflow using a synthetic version of a dataset, then switch to the real dataset for the final analysis. An administrator can also run the workflow on the restricted dataset and provide the results to the researcher without granting access to the underlying data.

• Work with standardized data models: Develop a workflow using one dataset that follows a standardized data model, such as OMOP, and apply it to another dataset using the same model.

To replace the dataset, open the More menu in the resource bar, and select Replace dataset.

Search for the dataset you would like to replace it with. If you do not have data access to the new dataset, you will be unable to view its restricted tables and will need to apply for access to see and use them in your workflow.

Then align the workflow's tables to the corresponding tables in the new dataset. This updates the workflow to use the new data while preserving the transforms and analyses you've already created.

3. Verify your workflow

Downstream nodes will be stale to indicate that an upstream change has been made. After switching the data source, review your workflow to make sure the new data is being used as expected.

In particular, check that:

• The expected tables are available.

• The variables used by your workflow are available.

• Variable names and data types are compatible.

• Any filters or other assumptions in your workflow still apply.

4. Run your workflow

Once you've verified the new data source, run your workflow to update its outputs. To update all nodes at once, select Run all stale nodes from the workflow's Map menu.

Always check that the resulting output looks as expected!

Next steps

Share and collaborate

All Redivis notebooks support real-time collaboration, allowing multiple editors to edit and run cells in a running notebook. When another editor is active in a notebook, you will see a colored cursor associated with them, similar to a Google Doc.

Share your workflow to work with collaborators in real time and make it public so that others can fork and build upon your work.

Cite datasets in your publications

If the work you're doing leads to a publication, check for proper citation information on the reference pages of the datasets used in your workflow.