Cleaning tabular data

Redivis datasets are a great place to host data for interrogating in a workflow, but you can also edit the underlying data.

Perhaps you found an issue with the source or want to restructure it before making it available to others. Redivis has all the tools you'll need to do this in a versioned, transparent, and step-by-step process.

1. Upload the raw data

If you haven't yet, upload and release the data you want to work to Redivis. A personal dataset or one belonging to an organization works just as well.

2. Add this dataset to a workflow

Create a new workflow and add this dataset to it.

If you want to share the data transformation process with others for transparency, you can make this workflow public in the share modal. Only people with access to the underlying data will be able to see the data in the workflow even if the workflow is public.

You can follow along in this workflow Demo tables edits.

3. Transform data

Select the table you want to make changes to and create a new transform.

Tip: you can reference the names of specific uploads within your table using upload pseudo-variables.

Use this transform to edit this table. Some common actions include:

Example edit 1: Rename and retype a variable

Create new steps to rename and retype any variables you'd like to update.

Example edit 2: Recode the values of a variable

Create a new variable with the same name. Keep this variable and Discard the original variable. Select the Case (if/else) method and create the conditions you want to recode to.

Example edit 3: Add a new variable

This example creates a new variable containing a date, but you could create any variable you want from the data in the table. Perhaps an aggregation that would be helpful to see with this data? Or the sum of multiple other variables?

You can transform this table using the graphical interface or SQL code. When you are done making edits, move all the variables you want to retain from the Discard section to the Output section. Replaced variables should be left behind, such as the store_and_fwd_flag from the example.

Validate that this new table looks correct by looking at the output table below this transform.

Continue making changes to other tables in this dataset using this workflow until you are satisfied with the results.

4. Re-upload the finished tables

After completing your changes, go back to the original dataset and create a new version.

Open any table you made changes to in the workflow, and select Import data.

Choose Replace for the merge strategy to replace the existing table with the new version.

Select Redivis as the data source. The table or file reference should contain:

1. Your username

2. The name of the workflow (underscores replace spaces)

3. The table name (underscores replace spaces)

For example: username.testworkflow.table_name

Once it's done uploading, validate that the data looks correct.

Close this table and repeat this process for any other tables you've edited.

After you have uploaded the edited tables, you can add the workflow's link to this dataset's documentation or release notes, detailing the changes. If you made the workflow public, anyone who has data access to this dataset can view the changes you made.

Now release this version! The new version of the data contains the edited tables. Anyone using this data in a workflow will see that this dataset has a new version next time they open a workflow. You can see the release notes and the updated table of the Demo tables dataset live.

Redivis stores data as compactly as possible to minimize storage costs. Only records new to all versions will increase storage needs. If you would like to delete the first version of the data, open the version modal in the dataset editor and select Delete version.

Next steps

Start working with your data

Once your dataset is released, bring it into a workflow to transform and analyze it, leveraging lightning fast tools from your browser.

Learn more in the Analyze data in a workflow guide.