> For the complete documentation index, see [llms.txt](https://docs.redivis.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.redivis.com/reference/workflows/transforms/transform-concepts.md). # Transform concepts ## Overview A transform is a series of steps detailing a data transformation that are executed in sequential order when the transform is run. These steps and a final variable selection create an output table that can be inspected and further transformed. {% hint style="info" %} **Transforms vs. notebooks?** There are two mechanisms for working with data in workflows: [transforms](/reference/workflows/transforms.md) and [notebooks](/reference/workflows/notebooks.md). Understanding when to use each tool is key to taking full advantage of the capabilities of Redivis, particularly when working with big datasets. Transforms are better for: * Reshaping and combining tabular and geospatial data * Working with large tables, especially at the many GB to TB scale * Preference for a no-code interface, *or* preference for programming in SQL * Declarative, easily documented data operations Notebooks are better for: * Interactive exploration of any data type, including unstructured data files * Working with smaller tables, though working with bigger data is possible * Preference for Python, R, Stata, or SAS * Interactive visualizations and figure generation {% endhint %} ## Steps The majority of building a transform is choosing and completing data transformation steps. Steps represent one specific action, such as [filtering rows](/reference/workflows/transforms/step-filter.md) and [joining tables](/reference/workflows/transforms/step-join.md), that are completed in the Redivis transform interface. Every step is a fundamentally a piece of SQL code that can be examined and could be reproduced by running in any environment with the same data. When running the transform, steps will be executed in order, building off of what directly came before them. Steps have additional options in the `⋮` menu to help in the building process: * Collapse / expand * Annotate * Disable / enable * Reorder (this might change the outcome) #### Output variables The final task in constructing a transform is to select which variables you want in the output table. Use the [variable selector](/reference/workflows/transforms/variable-selection.md) in the bottom pane of the transform to choose which to keep. ## Running transforms Transforms are iterative and are designed to easily check your progress as you work. As long as a transform is valid, you can select **Run** to generate an output table. Select this output table node in the workflow tree to see the results once the run is complete. #### Invalid state The transform interface prevents you from running a transform that is invalid. This is displayed with an invalid icon on the step or the part of a step where the issue is. Invalid alerts typially trigger when some crucial information is missing. You can follow the invalid icons to find the locations you need to update. #### History and revert Every time a transform runs, a snapshot of the transform at that point in time is saved. You can select **History** to view the log of these runs, and choose any snapshot to revert the transform to that point in time. Your transform will then be in the Edited state and no changes will be made to the output table. Once you run it, this configuration will become a new snapshot at the top of the transform history list. #### Checking the output Output tables don't just exist to store data, you can use the variable statistics generated in the [table node ](/reference/workflows/tables.md)to make sure your transform did what you expected it to do. Common things to check include the total number of rows and variables in your output table, distinct and null counts of a key variable, or even running a query on the **Query** tab of the table to filter by a particular value. If you find the table isn't what you want, you can go back to the transform to make changes and rerun it. Transforms were created to be iterative! ## Additional tools for working with transforms #### Name and rename By default transforms are named with a number based on the order they were created. Renaming transforms with descriptive names allows you and your collaborators can quickly find them later on. You can rename a transform by selecting the title of the node or by choosing the Rename option in the transform `⋮ More` menu. If you have not specifically renamed the connected output table, renaming a transform will also rename this output table. #### Change the source table All transform nodes have one source table. You can join in multiple tables, but the source table designation will affect how the joins execute. You can change the source table from the `⋮ More` menu in the transform. Note that if the new source table has different variables your transform might become invalid until you update it. #### Split and combine transforms All transforms can be split at the step level into two different transforms by selecting **Split** in any step's menu. Additionally, two transforms can be combined into one by right clicking on a table to **Remove** it. You might want to split a transform above a tricky step to see what the output table would look like at that point in the process. This can be a key tool in troubleshooting any issues and understanding what might be going wrong. After splitting a transform to check an output table, the next logical step might be to combine these two transforms back into one again. Or perhaps you have a string of transforms which you no longer need the output tables for and want to reduce the size of a workflow. #### Copy, paste and insert As you go, you may want to take certain data cleaning actions multiple times or move them around in your process. You can right click any transform in the workflow tree to see options for copying it or deleting it. Right click on any table node to paste the copied transform. You can insert transforms above other transforms. Steps and parts of steps can also be copied and pasted within the same transform or across multiple transforms. #### View SQL code All transforms generate SQL code, which will execute when the transform is run. To view the code for a particular step, select **View SQL** in the step menu. You can also see the SQL for the entire transform from within the `⋮ More` menu of the transform. You can convert this code into a SQL query step and run it to achieve the same outcome or edit the code directly. ## SQL Concepts Redivis uses SQL as the basis for data transformation for many reasons. It is designed for working quickly and efficiently with large tables, for ease of learning concepts and basic executions, and is highly reproducible. While Python, R, Stata, and SAS are commonly used in data analysis, they are not suited to working with large tables. Executing queries on millions of records can take hours and, in many cases, fail. We have provided a point and click interface for using transforms to make using a perhaps unfamiliar coding language easier. #### Fundamentals * Tables are data structures containing rows and variables. * Variables represent the fields in a table, with every row holding a value for each one. * Rows represent individual observations, with a value recorded for each variable. * SQL queries always operate on data from one or more tables and output a single data table. #### Building a query When you are using the interface or writing SQL code, it is best to start by figuring out what the output table should look like. What are the variables? What should each row of the table represent? Once you know what the output table should look like, you can work backwards and formulate the steps of a query to build the shape you'd like. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.redivis.com/reference/workflows/transforms/transform-concepts.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.