> For the complete documentation index, see [llms.txt](https://docs.redivis.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.redivis.com/reference/datasets/data.md). # Data Datasets can contain data in one of two formats, available on the corresponding tab of the dataset page. * **Tables** contain tabular data which can be quickly viewed and analyzed on Redivis using transforms and notebooks in a workflow. * **Files** can be any type of data with a basic inspection tool and available for analysis in notebooks. ## Tables All datasets that contain data will have at least one table, displayed on the **Tables** tab of the dataset page. Tables are created by data editors when they [upload tabular data](/reference/datasets/create-and-edit-datasets/import-sources.md), and users' access to these tables will be governed by the dataset's [access configuration](/reference/data-access/configuring-access.md). {% embed url="" %} Embedded view of a table with 2.7B records. Click around to interact! {% endembed %} #### Working with tables Tables can be explored from the dataset page, including the ability to view [variables and summary statistics](/reference/tables/variables.md), [cell contents](/reference/tables/cells.md), and run [one-off queries](/reference/tables/queries.md) against the table. These tables are then further utilized from within a [workflow](/reference/workflows.md). Learn more about [tables](/reference/tables.md) on Redivis. *For editors: How to import* [*tabular*](/reference/datasets/create-and-edit-datasets/import-tabular-data.md) *or* [*geospatial*](/reference/datasets/create-and-edit-datasets/import-geospatial-data.md) *data.* ## Non-tabular files Files are data entities uploaded to [datasets](/reference/organizations/datasets.md) on Redivis. Files are used to store non-tabular or unstructured data of any file type. You can view a dataset's files by selecting the **Files** tab of any dataset or [dataset node](/reference/workflows/data-sources.md) in a workflow.
#### Inspecting files To inspect an individual file, select it's name to launch the file viewer. Any file can be downloaded or referenced within a notebook, and many file types can be previewed directly within Redivis. Some files can be shown in more than one way. For example, an Excel workbook laid out as a spreadsheet or as a plain table. Use the **Preview** dropdown in the file viewer to switch between these views or to see any file's raw contents.
{% hint style="info" %} You can view [example files](https://redivis.com/datasets/yz1s-d09009dbb/files) on the Redivis Demo organization. {% endhint %} Redivis currently supports previews for the following file types: | Category | Format | Extensions | | ------------------------------------ | ------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | **Documents** | PDF | `.pdf` | | | Word | `.docx`, `.docm`, `.dotx`, `.dotm` | | | PowerPoint | `.pptx`, `.pptm`, `.ppsx`, `.ppsm`, `.potx`, `.potm` | | | Excel workbooks | `.xlsx`, `.xlsm`, `.xltx`, `.xltm` | | | Other spreadsheets | `.xls`, `.xlsb`, `.ods`, `.numbers`, `.dbf` | | | Markdown (including R Markdown and Quarto) | `.md`, `.markdown`, `.rmd`, `.qmd` | | | Jupyter notebooks | `.ipynb` | | | HTML | `.html` | | | TEI | `.tei` (and TEI documents saved as `.xml`) | | | Plain text | `.txt`, `.log` | | **Code** | Source code and configuration files, with syntax highlighting for 100+ languages | `.py`, `.r`, `.sql`, `.sas`, `.sh`, `.js`, `.java`, `.c`, `.cpp`, `.go`, `.rs`, `.jl`, `.json`, `.xml`, `.yaml`, `.toml`, `.tex`, and many more | | **Images** | Common image formats | `.png`, `.jpg`, `.jpeg`, `.gif`, `.webp`, `.avif`, `.apng`, `.bmp`, `.svg`, `.ico` | | | HEIC / HEIF | `.heic`, `.heif` | | | TIFF (including BigTIFF and Cloud Optimized GeoTIFF) | `.tif`, `.tiff` | | | JPEG 2000 | `.jp2`, `.j2k`, `.jpx`, `.jpf`, `.jpm`, `.j2c`, `.jpc`, `.mj2` | | **Audio and video** | Audio | `.mp3`, `.wav`, `.flac`, `.aac`, `.m4a`, `.ogg`, `.oga` | | | Video | `.mp4`, `.mov`, `.m4v`, `.webm`, `.ogv`, `.mpeg`, `.mpg`, `.3gp`, `.3g2` | | **Geospatial** | Vector data | `.shp` (or zipped as `.shp.zip`, `.shz`), `.geojson`, `.topojson`, `.kml`, `.kmz` | | | Raster data | GeoTIFF (`.tif`, `.tiff`) | | | GeoPackage | `.gpkg` | | | LiDAR and point clouds | `.las`, `.laz`, `.copc.laz` | | **Databases** | SQLite | `.sqlite`, `.sqlite3`, `.db`, `.db3`, `.s3db` | | | Microsoft Access | `.mdb`, `.accdb` | | **Scientific and array data** | HDF5 (including NeXus) | `.h5`, `.hdf5`, `.he5`, `.nxs` | | | NetCDF | `.nc`, `.nc4`, `.cdf` | | | MATLAB (v7.3) | `.mat` | | | NumPy | `.npy`, `.npz` | | | Zarr (zipped, including OME-Zarr) | `.zarr.zip`, `.ozx` | | | FITS (astronomy) | `.fits`, `.fit`, `.fts`, `.ftf` | | **Medical imaging and neuroscience** | DICOM | `.dcm` | | | NIfTI and other volumes | `.nii`, `.mgh`, `.mgz`, `.mha`, `.mif`, `.nrrd` | | | Brain surfaces (GIFTI, FreeSurfer) | `.gii`, `.mz3`, `.pial`, `.white`, `.inflated`, `.sphere`, `.smoothwm` | | | Tractography | `.tck`, `.trk`, `.trx` | | | Neurodata Without Borders | `.nwb` | | **Genomics and bioinformatics** | Sequences (FASTA, 2bit) | `.fa`, `.fasta`, `.fna`, `.faa`, `.ffn`, `.frn`, `.fas`, `.mpfa`, `.2bit` | | | Sequencing reads (FASTQ, Nanopore) | `.fastq`, `.fq`, `.fast5` | | | Read alignments | `.sam`, `.bam`, `.cram` | | | Variants | `.vcf`, `.gvcf` | | | Genome tracks (bigWig, bigBed) | `.bw`, `.bigwig`, `.bb`, `.bigbed` | | | Annotations and genomic tables | `.bed`, `.bedgraph`, `.gff`, `.gff3`, `.gtf`, `.narrowpeak`, `.broadpeak`, `.gct`, `.seg`, `.bim`, `.psam`, `.pvar` | | | Single-cell (AnnData, MuData, Loom, Seurat) | `.h5ad`, `.h5mu`, `.loom`, `.h5seurat` | | | Hi-C contact matrices | `.cool`, `.mcool` | | | Annotated sequences and plasmid maps (GenBank, EMBL, SnapGene, SBOL) | `.gb`, `.gbk`, `.gbff`, `.genbank`, `.embl`, `.dna`, `.sbol` | | | Phylogenetic trees (Newick, NHX, phyloXML, NEXUS) | `.nwk`, `.newick`, `.tre`, `.tree`, `.treefile`, `.contree`, `.dnd`, `.nhx`, `.phyloxml`, `.nex`, `.nexus` | | | Multiple sequence alignments | `.aln`, `.clustal`, `.sto`, `.stk`, `.stockholm`, `.msf`, `.phy`, `.phylip`, `.a3m`, `.afa` | | **Molecular structures** | Proteins and macromolecules | `.pdb`, `.ent`, `.cif`, `.mmcif`, `.mcif`, `.bcif`, `.gro`, `.pqr`, `.pdbqt` | | | Small molecules | `.mol`, `.mol2`, `.sdf`, `.sd`, `.xyz` | | **3D models and point clouds** | 3D models and scenes | `.obj`, `.stl`, `.ply`, `.gltf`, `.glb`, `.fbx`, `.dae`, `.3ds`, `.3mf`, `.amf`, `.usd`, `.usda`, `.usdc`, `.usdz`, `.vtk`, `.vtp`, `.wrl` | | | Point clouds | `.las`, `.laz`, `.copc.laz`, `.pcd`, `.pts`, `.ply`, `.xyz` | | **Archives** | Zip (including zip-based formats such as `.jar`, `.whl`, `.epub`, `.odt` and `.twbx`) | `.zip` | | | Tar (plain or compressed) | `.tar`, `.tgz`, `.tar.gz`, `.tar.bz2`, `.tbz2`, `.tar.xz`, `.txz`, `.tar.zst`, `.tzst` | Keep the following in mind when previewing files: * **Compressed files**: gzipped files (`.gz`, `.bgz`) are previewed as their uncompressed format, e.g. `calls.vcf.gz` or `data.csv.gz`. * **Files within archives**: zip and tar archives list their contents, and any file inside them can be opened and previewed like any other. * **Unrecognized extensions**: files with a missing or ambiguous extension are identified by their contents where possible (e.g. HDF5, Parquet, DICOM, or NetCDF files with a generic name). * **Large files**: many formats, including Parquet, Arrow, SQLite, HDF5, Zarr, TIFF, BAM/CRAM, VCF, and point clouds, are read in place, so only the part of the file you're viewing is loaded, even for very large files. If you have a file type that you think should be supported that isn't, please [let us know](https://redivis.com/contact)! #### Folders and index tables All files are assigned to a folder within the dataset. You can select the folder name in the right grey bar to filter this dataset's files by the folder they are in. Each folder has a corresponding index table that is present on the **Tables** tab of the dataset. These will match the folder name and have a **File index** label. In these index tables, each row of the table represents a file in that folder. | Variable name | Description | | ------------- | ------------------------------------------------------------------------------------------------------------------------------- | | `file_id` | A unique, system generated identifier for the file. Use the file\_id to reference and download specific files in your analysis. | | `file_name` | The name of the file. | | `size` | Size of the file in bytes. | | `added_at` | Timestamp for when the file was added to this folder. | | `md5_hash` | MD5 checksum of the file contents, encoded as a base64 string. |
#### Working with files By representing files within an index table, you can query and subset the files [within a workflow](/reference/workflows.md), while joining them with other tabular metadata. In this way, file index tables behave like any other tabular metadata on Redivis. To do deeper analysis of files, you can [load these files in a notebook ](/reference/workflows/notebooks/notebook-concepts.md)for further analysis. Consult the documentation for your preferred programming language to learn more: [Working with unstructured data files in Python](/reference/workflows/notebooks/python-notebooks.md#working-with-non-tabular-files) [Working with unstructured data files in R](/reference/workflows/notebooks/r-notebooks.md#working-with-unstructured-data-files) *For editors:* [*How to import files*](/reference/datasets/create-and-edit-datasets/import-files.md) --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.redivis.com/reference/datasets/data.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.