> For the complete documentation index, see [llms.txt](https://docs.redivis.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.redivis.com/api/client-libraries/redivis-js/examples/uploading-data.md).

# Uploading data

### **Create a new dataset**

```javascript
import redivis from 'redivis';

// Could also create a dataset under an organization:
// const dataset = redivis.organization('my_organization').dataset('some_dataset');
const dataset = redivis.user('your_username').dataset('some_dataset');

// publicAccessLevel can be one of 'none', 'overview', 'metadata', 'sample', 'data'
await dataset.create({ publicAccessLevel: 'overview' });
```

### **Create a table and upload data**

```javascript
import redivis from 'redivis';

const dataset = redivis.user('user_name').dataset('dataset_name', { version: 'next' });

// Create a table on the dataset. Datasets may have multiple tables
const table = await dataset.table('table_name').create({ description: 'Some description' });

// Upload a file to the table.
// You can create multiple uploads per table, in which case they'll be appended together.
await table.upload().create(
  './data.csv', // A file path (in Node.js), Blob, File, ArrayBuffer, ReadableStream, rows, or arrow Table
  {
    type: 'delimited', // Inferred from the file name if not provided
    removeOnFail: true, // Remove the upload if a failure occurs
    waitForFinish: true, // Wait for the upload to finish processing
    raiseOnFail: true, // Throw an error on failure
  },
);
```

### **Upload data from the browser**

```javascript
import redivis from 'redivis';

const table = redivis.user('user_name').dataset('dataset_name', { version: 'next' }).table('table_name');

// e.g., from an <input type="file">
fileInput.addEventListener('change', async () => {
  const file = fileInput.files[0];
  await table.upload().create(file, {
    onProgress: ({ bytesUploaded, totalBytes }) => {
      progressBar.value = bytesUploaded / totalBytes;
    },
  });
});
```

### **Upload rows**

```javascript
import redivis from 'redivis';

const table = redivis.user('user_name').dataset('dataset_name', { version: 'next' }).table('table_name');

// Arrays of row objects (and arrow Tables) are uploaded as newline-delimited JSON
await table.upload('my_rows').create([
  { id: 1, name: 'a' },
  { id: 2, name: null },
]);
```

### **Upload non-tabular files**

```javascript
import redivis from 'redivis';

const dataset = redivis.user('user_name').dataset('dataset_name', { version: 'next' });

// Non-tabular files must be uploaded to file index tables
const table = await dataset.table('my_files').create({ isFileIndex: true });

// Upload all contents in a directory (Node.js only)
await table.addFiles({ directory: '/path/to/directory/' });

// Upload specific files
await table.addFiles({
  files: [
    { path: '/path/to/file.png' }, // File name will be "file.png" (Node.js only)
    { path: '/path/to/other/file.png', name: 'other_file.png' }, // File name will be "other_file.png" (Node.js only)
    { data: 'Hello world', name: 'hello_world.txt' }, // Data can be a string, Blob, ArrayBuffer, or typed array
  ],
});

// In the browser, upload Files from an <input type="file" multiple>
await table.addFiles({ files: fileInput.files });
```

### **Upload data from an external source**

```javascript
// Assuming we get a reference to the table the same as above...

await table.upload('data.csv').create({
  transferSpecification: {
    sourceType: 'gcs', // One of gcs, s3, bigQuery, url, redivis
    sourcePath: 'my-bucket/path-to-my-file.csv',
    // sourcePath: 'https://example.com/data-file', (for sourceType == 'url')
    // sourcePath: 'project_name.dataset_name.table_name', (for sourceType == 'bigQuery')
    // sourcePath: 'owner_name.dataset_or_workflow_name.table_name', (for sourceType == 'redivis')
    identity: 'my_email@example.com', // The email associated with the data source
  },
});
```

### Stream data to an upload

```javascript
import redivis from 'redivis';

const dataset = redivis.user('user_name').dataset('dataset_name', { version: 'next' });
const table = dataset.table('table_name');

// Providing a schema with the initial request is optional (but recommended).
// If not set, the schema will be inferred based on the first batch of rows.
const schema = [
  { name: 'var1', type: 'string' },
  { name: 'var2', type: 'integer' },
  { name: 'var3', type: 'dateTime' },
];

const rows = [
  { var1: 'hello', var2: 1, var3: null },
  // dateTime must be in the format YYYY-MM-DD[ |T]HH:MM:SS[.ssssss]
  { var1: 'world', var2: 2, var3: '2020-01-01T00:00:00.123' },
];

// Reference each upload by its name, which must be unique amongst other uploads
//   for the current version of this table.
const upload = table.upload('my_stream');

// Create, or get a reference to an existing upload named "my_stream"
await upload.create({ type: 'stream', schema, ifNotExists: true });

const insertResponse = await upload.insertRows(rows);

// See REST API / uploads / insertRows
console.log(insertResponse);
```

### **Release a new version**

```javascript
import redivis from 'redivis';

const dataset = redivis.user('your_username').dataset('some_dataset', { version: 'next' });
await dataset.release({ releaseNotes: 'Initial release' });
```

### **Create a subsequent version on an existing dataset**

```javascript
import redivis from 'redivis';

let dataset = redivis.user('your_username').dataset('some_dataset');

// createNextVersion will throw an error if a "next" version already exists,
// unless the ifNotExists option is provided
dataset = await dataset.createNextVersion({ ifNotExists: true });
const table = dataset.table('table_name');

// By default, all new data is appended to the previous version of a table.
// If you'd like to replace the previous data, update the uploadMergeStrategy.
await table.update({ uploadMergeStrategy: 'replace' });

await table.upload().create('./data.csv', {
  // All additional options are optional; default values are shown here
  type: undefined, // One of stream, delimited, csv, ndjson, avro, parquet, orc, xls, xlsx, dta, sas7bdat, sav. Inferred from the file name if not provided.
  skipBadRecords: false,
  hasHeaderRow: true, // Only relevant for csv, xls(x)
  removeOnFail: false, // Remove the upload if a failure occurs
  waitForFinish: true, // Wait for the upload to finish processing
  raiseOnFail: true, // Throw an error on failure

  // The following are only relevant for delimited files:
  hasQuotedNewlines: false, // Allow newlines within cells. Setting to true will substantially reduce ingest performance.
  quoteCharacter: '"', // The character used to escape delimiters within cells. Generally a double quote in compliant CSVs.
  delimiter: undefined, // Explicitly set the delimiter, otherwise the delimiter will be automatically inferred.
});

// When all uploads have finished, release the next version
await dataset.release();
```


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.redivis.com/api/client-libraries/redivis-js/examples/uploading-data.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
