> For the complete documentation index, see [llms.txt](https://docs.redivis.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.redivis.com/reference/workflows/notebooks/compute-resources.md). # Compute resources Notebooks on Redivis provide a highly flexible computational environment. Notebooks can be used for anything from quick visualizations to training sophisticated ML models on a large corpus of data. Understanding the compute resources available and when to modify parameters can help you take full and efficient advantage of the high-performance computing resources on Redivis. ## Default (free) notebooks The default notebook configuration on Redivis is always free, and provides a performant environment for working with most datasets. The computational resources in the default notebook are comparable to a typical personal computer, though likely with better network performance. The default free notebook configuration offers: * 2 vCPUs (Intel Ice Lake or Cascade Lake) * 32GB RAM * 100GB SSD: * IOPS: 170,000 read | 90,000 write * Throughput: 660MB/s read | 350MB/s write * 16Gbps networking * No GPU (see [custom environments](#custom-compute-environments) below) * 6 hr max duration * 30min idle timeout (no code is being written or executed) ## Custom compute configurations When you need additional computational resources, you can choose a custom compute configuration for a notebook. This enables you to specify CPU, memory, GPU, and hard disk resources, while also giving you control over the notebook's max duration and idle timeout. In order to customize the compute configuration for your notebook, select **Edit compute configuration** in the notebook start modal or toolbar.

### Compute platforms Custom machine types are organized into five compute platforms, each designed for different types of workloads. General purpose and GPU configurations are suitable for most workloads, while the other platforms are intended for more specialized computational requirements.

Compute platform	When to use	Example workloads
General purpose	You need a balanced amount of CPU and memory.	Data exploration, statistical analysis, visualizations
Compute optimized	Your workload is primarily CPU-intensive.	Simulations, parallel processing, computationally intensive transformations
Memory optimized	Your workload needs to keep large amounts of data in memory.	Large in-memory datasets, memory intensive joins
Network optimized	Your workload depends heavily on network throughput.	Large data transfers, network-intensive processing
GPU	Your workload can take advantage of GPU acceleration.	Machine learning, deep learning, SPU-accelerated computing

{% hint style="info" %} If you're unsure which platform to choose, start with a general purpose machine. You can monitor resource utilization while your notebook is running and adjust the configuration if your workload requires additional resources. {% endhint %} ### Disk types Custom compute configurations can use different disk types depending on the selected machine type. When available, Local SSD is generally recommended because it provides high disk performance and is generally cost-effective. Other disk types may be appropriate depending on your machine type and workload.

Disk type	Example workloads	Why
Local SSD	• Training machine learning models that repeatedly read training data • Training large scale transformations that create temporary intermediate files • Processing large collections of images, text, or other files	Very fast local storage that is useful when the analysis frequently reads and write temporary data.
Networked hyperdisk	• Large analyses that need high-performance storage but cannot use local SSD • Workflows that process substantial datasets over an extended period	Provides high-performance networked storage when local storage isn't available or appropriate.
Networked SSD	• Statistical analyses on moderately large datasets • General data processing • Analyses that read and write data regularly but don't require maximum disk throughput	Good performance for general workloads without needing the highest disk performance.
Networked balanced disk	• Smaller or less disk-intensive analysis • Exploratory analysis • Workflows where CPU or memory, not disk access, is the primary bottleneck	A reasonable option when disk performance is less important.

Disk availability varies by machine type. When Local SSD is not available or is not appropriate for your workload, select the disk type that best matches your performance and storage requirements. Google Cloud provides additional information about [disk types and their performance characteristics](https://docs.cloud.google.com/compute/docs/disks). ### Provisioning model Custom compute configurations can use different provisioning models depending on the selected machine type. Standard provisioning provides regular access to available compute resources, while Spot provisioning uses excess Google Cloud capacity at a reduced cost. #### Spot VMs Spot VMs provide access to compute resources at a reduced cost by using excess Google Cloud capacity and can be up to 90% cheaper than the standard provisioning model. They can be a cost-effective option for workloads that are flexible and can tolerate interruptions, such as exploratory analysis, batch processing, and machine learning workloads that can resume from a saved state. For some machine types, Spot VMs are available as an optional lower-cost alternative to standard compute. Other machine types are only available with Spot VMs. Because spot VMs rely on available capacity, they may be interrupted or stopped when Google Cloud needs the underlying resources. When this occurs, the notebook will be shut down. Redivis continuously caches the notebook's `/out` directory, so save any computational state needed to resume your analysis there. ### Machine availability Machine availibility depends on current resource capacity in Google Cloud. Occassionaly, a requested machine type may not be immediately available. When this occurs, Redivis automatically retries provisioning the machine for approximately 30 minutes across available zones. If the machine cannot be provisoned after repeated attempts, an error is displayed. If availibility issues persist, consider selecting a different machine type or using Spot provisioning when available. ### Custom machine costs All custom machines have an associated hourly cost, charged by the second. This cost is determined by the then-current price for that machine configuration on Google Cloud. In order to run a custom machine, you must first purchase [compute credits](/reference/your-account/compute-credits-and-billing.md), and have enough credits to run the notebook for at least 15 minutes. If you run low on credits and don't have [credit auto-purchase](/reference/your-account/compute-credits-and-billing.md#auto-purchase-compute-credits) configured, you will receive various alerts as your credits run low. Ultimately the notebook will shut down when you are out of credits. ### Monitor resource utilization You can monitor your notebook's resource utilization using the metrics bar while the notebook is running. If CPU, memory, GPU or disk usage consistently approaches the available capacity, you may need to select a configuration with additional resources for that analysis. Resource indicators change color when usage approaches available capacity.

## Maximizing notebook performance All notebooks on Redivis use either Python, R, Stata, or SAS. While Redivis notebooks are highly performant and scalable, the coding paradigms in these languages can introduce bottlenecks when working with very large tabular data. If you are running into issues with performance we suggest: * Using [transforms](/reference/workflows/transforms.md) to clean and reduce the size of your data before analyzing them further in a notebook. When possible, this will often be the most performant and cost-efficient approach. * Adjust your programming model to load data lazily or on-disk to avoid exceeding memory limits. See suggestions for working with larger tables in [Python](/reference/workflows/notebooks/python-notebooks.md#working-with-larger-tables), [R](/reference/workflows/notebooks/r-notebooks.md#working-with-larger-tables), [Stata](/reference/workflows/notebooks/stata-notebooks.md#working-with-larger-tables), and [SAS](/reference/workflows/notebooks/sas-notebooks.md#working-with-larger-tables). * Adjust the compute resources in your notebook. This may help to resolve these bottlenecks depending on what is causing them! #### Some quick rules of thumb: * < 1GB: probably doesn't matter, use what suits you! * 1-10GB: probably fine for a notebook, though a transform might be faster. * 10-100GB: maybe doable in a notebook, but you'll want to make sure to apply the right programming methodologies. Try to pre-cut your data if you can. * \>100GB: You should probably cut the data first in a [transform](/reference/workflows/transforms.md), unless you really know what you're doing. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.redivis.com/reference/workflows/notebooks/compute-resources.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.