Hybrid Cloud for AI: Keeping Data On-Premises While Compute Bursts to the Cloud

AI teams face a practical mismatch: their datasets keep growing, while the specialized compute needed to train models can be costly and difficult to procure. Daniel Valdivia, an engineer at MinIO, describes one way to handle that tension: keep valuable data on infrastructure a company controls, then rent cloud compute for the periods when a workload needs it.

This approach, often called cloud bursting, depends on treating storage and compute as separate choices. Instead of keeping large datasets in the cloud indefinitely or buying enough accelerators to cover every possible peak, a company can store data locally and start cloud instances with GPUs or other specialized hardware for training runs. When the work ends, it can shut those instances down.

Valdivia has worked in engineering for more than 18 years. His earlier roles included development and project management at Mexvax and Expert Creations, mobile application development and programming analysis at Oracle, and senior application development at ServiceNow. He joined MinIO as an engineer in 2019, working with organizations planning storage and compute for AI workloads.

Put storage at the center of the design

Valdivia’s starting point is that data grows steadily and holds much of a company’s operational value. At petabyte scale, cloud storage charges can make it more practical to bring datasets back to a company’s own facilities, where they can be kept behind its firewalls. The decision depends on the company’s costs and requirements, but storing everything in the cloud isn’t automatically the economical choice as volumes rise.

That doesn’t mean every part of the system needs to move on-site. Compute hardware changes quickly, Valdivia argues: a costly server bought today may be overtaken by a more capable version within three years. Cloud capacity gives teams room to respond to temporary demand without building their entire infrastructure around peak usage.

For AI, this division can also address GPU availability. Organizations may not be able to obtain accelerators when they need them, and cloud instances offer another route to specialized compute. A team can retain private data on its own network, move or stream the data required for a training run, and pay for cloud compute only while the pipeline is active.

The economics turn on how well the pipeline uses that rented capacity. GPUs and tensor processing units can be expensive, so leaving them idle while data is prepared or decisions are made wastes paid compute time. Valdivia’s recommendation is to organize the workflow around object storage, performing suitable preprocessing before the intensive training stage begins.

Stream data through training

With this design, teams clean and prepare data, then return it to object storage in a form that training jobs can read. Compute workers fetch batches as needed, rather than relying on one local filesystem or a single machine to hold the working dataset. Valdivia gives the example of 10,000 GPUs streaming data from an object store, with the next batch loaded into memory as each worker is ready.

The broader point is architectural: a pipeline should support parallel access to shared data and make it possible to stop compute when a run finishes. A workflow built around a local filesystem can constrain teams to one experiment or pipeline at a time, he says. Cloud resources are more useful when the software can distribute work, read from shared storage and restart or recover through checkpoints.

Checkpoints also give training jobs a place to save progress and results. Models and intermediate outputs can be written back to object storage, where later stages can access them. Inference can use the same general pattern: deployment machines load models from storage and begin serving, without requiring every model to live permanently on each machine.

These choices have operational consequences. Teams need to design data formats, preprocessing and checkpointing around the training system they plan to use. They also need to consider network throughput and latency: streaming data from storage only helps if workers can receive it quickly enough to stay productive. The interview doesn’t provide benchmark results or cost comparisons, so the gains will depend on a workload’s data volume, access pattern, infrastructure and cloud pricing.

Choose placement by workload

For Valdivia, data protection is the primary consideration when choosing where to store information. The storage layer should tolerate failures, guard against data loss and fit the company’s budget. Compute can then be split according to its needs: traditional analytics tools such as Spark and Presto may run on-site, while AI jobs that need specialized hardware can use cloud capacity.

This mix can also suit irregular workloads. If demand spikes, a company can add cloud compute and remove it when the extra capacity is no longer needed. That can avoid buying infrastructure sized for occasional peaks, a trade-off that becomes more significant as organizations grow. The key is to make sure the workload can move to rented capacity without creating a more expensive data transfer or operational bottleneck.

Data movement deserves particular attention. Valdivia says cloud providers don’t charge for bringing data in, while moving it back out can incur egress costs. For repeated training over the same datasets, he describes placing MinIO storage at the edge of the cloud, close to the compute. Data can be sent there for training, and the cloud-hosted setup can be removed after the work is done, reducing transfer distance and latency.

That arrangement still requires teams to account for the cost and time of initially loading datasets, as well as any data they need to retrieve later. The source gives no specific pricing or transfer volumes; the practical calculation will vary by provider and workload. Repeatedly using the same data may make nearby storage attractive, while a short, infrequent job may call for a different balance.

Beyond AI training

The same pattern can apply to large analytics jobs that are too substantial or sporadic to run continuously on a company’s own machines. Valdivia points to Amazon Elastic MapReduce as one service through which users can start configured compute nodes, run an analysis pipeline and tear down the setup afterward.

That makes temporary cloud capacity relevant to traditional big data processing as well as machine learning and newer AI pipelines. In each case, the common feature is a workload that interacts with large datasets but doesn’t necessarily need dedicated peak capacity all the time.

Valdivia describes a car company operating 180 servers and a multi-petabyte environment with one person managing the setup. He presents it as an example of storage infrastructure becoming manageable with a small team. It’s a striking illustration, though the interview doesn’t provide details about the systems, workload or support arrangements behind it, so it shouldn’t be treated as a staffing benchmark.

Repatriation and storage choices

Valdivia says cost pressures had led many companies to return data from cloud services to on-site infrastructure in the year before the interview. He connects that trend with growing interest in large language models and generative AI, which require substantial datasets for companies to adapt them to their work.

He describes a shift from hybrid cloud adoption between 2020 and 2022 toward more companies bringing data back, enabled by storage software that can run across cloud providers and private infrastructure. In one example, a large streaming service moved from Amazon to its own systems. It used fast NVMe storage for recently updated programming and a separate system with spinning drives for material users might want later.

The example points to a tiered approach: keep frequently needed content on faster storage and less frequently accessed content on lower-cost drives. The interview doesn’t identify the service or quantify its savings, and it doesn’t establish that the design fits every streaming workload. It does show how control over storage placement can be part of a broader cost and performance decision.

Valdivia’s larger argument is that storage has become a basic layer of application infrastructure. Cloud services made it easier for teams to access capabilities they might not have had time to build themselves, but companies are now weighing those conveniences against the cost of holding ever larger datasets there. His “road” analogy captures the point: applications need storage in the way cars need roads, and teams want software that lets them build and manage that foundation on their own terms.

For AI practitioners, the takeaway is to plan storage, data preparation and compute as one pipeline. Keeping data on-site can make sense at scale; cloud accelerators can provide flexible bursts; and object storage can connect the two when workloads are designed to stream data and save checkpoints. The choice still depends on transfer costs, access speed, hardware availability and how often a workload runs.