The Present and Future of Cloud Storage: A Conversation With Qumulo

MASV and Qumulo recently announced an integration to provide high-performance, enterprise-grade cloud data ingest to the Cloud Native Qumulo (CNQ) cloud storage platform.

The new integration allows professionals with massive data needs to combine MASV’s blazing-fast, reliable transfers with Qumulo’s fully managed file management services on AWS.

Doesn’t get much better than that, right? Well, hang on: Maybe it can.

That’s because Qumulo is also full of extremely smart people with their fingers on the pulse of the latest trends surrounding cloud and data storage and performance. We recently sat down with Daniel Beres, a Technical Strategist at Qumulo, to get his take on where things are and where they’re headed when it comes to cloud storage and big data workflows.

Let’s get into it.

Note: This interview has been lightly edited for length and clarity.

• How does Qumulo help handle massive files in storage, especially for M&E and AI workflows?
• What are the biggest storage bottlenecks slowing down production teams today?
• Why is the MovieLabs 2030 Vision important for the future of production?
• What is Qumulo Cloud Data Fabric, and how does it help with security and performance?
• What trends are you seeing around on prem, fully cloud, and hybrid deployments?
• Why are hybrid workflows still so prevalent?
• What are some common misconceptions about cloud storage?
• How does Qumulo’s file system help improve the performance of S3?
• How is AI affecting storage and related workflows?
• How does Qumulo address the issue of cumulative scalability, specifically around training AI?
• Over the next five years, what do you think the trends in storage are going to be?
• What, in your opinion, makes MASV different than other file transfer solutions?
• C'est un Wrap...

How does Qumulo help handle massive files in storage, especially for M&E and AI workflows?

Daniel Beres: What Qumulo has done is looked at the industry as a whole and focused on different areas of it, such as M&E, AI, and other mass offerings of unstructured data. And that’s the key here: Unstructured data. When we’re looking at where storage has gone over the years, it’s shifted from database and structured information into something more file-based or unstructured.

And one of those key areas is M&E. If we look at what was shot with video recorders 20 years ago, the amount of data that they actually put onto storage wasn’t that much. But with 4K, 8K, and even more resolutions coming, we’re looking at massive amounts of data. It’s one thing to use file systems of the past, but by doing that you limit yourself—not just on performance, but scalability—and that’s where Qumulo is unique in the industry.

Blended Sense Customer Story

How MASV “unlocked” the full potential of Blended Sense’s video production shop—and helped it retain its most valuable freelancers through easy, fast, drama-free transfers.

Lire la suite >

What are the biggest storage bottlenecks slowing down production teams today?

DB: I think it’s the speed to get new assets into the editor’s hands, or the visual effects departments to start working on those as soon as possible. What we’re seeing in the industry is not so much just how much storage you have, but how fast you can bring that information into storage.

Why is the MovieLabs 2030 Vision important for the future of production?

DB: We can go back and see how security has affected different movie releases in the past, where certain elements have been leaked out to the press, and that has made a good or bad impression on customers actually going to see these different productions.

So what we’re looking at from a security standpoint is making sure that only those people that need access to those video assets, those elements, can have access to it. Or that when one studio shares information with another, they only share what they need between the two, so no two people can actually go out and do something that might compromise security.

What is Qumulo Cloud Data Fabric, and how does it help with security and performance?

DB: Cloud Data Fabric is our implementation of a global namespace. Global namespaces in the past have been a layer on top of file systems, whether they access file systems as blocks of information or as files of information. It really didn’t play well into being able to share data across the world at speed.

So we integrated our version of a global namespace into the fabric of our cloud storage, our Cloud Data Fabric. This allows you to access those blocks of information—not just files—anywhere in the world, whether it’s an edge location, whether it’s a centralized location, or maybe different studios. Each one of those can have direct access to the blocks you need to build out those assets for those creatives, whomever needs it.

What trends are you seeing around on prem, fully cloud, and hybrid deployments?

DB: We see a lot of hybrid installations out there. It’s not just that they need to be in the cloud. It’s not that they need to be on prem. They need to be in both. And by having that ability to be in both places at the same time, without having to replicate data, copy data, move that data around, and literally be able to store it where it’s needed most, means everybody has access to it. It really makes the Cloud Data Fabric we’ve implemented much broader in global reach.

Why are hybrid workflows still so prevalent?

DB: Here’s what we see: The hybrid nature of our customers using both cloud and on prem isn’t so much about having access to the data wherever it lies. They want to have access to the data where they’re at. And having that ability at any time is really what makes a hybrid installation of cloud data fabrics the way to go.

What are some common misconceptions about cloud storage?

DB: One of them is that it costs a lot of money to move to the cloud. And yes, if you take what you’re running on prem today and literally lift-and-shift directly into the cloud, you’re going to pay a lot more. You need to look at how cloud can actually give you the economies of what they’ve built out in their environments. One of those is the ability to scale on demand at any time based on your workflow. One of those could be a burst render: You don’t need 300 or 1,000 machines out there waiting for you to move data in and then actually use them, and then sit idle when you’re done. You can use the ability inside of the cloud to dynamically assign and build out these different types of scenarios for burst render. You can expand from 300 nodes to maybe 1,000 nodes at any time. This allows you to not spend the money when you’re not using those systems.

Cloud Native Qumulo does the exact same thing: If you only are using a certain amount of performance every day, don’t expand it. Don’t add additional nodes into our infrastructure. Literally just run it as-is until you need to do more heavy lifting, and at that point expand the performance side. Because the back end cloud storage is always going to be S3—which, by its nature, only expands when you use it and contracts automatically when you’re not.

How does Qumulo’s file system help improve the performance of S3 cloud storage?

DB: Cloud Native Qumulo helps remove some of the bottlenecks you might have with S3. Some of those could be the file-to-object ratio. A lot of customers will actually write one file to one object when they write it into S3— and that’s okay if you’re storing archive data, but when you’re looking at it from a transactional standpoint, say you’re rendering out a particular element, or maybe you’re doing online editing of different aspects inside a timeline. Well, you need that iterative process of being able to scrub through, move blocks around, change what a file is actually doing at any point in time, which an object-based system doesn’t allow you to do.

So we’ve broken that up. We’re actually writing blocks of information into S3. This is different. We do this on purpose, because not all blocks are the hottest blocks that you need access to at any time. We’re having those hot blocks stay in our caching mechanism that’s built into CNQ. Having those hot blocks available to anybody at any time keeps you from constantly going back and forth to S3 to grab those blocks of information. And we have this ability to understand how you’re using the file system—so it’s not so much about the blocks you’re using right now, but what are the next blocks, or the other blocks of information, not just the files, but the blocks inside of those files that you might need access to at the same time that you’re accessing the ones you’re using right now.

And why is that important? Well, you’re not constantly bringing in files and flushing them out and bringing files in and flushing them out again. You’re literally only using the blocks you need. Most of those are staying in cache, and when they’re in cache, other people have the ability to use those as well. So your renders go from hours down to maybe minutes.

How has AI affected storage and related workflows?

DB: What we’re seeing is this shift in big data away from the older way of doing batch analytics, into a more modern approach of ‘let’s teach the AI to go out and analyze the data for us.’ What’s great about AI is we’re now seeing unique ways of looking at our data from video footage. We’re seeing better ways to do cloud formations or waves in the ocean. All those are being generated from massive amounts of data that we’ve acquired over the years. Some of this is seismic activity. Some of those are waveform analysis. Some are different aspects that we’ve seen in other movies that are better rendered in newer productions.

But that AI implementation we’re seeing requires massive amounts of data, and that data is sitting in silos everywhere. Most of it needs to get into a central location so we can start accessing it and doing something with it. So where I see some of the challenges is with getting that data not just uploaded, but then the ability to access it once it has been uploaded.

How does Qumulo address the issue of cumulative scalability, specifically around training AI?

DB: If we go back about 10 years, when you had to scale up performance with cloud storage at the same time to be able to get that information out, one node meant more storage inside of what you’re doing, or more storage, meaning you had to have more compute power behind it. Disaggregating those two and separating them out based on your utilization needs allows us to build either the best-performing cluster, so you can access your data, or even the cloud storage for that. These go into a model that’s more cloud-centric, meaning that if you need more ability to perform, you need to go out and analyze data because you have thousands of nodes trying to figure out whatever the new model is, great. Scale that out in the cloud with as many different resources you need to do that. But when you’re done, scale them back. But because that storage is completely separated out from there, that storage is only the amount you need at the time to do what you’re doing.

In other words, if it’s only 100 terabytes of data, don’t add more nodes for 100 terabytes of data, because you don’t need them. If it’s only 10 terabytes of data, or it’s 10 petabytes, 100 petabytes, you don’t need to have all these performing nodes on the front to be able to access that data. It’s really based on how you want to get access to it, and how fast you want to get access to it.

Over the next five years, what do you think the trends in storage are going to be?

DB: That’s a tough one to say. But what I what I’ve seen happen till now, and where I see that direction continuing, is that unstructured data is going to get even larger in scale.

When we looking at our customers, whether they’re in the AI industry, whether they’re into oil and gas or healthcare, everybody is storing more and more information at higher resolutions so that they can better analyze or do something with that data. That just means more storage. That means growth in that storage, but it also means how fast can I get access to that data, and how fast can I upload that information from wherever I’m actually creating that data from? All those are going to play into the future. It’s going to be huge.

What, in your opinion, makes MASV different than other file transfer solutions?

DB: Where I see MASV taking it a little bit differently is around how they do it, and the information that they can assign to data as it is being uploaded. The other part of it is being able to distribute a single file to multiple places at the same time, if they need to, so stakeholders can work on whatever they just uploaded.

And from the aspect of what I’ve seen in the industry, having this ability to—if you are connected by a cellular device out in the field, and you need to get a file up, and making sure it gets delivered completely—you’re not just waiting, and you make sure it’s actually up in the cloud or wherever you need to deliver that to. MASV has really taken the biggest step in that direction, along with making sure that those files get delivered correctly and to the right individuals when they need to be.