Automated File Transfer with MASV and Docker
If you’re the type of person who works with large files on the regular, you will benefit from automated file transfer. You could be an editor who needs to improve their video workflow (more on this below); a photographer sending hi-res images to clients; or even a game developer packaging up and sharing FBX or OBJ files.
Automated file transfer is about convenience. Moving files and waiting for uploads/downloads eats away at the day. My philosophy is, no task that is performed multiple times a day should have to be manually completed (if you can avoid it).
A Quick Intro
My name is David Postlethwaite and I love writing code to do my job for me. As a former DevOps Engineer—now Developer Advocate at MASV—my day to day consisted of writing the automatic workflows for code building, code deploying, running integration tests, and optimizing images for cloud delivery. One of the main reasons I hopped on at MASV was because they embraced that same automation methodology. That includes the wonderful TransferAgent tool that the team has built with our automated file transfer capability.
What is MASV TransferAgent
MASV is a file sharing service focused on fast delivery of heavy files and automated file transfers. We have a web app, a desktop client, and an API. Users kept reaching out to use our API, and over time we realized that we needed an easier solution for people who just wanted to get going. That was when we created TransferAgent.
The MASV TransferAgent is a tool designed to abstract our web API for our users. It supports uploading and downloading multiple packages at the same time without our users having to figure out queues, bandwidth limitations, and all sorts of funky filesystem quirks. TransferAgent worked so well, we decided to build our desktop client around it. Making it so we can have a quick turnaround time for our users and the best possible version of our tooling out in-market.
What are Docker Containers?
What is a Container? | App Containerization by Docker
One of the things that a new automator will run into is Docker and containerization. Below is a blurb from Docker:
A container is a standard unit of software that packages up code and all its dependencies so the application runs quickly and reliably from one computing environment to another.
You can think of a container as its own self-contained application. As long as there is a container engine (in our case Docker) that self-contained application will run the same way every time no matter the hardware it is on. Your team and mine no longer have to worry about incompatibilities between Windows, Mac, and Linux. Containers let us focus on building the best application. Your team only has to care about how to use the application (and not where you are running it).
What drew us to adopting containers is how they can solve compatibility issues between platforms and improve workflows. For example, before implementing containers, a company would have three separate on-premise servers running their website. Each of those servers is managed manually by a sysadmin. One day, a bug is introduced into the application that requires a vendor package update and manual configuration files to be changed. The sysadmin goes in and makes all the changes on each server, but misses a single line in one of the configuration files. Now, only two out of three servers are working.
Now, imagine if these three servers were running Docker with the website inside of a container. A developer pushes out a new feature requiring a new dependency. Previously, this would break the server as it is missing the dependency. Instead, the Docker image crashes and refuses to run. Allowing for an easy revert to the previous version. The fix for all three servers is now to change a file in one place and rebuild the Docker container. Then all the servers can utilize the proper version.
MASV + Docker
“Can we not?” is a phrase I’ve heard before when talking about minimizing the amount of work required. It’s essence is figuring out if we are over engineering something. By adding Docker builds, we can direct our users to all use the same version of TransferAgent. With most of our users directed towards the Linux version, it allows bug reports to mostly target one operating system as Docker runs on Mac, Linux and Windows. In other words, Docker allows us to reduce our bug vector.
Note: We are not removing any builds for the platforms we currently use and support.
Docker containers are incredibly easy to update. You can stay on your `latest` container tag, which will download the newest version when you restart. Or you can even follow our semver pattern and update as you need.
If you’re hoping to work on building an app or script around our TransferAgent app then you don’t need to worry about compatibility issues on the TransferAgent side. An easy transfer between Mac, Linux, and Windows is what we hope to provide.
Video Editing and Automation
Video editors are a growing group of professionals looking at automating their workflows. Docker can support all sorts of tools from FFmpeg, handbrake, and even TransferAgent. With the right scripts and files in the right place, these tools can make rendering, finishing, and final automated file transfer a breeze.
If you’re a video editor, videographer, or any sort of tech addict, you probably have some extra hardware laying around for you to put Docker on. Be it a Raspberry Pi, your own home NAS, or even some old laptop you chuck in the corner.
We hope you stay tuned for the next article where we get TransferAgent up and running on Docker and think of us for your next file transfer.