MASV Transfer Agent

MASV Transfer Agent (masv-agent for short) is a cross-platform service that manages file transfers to and from MASV network. Interaction with the service is performed via REST API calls. It supports uploading and downloading multiple packages simultaneously, while abstracting away filesyetem interactions. The agent is the beating heart of our desktop app that’s been used in production with hundreds of terabytes of data getting transferred through it, reliably and quickly.

The agent is stateful – it stores metadata about transfers and user sessions in order to carry out its responsibilities and survive interruptions and shutdowns.

Download masv-agent

You can download the latest version of pre-packaged binaries of masv-agent for your operating system using the links below:

The zip package contains the masv-agent binary (versioned), along with a swagger API spec file that documents all the agent API end points.

If you need a binary build for a different CPU architecture (e.g. i386 or arm), please contact [email protected].

Running the agent

To start the agent, simply run the masv-agent binary. By default, the service will attempt to start an http server listening locally on port 8080. You can customize the listening behaviour, like using different listeneing schema, specifying listen IP address and changing port by passing -listen argument. For example:

masv-agent -listen http://0.0.0.0:8888

The following listening schemas are supported:

  • Plaintext http (no TLS). For example: masv-agent -listen http://0.0.0.0:8123
  • HTTPS, using a self-signed TLS certificate. The certificate is generated at runtime each time the agent is started, so it does not persist. For example: masv-agent -listen https://localhost:8443
  • Unix sockets (Linux and MacOS). For example (notice the triple slashes for absolute path): masv-agent -listen unix:///path/to/socket/file.sock
  • Windows named pipes (Win 7 and later). For example masv-agent -listen "\\.\pipe\name-of-pipe"

More customization is available via command line arguments, which can be viewed withmasv-agent -h. You can check which version of the agent you have by running masv-agent -version

The following listening schemas are supported:

  • Plaintext http (no TLS). For example: masv-agent -listen http://0.0.0.0:8123
  • HTTPS, using a self-signed TLS certificate. The certificate is generated at runtime each time the agent is started, so it does not persist. For example: masv-agent -listen https://localhost:8443
  • Unix sockets (Linux and MacOS). For example (notice the triple slashes for absolute path): masv-agent -listen unix:///path/to/socket/file.sock
  • Windows named pipes (Win 7 and later). For example masv-agent -listen "\\.\pipe\name-of-pipe"

More customization is available via command line arguments, which can be viewed withmasv-agent -h. You can check which version of the agent you have by running masv-agent -version

User session, sign in and sign up

Some actions require a valid user session, for example, sending a package to an email recipient, or creating download automations. Other actions like uploading to a portal do not require a user session. When you try to perform an action that requires a session without being logged in, the service will respond with a 401 Unauthorized status code and the following error body:

{
  "code": 262146,
  "message": "session is nil"
}

Get current user session

To see which user is currently signed in to the service, use the following request:

curl -X GET http://localhost:8080/api/v1/login

The response will look like the following:

{

  "user": {
    "email": "[email protected]",
    "id": "01CWEEY5PT3535ETT4TH9NYZVP",
    "name": "Firstname Lastname"
  },
  "teams": [
    {
      "custom_expiry_default": 30,
      "custom_expiry_enabled": true,
      "download_limit_default": 12,
      "download_limit_enabled": true,
      "id": "01CWEEY60MREFF7PPYZ82QSQ9J",
      "max_email_recipients": 30,
      "name": "Acme",
      "subdomain": "acme"
    },
    ...
  ]
}

Where:

  • user object contains logged-in user attributes
  • teams is an array of all teams that the user belongs to, with details like limits and general settings as configured for that team

You may also retrieve the teams that the current signed-in user belongs to, as follows:

curl -X GET http://localhost:8080/api/v1/teams

Sign in

To sign in with an existing MASV account:

curl -X POST -H "Content-Type: application/json" http://localhost:8080/api/v1/login -d '{"email":"[email protected]","password":"top_secret_password"}'

The response body will look exactly the same as that in the previous section (get user session).

Sign up

To sign up for a new MASV account:

curl -X POST -H "Content-Type: application/json" http://localhost:8080/api/v1/signup -d '{"email":"EMAIL","password":"PASSWORD","name":"FIRSTNAME LASTNAME"}'

Note that a successful sign up will automatically sign you in, so an additional sign in is not required.

Uploads

MASV has two types of uploads (or packages sent):

  • Team uploads: Packages sent to email recipient or packages that can be shared via a link.
  • Portal uploads: Packages sent to a specific MASV portal, not necessarily owned by the logged-in user.

Regardless of the type of upload, all uploads are managed in the same way – the only difference is the way they’re created. Each upload can be in one of the following states:

  • transferring: the upload is currently transferring (uploading) data.
  • paused the upload is paused by the user.
  • idle: the upload has finished uploading data, and can accept additional files or finalized.
  • complete: the upload has been finalized and the package has been sent to the desired destination.
  • error: a fatal error was encountered and the upload had to stop. This might or might not be recoverable, depending on the type of error.

Send a team package

Team packages are packages that are meant to be sent to email recipient or shared as a download link. A valid user session is required to initiate sending a team package.

The following command will create a team upload:

curl -X POST -H "Content-Type: application/json" http://localhost:8080/api/v1/uploads -d '{
  "subdomain":"TEAM_SUBDOMAIN",
  "paths":["/path/to/file/or/folder", "/path/to/another/file/or/folder"],
  "package_name": "Optional package name",
  "package_description": "Optionali package description",
  "recipients": ["[email protected]", "[email protected]"],
  "password": "optional_download_password"
}'

Where:

  • subdomain is the subdomain of the team that the currently logged-in user belongs to, from which the package will be sent.
  • paths is an array of files or directories paths to be sent. Paths should be absolute. For directories, the agent will traverse them and automatically pick up all files inside.
  • recipients is an optional array of recipient emails.

If successful, the agent will respond with a JSON object that indicates the newly created upload_id that can be used to query or modify the upload state.

Upload a package to a portal

The agent can also upload packages to any MASV portal. This functionality does not require a user session since portal uploads do require login.

To create a portal upload:

curl -X POST -H "Content-Type: application/json" http://localhost:8080/api/v1/portals/uploads -d '{
  "subdomain":"PORTAL_SUBDOMAIN",
  "sender_email":"[email protected]",
  "paths":["/path/to/file/or/folder", "/path/to/another/file/or/folder"],
  "access_code": "optional_access_code",
  "package_name": "Optional package name",
  "package_description": "Optionali package description"
}'

Where:

  • subdomain is the desired portal subdomain. For example, if the portal URL is https://acme.portal.massive.app, then the subdomain will be acme
  • sender_email is the sender’s email address
  • paths is an array of files or directories paths to be sent. Paths should be absolute. For directories, the agent will traverse them and automatically pick up all files inside.
  • access_code is the portal access code if the portal was password-protected

If successful, the agent will respond with a JSON object that indicates the newly created upload upload_id that can be used to query or modify the upload state.

View upload status

As mentioned earlier, all uploads, regardless of their type, can be managed in the same way after they’re created.

To list all uploads managed by masv-agent, call:

curl -X GET http://localhost:8080/api/v1/uploads

In order to view the full details for a specific upload (including individual file states), use the following request:

curl -X GET http://localhost:8080/api/v1/uploads/{UPLOAD_ID}

Manage uploads

The agent will start uploading file data at the time of upload creation. Once all files are transferred for a given upload, it will transition to the idle state. It is up to you to finalize the upload and get the package delivered, by posting a finalize request to transition the upload to the complete state.

To pause an upload:

curl -X POST -H "Content-Type: application/json" http://localhost:8080/api/v1/uploads/{UPLOAD_ID}/pause

To resume a paused upload:

curl -X POST -H "Content-Type: application/json" http://localhost:8080/api/v1/uploads/{UPLOAD_ID}/resume

When an upload has finished transferring all files (transitioned to idle state), it can be finalized as follows:

curl -X POST -H "Content-Type: application/json" http://localhost:8080/api/v1/uploads/{UPLOAD_ID}/finalize

To delete an upload, regardless of what state it was in:

curl -X DELETE http://localhost:8080/api/v1/uploads/{UPLOAD_ID}

You can generate a shareable download link for team uploads that can be shared with any desired recipient.

To create a link, the upload has to be in the complete state and the upload type should be team. Simply call:

curl -X POST -H "Content-Type: application/json" http://localhost:8080/api/v1/uploads/{UPLOAD_ID}/link

The agent will return a JSON object similar to the following:

{
  "email": "[email protected]",
  "expiry": "2020-12-31T23:59:59.999",
  "locked": false,
  "id": "01ECSWWC8R6J1N8Y46S094CRGT",
  "secret": "dDaNpsSTqlRDnTBe"
}

To construct a download link from the information above, simply plug in the id and secret values from the response to the following URL: https://get.massive.app/{id}?secret={secret}

Downloads

The agent can download any MASV package providing that you have a valid download URL in the format https://get.massive.app/{ID}?secret={SECRET}. For password-protected downloads, you also need to provide the password. The agent rebuilds the directory structure of the package as sent by the sender. Initiating and managing downloads does not require a user session.

Please not that cross-platform downloads can cause issues with special characters not being accepted by certain platforms, so the agent will remove any special characters from file names if they were not allowed on the target platform.

Similar to uploads, downloads transition through different state throughout their lifecycle, with the exception that downloads do not go through an idle state, but transition to complete immediatel once all data is transferred. Downloads can be in one of the following states:

  • transferring: the download is currently transferring (downloading) data.
  • paused the download is paused by the user.
  • complete: the download has been completed.
  • error: a fatal error was encountered and the download had to stop. This might or might not be recoverable, depending on the type of error.

Download a package

To initiate a package download, simply supply the full download link URL to the agent along with a destination folder in your computer to store the downloaded files:

curl -X POST -H "Content-Type: application/json" http://localhost:8080/api/v1/downloads -d '{
  "url":"https://get.massive.app/LINK_ID?secret=LINK_SECRET",
  "dst_folder":"/destination/folder",
  "password": "optional_password"
}'

Where:

  • url is the full download link sent to you by the sender or obtained from the web UI.
  • dst_folder is the destination directory to download files into.
  • password is the download password, which is only required if the link was password-protected.

If successful, the agent will respond with a JSON object that indicates the newly created download_id that can be used to query or modify the download state.

View download status

To list all downloads managed by the agent:

curl -X GET http://localhost:8080/api/v1/downloads

In order to view the full details for a specific download (including individual file states), use the following request:

curl -X GET http://localhost:8080/api/v1/downloads/{DOWNLOAD_ID}

Manage downloads

To pause a download, call:

curl -X POST -H "Content-Type: application/json" http://localhost:8080/api/v1/downloads/{DOWNLOAD_ID}/pause

To resume a download:

curl -X POST -H "Content-Type: application/json" http://localhost:8080/api/v1/downloads/{DOWNLOAD_ID}/resume

To delete a download, regardless of what state it was in:

curl -X DELETE http://localhost:8080/api/v1/downloads/{DOWNLOAD_ID}

Please note that deleting a download this way does not automatically delete the completed files from the file system. If you want to delete a download record and delete all files from the filesystem at the same time, you can do so by calling:

curl -X DELETE -H "Content-Type: application/json" http://localhost:8080/api/v1/downloads/{DOWNLOAD_ID} -d '{"remove_files": true}'

Automations

MASV agent allows you to automate certain tasks to save time and support more robust workflows. At the time being, the agent has one type of automations built-in: automatic download of packages uploaded to portals. More automations are being developed and will be available in the near future.

Portal download automation

This automation allows the agent to automatically download packages uploaded to a certain portal. The portal has to be owned by the team to which the logged-in user belongs to.

This automation automatically picks up new packages uploaded to the specified portal and queues them up as a new download. The downloads can be managed in the same way as other user-initated downloads are handled, and they go through the same state transitions.

To create a portal download automation, simply call:

curl -X POST -H "Content-Type: application/json" http://localhost:8080/api/v1/automations -d '{
  "name": "AUTOMATION_NAME",
  "subdomain": "PORTAL_SUBDOMAIN",
  "dst_folder": "DESTINATION_FOLDER",
  "effective_time": "2010-12-12T23:59:59-05:00",
  "enabled": true
}'

Where:

  • name is a custom automation name.
  • subdomain is the subdomain of the portal from which uploaded packages will be automatically downloaded. The portal has to be owned by the team of the signed-in user.
  • dst_folder is the destination folder where all packages will be downloaded to.
  • effective_time is a string date/time in RFC333 format which indicates the start date/time after which uploaded packages will be downloaded. Setting it to a date/time in the past will cause the agent to download all packages uploaded after that point in the past until now, and subsquently, any future packages.
  • enabled indicates whether the automation will be enabled immediately or not

Successfully creating an automation will return an automation_id that can be used to manage the automation.

Manage automations

To list all managed automations, call:

curl -X GET http://localhost:8080/api/v1/automations

To modify an existing portal download automation (in this example we disable the automation without deleting it):

curl -X PUT -H "Content-Type: application/json" http://localhost:8080/api/v1/automations/{AUTOMATION_ID} -d '{
  "name": "NAME",
  "subdomain": "PORTAL_SUBDOMAIN",
  "dst_folder": "DESTINATION_FOLDER",
  "effective_time": "2010-12-12T23:59:59-05:00",
  "enabled": false
}'

To delete an existing portal download automation, use:

curl -X DELETE http://localhost:8080/api/v1/automations/{AUTOMATION_ID}