Fast, unlimited & cheap cloud storage with "duck" (WebDAV)

So, WebDAV is this marvelous (ok, not quite, but somewhat cool) protocol that lets you access file resources over HTTP(S). Actually it's an extension of the HTTP protocol. Technicalities aside, WebDAV can be used as a way of expanding your current computer's storage capabilities in a virtually unlimited way for fairly cheap, thanks to the magic of the cloud.

WebDAV perks

Thing is, with the recent advents in personal computing our devices are smaller and smaller every day, with less and less storage. But, in reality you don't need to pay a fortune for storage anyways, the cloud is a cheaper way of storing things and most of the time it is safer (e.g. reliable), too. The practicality of WebDAV in this scenario is that (when using it) you don't need to install a proprietary application to use it, neither you're required to download all the files into your hard drive all at once or sync them in the background forever, WebDAV works onDemand. So, in summary:

• Storage is decreasing on our devices in favor of the cloud
• Cloud storage is cheap, reliable and easy to use
• WebDAV is a very practical way to use cloud storage

NOTE: Many cloud storage providers out there support WebDAV, either out of the box or via community additions. However on this post I'll center around Yandex Disk for my examples as it is free to start with, supports WebDAV natively and it's very cheap to expand further on.

The problem with WebDAV

WebDAV is pretty cool. However it can be incredibly slow, no matter where or when you use it. The question is: is there a way to get a faster WebDAV? yes, there is! it's called 'duck'. Duck is a multi-platform CLI for managing connections to remote servers, (actually is the CLI for CyberDuck, the famous MacOS application) and it works perfectly for many reasons:

1. It is cross-platform (Windows, Linux, OS X/Mac...)
2. It supports all the use cases you might expect from a storage manager
3. It is FOSS (Free and OpenSource Software)
4. It is incredibly fast (thanks to parallel processing)
5. It supports various providers directly (along with their WebDAV integrations too!)
6. It supports other protocols besides WebDAV, obviously

And some more.

How to install and use 'duck'

I'll skip the installation part, their website is very comprehensive on that department; About how to use it though:

1) Set ENV Variables

Assuming you have an account with a cloud storage provider that supports WebDAV (e.g. Yandex Disk), I recommend you to set the following environment variables on your system:

• WEBDAV_USERNAME your_username
• WEBDAV_PASSWORD your_password
• WEBDAV_DOMAIN webdav.yandex.com
• WEBDAV_URI davs://$WEBDAV_USERNAME:$WEBDAV_PASSWORD@$WEBDAV_DOMAIN

Personally I use fish both on OS X and Linux so I just had to set them at ~/.config/fish/config.fish this way:

• set -x WEBDAV_USERNAME username
• set -x WEBDAV_PASSWORD password
• set -x WEBDAV_DOMAIN webdav.yandex.com
• set -x WEBDAV_URI davs://$WEBDAV_USERNAME:$WEBDAV_PASSWORD@$WEBDAV_DOMAIN

You could set them on your .bashrc or .zshrc accordingly if that's the case. These give us a little more freedom to use the duck command loosely enough without writing a lot. Note that our WEBDAV_URI does not define a path, that's on purpose, so we can navigate our tree directory on the remote server more freely. 

2) Basic duck commands (according to our setup above)

Upload Files

• duck --upload $WEBDAV_URI/ example.txt --parallel 4 --retry 3

List Files

• duck --list $WEBDAV_URI/ --retry 3

Update Files

• duck --upload $WEBDAV_URI/ example.txt --parallel 4 --retry 3 -e overwrite

Delete Files

• duck --delete $WEBDAV_URI/example.txt --parallel 4 --retry 3

Download Files

• duck --download $WEBDAV_URI/example.txt example-downloaded.txt --parallel 4 --retry 3

These right here are the basic CRUD (create, read, update, delete) operations listed with a very simple example case, but duck is much more capable: it can handle globs, work with directories, syncronize folders, copy data between servers, helps us editing files on our local machine via a desired app and then re-uploads them, etc... You can read more about how to use duck on their wiki page.

File encryption

So, the cloud is cool right? but what about privacy? let's say you want to upload things so no one else can open them without your permission. (After all, you can't trust your cloud provider blindly); Another command line utility that can help us across all platforms with that is p7zip:

1) Install p7zip

• # apt-get install p7zip-full (Ubuntu Linux)
• $ brew install p7zip (OS X/Mac)

I'm sure as with duck, it is available via chocolatey or something like that on Windows too.

2) Encrypt your files

NOTE: This method does not work for backups, as the 7zip format does not preserve owner/permissions metadata on files, check out the 7z manpage for more information about how to workaround this problem.

To encrypt/compress a file (and/or folder) before uploading it to the cloud you could use these commands (for example):

• 7z a -mhe=on -p file.txt.7z file.txt
• 7z a -mhe=on -p yandex.folder.7z yandex/

The bold argument in there is the resulting archive's name. You could name it whatever you want; I think p7zip will add the .7z extension automatically at the end also (if you skip it from the param). You'll be asked for a password and some processing later a new, (encrypted and compressed) 7zip archive file will be returned. One could customize this workflow, (add some automation and so on) via a shell function of sorts, the sky is the limit here, let's see a fish example for this implementation:

• https://gist.github.com/Jmlevick/8c0174a558cb09ddf74d62aad4291d5e

So there you have it, a quick painless way to manage your WebDAV files via the command line across platforms (uploading your files and folders securely too!) Enjoy.