21

Situation:

  • User has several folders worth of GBs which they want to keep in sync on a cloud service (for example Mega or Dropbox) in an encrypted form;

  • User does not want to sync a single encrypted volume, because this would mean, obviously, re-uploading the single volume every time even the smallest file in it changes;

It is clear that this rules out, for different reasons, VeraCrypt or EFS.

What should be the solution?

Community
  • 1
nico
  • 341
  • 1
  • 2
  • 9
  • 1
    Much better question, and resolves the X/Y problem. – schroeder Feb 09 '17 at 12:30
  • Are you cool with an initial, one-time upload of an encrypted volume? – schroeder Feb 09 '17 at 12:31
  • OK, I'm happy you're happy :)) Yes I assume the initial upload is necessary. I've been doing this with a rar encrypted file in the past which I would create via a script, but then syncing meant overwriting it each time, which eventually became impossible. – nico Feb 09 '17 at 12:34
  • Would you accept a solution that encrypted your files locally before uploading them, e.g. a Dropbox service which zero-knowledge about your files on the server side? – Out of Band Feb 09 '17 at 13:19
  • @Pascal, sorry I missed your comment. I normally use Mega which allegedly encrypts files on their servers... However what I want is having control on the encryption locally, so what you hint at seems ideal. Please elaborate. )) – nico Feb 09 '17 at 13:48
  • My occasionally-updated cloud comparison suggests Seafile has client-side encryption: https://github.com/pjc50/pjc50.github.io/blob/master/secure-clouds.md : https://www.seafile.com/en/help/encrypted_libraries/ – pjc50 Feb 09 '17 at 14:10
  • @nico MEGA doesn't allegedly encrypts files on their servers, it encrypts files *in the client*. But I wouldn't recommend using it because they "white their own crypto" (like using AES in a key stretching function). – Gustavo Rodrigues Feb 09 '17 at 17:02
  • I don't know whether it works well with Dropbox, but you could try: http://duplicity.nongnu.org/duplicity.1.html – Carsten S Feb 09 '17 at 17:12
  • @Carsten: I could, but I'm under windows. A ready-made PowerShell script would suit me though. :)) – nico Feb 09 '17 at 18:02

9 Answers9

14

this would mean, obviously, re-uploading the single volume every time even the smallest file in it changes;

It is clear that this rules out, for different reasons, VeraCrypt or EFS.

That's not true. VeraCrypt does not re-encrypt the whole encrypted volume on each change of the files inside and Dropbox does not upload the whole volume if only a small part of it changes:

Dropbox tries to achieve the best possible performance by trying to be as smart as possible about what data it transfers. It only uploads changes made to the file instead of the entire file.

Community
  • 1
techraf
  • 9,141
  • 11
  • 44
  • 62
  • 1
    Maybe. Normal file change patterns differ from filesystem change patterns. Maybe the Dropbox algorithm won't be able to handle change patterns occuring on a filesystem container efficiently. You'd need to test it in order to see if the sync algorithm Dropbox uses is smart enough. – Out of Band Feb 09 '17 at 13:17
  • I'm using this pattern for several years. What kind of tests do I need to perform to write an answer on SE according to you? It's really not a space technology. Git does it, Bittorrent does it, most of the software that synchronises files across servers do. – techraf Feb 09 '17 at 13:18
  • Well, I guess if you've been using it for years and Dropbox hasn't been constantly syncing and producing gigabytes of traffic, then that answers the question :-) – Out of Band Feb 09 '17 at 13:24
  • @techraf I believe all this is true, but that certain conditions exist. That's why I didn't promote the idea. If you can confirm, then great. – schroeder Feb 09 '17 at 13:40
  • @techraf. Unfortunately I don't use Dropbox but Mega which, in my experience, overwrites large files and cannot just change bits and chunks of them. At least it does so with RAR files but maybe that was the problem. We are talking about a 2GBs container file or archive here, are you sure Dropbox would upload only the little 10kbs that have changed? If so, that's pretty good and it does make a bit of a difference for me. Though it would be great to be not so dependent on a specific cloud service. I have to do some testing I guess. Too bad it takes me three days to upload a file that size... – nico Feb 09 '17 at 13:41
  • 4
    @nico Sorry, but if you don't use Dropbox then don't write the title "Keep encrypted files in sync on a **dropbox service**" and don't ask "(for example Mega or **Dropbox**)". You asked a question and I answered the question. – techraf Feb 09 '17 at 13:47
  • 1
    @techraf, you are right. I've changed the title as soon as I read your comment. Mistakenly I wanted to avoid mentioning Mega since I didn't want people to tell me that they encrypt their files anway. Nothing wrong with your answer though because I can just as well change my cloud provider if it's worth it. I will test both with an encrypted volume, in fact. – nico Feb 09 '17 at 13:51
  • 2
    @nico: If you're dead set on not using Dropbox, you can roll your own by using the rsync command line tool (available both for Unices and Windows) to periodically rsync your Veracrypt container to the destination. You'll need a virtual cloud server with enough storage, which will cost you about 5-10 $ a month, to rsync your files to. Unfortunately, there won't be a ready-made up to use on your phone or tablet... – Out of Band Feb 09 '17 at 14:20
  • Here's what Mega says: Dear MEGA user, thank you for using MEGA. MEGAsync can't do delta updates (upload only the modified data of a file) yet so it will upload the whole file every time it's modified. – nico Feb 09 '17 at 15:13
  • 3
    Yes, but Dropbox (and service with delta uploads) will work for you. When you pack the files using RAR, the whole archove will be different in the end. If you modify only one file in the VeraCrypt container, only some blocks will be different and the application will only sync those block which were affected by the change. – JohnEye Feb 09 '17 at 16:38
4

You could use one of the zero-knowledge file sync services that exist online. Examples would be http://www.spideroak.com, https://www.boxcryptor.com/en and so on. Pick any one that earns your trust.

Basically, they offer a Dropbox-like service, but they promise to encrypt all your files locally, on your own machine, using a password that never leaves your computer, before the files are uploaded to their servers.

Some of them also offer web access to your files, but I'd probably steer clear of that, since any solution that works on the web will likely be implemented using javascript, which can't protect your credentials.

Of course you'll never be completely sure that these services don't have a backdoor somewhere in their software, but that's the price of using a finished product, instead of rolling your own.

https://librevault.com/ might be a way to go if you want to inspect source code to make sure there's no backdoors, but it doesn't look like it's ready for prime time yet. Maybe there's other zero-knowledge open-source solutions around. If there were, I'd favour one of these.

Out of Band
  • 9,150
  • 1
  • 21
  • 30
  • 1
    Unfortunately I don't trust easily, which puts me a bit in a bind here. Ideally, IMO, you should rely on one entity to encrypt, and a completely different entity to upload. – nico Feb 09 '17 at 14:11
  • Then with your requirements I don't think there's a solution except what @schroeder suggests - e.g. rolling your own encryption. FWIW, I still think using an encrypted Veracrypt volume over Dropbox or any other file sync service that does partial file syncs, like techraf suggests, would be the way to go. – Out of Band Feb 09 '17 at 14:15
  • Yes I think I will try that way first, and if it fails I will go Schroeder's way. – nico Feb 09 '17 at 14:20
  • 3
    Actually, Boxcryptor works that way: it doesn't handle the file syncronization, just the encryption. You point it at a folder under your dropbox/OneDrive/whatever root, setup your keys. The client regular software handles the sync and it handles the encryption. That said, unless you're willing (and able) to perform a full code review yourself of all software involved, you're going to have to trust SOMEONE at one point. – Stephane Feb 09 '17 at 15:13
  • @Stephane: I am usually prone to trusting open source projects which are participated by enthusiasts. But Boxcryptor does seem a very good possibility, and I did consider it. But then I discarded because it doesn't support Mega which is where I have my files now. Then I started thinking: why this software should support only certain providers and not others, when all it does is encrypting files in a folder?? This would never happen in an open source project. – nico Feb 09 '17 at 16:03
  • Boxcryptor might work with Mega, too. Have you asked them? They ask you to contact them if your provider isn't listed, and they'll check for you. – Out of Band Feb 09 '17 at 16:19
3

WinSCP can synchronize a folder to (individually) encrypted files on the cloud. It also uses SFTP instead of the often flaky synchronization programs offered by cloud storage providers. https://winscp.net/eng/docs/file_encryption

FTPS is not yet supported (https://winscp.net/forum/viewtopic.php?t=26434).

The great feature of WinSCP is that the encryption code is well documented, a standalone decryption tool is also provided.

gueste
  • 31
  • 1
2

You could use a "shadow copy" design pattern where you copy all new files (and delete files) to a duplicate location, then encrypt the individual files in the shadow copy. The backup/sync solution is aimed at the encrypted shadow copies.

This can be done with scripts, though comes with risks and inefficiencies.

Using this pattern, your local email client is unaffected, and you only backup/sync encrypted changes.

schroeder
  • 123,438
  • 55
  • 284
  • 319
  • ..you need to be a bit more detailed that this. What piece of software would I use to encrypt the files individually in the copied folder? Obviously I need something that can be called via script (batch file) and can recursively encrypt the content of several folders? As a sideline, can I say that it seems to me incredible that there is not a comprehensive software to target this particular need, which after all must be quite common. – nico Feb 09 '17 at 12:41
  • 1
    You could use a wide range of software, including PowerShell. The market is chock full of encrypting programs, many accessible from the commandline. As I say, there are risks and inefficiencies in the approach and most people trust the cloud service's encryption and if they don't, spin up their own file server for the job. – schroeder Feb 09 '17 at 12:44
  • Or they use an online mail service where the users maintain the encryption keys. – schroeder Feb 09 '17 at 12:45
  • The people who have this problem are people dealing with massive amounts of data that makes it infeasible to transfer in bulk. Mostly, those people are large companies. They have tools to do what you want, but it's basically what I laid out. – schroeder Feb 09 '17 at 12:46
  • Can you please suggest which software would _you_ use to achieve this? I'm not sure this is relevant only for companies. I'm sure there's plenty of individuals who want to upload files they have encrypted locally without relying on a mounted volume. That's why there are services for sale to target this, as described [here](https://www.nextofwindows.com/top-3-free-encryption-tools-to-protect-your-data-stored-in-the-cloud). Too bad they want my money and/or put the process out of my control. – nico Feb 09 '17 at 12:55
  • Also, you suggest Powershell but I believe Powershell relies on installed certificates to encrypt binary data, not sure it's the best way for this situation. – nico Feb 09 '17 at 12:59
  • 1
    @nico a certificate is just a pair of keys, it's quite easy to generate and allow you to send your public key so someone can transmit you an ecrypted file you only can decrypt. One in many example tool is https://www.gpg4win.org/ you can get some information and inspiration from there. – Tensibai Feb 09 '17 at 13:03
  • @Tensibai, thanks for the suggestion GNUPgP could be a way. I think the answer to my question goes also in the direction of something like AxCrypt, which has a commandline and encripts individual files. – nico Feb 09 '17 at 13:32
  • @nico as said, it's one in many. ATEOTD I think a per file encryption is the way to go if you don't trust your provider. – Tensibai Feb 09 '17 at 13:39
  • @nico Frankly, I'd use PowerShell. Not sure what the problem is with certificates. Plus, you don't have to use certificates. – schroeder Feb 09 '17 at 13:39
  • @Schroeder, just something I read [here](https://gallery.technet.microsoft.com/scriptcenter/PowerShell-Encryption-45709b87). You're right, certificates are not the problem anyway. I'm not too familiar with Powershell though. – nico Feb 09 '17 at 13:44
2

I asked this question a few years ago on Superuser. The solution I came up with at the time was to encrypt the files using BoxCryptor (which does exactly what you're asking for), then upload the encrypted files onto Dropbox.

Note that this approach leaks some information, such as how many files you have and how large they are. Also, the file-names are only encrypted in the paid-version.

I've since moved my files to SpiderOak, after making a comprehensive spreadsheet of backup services†. It's much more convenient, and the zero-knowledge backup ultimately provides the same (or better) security. I now only use Boxcryptor for things I rarely need to decrypt, like tax documents.

† Note: This spreadsheet is several years old, and the information may be outdated, especially the pricing

Community
  • 1
BlueRaja - Danny Pflughoeft
  • 1,943
  • 13
  • 18
  • Yes, but if you want to encrypt files on your local drive then maybe SpiderOak isn't that much more convenient. – zylstra Oct 20 '17 at 23:32
1

This is a problematic question, requiring an overly detailed explanation. Assuming you're talking about Encrypted Volumes as 1 file objects.

The reasoning can boil down to one tiny little detail concerning file encryption. Basically, a 'good' encryption methodology will encrypt the file in its completeness, and not in 'chunks'. Therefore, any small changes (one byte) would generate an entirely new file content.

If you're proposing solutions that get around this fact by chunk-encrypting files, then you're actually missing the whole point of long-term encrypted storage.

That's the philosophical answer to the technical problem above. Therefore, what you need to consider is re-organizing your file volumes to minimize size and changes to a subset of encrypted files which would be easier to synchronize and move around.

But, technically speaking, you cannot synchronize part of an encrypted file. It goes against good security practices if you actually do. I would seriously study the underlying cryptography before investing in a solution that allows incremental updating of encrypted files. I have yet to see a "good" solution to this problem that doesn't sacrifice security.

selt
  • 11
  • 1
  • How about encrypting each file individually, and uploading them one by one (rather than uploading a single encrypted file container)? I'm not thinking about large files here, but rather a large folder. What would be the best way to achieve this? – nico Feb 10 '17 at 05:23
1

I'm afraid your assumptions are incorrect. VeraCrypt DOES in fact use a format that is delta-change friendly. This means that you can use it perfectly well with DropBox and similar cloud storage. I know this because I do! I have some large volumes that are synchronised to the cloud.

The important thing is to ensure that you are using a cloud storage sync client that uses delta-processing (e.g. it checks for changes at a block level and only synchronises changed blocks) - which I suspect all do now, that wasn't always the case.

Julian Knight
  • 7,092
  • 17
  • 23
1

If you can "mount" the cloud service as a local path, borgbackup (BSD license) is a pretty good solution. It's a python based tool that makes encrypted backups. The server sees only large-ish blobs of "data", so your file structure or meta data is also not revealed. (I say "mount", because borg requires the server also to run borg, which for simple file hosting on cloud is not possible).

If you cannot do that, you can investigate "duplicity" (http://duplicity.nongnu.org/; GPL license if I recall), which does not require anything specific on the cloud, and supports about 20 different cloud providers so far. I used duplicity for years, and only switched to borg because borg has better support for pruning old history (even automatically, if you wish). Duplicity scores badly on that front.

But for cloud I currently use rclone (MIT license), which supports an even larger number of cloud providers, and can encrypt locally to any of them (if I understand things correctly; I only used google drive so far). Rclone works very much like rsync, and is pretty darn fast too, in my experience. Rclone of course reveals file metadata like file size and directory structure. I do not care about that so it's OK for me; if you do care, then it won't work for you.

0

I use cryptomator for this and have been mostly satisfied. If anything, it's written in Java, so it'll pull in 280mb of dependencies on java runtimes, and the gui won't integrate especially well with your desktop or scale nicely on HDPI.

Other than that it does a good job (note that I haven't audited the encryption code or anything). It's free, opensource and will encrypt file names.