Author Topic: Do it yourself dropbox (Read 27158 times)

kartal · « **on:** March 08, 2011, 11:19 PM »

Hey people

I have managed to create a rig that lets me share my files(via local or distant connections). It is not a simple setup but I think it works well. When I have time I will write a a short tutorial about it. Basically I am using SSH(on Cygwin) and HFS(http://www.rejetto.com/hfs/). I send myfiles to Ssh server and share them with Hfs while I am creating the url for the file on the fly with simple bash console scripts(generated when the file is sent). I endup giving away a url that is similar to any web based sharing tools. I can send files within my network or from outside world.

Does anyone else created a do it yourself dropbox?

I personally like to make life harder for myself so that I can enjoy all the freedoms that comes with custom workflows and methods and I am always interested in what others have to offer as far as custom stuff goes. I do not like to put my data on someone else`s server so I rigged up my own

What is cool is that I can send folders, zip them before sending, pipe other stuff, and even resume files (will use rsync) etc. And I have no MB limit

housetier · « **Reply #1 on:** March 09, 2011, 02:06 AM »

I have several times thought about creating my own such synchronization service. But I have never actually started on it, that is, I never got past sshfs.

I want to have my personal service because I don't trust 'them'. Sometimes there terms & conditions are not really in my favor (all your stuff belong to us, but you are still liable). Having a system that works for clients on different platforms -- linux, windows, mac, mobile os -- would be awesome.

f0dder · « **Reply #2 on:** March 09, 2011, 06:22 AM »

housetier: check out SpiderOak - while even the best providers might disappear without notice, at least the SpiderOak guys have no way to access your files. Client works on multiple OSes.

nudone · « **Reply #3 on:** March 09, 2011, 08:21 AM »

Are we saying that DropBox isn't to be trusted. Not wanting to start an argument, just wondering if I should stop using DropBox and move to SpiderOak.

Eóin · « **Reply #4 on:** March 09, 2011, 08:36 AM »

I know I personally never put anything private into my Dropbox folder.

argv · « **Reply #5 on:** March 09, 2011, 08:55 AM »

And I have no MB limit
-kartal (March 08, 2011, 11:19 PM)

That's hard to believe

I know I personally never put anything private into my Dropbox folder.
-Eóin (March 09, 2011, 08:36 AM)

Personally I'm using a TrueCrypt disk in the dropbox folder for really private data

kartal · « **Reply #6 on:** March 09, 2011, 09:11 AM »

I have several times thought about creating my own such synchronization service. But I have never actually started on it, that is, I never got past sshfs.

I want to have my personal service because I don't trust 'them'. Sometimes there terms & conditions are not really in my favor (all your stuff belong to us, but you are still liable). Having a system that works for clients on different platforms -- linux, windows, mac, mobile os -- would be awesome.
-housetier (March 09, 2011, 02:06 AM)

My first post was about making data available to outside world on demand(securely and freely). Since you are also wondering about syncing ,I use Unison+SSH for unidirectional syncing(multiple platforms) and works great. I sync GBs of data a day sometimes among 3-4 computers for personal and professional use. I sync big files, text files, video files etc.

@argv
Not sure if you are being sarcastic about the data limit, but really I have alot of empty space, and I can buy new hd as I needed.

Also syncing Truerypt container might be an ok idea for small data, but really you need to sync the container file every time you change something in the container disc(save a new text file ), for anything bigger than couple hundred MB wont be feasible for my use.

f0dder · « **Reply #7 on:** March 09, 2011, 09:20 AM »

kartal: perhaps he's using block-based rather than file-based syncing?

mwb1100 · « **Reply #8 on:** March 09, 2011, 09:25 AM »

I think that Unison uses rsync to perform the transfer, so entire files aren't transferred, only the different bits (as detected by rsync).

kartal · « **Reply #9 on:** March 09, 2011, 09:29 AM »

I think that Unison uses rsync to perform the transfer, so entire files aren't transferred, only the different bits (as detected by rsync).
-mwb1100 (March 09, 2011, 09:25 AM)

Yes Unison is using rsync by default, and I found it to be quite fast since all the clients also acts as servers(Unison always has first hand access to local files).

This is not like syncing files over FTP, Unison knows which files has changed and it can detect moved files.

timns · « **Reply #10 on:** March 09, 2011, 09:41 AM »

I achieved something similar on our own Windows servers using VPN, auto-login scripts and a shared drive.

kartal · « **Reply #11 on:** March 09, 2011, 01:05 PM »

kartal[/b]: perhaps he's using block-based rather than file-based syncing?
-f0dder (March 09, 2011, 09:20 AM)

With dropbox?

f0dder · « **Reply #12 on:** March 09, 2011, 02:51 PM »

kartal[/b]: perhaps he's using block-based rather than file-based syncing?
-f0dder (March 09, 2011, 09:20 AM)
With dropbox?
-kartal (March 09, 2011, 01:05 PM)

I would have thought any sane service to only transfer modified blocks - SpiderOak certainly does.

jgpaiva · « **Reply #13 on:** March 09, 2011, 06:30 PM »

kartal[/b]: perhaps he's using block-based rather than file-based syncing?
-f0dder (March 09, 2011, 09:20 AM)
With dropbox?
-kartal (March 09, 2011, 01:05 PM)
I would have thought any sane service to only transfer modified blocks - SpiderOak certainly does.
-f0dder (March 09, 2011, 02:51 PM)

Actually, a friend of mine found out that DropBox does even more than that. It only transfers new blocks among all of its members' blocks. That is, if you try to upload a known large file (say a Ubuntu DVD or some known media), it'll upload instantly (regardless of the speed of your connection).

f0dder · « **Reply #14 on:** March 10, 2011, 07:07 AM »

I would have thought any sane service to only transfer modified blocks - SpiderOak certainly does.
-f0dder
Actually, a friend of mine found out that DropBox does even more than that. It only transfers new blocks among all of its members' blocks. That is, if you try to upload a known large file (say a Ubuntu DVD or some known media), it'll upload instantly (regardless of the speed of your connection).
-jgpaiva (March 09, 2011, 06:30 PM)

Good for speed, and storage space reduction at DropBox's servers, bad for privacy.

worstje · « **Reply #15 on:** March 10, 2011, 07:22 AM »

I guess there might be a privacy element involved with such caching if the extra information involved remains even after the last instance of the data was removed from Dropbox servers. But seriously, practically, what is the privacy leak in this case? The most devious usecase I can think of is some corporation querying dropbox for all possible files they own to see if it was ever uploaded... and even then it does not reveal who has the file. Hell, technically said corporation would probably be liable for stuff like reverse engineering the Dropbox protocols and whatnot, so it would legally be unusable information.

I use the almost exact same technique for JottiQ. Jotti's malware scan has an internal database of files it has already scanned, which allows me to avoid uploading a file a second time.

So, in a nutshell: I am willing to share some file with the service (Jotti / Dropbox), so I would be sharing the private information I worry about with the service to begin with. Thus the service obtains no private information it wouldn't have without such an optimization.

That make sense?

f0dder · « **Reply #16 on:** March 10, 2011, 07:58 AM »

SpiderOak has a blog entry somewhere that lines out some security flaws (some which aren't all tinfoil hat) with a scheme that shares blocks between users - too lazy to search for it now, sorry, but it shouldn't be too hard to track down if you're interested. No reverse-engineering is needed, if global hashes are stored all the Evil Corp^TM needs is a subpoena.

From the SO blog posts, it sounds like their method and reasoning for it is pretty well considered. Zero-Knowledge ftw.

jgpaiva · « **Reply #17 on:** March 10, 2011, 08:02 AM »

Actually, a friend of mine found out that DropBox does even more than that. It only transfers new blocks among all of its members' blocks. That is, if you try to upload a known large file (say a Ubuntu DVD or some known media), it'll upload instantly (regardless of the speed of your connection).
-jgpaiva (March 09, 2011, 06:30 PM)
Good for speed, and storage space reduction at DropBox's servers, bad for privacy.
-f0dder (March 10, 2011, 07:07 AM)

Yes, no question there. It also means that they must store all data unencrypted on their side, or they wouldn't be able to serve you a file someone else uploaded.

kartal · « **Reply #18 on:** March 10, 2011, 09:07 AM »

I have nothing to hide! But I prefer SSH and Unison to sync and store my files, because those are my files(at least I own some of them) that sit on my paid hardware and I like to have control over my own system. I know people are going nuts over cloud stuff, I think it is pretty neat and wonderful as well. There can be amazing uses for the future of man kind, but I prefer to rely on my own ideas and workflows. This way of working also makes me learn and implement new stuff. So I do not become a one way techno consuming machine as well.

You know the cloud can get cloudy, rainy and stormy, then what do you do?

JavaJones · « **Reply #19 on:** June 16, 2011, 10:59 PM »

Just ran across this today: http://owncloud.org/index.php/Main_Page
ownCloud is still in development but it looks promising. You need to run your own LAMP stack to host the sync server which then of course has a web UI and there are desktop clients in the works for Win/Mac/*nix. Hopefully mobile apps eventually too. It's a bit too early in dev for me to want to play with it but I'd be interested to hear anyone else's experience.

- Oshyan

40hz · « **Reply #20 on:** June 17, 2011, 09:18 AM »

Maybe I'm missing something, but wouldn't it be a lot easier to just run a copy of the Amahi server and be done with ?

On a related note, the Linux Action Show podcast had and episode where they discussed how to go about setting up your own personal cloud. Some very good ideas and solid recommendations in there if you can put up with the silly banter and general loose format of the show. I do put up with it because it's a good show even though Chris Fisher can really get on my nerves from time to time. (Hey Chris? Comedy is a serious business... best leave it to the pros unless you really are funny.)

The episode can be downloaded/watched/listened to here.

Note: apparently a bunch of us blipped Bryan about Amahi since we felt he might be reinventing a wheel. He's agreed there's a good chance he might and plans on doing a show on Amahi in the near future. He's put his project on semi-hold until he checks it out.

JavaJones · « **Reply #21 on:** June 18, 2011, 05:32 PM »

I don't know enough about ownCloud or Amahi to say with any confidence, but Amahi strikes me as a significantly "heavier" system and, while more flexible, perhaps overkill for someone looking for a simple Dropbox-like system. That being said it's definitely a cool option in general.

- Oshyan

40hz · « **Reply #22 on:** June 18, 2011, 07:58 PM »

I kinda see something like Amahi as less overkill in that all the heavy lifting and configuration has been done for you. So from an enduser perspective it's easier to install and administer. The beauty of it is you only need to use the parts you want. And it's GUI based so you don't need to do all the 'science faire' stuff you'd need to do if you rolled your own.

Sometimes it's easier to go up a level in complexity in order to gain a quantum leap in day-to-day simplicity of operation. This is something I go through with my clients from time to time. Sometimes the simpler something is, the more complex it is to actually use. And vice versa.

Just my 2¢ anyway.

JavaJones · « **Reply #23 on:** June 19, 2011, 02:01 AM »

Fair points. I haven't tried either so it's all hypothetical to me. But Amahi does look interesting...

- Oshyan

40hz · « **Reply #24 on:** June 19, 2011, 06:04 AM »

I tried Amahi back a few years ago and I was very impressed by it. And its developers haven't been sitting on their hands since then.

Nice thing about it is it only requires Fedora, so if you have a spare 32-bit box floating around, you're good to go. Which is convenient since all the new Windows servers have gone over to 64-bit architecture.