Can we compare file transfer protocols?

[superboyac]

I'm trying to get my head around the different file transfer protocols.  I'd like to discuss the pros/cons of them here.  I'm planning on doing some more sophisticated backup procedures next year, along with some business plans, and so forth.  I'd like to create a closed system of file transferring for myself and my friends/business partners that will work well.  By closed, I mean not using other people's servers or services, other than the ISP.

My goal (please read and think about carefully):
Whatever I use, I'd like it to mimic regular, local file management as much as possible.  This means minimal connection/reconnection issues, minimal security obstacles AFTER the initial setup.  Drives, folders, files can be setup to behave just like local files.  In other words, moving and copying and renaming is not quirky just because it's not local.  dragging/dropping works well.  The files/folders can appear in a regular file management program without too much fussing with it.

So, I'm pretty familiar with ftp.  ftp is good and all, but not very secure obviously.  I also don't like how with ftp there's always the reconnection, losing connection issues which are constantly interrupting your workflow.  Even the best ftp programs struggle with this.  I think it's just inherent to ftp's nature.  And it's difficult to copy/move files from local to server folders.

I have limited experience with the other ftp-like services like sftp and secure ftp.  Most of my complaints are the same as with regular ftp.  The only extra complaint I have is with the initial setup and connection.  With the added security, there are more things to do in the beginning like passwords, ports, settings, authentication settings, certificates, etc.  But once it's set up, it's pretty much just like ftp.  So I prefer the added security, but still would like something more elegant.

VPN is something I have recently been exposed to.  It seems to me to be even more of a headache than the ftp varieties.  Certificates, servers, setup is pretty difficult.  I just tried it out at my work (from home) using Cisco's, and it's pretty slow and very unstable as far as losing connection.  It makes it unusable to me.  I'd like something that, once connected, can stay connected with minimum hassle and without me knowing about how it's retaining that connection.  It bothers me to see a dialog or tray icon keep popping up every two minutes while it loses and reconnects to the server.  Very annoying.

Why do all these things have such hard times staying connected?

So what else is there?  Right now, if I had to choose, I would go for sftp.  It comes closest to what I want.  I just wanted to explore all the alternatives.

[Shades]

Personally, I use a VPN connection to create an intercontinental connection that is actually quite solid (from Paraguay to the Netherlands). With Cisco's VPN software. So my experience is totally different than yours, I'm afraid.

Today (Paraguay is in its rain season) rain fell for an hour resulting in blanking streets with 10 cm of water. In that hour the wind was blowing that amount of water horizontally. I use an wireless antenna to connect to my provider, but all the bad weather did not have any effect on my VPN connection.

VPN is quite hard to set up properly, but after that work nice...as long as nothing (dramatically) changes in either software or hardware on both ends. 

 

[steeladept]

I am no network expert, but I really think you are ignoring a lot of the reasons WHY the disconnections occur.  Moreover, you are not considering the difficulty in even setting up a basic connection, let alone what you are looking for.  FTP as a protocol, from my understanding, is a point to point connection for a file or files.  You define what you want sent and it sends it.  It builds the header file, sends the data, and reconstructs it on the other end.  Once that is done the session is closed.  If it doesn't close, the other end doesn't know the transmission is complete and it never rebuilds the bits in the correct order. (Maybe someone else can explain it better/correctly).

Other protocols, such as HTTP will appear to keep the connection open, but in reality just burst data in a similar manner.  The problem here is latency and data control.  Data can not handle disruption of the data stream without corruption.  Even on a LAN this can be an issue.  This becomes even more severe when you go from system to system and perhaps (even probably) from system structure to system structure.  Sure the protocol is the same, but system requirements make implementation(s) different and therefore, potentially, less than perfectly compatible.

Then you have multiple networks to deal with, any one of which can break (or be broken) at any time outside of your control.  I work as a Network Monitor (among other duties) for my company and it is a rare day when we don't have some sort of WAN outage, not to mention the numerous LAN outages somewhere in our network.  That is why there is so much redundancy built in, and to build that in costs a LOT of money.

So what does this all have to do with your request?  Well first off, it means ANY solution that fits what you want is very likely going to be rather complicated to set up.  Moreover, how much control do you need/want and how much would you want/need for the other users?  From my initial readings I would say VPN is likely the way for you to go, but perhaps something like HTTP using a web sharing interface (like SkyDrive, et. al.) would work better?  Does it have to integrate directly into your file manager?  You seem to imply it should.  If so, then the VPN is again the way to go (or something REALLY crazy like VDI, but that is way over the top just to integrate file sharing...).  Perhaps just having a drive letter assigned to a file would work with something like DropBox?  Don't know if you can do that, but there are some services out there you can (or at least could - I used to have one but can't remember the service anymore - I think it was SkyDrive before they got bought by Microsoft, but I could be way off base).  Even if that works, though, it still would not have an always on connection like a LAN drive.  If you are not fond of even that kind of speed and insist that it should be instant, like a local folder, well then I give up.  I can't even imagine any system capable of that across the internet without dedicated off-line wires going to each computer in the network.  Again, I am no expert, and I may have a lot of this slightly (or not so slightly) wrong, but I am sure others can correct this.  Never the less, I can say the options I presented are as close as I can come to a suggestion and why I think you are over-simplifying your request.

[superboyac]

Thanks for the discussions so far.  OK, it sounds like maybe I've just had bad experiences with VPN.  I wouldn't mind at all for VPN to be the solution.  What are the pros/cons of VPN vs. FTP (any flavor)?  I'd like to know.

I don't mind if it's complicated to set up, as long as once its set up, it works with minimum headache.  Also, yes, I do prefer instant folder/file access, but I realize that there is going to be some latency and stuff.  That's fine.  I'm not intending to use it as a reliable streaming service or anything.  More like file management and backup.  And I don't want to use a service like Dropbox or skydrive because the file transfer sizes I'm talking about are many gigabytes.  I don't want a middle man, either.  I use dropbox now, and it's good for my personal little files and stuff, but not for my large databases and collections of stuff.  nor do I want those companies even dealing with that stuff of mine.

It sounds like VPN would be worth exploring further.  Maybe my company just hasn't implemented it optimally.  I wouldn't be surprised, our IT hardly has a sense of user-friendliness, which is typical I suppose.

[steeladept]

Okay, I will give it a go:

FTP -
Pros - Quick and easy to set up.
        Basic command line client built into most (all?) Operating Systems
        Fairly easy to navigate and manage files
        Secure protocol extensions available
        May be used for direct point to point connections

Cons - New connection per transfer
          No real flow control e.g. easily corrupted files if transfer is interrupted (though there are some programs to
            mitigate this through resuming techniques).
          Not all clients support all protocol variants, especially secure variants

VPN -
Pros - Secured and/or Encrypted connections (by definition, otherwise it isn't "Private")
        Can be set up to break all other connections (prevents bridging attacks on a network)
        Full access to ALL domain resources once authenticated (more than just file transfers)
        Fairly reliable connection once established (compared to other remote access technologies)
        VPN servers built into many routers, both commercial and personal/consumer grade to minimize difficulties

Cons - Very difficult to set up, even worse to set up properly
          Requires a separate client that may not even be supported on all OS's
          Can be difficult to troubleshoot where the connection issue exists (client, OS, internet, or Server)
          Requires a Domain (or other network authentication mechanism - not built into VPN)
          VPN server required to control connection to network (though one built into routers can be used for
              small or basic services as stated in Pro section).


I think this covers most or all that I can think of.  VPN has many more Pros and Cons, but that is to be expected since it provides more services/features. 

[superboyac]

Thanks steel!  I really appreciate that information.  That's going into my database.  I'm willing to give VPN a try.  It sounds like it may be a pain to setup, which I'm ok with, and after that it can be easy to use.  I guess my next step is to find a good, well written setup procedure that won't require me spending hours and hours googling and reading a bunch of forums for answers.  If you come across any, please send them my way.

[4wd]

Does this count?

Or do you specifically not want to use third party software?

If nothing else it will give you an idea of a rather easy OpenVPN set up.

[f0dder]

You should give a few more details on your scenario, Superboy.

Is this intended as more of a backup scenario, where individual users have access to their own "backup repository", and will usually be pushing changes and only be pulling sometimes? (pull would be either restoring from backup, which is hopefully seldom , or for synchronizing between machines). Or are you looking at having several users share access to a "file repository"?

FTP sucks bigtime, it's really a retarded protocol; excusable because it's so ancient and people didn't know better back then, but completely and utterly unsuable for a lot of common workflows. It sucks for a big amount of small files because you open/close a data connection for each transfer, which will get your performance killed because of latency and the slow-start property of TCP. It's OK when you need full transfer of a few big files, but sucks if you need to update changed parts of a big file (can't be done with FTP itself, needs separate server and client stuff to locate changed parts).

SFTP is just FTP with an SSL layer, so it still sucks just as much as FTP. Also, you'll have to be a bit careful with the settings, since it's possible to switch between encrypted and plaintext... you risk setting up encrypted login, but having the rest of the connections going in plaintext.

SCP is basically normal "cp" (ie. the unix "copy" command) running in a SSH tunnel. SSH == good, bigtime - while it's not 100% perfect, it's stood the test of time, and has had flaws worked out. I haven't looked at how the "cp" parts is done, so I assume it just is pretty much the copy command run through SSH, which means "dumb" full-content transfer of files... but since it's run through a SSH tunnel, you don't get FTP's retarded new-connection-per-file behaviour. A GUI client will probably send one copy command per file it's receiving, which will still have *some* latency overhead for smaller files, but not nearly as much as FTP.

HTTP is better, since you get keep-alive connections... but there's some protocol overhead, and the issue of server setup and rights management. I think you'd want to look at WebDAV stuff for that, but it's not something I have experience with; I'd expect it to be better performing than FTP, though.

For VPN, I assume you mean a VPN connection combined with regular Windows explorer style file access. Not somethind I'd personally like - it's secure enough if you use a decent VPN, but the CIFS/SMB protocol Windows uses for remote file transfers was made for LANs and isn't too hot for internet connections, there's too many roundtrips. It works, but since the user interface offered is the standard Explorer file manager, you kinda expect local speeds (even when you know you're accessing the net, your subconscious mind associates the standard interface with local speeds), but that's definitely not what you'll be getting.

So, back to usage. I assume you don't want clients to be viewing/editing data directly on the remote location - VPN+CIFS would allow them to this, but it's something your probably really really really don't want... performance is awful, especially because most programs are designed for local file access patterns - they might be reading and writing small blocks and seeking all over the place, which is pretty awful for remote access patterns... and things often get really nasty if the connection is lost.

If big files are involved, and two-way synchronization is needed, you should probably be looking for a solution that can handle partial updates (and then you still have to realize that some big binary blob formats are modified in a way that partial updates can't even be done - shame on those formats). On top of that, you need to carefully consider the problems involved if multiple clients have access to the same repositories.

A solution that's pretty good in effiency would be SSH+rsync... yum for the performance benefits. It's definitely not user-friendly with a vanilla setup though

[JavaJones]

+15 for f0dder's comments. VPN is not a "file transfer protocol" per se, so you need to consider what you'd use along with it for file transfer, e.g. "Explorer", which as he pointed out is highly problematic for non-local data. rsync may not be the easiest thing to setup, but it's the most directly applicable to what *seems* to be your scenario (based on the few details provided thus far).

- Oshyan

[superboyac]

Thanks fodder!  That is very very useful.  OK, I believe you that VPN is not the way to go.  But what is?  I'm not really sure what SSH+rsync is, but even if it is good for performance, I tend to hesitate when it's not something easy to setup.  I like buttons and dropdowns and such.  I hate when configuration is done using text files and programming language.

I can clarify how I want to use this:
I have a server at home.  Many many gigabytes, terabytes of stuff is on it and it's not rare for the contents to change and shift around a lot (several gigabytes worth) every week or month.  I mention that because most of the "easy" solutions available online like Dropbox and stuff are normally limited to a few gigabytes worth of stuff, and what I'm talking about is way more than that.
Now, I have some business partners and friends and family that I want to be able to share some of my server contents with.  Both ways transferring (uploading and downloading).  Ideally, I'd like my server to appear as an integrated folder or drive in their explorer.  That's the goal.  That way, all other programs on their computer could access the files as they would any other local file.  And when I need to perform backups using SFFS or whatever I use, I can just say backup the "N:" drive which would be the server.  No messy configuration files, no crazy programming language jargon settings.  So how do I do that?

FTP just doesn't work well for that because of that connection issue you and several others have mentioned.  I accept that VPN is not really the solution for this either.  But what is?  I've tried things that use the web interface to make it easy to transfer files...like HFS, which is really great for that.  I'm using that now, and it's ok, but it's limited because you can only access it through the web.  I want to have a network folder on other people's computers the same way as I can add network folders through my work's intranet as mapped drives.  Something like that.

Maybe that's the question.  How do I map a network drive that is NOT intranet, but outer-net (WAN, i suppose)?

[superboyac]

Does this count?

Or do you specifically not want to use third party software?

If nothing else it will give you an idea of a rather easy OpenVPN set up.

mmm...that's an good thread.  yes, that is along the lines of what I'm looking for.  Freenet sounds interesting if not for the difficulty in setting it up.

[superboyac]

What do you guys think of the Logmein/Hamachi thing?
https://secure.logmein.com/products/hamachi2/

[f0dder]

I want to have a network folder on other people's computers the same way as I can add network folders through my work's intranet as mapped drives.  Something like that.

Maybe that's the question.  How do I map a network drive that is NOT intranet, but outer-net (WAN, i suppose)?

The short answer: you don't. It's exactly the situation a VPN would allow, but it just doesn't work in real life - the reason it works for your company is because it's on a LAN, with decent bandwidth and latency.

What do you guys think of the Logmein/Hamachi thing?
https://secure.logmein.com/products/hamachi2/

Hamachi is a VPN, even if it's a bit different from the typical "dial in to your company's VPN server" style of VPN. It's very good for playing pirated games that won't work in internet mode but can be played "on your LAN", as well as legitimate VPN uses... but it's not going to be any better for your needs than other VPN software.

If you want stuff to work properly, you have to give up the "I want it to appear as a drive" requirement, really. There might be some products that use explorer extensions to do some degree of drive mapping (working in at least explorer) using different network protocols, but imho the sensible thing to do is getting some solid software that will do the kind of file sync you want.

That way, all other programs on their computer could access the files as they would any other local file.

This is just a recipe for disaster - it barely works when doing it on a LAN, you don't want to do that on a WAN.

[superboyac]

Thanks fodder.  If that's the case, and what I'm asking for is either extremely difficult or impossible, then I think I'm just going to stick with sftp or something.  I can use it to back up my files from several computers outside of my home, and I guess I'll only use full access mapped drives on the few computers in my home.

VPN doesn't even really work for my company.  I mean, I can sort of connect to it, but it didn't seem like I could do much.  The connection was too intermittent and hectic to really sit down with it and start working on files.  The best thing i could do was copy the file I wanted to over on my local computer, but that's it.  But if that's the case, it's no different than a ftp server with more headache.

[4wd]

HTTP is better, since you get keep-alive connections... but there's some protocol overhead, and the issue of server setup and rights management. I think you'd want to look at WebDAV stuff for that, but it's not something I have experience with; I'd expect it to be better performing than FTP, though.

If big files are involved, and two-way synchronization is needed, you should probably be looking for a solution that can handle partial updates (and then you still have to realize that some big binary blob formats are modified in a way that partial updates can't even be done - shame on those formats). On top of that, you need to carefully consider the problems involved if multiple clients have access to the same repositories.

Taking just the relevant parts from your post f0dder, it almost sounds like Syncrify fits the bill.  From the people who brought you DeltaCopy, (rsync for Windows).

They have a free Personal version which should help you determine whether you want to part with cash for the Professional version.  If you only want one user then maybe the Personal version running over a VPN, (to take care of the encryption), will be adequate for your needs superboyac.




If you can get by without delta copying, then there is also Synaman, (also has a free Personal edition), on which you can transfer over SSL.

[JavaJones]

Are you willing to pay for a solution? If so, then virtually "perfect" options exist in e.g. Dropbox (commercial), Sugarsync, or one of the many other multi-sync/backup solutions. Many of them do pretty much exactly what you're describing, in a seamless, easily setup, easily updated way, that generally handles conflicts and autoupdates well. No other system I'm aware of is going to be as seamless as that. That's probably why they can charge for their quality of service.

- Oshyan

[fenixproductions]

@superboyac
If your server contains WinXP (at least) I would recommend to give Hamachi a try.

What I can see at home: it allows to create LAN network with access granted to many users across globe. I can observe it working for simple file sharing (default Guest networks accounts) but it should be possible to set it up like normal LAN with full users control.

[superboyac]

I'm feeling like I'm just going to do ftp or sftp and try that for a while.  If I feel a strong need to do something more, I'll try other things.  It doesn't seem like there's exactly the thing I'm looking for available, or easily available.  I appreciate all the suggestions, I just don't have much time right now to look into things that don't involve buttons and nice, simple dialogs.

[f0dder]

What I can see at home: it allows to create LAN network with access granted to many users across globe. I can observe it working for simple file sharing (default Guest networks accounts) but it should be possible to set it up like normal LAN with full users control.

Hamachi works, but again - the Windows SMB/CIFS protocol really isn't suited for usage across a WAN. superboyac mentions that "And I don't want to use a service like Dropbox or skydrive because the file transfer sizes I'm talking about are many gigabytes.", I'd definitely not want to do that via SMB/CIFS across a WAN

[JavaJones]

I don't really understand, SFTP isn't going to be particularly easy or automated to use for bi-directional syncing, nor does it easily handle differencing and limiting unnecessary file transfer. All of the Dropbox-like tools are quite simple to use and automated, and in many cases can transfer only changed parts of large files, for example. And they come with "buttons and nice, simple dialogs". The only drawback is they're not free...

- Oshyan

[mwb1100]

SFTP is just FTP with an SSL layer, so it still sucks just as much as FTP

SFTP is not FTP over SSL (that's FTPS).  Confusingly, SFTP is unrelated to FTP (except that it's a file transfer protocol). One advantage it has over FTP is that it uses a single data connection for transfers. there's no need for it to setup and tear down a connection for each file transferred.

[f0dder]

SFTP is just FTP with an SSL layer, so it still sucks just as much as FTP

SFTP is not FTP over SSL (that's FTPS).  Confusingly, SFTP is unrelated to FTP (except that it's a file transfer protocol). One advantage it has over FTP is that it uses a single data connection for transfers. there's no need for it to setup and tear down a connection for each file transferred.

Ah, thanks for clearing that up - SFTP is going to be a bunch better than FTP/FTPS, then.

[fenixproductions]

@f0dder
I've read what is written in posts above and I don't see "the best of all" solution. Because of that, it would be rather better to give each one of them a try and check which has the lowest issues count.

[superboyac]

Ok everyone, first of all, it doesn't matter to me what the differences are between SFTP and FTP FTPS, and what not.  I'm not saying that to be an asshole.  I'm saying that because no matter which one it is, ftp clients still treat them the same way minus different setting in the preferences.  So in the end, it doesn't matter to me.

My ideal goal, once again, is to somehow have a drive or folder that I can access from ANYWHERE as a drive or folder.  Not a Dropbox button, not a separate client software access, and so forth.  From the discussion here, I can see that what I want is either extremely difficult or impossible.  Fine.  In that case, I'm going to find the alternative that is closest to what I want.  Dropbox is not the answer because there is no way in hell I'm going to upload my stuff to someone else's central server and pay for it on top of that.  It's not even about the money.  It's about the fact that I want the server on my property.  I want direct access to that server.

The reason why I think I'm going to go with sftp is because it's a common enough thing where most file management type of activities can accomodate it because it's probably already built into their features.  It's not a new, proprietary thing like a Dropbox.  So, for example, I can use SFFS to backup files using SFTP because it's already built into it.  I can also use Dopus, or Total Commander to access the server from anywhere directly.  This is kind of stuff I'm looking for.  My only complaint with it was the weird, intermittent way it connects, which everyone here has so graciously explained all the reasons why and how that happens, which is quite informative.  But nevertheless, the bottom line is that ftp is pretty good if not for the connection thing.

And just for the record, I want to say that I have never seen what I am asking for done in the way that I am describing.  So I realize it's quite a picky thing I'm asking for.  I just find it odd that something like this is not easily done, but I guess I feel that way about a lot of things.  I know you guys think I'm all stubborn and difficult about all of this, but my life is pretty hectic right now, and on top of that, I can feel myself drifting away from this computer stuff that I have been knee deep in for half of my life.  I like 40hz' post a day or two ago where he talks about the different kind of computer geeks.  There are those who are IT people for a living, so it's not a problem for them to do things with configuration files and complicated setups and so forth.  Then there are those (like me) who are extremely good with computers, but we're not programmers, so there's a subtle difference in the things I prefer or am willing to try.  I am in the extreme minority, both among the people here, and among normal non-computer geeks.  Non-computer people lump me in with the IT guys, and the IT guys think I'm a non-computer people.  The truth is, I am beyond an expert when it comes to using software.  I can use the shit out of software and talk for days about every little button and feature.  What I don't do is all this behind the scenes stuff.  Things involving ini files, scripting, databases.  I'm not even that concerned about HOW things work.  Like, i don't care why java applications are slow...I just care that they are slow.  Most of you guys get into why it's slow, and how it shouldn't be slow, and then the acronyms start flying and the technicalities start being thrown around.  That's cool, it's just not for me.  but that doesn't mean I'm an computer idiot and I need Apple things or else I won't be able to make my way around.  I just want to get stuff done.  I like documented procedures for specific things.  I don't like searching through forums (anymore) and trying out 20 things until they work.  I don't even have the time for it anymore.

Last year, I would spend an hour fiddling around with css code to create a padding around the pictures on my site that I felt was the proper amount of padding.  I'm done with that shit, I'm tired of it.  I want a button that says "padding".  I want to go there, type a number, hit OK, and see the padding changed.  I DON"T want to go into a css file, change some things, save it, upload it to the server, reload the webpage, check if it works, rinse and repeat.  I'm telling you, i don't have time for this stuff anymore, and my brain is going to explode trying to figure that out on top of figuring my career stuff out, on top of figuring music out, on top of figuring out sports, etc.  I'm cutting the pseudo-programmer mindset out of my brain.

What I really want is to hire a programmer for myself.  That's weird, I know.  but I want someone local to me that I pay a certain salary, and I just say, "can you make such and such" and he does.  The problem is, I want it for personal reasons, not really for any kind of business or money-making reasons.  So at this point, I can't justify it.

Anyway, that's what I'm going through.  I know, believe me I know, that I've been a pain in the ass around here.  I'm sorry.  I know everyone is just trying to help, so I'm sorry.

[Stoic Joker]

My ideal goal, once again, is to somehow have a drive or folder that I can access from ANYWHERE as a drive or folder.

An FTP server can be added (as a location) to Network Places and it just appears/acts like a normal folder. This has been an option in windows since the 9x days. Mind you it's not always real smooth, but it seems awful close to what you're after (hit remote folder X from anywhere).

In XP and below it appears as the Add Network Place menu option. - I'm not sure about Vista - In 7 it appears as a "Connect to a Web site..." link at the bottom of the Map Network Drive dialog.