Not-so-mini review of CrashPlan backup software

[mouser]

fyi: Crashplan having a sale currently: https://www.crashplan.com/bigsale/

[mwb1100]

Thanks! 

It's still at a very good discount (59%), but I wish I learned about this a couple days ago:

How the Sale works -

On Friday, Nov. 23 at 6:00AM CST, CrashPlan 1-year plans will be FREE. Every two hours after that, the price will steadily increase until Monday Nov. 26, 2:00PM CST, when your purchase will be discounted 42%. The discount will remain at 42% until the sale ends, 11:59PM CST Monday.

[JavaJones]

Well, I thought I should report here that I finally stopped using CrashPlan. The memory use stayed around 1.5-2GB permanently, never improving over the years I continued using it! This is just a really bad sign IMO. But I couldn't be sure if there was something wrong until I tried another service, and as outlined in my original review, my needs were somewhat specific: affordable for large data sets (I'm now backing up nearly 3TB, up from just under 2TB when I wrote the review), a "seeded" backup option, and a local backup option in the same client, primarily.

I finally found iDrive and to my surprise it seems to meet all my needs. It has also been running on my full nearly-3TB data set for a few weeks now and memory use has hovered around 150MB! So clearly CrashPlan is doing something wrong, wrong, wrong.

Not only does iDrive do online and local backup - like CrashPlan - it *also* has a sync option and a full-disk image function (you can't schedule it though), it supports mobile devices too, it can backup Facebook and some other social media (many people here probably don't care, but it's an interesting feature), *and* it has online file sharing, which Crashplan and many other backup-oriented online storage systems don't support (so I don't have to keep using the free and highly space-limited Dropbox). It may be a security risk, and I would guess that in general CrashPlan's security is superior to iDrives, but for my needs iDrive is sufficient. Oh yeah, and they have a "seeded" backup and restore option too (called iDrive Express). It's kind of a Swiss Army Knife of features, which might be good or bad depending on your perspective, but I can say that at lesat memory use doesn't suffer because of it!

There are two caveats to the iDrive services that I should point out. First is that it's more expensive than CrashPlan. The amount of data I have necessitated the "10TB" plan, with is $375yr (holy crap, what!?). So actually iDrive would not have been an option for me at all had it not been for the fact that they have all kinds of deals out there. This is the one I took advantage of: https://deals.androi...-cloud-backup-bundle
It has been set to expire "within a few days" for months now, it seems to be perpetually on sale essentially.

This is an odd one, $97 gets you both an iDrive Wifi (a local hard drive for backup that you can connect to wirelessly, great for people who only have laptops in their homes), *and* a 1 year subscription to the 10TB plan at iDrive. What's odd about it is it would be a huge discount - and worthwhile for me and probably many others - if it were *just* the 10TB plan, but it also includes the wifi device. Mine is actually sitting unused on my desk, it was the 10TB of online data I was after.

The other thing that's odd to me about this deal is I contacted them before finding this deal saying I would happily pay for a 3 or 5TB plan that was priced in-between the 1TB and 10TB plans they already have, and that the 10TB option was way too expensive for me and much more space than I needed anyway. A sales rep got back to me but even after repeated back-and-forth and knowing I was a potential CrashPlan convert, he didn't offer me a price that was even remotely appealing. I had written iDrive off and was just going to stick with CrashPlan until I found this deal. It's quite strange to me then that the sales rep wouldn't offer me even a *decent* deal, let alone this frankly kind of crazy discount (80% off).

So you pay $97 the first year basically, and then it's a discounted $59 rate for the 2nd year. After that it's the full price, which I don't think I'll be willing to pay. But it gives me 2 years to decide if I like the service and to find something better if it doesn't justify the price when that 3rd year renewal comes due. In that time they'll hopefully realize a 3 or 5TB mid-tier plan also makes sense, or bring down the cost of their 10TB as competition from others continues. 2 years from now storage costs could be a fair bit less, or at least that's my hope.

Anyway, the other issue with their service is that in order to get the iDrive Express you have to fill out an online form. It's not something you can do from within your online account, for some reason, so you have to fill out some details including your address. For some odd reason the form on their site never worked for me, it always told me that I had to fill in the "State" field, which of course I had filled in every time to no avail; it kept giving me that error. I tried on multiple browsers, then multiple computers, and it never worked. I contacted their support several times about it and they said to try again and couldn't understand why it didn't work, but I'm honestly not confident they even tried it themselves, so it may well be legitimately broken. I finally called in to resolve the issue, which they insisted on (they wouldn't do it over email), and while the support rep and overall experience was not particularly good, I did finally get everything resolved. The iDrive "express" shipment took quite a while to arrive, but it's free (I think they have faster shipping options, but I was still using CrashPlan and not in a hurry at this point).

iDrive itself is fine. Its interface is a bit quirky, but no worse than CrashPlan, and according to their support the client is native, not Java. It certainly seems to respond and perform better than CrashPlan's did, particularly in the area of memory use. Bandwidth usage has been fine, in fact it saturated my upstream cable bandwidth until I throttled it (the throttle percentage is relative to your LAN connection speed, it seems, and not in some specific unit of measure unfortunately, e.g. Kb/s, so I had to set it to 10% or something of my 100mb LAN port; not sure if this also affects actual LAN transfer or just WAN). I don't use the social media backup yet, nor have I tried online file sharing or file sync. I just started using local backup, but have been using the online since my "Express" transfer finished a week or so ago. I haven't tried a restore yet, I know I should test that and I will soon.

Overall I'm fairly happy with iDrive and plan to stick with it for at least 2 years at this point. For those with 1TB of data or less, their standard plan is competitively priced and the breadth of features combined with reasonable resource use is, IMO, unmatched. Even if you do have "only" 1TB of data, iDrive Express ("seeded backup and restore) is still very worthwhile, and many other services don't offer it despite offering "unlimited" space, a fact which seems quite inconsistent, even disingenuous, to me.

So has anyone else had experience with iDrive? Anyone still using CrashPlan?

- Oshyan

[4wd]

I also recently started using iDrive, I still have an account with Backblaze which I'll let expire when it comes up, (the Backblaze "we'll delete your data if your computer doesn't log in within 6 months" policy is a real killer since I've been OS for 6 months or more at a time every two or three years).

Installed the iDrive program a few weeks ago, I'm only backing up ~270GB (which is still uploading) so memory usage of the three components is ~30-40MB.  It also means I'm only on their standard plan, (1TB), but it only cost US$14.95 for the first year which I got by registering for a free account and installing the software.  Less than 24 hours later I get an email offering the discount so I thought I'd try it until I find something better/else.

Accessing my files is pretty easy using their Android app or via a browser, plus I can back up all my devices to the one account without paying any extra: 3 computers; 2 tablets; 1 phone - (the total is still well under 1TB).

The big plus compared to Backblaze is that I choose which files I want to backup, not some idiot sitting in another country.

[wraith808]

Have you tried Roll Your Own solutions with S3 or Glacier backing it up?  That's what I'm starting to look into now.

I might do a mini-review roundup when I've solidified... but that's a bit away.  Some that I'm looking at:

StrongSync
Cloudberry Backup
CloudBerry Box
Duplicati
FastGlacier
Jungle Disk -  Not sure if they still have the desktop version publicly available.
S3 Backup

These last items are command-line tools.  The command line option, while more than I wanted to get into, is becoming appealing since I can link it to my current backup solutions:

S3Sync.rb - The author isn't maintaining it anymore, so not sure I want to get into this.
S3 Sync
S3 Tools

A couple of articles:

http://forum.synolog..._Server_to_Amazon_S3
http://eberly.org/20...azon-s3-using-rsync/ (Of course, s3fs is in an uncertain state, so I'm going to have to deal with alternatives... maybe)

Of Course, there's also Tarsnap- but I'm not going that hardcore: https://www.tarsnap.com/

[JavaJones]

Roll your Own isn't an option for me due to the sheer size of my data set (unless Amazon offers a service where you can send them a hard drive and they'll stick it on Glacier storage - do they?). Even if such an option were available, it would have to have a clear price advantage and require minimal maintenance, otherwise it really wouldn't be worth it for me. But I'm *definitely glad* such options exist. Can't let the big backup players get too complacent. When storage is available so simply and cheaply direct to the user, the backup companies have to compete on features and price, which is good.

- Oshyan

[wraith808]

I've only just gotten started, but they seem to have a couple of things in the faqs about it:

https://aws.amazon.com/glacier/faqs/
https://aws.amazon.c...m/importexport/faqs/

[Deozaan]

Glacier sounds pretty cool if your backup storage needs are relatively small. Doing some rough calculations, Glacier would cost about $370/year for 3TB of data. iDrive is $375/year for up to 10TB of data (not counting the discount). Considering JavaJones' current usage and needs, plus the greatly discounted 2 years of service, I don't think Glacier would offer him any benefit over iDrive.

[wraith808]

Glacier sounds pretty cool if your backup storage needs are relatively small. Doing some rough calculations, Glacier would cost about $370/year for 3TB of data. iDrive is $375/year for up to 10TB of data (not counting the discount). Considering JavaJones' current usage and needs, plus the greatly discounted 2 years of service, I don't think Glacier would offer him any benefit over iDrive.

-Deozaan (May 18, 2015, 04:04 PM)

Cool thing about amazon... you don't have to rely on rough estimates.

http://calculator.s3...onaws.com/index.html

$30.72 per month for the storage.  Unless there's a catastrophic need to retrieve all the data, that should be pretty accurate, within the number of requests- with 4GB/1000 requests, it goes up to $30.77.  If it's truly catastrophic, you get it out the same way you got it in.

What I like about it is the size of Amazon, and the redundancy.  That's why Amazon can get away with charging more IMO.  And the real problems that I've had with RYO solutions is the infrastructure.

But yes, each service is going to have it's own uses.


... but woe to you if you get over 10TB.  iDrive scales into business at that point.  I guess their redundancy and such is better for business.

[JavaJones]

Yeah, plus $80 for the initial data import, which is free with iDrive.

I certainly think Glacier and a RYO solution is a great option to have. For those more security concerned, or who want more specific setups, etc. But indeed for my needs iDrive seems to be a better deal *and* have more features, be easier to setup and manage, etc.

- Oshyan

[JavaJones]

Update on iDrive: All is not so rosy.

I have run into a couple of concerns and issues over the past week or two that I thought I should mention here.

First off I get notifications for updates to the client but trying to use the built-in update function never works. It downloads to 100%, then just sits there and never applies an update. This may be a security setting issue on my end. It's a minor but consistent annoyance. Downloading the application from their server manually and then just reinstalling it seems to work fine to update though.

Second - and perhaps most importantly - their system's support for deduplication seems to either be extremely minimal/simplistic or *non-existent*.

This is normally considered as much of a benefit to the host/service provider as to the user in that it can help them reduce data use on their end *if* it's done *between* people's accounts (i.e. if you have a large movie file downloaded and Joe Schmoe who also uses iDrive does, then they only need to store a single instance of the file on their server). This is an obvious privacy and security risk and it does not appear that iDrive does this (which is good for you, the user). For the user the main benefit is reduced bandwidth use if the file can be found to already exist on their server.

If deduplication is done *within a single user's account* it can still help with bandwidth, i.e. if you *move* files around on your system, it does not have to re-upload them. Unfortunately iDrive does not seem to handle this, or doesn't support it very well. What happened is I have 2 physical hard drives being backed up to the same location at iDrive (and locally). I had 150GB of photos on one drive when I did my original iDrive Express "seeded" backup that I sent to them. I later moved that 150GB of photos to the other drive, where all the other pictures live; it's a sort of archive, whereas the main drive is an SSD and a "working drive". So I do such file transfers semi-frequently. iDrive then had to *re-backup* all 150GB of photos (re-upload), even though they were the same exact photos and had simply been moved from one drive to another!

I consider this to be a pretty big drawback, at least for my usage patterns. I will seldom be moving such a large amount of data at once, usually it's more like 10-15GB, and that can re-upload fairly quickly. But it's still a needless inconvenience and evidence of an extremely simplistic - frankly naive - backup client. It is quite honestly not confidence inspiring, although I have no other specific reason to believe that their backup system is shoddy. It just seems like such a big, important, and obvious thing, one wonders what else they're overlooking...

It also may help explain why CrashPlan - which uses fairly sophisticated deduplication, block-level even, I believe - might be using so much more memory than iDrive. It seems reasonable to conclude that CrashPlan is just doing a lot more with the data locally, i.e. it also does compression, whereas iDrive seems to do so minimally if at all (the local backup size is roughly equivalent to the original data size on disk), or to only do it for remote backup and not local. I would really like to see both compression *and* deduplication and I consider both it to be an important features of any modern backup solution. The fact that iDrive appears to be missing both this (to some degree) is worrisome. They do claim to have compression and incremental backup on their features page, but I assume both are it is referring to remote backups, and apparently "incremental" doesn't cover my scenario of moved-but-not-modified files. [edit: I later found that compression *is* working]

Lastly, when I contacted support to discuss this issue they were not exactly impressively knowledgeable. The rep did eventually tell me several times that the data would have to be re-uploaded, but couldn't explain why and didn't seem to understand that this was a missing *feature* (and an important and relatively standard one). He did say he'd open a feature request for that, but had no information about whether such a request was common. Visiting their (surprisingly hard to find from their site) forums I don't see any other mention of it, which is surprising. Maybe my usage pattern is unusual.

Anyway, all that being said I'm still sticking with iDrive for a while, both because I have invested money in it, and because CrashPlan was not without its (significant) problems either. As I've mentioned, I have such a large amount of data that a "seeded" backup service is necessary, and the per-GB storage cost needs to be reasonably low. iDrive and CrashPlan are the only 2 services that seem affordable for my needs at present, but I'd love to hear about other options if anyone is aware of one that fits those specific needs.

Last but not least if you're still interested in iDrive after my concerns expressed above, and if you don't need 10TB of space like I do, this deal might be more of interest (I haven't tested it to see if it's still live):
https://www.idrive.c...eals/pd/backupreview

- Oshyan

[4wd]

First off I get notifications for updates to the client but trying to use the built-in update function never works. It downloads to 100%, then just sits there and never applies an update. This may be a security setting issue on my end. It's a minor but consistent annoyance. Downloading the application from their server manually and then just reinstalling it seems to work fine to update though.

-JavaJones (May 25, 2015, 02:34 PM)

FWIW, I think I've had 5-6 updates in the last two weeks or so and all of them have downloaded and installed OK.  I've had to reallow some of the components through WFwAS but other than that the updates have worked.

Last but not least if you're still interested in iDrive after my concerns expressed above, and if you don't need 10TB of space like I do, this deal might be more of interest (I haven't tested it to see if it's still live):
https://www.idrive.c...eals/pd/backupreview

That's the deal I got but I didn't have to fill out a page to get it, just sign up for a free account and install the software.  I got offered the deal after about a day by email.

[JavaJones]

4wd, glad to hear the updates work for you. Must be some configuration issue on my end. I'll have to look into it.

- Oshyan

[JavaJones]

Update 2 on iDrive: Another problem found.

The impression of iDrive being a feature-rich but fairly naively implemented application continues to solidify for me. The latest issue is that backups take a *really long time* (about 14 hours on an overclocked 4.2Ghz i7 2700k with 32GB of RAM). Now of course we're talking about 3TB of data, but these are incremental backups. The problem appears to be that iDrive is very bad at identifying what data has changed and so it basically scans through *all data in the backup set* and then updates anything new or changed that it finds. You can see in this screenshot of the log that it's actually only backing up a mere 47 changed files! But it still took 14hrs to do it because there are 1.3 million total files.



I'm fairly certain there are more effective approaches, such as using the NTFS file table to look for changed files, doing active file system monitoring (performance considerations?), etc. And of course the tremendous length of time is due largely to the sheer size and number of files in my backup. Still though, I don't think this should be considered normal and expected behavior, right? If I recall correctly from my time with CrashPlan, this kind of thing was not really a problem. But this may go into the pile of evidence suggesting that CrashPlan's high memory use had a bit more reason to it than I'd expected, i.e. if it's doing more work in monitoring the file system for changes, deduplication, etc. than iDrive is. It's still an open question whether the memory trade-off is worth it for those features...

I should note that there is a separate "continuous data protection" option that may help with my particular needs, but it's a bit of a workaround. Here's their page on the feature (emphasis mine):
https://www.idrive.c...uous_data_protection

The Continuous Data Protection feature allows IDrive to automatically recognize the changes to files present in your backup-set and back them up in real-time...

What's odd is that this indicates the technology I'm talking about *does* exist in iDrive, it's just not used for the main backup. So why not just use this Continuous Data Protection *instead* of the main backup? Well, the big limitation is it does not consider or backup files larger than 500MB. As their FAQ indicates:

CDP is not a replacement for the traditional schedule backup feature but works along with the scheduled backup to provide timely protection for your data.

Now thinking about my particular backup needs, dealing with files greater than 500MB in size is fortunately not actually that common. So my current thinking is I will enable this CDP feature and see how well it works, and lower the frequency of the full backups to once or twice a week instead of daily as I have it now. If I'm right about not dealing with large files that much, then this should give me a good level of protection through the week, and then reserve the once-weekly full backup to basically just catch any missed big file changes. It will still take a long time, but at least it will only be once a week. The one big exception is my Lightroom catalog, which is currently around 1.6GB in size. This is a pretty important one for me, so I need to think about how much this puts me at risk... I'm also not sure whether this CDP option works for both online and local backup.

Does anyone else deal with big quantities of data?

I think once I've worked out all the kinks and addressed all my own questions I'll try to write a separate "mini" review of iDrive to distill these experiences down into something more immediately useful and accessible.

- Oshyan

[f0dder]

So clearly CrashPlan is doing something wrong, wrong, wrong.

-JavaJones (May 17, 2015, 07:46 PM)

Java? :-)

I'm fairly certain there are more effective approaches, such as using the NTFS file table to look for changed files, doing active file system monitoring (performance considerations?)

-JavaJones (May 27, 2015, 02:30 PM)

If I were to design a backup solution, I'd definitely use a combination of USN Journal scanning, if available, on program start-up, combined with filesystem notification events.

[JavaJones]

Yes, Java. *sigh*

iDrive is still chugging along, doing... *OK*. It's better than CrashPlan... mostly. "Better enough". I *have* seen them fix one or two issues I had, for example with auto-update not working, so that's encouraging. And supposedly they're working on better change detection and stuff. So we'll see...

- Oshyan

[4wd]

Yes, it's still chugging along here also on my main machine - I suppose I could have taken them up on the initial HDD load, (free for Australia also, which is good), but it wasn't really critical in my case.

It's definitely good to be able to back up everything to the one account: 3 PCs, 1 server, 2 Android tablets, 1 Android phone, and 1 Windows phone.
And be able to access anything, across all of them, from any device.

[JavaJones]

Yes, the "universal backup and access" is indeed nice. I only have 1 PC and 1 Android device (plus Facebook), but that could easily grow. Nice to have a service that will accommodate that...

Honestly I think if they can gain more business and choose to invest the revenue in improving support and dev resources they could have a class-leading solution.

- Oshyan

[jaden]

Thanks for the pointer to iDrive. I'm looking for an alternative to Crashplan.

I got the 4 year plan for the discount, with 2.9 TB on my machine and another 2 TB on my wife's, but I'm far from happy and the 4 years is nearly up. I'm not a fan of the excessive RAM use, but I really dislike how it starts the backup over from 0% after doing a version scan. It's taking forever on my 10Mbps upstream connection.

I also dislike how changing the backup set causes it to start over from scratch. A few weeks ago it was at around 95%, but after excluding my Chrome profile to speed things up, it started over and now I'm at 24%.

However, as you said, alternatives like BackBlaze aren't a viable option due to policies like only retaining deleted files for 30 days.

Using Amazon Glacier or Google Nearline is cost prohibitive, but they're getting there. I would love to see a service using Nearline with some deduplication and a lean client. I have nothing against Java per se, but CrashPlan clearly hasn't optimized their client for memory usage.

I'm thinking about trying iDrive but the discount expired and I know very little about them. For all of Crashplan's flaws, I was fairly confident they wouldn't be disappearing any time soon.

I've looked at SpiderOak's 5TB option for $279/year or $25/month. It's still pricey and not exclusively focused on backups, but the site says they store every version of your file.

I'd love to hear of other alternatives.

[JavaJones]

Without the discount iDrive's choice of 1TB or 10TB doesn't really work, I feel. 10TB is both too much space *and* too expensive. And their sales team doesn't seem that interested in offering an in-between product. Since that deal is now expired, I don't know what other options you have for long-term affordable renewal. I can give you a referral code that will give you 50% off your first year, which makes 10TB $250, but that doesn't help you in subsequent years unfortunately. Here it is if you want it though:
https://www.idrive.com/?uid=S1B6C9L6O4

As to the issues you describe with CrashPlan, I think iDrive has some of its own, a few of which I've described in this thread. Its implementation of incremental backup seems similarly problematic for example (although it may in practice be better than your experience with CrashPlan). Also, with the amount of data you have, their "express" service is going to be a cost and time savings over almost any other service. Still, I'm afraid I can't *unreservedly* recommend iDrive. It is working acceptably for me, and I definitely like the memory use improvements, but there are still issues. Hopefully they will continue improving their client as I do think they have a ton of potential.

- Oshyan

[jaden]

Thanks for the referral code. I'm hesitant to jump into iDrive due to the cost and the issues you wrote about, but I'm not seeing much else out there.

I would have thought someone would have seen this need and filled it, but there doesn't seem to be a really good solution for our storage needs at a reasonable price point.

I keep hoping CrashPlan will improve their client when I see an update, but so far I've been disappointed.

[JavaJones]

I was with CrashPlan for years, like you, and client improvements seemed to be minimal. I think they might have to transition off of Java entirely in order to fix the memory use issue, for one thing. And for me it was really starting to be unacceptable, 2GB+ at all times with CrashPlan in the background!

In my relatively short time using iDrive I have at least seen one bug fixed, which had to do with automatic updates not working on some people's machines (including mine). On the other hand perusing their forums you see a lot of unsatisfying answers to questions, a lot of "contact us directly for support" diversion (and you never know how the issue was resolved then, if at all), and a number of complaints about similar issues. In fact, I would suggest browsing the support forum of *any* backup application you are considering. It can be eye-opening.

- Oshyan

[f0dder]

Using Amazon Glacier or Google Nearline is cost prohibitive, but they're getting there. I would love to see a service using Nearline with some deduplication and a lean client. I have nothing against Java per se, but CrashPlan clearly hasn't optimized their client for memory usage.

-jaden (July 29, 2015, 01:18 PM)

Kinda impossible to do for Java. You need to reserve a heap that's large enough to handle your largest theoretical load, plus some extra for garbage collection bookkeeping. This can be grown dynamically, but I don't think the JVM ever shrinks the operating system allocated heap. Makes sense for server loads, isn't always a nice feature on client machines though.

[jaden]

Kinda impossible to do for Java. You need to reserve a heap that's large enough to handle your largest theoretical load, plus some extra for garbage collection bookkeeping. This can be grown dynamically, but I don't think the JVM ever shrinks the operating system allocated heap. Makes sense for server loads, isn't always a nice feature on client machines though.

-f0dder (July 31, 2015, 06:13 AM)

You're right, Java memory usage does tend to go up and to the right, but there has been some experimental efforts such as DeallocateHeapPages. A fellow coworker of mine was able to vastly reduce memory usage in a Java app he had written after profiling it to find the main problem points.

[dr_andus]

Hey JavaJones and other iDrive users,

sorry for resurrecting this old thread, but what's your verdict on iDrive now, with the benefit of hindsight? Has it worked out for you?

P.S. Does any of you use the iDrive Wi-Fi device as well by any chance? Is it worth the $89.99, or can that function be emulated with a cheaper alternative?