Messages

76

Announce Your Software/Service/Product / Re: Bvckup 2

« on: January 28, 2013, 04:13 AM »

Apologies for not replying sooner. I was pushing out the major website redesign.

--

It would seem that the difference between depth- and breadth-first are pretty small for the warm-cache tests.

I was seeing consistent 10-15% speed up when with depth-first. Your data seems to be supporting this to a degree too. Another thing was that NOT using FindExLargeFetch in warm-cache scenario results in 20-30% speed up (from 1000 ms to 800 ms). I added a command-line argument to control this in more recent versions of bvckup2-demo2.exe

I guess the 16 consecutive runs should be processed into {min,max,avg,mean} values - should be easy enough to do the processing, but how to handle the rendering?

I'd take the 80% median (sort and then trim 10% on each end) and then average it out. Also, as a bonus, if the standard deviation is high, then there's too much volatility and so the sample is not representative. With regards to visualizing, let me see what I can do. JS + Canvas might be the simplest option.

Also, if I find a way to automate the cold-cache testing (suggestions would be very welcome!)

I think the key to unlocking this problem is called shutdown.exe :)

77

Announce Your Software/Service/Product / Re: Bvckup 2

« on: January 14, 2013, 03:23 AM »

Sounds weird that you got such discrepancies - did you manage to make it somewhat reliable? Obviously there's going to be some fluctuation, but even 74 vs 24 seconds sound... interesting.

-f0dder (January 13, 2013, 06:28 PM)

I did. Don't remember what exactly it was, but I disabled few more things so to prevent any voluminous disk activity on boot and it helped. The numbers I posted afterwards are consistent across several reboots.

How are you doing the threading? Threadpool of workers that fetch a path from a queue, and while processing a path add found directories to the queue and "do whatever" with files? (Guess it could be somewhat difficult controlling breadth vs depth scanning reliably that way?) Or a different level of granularity?

Yep, that's pretty much how it works, and indeed the depth/breadth order is not very strict, but it's not way too off either.

With regards to getting 3x speed up on a warm cache - what's not to like? :) Arguably, this is the use case for real-time backups, with an operational profile consisting largely of frequent scans of small number of selected locations.

-apankrat (December 27, 2012, 12:10 PM)

Oh, it's not that it's not something to like, I was just thinking that as a user, the absolute time difference isn't that big - and it will be dwarfed by syncing even a few files. (I'm pretty sensitive to timing myself, so it's something I might notice - but I'm thinking regular users here).

I agree, but the difference between something like 2s and 500 ms is a significant one (from the usability perspective). The difference is going to be even bigger when scanning somewhat more populated disks than my wife's laptop :)

As a programmer, I think it's a cool achivement, though... and it wasn't something I had expected - guess I should revisit that old benchmark code :). And I can't see much reason not to do the threading, at least when keeping the number of threads sane. I pondered a bit about the whole thing; there's some increase in CPU usage and thus speedstep, power consumption and heat (and potential noise from CPU fan), but I don't think any of those are realistic concerns, given the short running time.

Then there's additional memory consumption because of threads - but when sticking with a sane number of threads, that shouldn't be a problem either, neither for user- nor kernelmode memory.

So I don't see a reason to not do the threading, with the data given so far. And the 1.5x cold speedup is nice, hope I can duplicate that on my testbox! :)

Looking forward to seeing what you get. In the actual app there's a config option that controls the size of a thread pool. If threading is perceived harmful, one can always turn it down or disable altogether.

78

Announce Your Software/Service/Product / Re: Bvckup 2

« on: December 28, 2012, 02:52 PM »

Scanned C:\ with about 100K files in 20K directories.
HDD + W7/64bit + 4 cores.

[Select]

                  COLD       WARM

   1 thread   -  132 sec    2.55 sec
   4 threads  -  102 sec    1.24 sec
   8 threads  -   96 sec    1.32 sec
  64 threads  -   88 sec    1.44 sec
 256 threads  -   84 sec    1.52 sec
 512 threads  -   82 sec      -

In other words, parallel scanning yields ~2x speed up for warm caches, ~1.5x speed up for cold caches

79

Announce Your Software/Service/Product / Re: Bvckup 2

« on: December 27, 2012, 12:10 PM »

I've tried running few cold-cache test and got numbers that are wildly different - from 131 seconds to 70 to 24. This is on a box that was shutdown, then powered on (with network connection disabled, with Windows Search and Windows Update services disabled, virtually no active tasks in the Task Manager and no resident antivirus/malware apps... so theoretically there's nothing that would actively populate disk index cache on boot). I will see what I've missed in a bit and re-run the tests.

With regards to getting 3x speed up on a warm cache - what's not to like? Arguably, this is the use case for real-time backups, with an operational profile consisting largely of frequent scans of small number of selected locations.

80

Announce Your Software/Service/Product / Re: Bvckup 2

« on: December 27, 2012, 09:49 AM »

Updated verbose mode to dump the timing profile for FindFileFirst/Next/Close calls.
Please re-grab the .exe's from the original links.

The verbose output looks something like this now -

[Select]

------ config ------
Location:    n:\
CPU cores:   8
Threads:     16
Depth first: Yes

----- scanning -----
 32625 ms |      97964 files,      13192 folders

------ stats -------
FindFirstFileEx:
     0 ms        2054  |    10 ms           1  |   100 ms           -
     1 ms        1070  |    20 ms        1754  |   200 ms           1
     2 ms         859  |    30 ms         972  |   300 ms           -
     3 ms         708  |    40 ms         548  |   400 ms           1
     4 ms         421  |    50 ms         558  |   500 ms           1
     5 ms         242  |    60 ms         409  |   600 ms           -
     6 ms         143  |    70 ms         425  |   700 ms           -
     7 ms         175  |    80 ms         405  |   800 ms           -
     8 ms         211  |    90 ms         315  |   900 ms           -
     9 ms         221  |   100 ms         307  |  1000 ms           -

FindNextFile:
     0 ms      135181  |    10 ms           -  |   100 ms           -
     1 ms        1186  |    20 ms         127  |   200 ms           -
     2 ms          60  |    30 ms         201  |   300 ms           -
     3 ms          12  |    40 ms         151  |   400 ms           -
     4 ms           6  |    50 ms         149  |   500 ms           -
     5 ms          12  |    60 ms         139  |   600 ms           -
     6 ms           5  |    70 ms          72  |   700 ms           -
     7 ms           9  |    80 ms          58  |   800 ms           -
     8 ms           8  |    90 ms          43  |   900 ms           -
     9 ms          16  |   100 ms          22  |  1000 ms           -

FindClose:
     0 ms       12412  |    10 ms           -  |   100 ms           -
     1 ms         756  |    20 ms           1  |   200 ms           -
     2 ms          16  |    30 ms           -  |   300 ms           -
     3 ms           1  |    40 ms           -  |   400 ms           -
     4 ms           3  |    50 ms           -  |   500 ms           -
     5 ms           -  |    60 ms           -  |   600 ms           -
     6 ms           -  |    70 ms           -  |   700 ms           -
     7 ms           1  |    80 ms           -  |   800 ms           -
     8 ms           -  |    90 ms           -  |   900 ms           -
     9 ms           -  |   100 ms           -  |  1000 ms           -

------ result ------
16 threads, 32625 ms

"20 ms   1754" means there were 1754 calls that took between 20 and 30 ms.

"500 ms   1" means there was 1 call that took between 500 and 600 ms.

81

Announce Your Software/Service/Product / Re: Bvckup 2

« on: December 27, 2012, 05:23 AM »

Got any benchmark code you're willing to share? I'd be interested in trying it out on my own system, I'm afraid I didn't keep the stuff I wrote back then (and there were no threaded versions anyway).

-f0dder (December 19, 2012, 06:01 PM)

Will do in a bit. I assume the command-line version is OK?

-apankrat (December 19, 2012, 03:50 PM)

Sure thing. Would be nice with some source as well, but I can understand if it'll be too time-consuming to remove dependencies on code you want to keep private :)

-f0dder (December 20, 2012, 02:03 PM)

Oki-doki.

bvckup2-demo2.exe - 32bit version
bvckup2-demo2-x64.exe - 64bit version, for nerds and rich kids

It's a console app, takes a path to be scanned and few other parameters:

[Select]

Syntax: bvckup2-demo2.exe [-v] [-D | -B] [-t <threads>] location-to-scan

  -v        verbose
  -D        depth first, scan child then sibling directories   (default)
  -B        breadth first, scan sibling then child directories
  -t        number of threads to use

  Thread count defaults to the number of CPU cores if not specified.

Scanning with all defaults should yield something like this -

[Select]

C:\Temp>bvckup2-demo2.exe C:\
8 threads, 840 ms

In a chatty mode it will look something like this -

[Select]

C:\Temp>bvckup2-demo2.exe -v C:\

------ config ------
Location:    C:\
CPU cores:   8
Threads:     8
Depth first: Yes

----- scanning -----
   840 ms |     104024 files,      28599 folders


------ result ------
8 threads, 840 ms

I'll run some tests and post my numbers. f0dder, if you have time, let's see yours too :)

82

Announce Your Software/Service/Product / Re: Bvckup 2

« on: December 21, 2012, 05:59 AM »

Thanks for those who submitted the error reports. There were two problems (one with the demo throwing an exception on older XP boxes - worked out to be an incorrect API documentation, and the other one was with UAC-less setups - the program didn't realize it had full admin rights as is). Both are fixed now, thanks a lot for helping. Please re-test at will if still interested.

83

Announce Your Software/Service/Product / Re: Bvckup 2

« on: December 20, 2012, 01:36 PM »

Hm, you might have a point wrt. warm cache querying - but have you tested the code across several OSes, especially pre-Vista? That's when Microsoft started doing a lot of work on lock-free data structures and algorithms in the kernel. Have you tested on XP and below?

-f0dder (December 19, 2012, 06:01 PM)

I tested on XP, but I foolishly lent my copy of Win 3.11 to someone so 'm afraid it's going to stay just the XP for now.

I tried depth- vs. breadth-

-apankrat (December 19, 2012, 03:50 PM)

Consistent 10% difference

Got any benchmark code you're willing to share? I'd be interested in trying it out on my own system, I'm afraid I didn't keep the stuff I wrote back then (and there were no threaded versions anyway).

Will do in a bit. I assume the command-line version is OK?

What about Symlinks, Hardlinks and Junctions? Do you handle those correctly, and have they given you much headache?

You bet they did, but I'd like to think I have them sorted out. See here.

I skipped the hardlinks though. That'd be chasing a very far end of the tail of the demand curve, I just don't have time for this now.

84

Announce Your Software/Service/Product / Re: Bvckup 2

« on: December 19, 2012, 03:50 PM »

Have a little faith, will you? :)

First of all, in a lot of cases scanning will effectively be equal to fetching data from the disk index cache. Especially, if I am to repeatedly scan the same spot in the file system. In this case, interrogating the API in parallel manner (read - using multiple threads) gives quite a bit of a boost as the performance is actually CPU-bound. For example:

[Select]

 Scanning C:\...
  1 threads -  2411 ms
  4 threads -  1021 ms
  8 threads -   871 ms

This is with warmed up cache. C:\ was scanned in full immediately before this test. Interestingly enough, playing with the order, in which sub-directories are queued for scanning, can speed things up by additional 5-10%:

[Select]

 Scanning C:\...
  1 threads -  2293 ms
  4 threads -   955 ms
  8 threads -   777 ms

Secondly, similar speed benefits apply in case of a cold cache. If you look at a sequential scanning with one thread, there will always be time when the OS is not doing anything for us, because our app is busy copying what it got from the OS into its own data structures. So if we have 2+ threads pulling at the API, it eliminates these idle OS times. Also, I've been messing with this stuff for a while now and I haven't seen the disk performance getting killed, nowhere close to it. I am guessing there's quite a bit of aggressive caching and read-ahead activity happening, because that's basically what I would've done if I were implementing this part of the OS.

Thirdly, the problem of the OS sitting idle becomes even more pronounced when you do an over-the-network scan. The presence of network latency translates into 80% of the time spent in the API call wasted on waiting for a network reply. You keep firing multiple calls in parallel - you fill up the OS request queue on the other end more evenly. The speed ups I saw when testing against my NAS over the WiFi were in the order of 20-50 times.


I got the point re: marketing speak though. I will try and back it up with the graphs :)


With regards to the MFT/USN - I really don't want to descent to that level. I considered using USN, for example, for move detection and it is - basically - a support hell. As much as I love troubleshooting NTFS nuances, this is just not my cup of tea.

85

Announce Your Software/Service/Product / Re: Bvckup 2

« on: December 19, 2012, 04:27 AM »

Aye... good points, gentlemen. I am just not feeling this name+version approach, the "2" somehow sticks out and I wish it wasn't there :)

86

Announce Your Software/Service/Product / Re: Bvckup 2

« on: December 19, 2012, 03:50 AM »

Why not?

87

Announce Your Software/Service/Product / Re: Bvckup 2

« on: December 19, 2012, 03:28 AM »

The first tech demo is out, and I feel obligated to dedicate it to DNF for being such a great inspiration.

http://bvckup2.com/news/18-12-2012

88

Living Room / Re: Why did it never occur to me.. You can wash a keyboard in water.

« on: November 23, 2012, 01:57 AM »

You should try baking next

BTW, I'd blow dry the keyboard though. Where I am at, the water is really hard. Even a simple drop of water dries into a visible circle, so letting the keyboard dry naturally is likely to result in a lot of extra calcium in the insides.

89

General Software Discussion / Re: Small free form db/notes software

« on: November 22, 2012, 03:39 AM »

A spreadsheet software like Excel fits the bill. A bit of an overkill, but not if you already have Office installed.

90

General Software Discussion / Re: Thoughts on this sort of UI styling?

« on: November 21, 2012, 03:52 PM »

Alright, what about this sort of approach? Less decoration, more layout, more contrast and a bit more texture.

91

General Software Discussion / Re: Thoughts on this sort of UI styling?

« on: November 20, 2012, 01:50 PM »

Paul, thanks for a thoughtful comment, but I'm afraid you got off a wrong assumption. This is not a program that is meant for repetitive use, it's a single-run disposable. It will go to the people on the mailing list, so it's probably OK to assume that whoever downloads and runs it would also have some interest to read through the explanatory text. With regards to the redundancy - I see your point, but I disagree. I am all for minimalism in the UI (see, for example, here), but this is not a place to overdo it.

Jibz, very good iteration, thanks. I ran this screenshot by /r/software (here) and the window being very tall was the #1 remark. I also have to concur that it looks rather nice without the texture. I had the texture put in first, and then added all other decorations. Never occurred to me to try and take it away.

40hz, Stoic Joker, f0dder - no, not buttons... but I suspect you suspect that already :)

--

In any case, I was looking to check if this sort of customization level is OK with everyone, and it looks like it is. So I got my answer.

Also, having slept on it, I am now reworking this particular proggy from being a "Test" into a "Demo". With this I can trim some text and simplify the layout.

92

General Software Discussion / Re: Thoughts on this sort of UI styling?

« on: November 19, 2012, 10:57 AM »

Thanks, boss :)

I just can't bring myself to use unmodified UI controls. My eyes bleed, and I don't like that. Unfortunately all this designy stuff comes at the extra expense of real estate that is consumed by the whitespace, padding, margins, etc. What I'm trying to do here is to flesh out and evaluate the visuals and overall styling. This is the hard part. Reducing margins is a piece of cake.

93

General Software Discussion / Re: Thoughts on this sort of UI styling?

« on: November 19, 2012, 10:35 AM »

Thank you, gentlemen.

Jibz, the window frame came with Windows. Unfortunately there's not much I can do about it without going overboard with customizations (and I'd rather not do that).

Renegade, aesthetics are as important, spit it out :) Agree on all other points, thanks.

tomos, agreed on line swap, already implemented.

94

General Software Discussion / Thoughts on this sort of UI styling?

« on: November 19, 2012, 02:21 AM »

Hi guys,

I am in the process of designing a UI for a Windows app, and the stylistic/visual side of it is proving to be quite a task. Unlike 5-10 years ago, the spectrum of employed design choices is now pretty wide. Native UIs are still here, but they are being supplemented by better looking UIs like those of Chrome, Firefox, Evernote, Acronis and others. There are also UIs that go for full customization and resemble more a web app than a Windows program, e.g. Kaspersky and iTunes.

What I'm trying to do is to find the balance between sprucing app the UI and still retaining the native look and feel. I have sketched out few options, and I'd like to get an opinion on one of them.



Does these customizations come across (even in a slightest way) as annoying or inappropriate? I'll take all opinions... i.e. not just from the UI purists :)

Thanks

95

General Software Discussion / Re: Antivirus-less setup

« on: September 28, 2012, 01:39 PM »

Apparently, EMET 3.5 is available - http://www.microsoft...etails.aspx?id=30424

96

General Software Discussion / Re: Antivirus-less setup

« on: September 28, 2012, 01:20 PM »

I gotta toss this EMET thing on top of my to-do list. It does look quite interesting. Any common hiccups to watch/look for in a network sized rollout of this that you know of?

-f0dder (September 28, 2012, 11:01 AM)

Any tech papers on how it works? I'm guessing it's doing some sort of ASLR and perhaps something else?


(edit) Ah, nevermind. Here's the list of what it does -

 

97

General Software Discussion / Re: Antivirus-less setup

« on: September 28, 2012, 09:24 AM »

Uh, why not simply disable the js, activex, and similar garbage in the mail client? Even using plain text instead of rich text formats is a viable option.

-eleman (September 28, 2012, 08:33 AM)

Because a properly crafted email may exploit a vulnerability in the mail client upon arrival, well before it's viewed.

98

General Software Discussion / Re: Antivirus-less setup

« on: September 28, 2012, 08:29 AM »

what other security measures do you take  (if any) ?

-tomos (September 28, 2012, 07:23 AM)

I rarely install anything. Whatever I install, I check the signatures and run it through online multi-scanners. I have all but TCP/IP switched off on all interfaces and I'd test the box now and then with rootkit and antivirus scanners. The reason I decided to bring this subject up is that I had my mail client crash while opening an email. The email was not malicious, but it made me realize that the mail client really needs to be sandboxed or isolated. Similarly, the browser needs to be isolated too, but it's of a lesser importance, because I can usually control where I surf (while with emails I don't control what I receive).

I wish there were an equivalent of chroot jail on Windows, but there's not. So what I ended up doing is setting up the mail client in a VM. It's not the most convenient option, but it's a very simple one.

Does anyone have any experience with Sandboxie or something similar? Sounds right, but not sure how well it works.

99

General Software Discussion / Re: Antivirus-less setup

« on: September 28, 2012, 03:03 AM »

@TaoPhoenix - Good point, forgot to mention that all network interfaces are locked down in some form or fashion. Basically, there are few ways to get the machine infected with no interaction - mail, browser and network services. If these are sandboxed, the rest of the machine doesn't really need an always-on AV, does it?

@eleman - Ack, same here wrt virustotal.

100

General Software Discussion / Antivirus-less setup

« on: September 28, 2012, 02:30 AM »

Does anyone here run their primary machine without a resident (always on) antivirus? Not because of the ignorance, negligence or laziness, but with it being a conscious choice. Simply be careful with installing and running 3rd party code (and have a dedicated VM for testing and trying random stuff).

Just curious if I'm an odd one out :-)