"Enable advanced performance" - the optimization that isn't

[f0dder]

I used to think this setting was a decent performance thing, and used to enable it. My impression, based on various faulty wannabe-tech-knowledgy blogs was that "Enable write caching" was only the usual filesystem caching, whereas "Enabled advanced performance" meant the actual disk write cache (ie., the on-disk memory buffer, usually 8, 16 or 32 megabytes).

[attachimg=#]

It's a while since I learned the true meaning, and turned off the setting in shock, and then didn't think much more about it. But after yesterday's slashdot article Apps That Rely On Ext3's Commit Interval May Lose Data In Ext4 (and especially a lot of the moronic comments, sigh) I remembered this setting again, and thought it would be a decent thing warning about it here at DC.

Basically, what it does is making the Windows API function FlushFileBuffers() do nothing. This API is meant to flush the OS's write cache for a file to disk, and is the only way to guarantee that your data goes to disk in a modern OS. Enabling the setting probably won't affect a lot of software positively (since not a lot of software actually uses FlushFileBuffers()), but for things like databases this is crucial to ensure data consistency across crashes.

Read the technet link for a more in-depth description, and go turn that checkmark off if you have it enabled

[Darwin]

Interesting, thanks f0dder. I checked my Vista 64 bit install and that setting had been enabled, so I disabled it. My XP Pro 32 bit install doesn't even offer the option. I'm not sure if I set that option myself or not - though I recognised it, so it must have been me!

[f0dder]

I definitely have had it enabled

IMHO it's wrong of MS to offer this as part of the GUI (people that "need" this should be forced to edit the registry!), and the description is very very vague and misleading.

"Enable write caching on disk" is good though

[Darwin]

"Enable write caching on disk" is good though

I left that one enabled... I probably stumbled across this "tweak" around the time I upgraded my HD in early January. It's the wording that's so familar AND the fact that it was enabled - MS doesn't ship Vista with it enabled out of the box, correct?

[f0dder]

"Enable write caching on disk" is good though

I left that one enabled... I probably stumbled across this "tweak" around the time I upgraded my HD in early January. It's the wording that's so familar AND the fact that it was enabled - MS doesn't ship Vista with it enabled out of the box, correct?

I would be very surprised if they do - it would've been all over the media (if more people knew what the setting does, anyway ). It definitely wasn't enabled when I installed Vista64 on my laptop.

[Edvard]

Boy did I have to dig for that one on my XP box at work!

So until 2.6.30 gets the patches, us Linux folks should tweak our writebacks:

Unfortunately, the default VM tuning parameters, which can be controlled by /proc/sys/vm/dirty_expire_centiseconds and /proc/sys/vm/dirty_writeback_centiseconds, means that in practice, a newly created file won't hit disk until about 45-150 seconds later...

or

Another solution is to make sure your system is reliable. :-)

but then again...

The final solution, is we need properly written applications and desktop libraries. The proper way of doing this sort of thing is not to have hundreds of tiny files in private ~/.gnome2* and ~/.kde2* directories. Instead, the answer is to use a proper small database like sqllite for application registries, but fixed up so that it allocates and releases space for its database in chunks, and that it uses fdatawrite() instead of fsync() to guarantee that data is written on disk.

A Registry for Linux?

Do I smell torches? and are those pitchforks?

[f0dder]

Well, imho the writeback delay for EXT4 is a bit on the insane side, but it's still the other software that's broken, not the filesystem. That this huge delay smokes out some of the bad software (especially when it's as high-profile as KDE!), I think it's worth keeping for a while. It's amazing when developers don't read the documentation, that you don't get any sync guarantees without a fsync call is pretty well-documented

Btw, the database approach isn't recommended as a fix for the stability issue, but to avoid the performance penalty of fsync'ing a lot of small config files. But it would definitely be a good idea, the zillion small config of *u*x is retarded

[40hz]

Well, imho the writeback delay for EXT4 is a bit on the insane side, but it's still the other software that's broken, not the filesystem. That this huge delay smokes out some of the bad software (especially when it's as high-profile as KDE!), I think it's worth keeping for a while. It's amazing when developers don't read the documentation, that you don't get any sync guarantees without a fsync call is pretty well-documented

That's an excellent point. I'm 100% with f0dder on this one.

A lot of application programmers tend to blame the operating system for their coding errors and general sloppiness. The complaints about EXT4's write-delay reminds me of those Windows developers who use undocumented system calls and then complain when Microsoft changes something which breaks their app.

The open source people have no excuse for not reading the docs and looking at the underlying code they're calling - because it's available. That was the whole point of open source.

Here's part of developer Ted Tso's comment up on Cannonical's Ubuntu Launchpad about the problem:

Link: https://bugs.edge.launchpad.net/ubuntu/+source/linux/+bug/317781/comments/45

So, what is the problem. POSIX fundamentally says that what happens if the system is not shutdown cleanly is undefined. If you want to force things to be stored on disk, you must use fsync() or fdatasync(). There may be performance problems with this, which is what happened with FireFox 3.0[1] --- but that's why POSIX doesn't require that things be synched to disk as soon as the file is closed.

[1] http://shaver.off.net/diary/2008/05/25/fsyncers-and-curveballs/

So, why wasn't this a problem before in the past? Well, ext3 by default has a commit interval of 5 seconds, and has data=ordered. What does this mean? Well, every 5 seconds, the ext3 journal is committed; this means that any changes in since the last commit are now guaranteed to survive an unclean shutdown. The journalling mode data=ordered means that only metadata is written in the journal, but data is ordered; this means that before the commit takes place, any data blocks are associated with inodes that are about to be committed in that transaction will be forced out to disk. This is primarily done for security reasons; if this was not done, then any newly allocated blocks might still contain previous data belonging to some other file or user, and after a crash, accessing that file might result in a user seeing someone else's mail or p0rn, and that's unacceptable from a security perspective.

However, this had the side effect of essentially guaranteeing that anything that had been written was guaranteed to be on disk after 5 seconds. (This is somewhat modified if you are running on batteries and have enabled laptop mode, but we'll ignore that for the purposes of this discussion.) Since ext3 became the dominant filesystem for Linux, application writers and users have started depending on this, and so they become shocked and angry when their system locks up and they lose data --- even though POSIX never really made any such guaranteed. (We could be snide and point out that they should have been shocked and angry about crappy proprietary, binary-only drivers that no one but the manufacturer can debug, or angry at themselves for not installing a UPS, but that's not helpful; expectations are expectations, and it's hard to get people to change those expectations, even when they aren't good for themselves or the environment --- such as Americans living in exburgs driving SUV's getting shocked and angry when gasoline hit $4/gallon, and their 90 minute daily commute started getting expensive. :-)

Maybe, when the whiners stop whining, they'll come to realize it's a blessing in disguise.



---

BTW: I couldn't get into the Slashdot link. (Possibly because it's been slashdotted? ) but here's a link to the original article Slashdot is quoting in case anybody's interested.

http://www.h-online.com/open/Possible-data-loss-in-Ext4--/news/112821

While you're in there, take a peek around the rest of Heise's H-Open website. It's a great resource.

[attach]

[Dirhael]

Turns out that this feature have been renamed and given a new description in Windows 7, perhaps to avoid confusion?

[f0dder]

Turns out that this feature have been renamed and given a new description in Windows 7, perhaps to avoid confusion?
(see attachment in previous post)

- about time. Still shouldn't be in the GUI though, and the description is still missleading.

[40hz]

Turns out that this feature have been renamed and given a new description in Windows 7, perhaps to avoid confusion?
(see attachment in previous post)

- about time. Still shouldn't be in the GUI though, and the description is still missleading.

Yeah. Would that more technology had functional names like "Space Shuttle" and "motion detector." What they're named is what they do.

I don't have a problem with potentially dangerous settings being controlled via a GUI. The most I would suggest is that they require Administrator privileges, and possibly a second (i.e. 'Are you sure?') confirmation before the new setting takes effect.

There's a limit to how much we should be expected to shield fools from acting foolishly.

[f0dder]

There's a limit to how much we should be expected to shield fools from acting foolishly.

Not really, no - if it hadn't been a GUI item, there would be that much misinformation about it on the net, I bet. (OK ok, eventually some moron would stumble across it with a hexeditor on a .sys file or some obscure technet page... but at least a technet page could have described the bloody thing).