Main Area and Open Discussion > General Software Discussion

"Enable advanced performance" - the optimization that isn't

<< < (2/3) > >>

Edvard:
Boy did I have to dig for that one on my XP box at work!

So until 2.6.30 gets the patches, us Linux folks should tweak our writebacks:
Unfortunately, the default VM tuning parameters, which can be controlled by /proc/sys/vm/dirty_expire_centiseconds and /proc/sys/vm/dirty_writeback_centiseconds, means that in practice, a newly created file won't hit disk until about 45-150 seconds later...
--- End quote ---
or
Another solution is to make sure your system is reliable. :-)
--- End quote ---
but then again...
The final solution, is we need properly written applications and desktop libraries. The proper way of doing this sort of thing is not to have hundreds of tiny files in private ~/.gnome2* and ~/.kde2* directories. Instead, the answer is to use a proper small database like sqllite for application registries, but fixed up so that it allocates and releases space for its database in chunks, and that it uses fdatawrite() instead of fsync() to guarantee that data is written on disk.
--- End quote ---

 :huh: A Registry for Linux?

Do I smell torches? and are those pitchforks?

f0dder:
Well, imho the writeback delay for EXT4 is a bit on the insane side, but it's still the other software that's broken, not the filesystem. That this huge delay smokes out some of the bad software (especially when it's as high-profile as KDE!), I think it's worth keeping for a while. It's amazing when developers don't read the documentation, that you don't get any sync guarantees without a fsync call is pretty well-documented :)

Btw, the database approach isn't recommended as a fix for the stability issue, but to avoid the performance penalty of fsync'ing a lot of small config files. But it would definitely be a good idea, the zillion small config of *u*x is retarded :)

40hz:
Well, imho the writeback delay for EXT4 is a bit on the insane side, but it's still the other software that's broken, not the filesystem. That this huge delay smokes out some of the bad software (especially when it's as high-profile as KDE!), I think it's worth keeping for a while. It's amazing when developers don't read the documentation, that you don't get any sync guarantees without a fsync call is pretty well-documented :)
-f0dder (March 12, 2009, 04:27 PM)
--- End quote ---

That's an excellent point. I'm 100% with f0dder on this one.

A lot of application programmers tend to blame the operating system for their coding errors and general sloppiness. The complaints about EXT4's write-delay reminds me of those Windows developers who use undocumented system calls and then complain when Microsoft changes something which breaks their app.

The open source people have no excuse for not reading the docs and looking at the underlying code they're calling - because it's available. That was the whole point of open source.

Here's part of developer Ted Tso's comment up on Cannonical's Ubuntu Launchpad about the problem:

Link: https://bugs.edge.launchpad.net/ubuntu/+source/linux/+bug/317781/comments/45

So, what is the problem. POSIX fundamentally says that what happens if the system is not shutdown cleanly is undefined. If you want to force things to be stored on disk, you must use fsync() or fdatasync(). There may be performance problems with this, which is what happened with FireFox 3.0[1] --- but that's why POSIX doesn't require that things be synched to disk as soon as the file is closed.

[1] http://shaver.off.net/diary/2008/05/25/fsyncers-and-curveballs/

So, why wasn't this a problem before in the past? Well, ext3 by default has a commit interval of 5 seconds, and has data=ordered. What does this mean? Well, every 5 seconds, the ext3 journal is committed; this means that any changes in since the last commit are now guaranteed to survive an unclean shutdown. The journalling mode data=ordered means that only metadata is written in the journal, but data is ordered; this means that before the commit takes place, any data blocks are associated with inodes that are about to be committed in that transaction will be forced out to disk. This is primarily done for security reasons; if this was not done, then any newly allocated blocks might still contain previous data belonging to some other file or user, and after a crash, accessing that file might result in a user seeing someone else's mail or p0rn, and that's unacceptable from a security perspective.

However, this had the side effect of essentially guaranteeing that anything that had been written was guaranteed to be on disk after 5 seconds. (This is somewhat modified if you are running on batteries and have enabled laptop mode, but we'll ignore that for the purposes of this discussion.) Since ext3 became the dominant filesystem for Linux, application writers and users have started depending on this, and so they become shocked and angry when their system locks up and they lose data --- even though POSIX never really made any such guaranteed. (We could be snide and point out that they should have been shocked and angry about crappy proprietary, binary-only drivers that no one but the manufacturer can debug, or angry at themselves for not installing a UPS, but that's not helpful; expectations are expectations, and it's hard to get people to change those expectations, even when they aren't good for themselves or the environment --- such as Americans living in exburgs driving SUV's getting shocked and angry when gasoline hit $4/gallon, and their 90 minute daily commute started getting expensive. :-)
--- End quote ---

Maybe, when the whiners stop whining, they'll come to realize it's a blessing in disguise.



---

BTW: I couldn't get into the Slashdot link. (Possibly because it's been slashdotted? ;)) but here's a link to the original article Slashdot is quoting in case anybody's interested.

http://www.h-online.com/open/Possible-data-loss-in-Ext4--/news/112821

While you're in there, take a peek around the rest of Heise's H-Open website. It's a great resource.

  :-*

Dirhael:
Turns out that this feature have been renamed and given a new description in Windows 7, perhaps to avoid confusion?

f0dder:
Turns out that this feature have been renamed and given a new description in Windows 7, perhaps to avoid confusion?
 (see attachment in previous post)
-Dirhael (March 13, 2009, 09:23 AM)
--- End quote ---
:up: - about time. Still shouldn't be in the GUI though, and the description is still missleading.

Navigation

[0] Message Index

[#] Next page

[*] Previous page