32-bit Windows and the dreaded (and misunderstood) 4GB RAM limit

[Darwin]

OK, cards on the table: the peon that can't understand the 4GB RAM limit is writing this post. 'Nuff said. What I'd like an answer to, if anyone cares to take a crack at it, is whether or not my computer will have access to 4GB of RAM (even if Windows can't see it). I've seen conflicting comments on this issue, hence my lack of comprehension. I'm hoping a hardware guru like f0dder can enlighten me a bit...

FWIW I could not resist a 2GB Kingston DDR2 SODDIM for $35. I have a Vista notebook that came with 512MB of dedicated graphics memory and 3GB of RAM (one 2GB SODDIM and one 1GB SODDIM). I swapped out the 1GB chip and put the new Kingston stick in and rebooted. Windows Customer Experience (or whatever it's called) reports that 4GB of RAM are installed. System Properties and Task Manager both report 3070MB... I had hoped I'd be one of those lucky people that gets 3.25 or even 3.5 GB of recognised RAM. C'est la vie. Unless there's a compelling reason to stick with 4GB of RAM in this machine (like it'll still be benefiting me in the background), I'll swap the new 2GB stick out. Coincidentally, I have a friend staying with us for two weeks with an XP notebook that takes DDR2 and it's only got a gig of RAM. 2.5GB of RAM should give XP wings!

Oh, one other thing. Enabling PAE on a 32-bit system - good idea or bad idea. Discuss.

My gut says bad idea, but I'd love to hear other opinions.

Darwin quits the field with apologies for a rambling and incoherent post...

[Carol Haynes]

I would expect you to have 4Gb - 512Gb of video RAM = 3.5Gb visible to Windows - so the question is where has the other half a gig gone?

I can only presume that you have nealy half a gig of other memory mapped IO devices on your system but it seems unlikely.

What other devices have you got that might use mapped memory? Check you device manager and check for memory mapped devices and see what has gobbled up you new memory!

From what I read (and its been a while so I may be wrong) enabling PAE in 32-bit XP doesn't actually do anything at all.

[tomos]

Windows Customer Experience (or whatever it's called) reports that 4GB of RAM are installed. System Properties and Task Manager both report 3070MB... I had hoped I'd be one of those lucky people that gets 3.25 or even 3.5 GB of recognised RAM

I thought Vista recognised "all" ram
I mean I thought the problem with not recognising ram was only with xp ? 

[Lashiec]

Whoo, I'm having a déjà vu sensation with this thread :-D

The limit of memory supported by Windows 32-bit is a minefield. Even if people with less than 4 GB suffer it, something I was unaware of before. Mark Russinovich published a good write-up of how Windows works in this respect. Of particular interest is the section titled "32-bit Client Effective Memory Limits". Highlights:

While 4GB is the licensed limit for 32-bit client SKUs, the effective limit is actually lower and dependent on the system's chipset and connected devices. The reason is that the physical address map includes not only RAM, but device memory as well, and x86 and x64 systems map all device memory below the 4GB address boundary to remain compatible with 32-bit operating systems that don't know how to handle addresses larger than 4GB. If a system has 4GB RAM and devices, like video, audio and network adapters, that implement windows into their device memory that sum to 500MB, 500MB of the 4GB of RAM will reside above the 4GB address boundary.

(...)

The result is that, if you have a system with 3GB or more of memory and you are running a 32-bit Windows client, you may not be getting the benefit of all of the RAM.  On Windows 2000, Windows XP and Windows Vista RTM, you can see how much RAM Windows has accessible to it in the System Properties dialog, Task Manager's Performance page, and, on Windows XP and Windows Vista (including SP1), in the Msinfo32 and Winver utilities. On Window Vista SP1, some of these locations changed to show installed RAM, rather than available RAM, as documented in this Knowledge Base article.

(...)

Even systems with as little as 2GB can be prevented from having all their memory usable under 32-bit Windows because of chipsets that aggressively reserve memory regions for devices. Our shared family computer, which we purchased only a few months ago from a major OEM, reports that only 1.97GB of the 2GB installed is available.

For example, I have 2 GB of RAM installed, but after reading the article I checked the system, and turns out I don't even have 1.5 GB available.

Erm, not, forget about this line, I looked in the wrong place

[Josh]

Its not a matter of what OS, its the platform. 32 bit platforms, by nature, can only address up to 3.5 GB of ram. So yes, Vista can SEE it, but it can't map it.

[Darwin]

I would expect you to have 4Gb - 512Gb of video RAM = 3.5Gb visible to Windows - so the question is where has the other half a gig gone?

My thinking exactly!

What other devices have you got that might use mapped memory? Check you device manager and check for memory mapped devices and see what has gobbled up you new memory!

Right - I'll check this out... AFAIK, there shouldn't be anything else using mapped memory. However, I am *guessing* that my ethernet cards, sound card (which is integrated), keyboard, touchpad, built-in webcam, may be contributing to this. I'll take a look when I get a chance (I'm on my XP machine at the moment).

From what I read (and its been a while so I may be wrong) enabling PAE in 32-bit XP doesn't actually do anything at all.

This is more or less my take on it as well. Once again - I've seen a lot of conflicting reports. The people I trust say it's a waste of time though (can't remember who those people are, off-hand).

Thanks for taking the time to answer, Carol, and everyone else.

[tomos]

another question:
if this memory is not available to the 32-bit OS - does that mean it's also unavailable to programmes on the computer ?

[Carol Haynes]

Yes - it is simply mapped to devices in the memory map and the actual physical memory is unused.

[Paul Keith]

I'm a newb at this thing. What platform can support 4gb? I thought Vista was 64-bit?

[Carol Haynes]

XP and Vista both come in 32-bit and 64-bit 'flavours'. As I understand it there are restrictions in 64-bit XP to maintain software compatability - though you may be able to enable PAE to extend the addressable memory. I haven't tried Vista 64 but I think it supports up to 8Gb.

The big problem is application compatibility. If software is written and compiled to run on 32 bit windows none of the memory addresses have enough bits to address more than 4Gb of physical memory because they are all 32 bit addresses. If software is compiled for 64 bit windows then it has 64 bit addresses and can address a huge amount of RAM - though MS have restricted the physical amount of RAM that can be actually used artificially. Some server products are less restrictive.

That's a simplistic view (I am simple) and may or may not be wholly accurate but it gives the idea.

[justice]

From http://msdn.microsoft.com/en-us/library/aa366778.aspx#physical_memory_limits_windows_vista:
[attach]

[Paul Keith]

So in the end from the gist of it, I take it now would be the worst time to upgrade your pc?

[justice]

I'm sure any pc you buy now can run 64 bit windows vista. I heard very good things about software compatibility, 64bit vista can still run 32bit programs just fine. So if you think of buying a new pc, just ask in the shop if it's possible to get 64bit vista on it preinstalled and they will make sure your drivers are up to date etc. You'll be set for the future.

If you don't plan to get more than 4gb of ram in the life of your new pc then 32bit is fine too, i don't see 32bit software will die off very quick seeing as how many people are still using XP etc. That's what I reckon, feel free to expand.

[Darwin]

Here's a MS KB about this: http://support.microsoft.com/kb/946003/

Thanks for all of your feedback, everyone. I'm leaning toward making a gift of this RAM to my friend and moving on. I can't see any benefit to be derived from retaining 4GB vs 3GB on this system - the amount available to windows is identical in either scenario. I have another friend with a 32-bit XP Pro system with 4GB RAM and virtually identical specs (he has a much higher spec processor than me but other wise same soundcard, videocard, etc.) and his reports the same amount of installed RAM.

C'est la vie... a fool and his money... 

[Darwin]

Of course, I could always dual-boot my machine with Linux and use all of that glorious RAM...

[f0dder]

Its not a matter of what OS, its the platform. 32 bit platforms, by nature, can only address up to 3.5 GB of ram. So yes, Vista can SEE it, but it can't map it.

Wrong, it's all about the OS (well, and partially BIOS/chipset - more about that in a sec).

What happens, by default, when you plug in 4 gigabytes of RAM in a system, is that it gets mapped physically from 0-4GB - this is the 32bit address space. The problem is that various hardware devices (not just physical expansion cards, but also on-motherboard devices, some CPU structures etc.) are memory mapped - this means that when requests to certain physical memory ranges are generated, your chipset sends those requests to the hardware devices rather than the memory chips, thus shadowing the memory.

Decent chipsets support memory remapping, which relocates RAM addresses - basically "anything goes" I guess, and I've seen many different remappings. But the most typical seems to be relocating entire RAM sticks at a time. With a lot of physical memory in your machine, lots of memory is going to be moved above the 4GB mark, into 64-bit address space. A request for physical address 6GB might then go to RAM address 2GB, while a request for physical address 2GB might be remapped to your GPU memory.

Phew. So far, so good.

When you're working with a 32-bit OS, applications can only see the 32bit address space - no way around that. Under Windows, there's furthermore a 2:2 split of the address space, meaning that each individual application can only access up to 2 gigabytes of memory, the other 2 gigabytes of address space are reserved for kernel use (including, but certianly not limited to, filesystem cache). With a special boot option, you can change this into a 3:1 mapping, allowing individual applications to access up to 3 gigabytes of memory (but only if they're linked with "large address-space aware" linker flag).

With PAE enabled, a 32-bit OS can use up to 64 gigabytes of memory. Each invidiual application can still only see the 32-bit address space, but each address space can be mapped to different portions of physical memory. This has been possible ever since the Pentium Pro. Also, with OS support routines like windows AWE, applications can be allowed to map "windows" to other parts of the system memory, effectively giving you access to lots of memory. It's more complicated to program for than a flat 64bit address space, but it's certainly doable - and it's quite similar to how 16-bit programs used XMS/EMS memory back in the days.

Now, the million dollar question is, if this is possible... why doesn't XP support it? Funny thing is that before the service packs, XP supported 4GB of physical memory. This means that with vanilla XP and PAE enabled (and BIOS/chipset memory remap) you didn't get those meager 3.25GB of memory, but the full 4GB. With SP1, this was changed to 4GB of address space.

Microsoft claims it did this because of buggy drivers, and I guess that's at least half the truth - on Windows, even 32bit drivers have always had to use 64bit addresses when dealing with physical memory kernel APIs, but a lot of moronic driver writers have probably ignored this. Another reason could be that MS wants you to buy 64bit and/or server editions if you want to use lots of memory (32bit server OSes support >4GB memory through PAE).

[mouser]

I want to emphasize something that f0dder mentioned briefly, since i have had some personal experience getting burned by it.

Some motherboard bioses, in combination with other hardware installed on your pc (most notably graphics cards), will find themselves unable to access even 4gb except when using a 64bit OS.

My Asus motherboard when combined with the graphics cards i had, could only access 2gb of ram from Windows XP Pro, even though i had 4gb installed.  This was due to memory mapping issues that i struggled with for a long time and then gave up on.  Some motherboards can handle it via remapping tricks, some cant it seems.

I had to upgrade to Win XP Pro x64 in order to access the full 4gb, which i have mixed feelings about (wouldnt recommend it unless you really need it).  If my motherboard had been able to access 3+ gb i would have stuck with Win XP Pro 32bit.

[f0dder]

mouser: without remapping, you would probably have gotten 3.25gig or whatever - with remapping, you'll likely be down to 2gig because entire memory modules are (usually) remapped at a time. Your board/BIOS probably did remapping without an option to turn it off.

[mouser]

there is a remapping option in my bios but both settings yielded failure.  i spent a lot of time messing with it until i gave up.  in one setting the bios saw the memory on post but windows didn't, in the other, even the bios couldn't see it.  i also tried putting the memory in different slots, etc. no luck.  could have been because of my 2 graphics cards, who knows.

[f0dder]

in the other, even the bios couldn't see it.

That sounds buggy!

Except if the "other" setting was no remapping, and the BIOS didn't report the memory because it was gobbled up by your video cards. Those can consume quite an amount of address space - my guess would be the amount of GPU video memory + a little extra for memory remapped registers.

If you want to stay with 32bit Windows but still have your RAM and eat it, you could always install one of the server editions and transmogrify it to a more workstation feel, but personally I prefer 64bit

[Darwin]

So... bottom line: I might as well give the new stick of RAM to a friend, right? It causes me near physical pain to think about that 1GB SODDIM gathering dust (I don't want to give THAT away because at some point I may need to get warranty work done on the notebook and want to be able to return it to factory spec before doing so) unless I'm benefiting from my capriciousness  I'd rather write off the $35 and let someone else actually benefit from it...

[cranioscopical]

Some motherboard bioses, in combination with other hardware installed on your pc (most notably graphics cards), will find themselves unable to access even 4gb except when using a 64bit OS.

And, for those of us without in-depth knowledge of the issue, it's very hard to tell in advance what combination will or will not work.
The mobo that I recently installed has an entry on the ASUS site warning that more than 3Gb RAM will likely result in LESS than 3Gb being recognized (ASUS P5E3 Premium).

After some deliberation, I decided that was a CYA disclaimer and went ahead with 4Gb... and I was lucky, it can see the typical 3Gb+.
There was a feeling of unease, though, as I did have to make that choice in the knowledge that I might need to buy a different mobo if the P5E3 didn't work out. Nobody that I found was able to give a firm assurance that this board would act as I wanted.

[f0dder]

Just move to 64bit OSes already, and put more pressure on those few lame-ass software writers who can't get their software 64-bit compatible. FYI, that does not necessarily mean rewriting the software to compile cleanly for 64-bit (which shouldn't be much effort unless you're following bad coding practices), it simply means not doing mindlessly stupid and dirty tricks.

Drivers and shell extensions are some of the only software that needs to be rewritten for 64bit platforms... and of course plugins for apps that go 64bit. Other than that, non-dirty 32bit software works like a charm on 64bit OSes.

[nontroppo]

The larger your graphics card the less memory you can use, an 2008 alienware gaming rig with 4GB of recommended RAM and a 1GB graphics card ends up with only 2.2GB useable!

I don't know if this affects windows PCs, but my macbook only allows 3GB addresable space (a chipset limitation), but putting 4GB using paired 2GB DIMMs rather than 3GB of 2GB+1GB allows faster graphics performance (something to do with 128bit addressing that is enabled by paired DIMMs). Perhaps this may affect some PCs too?

Linux and OS X support PAE with their 32bit variants AFAIK. I have a most wonderful 10GB in my desktop workstation; makes using Adobe suite along with Matlab while virtualising Windows XP a trivial experience 

[Cavalcader]

One thing I've been finding around the Web is the concept of increasing user mode addressing to 3 gigs instead of the default 2 (for use by applications), leaving the Vista kernel 1 gig instead of 2 (presuming a total of 4 gigs).

BCDEDIT /set increaseuserva 3072

XP has its own method for doing the same thing, and of course you need admin privs. Anybody know of any drawbacks for this one?