Linus torvalds about PAE

PAE really really sucks.

The biggest single reason to go 64-bit is exactly because of physical address space. Your virtual address space needs to bea multiple of the physical one: when you hit 1GB of RAM, 32-bit virtual memory is no longer acceptable. You literally do need more virtual memory than physical.

PAE turned that very simple fact on its head, and screwed things up royally. Whoever came up with the idea was totally incompetent, and had forgotten all the DOS HIGHMEM pains. There’s a damn good reason why we left the 286 behind, and started using 386′s, instead of having HIGHMEM crap with windows into a bigger physical space.

Repeat after me:

virtual space needs to be bigger than physical space. Not “as big”. Not “smaller”. It needs to be bigger, by a factor of at least two, and that’s quite frankly pushing it, and you’re much better off having a factor of ten or more.

Anybody who doesn’t get that is a moron. End of discussion.

Reasons for why you need a bigger virtual space:

– you need to map that physical memory somehow, and no, tiny windows into the physical memory simply do not cut it! If you cannot have normal pointers to the physical space, it immediately means that you need to jump through serious hoops to get there.

– you additionally need to be able to remap things in alternate ways (ie user space) or make space for non-memory issues (virtual page tables, IO, you name it)

Ergo, a factor-of-two is a requirement. PAE was a total and utter disaster.

Yes, Linux supported it, and probably did so better than anybody else. But “better than anybody else” still wasn’t very good. Because you couldn’t use normal pointers to point to arbitrary physical memory, all the memory that couldn’t be accessed directly (ie anythign that didn’t fit in the virtual address map, which also had the user space memory in it) was basically limited to “special uses only”.

So you could allocate user pages in it, but you had huge problems with things like internal kernel data structures, which can be the bulk of your memory needs under some (not that unusual) loads. Directory caches, inodes, etc couldn’t use it, and in general it meant that under Linux, if you had more than 4GB of physical memory, you generally ran into problems (since only 25% of memory was available for normal kernel stuff – the rest had to be addressed through small holes in the tiny virtual address space).

I’m not at all surprised that Windows didn’t push PAE either. It was a total braindamage. I bet they supported it in the server offerings just because they had to, and I bet they did a much worse job of it than Linux did, and the reason you can do that with servers is that the loads are much easier, and you can expect maintainers to set magic config entries to tweak it to make it appear to work well for any particular load, when in reality it is fragile as hell and works only with duct-tape and prayers.

That kind of “duct-tape and prayers and lots of specialized knowledge about the load” is simply not possible in a desktop environment. Yeah, users have prayers, but they lack the duct-tape and the knowledge to work around the problems.

And dammit, in this age and date when almost everybody has a gigabyte of RAM in any new machine, anybody who still thinks that “not that many people need 64-bits” is simply not aware of what he’s speaking of.

Go back and play with HIGHMEM.SYS on a 286, and stop blathering crap. When you’ve spent the last ten years of your life working with HIGHMEM.SYS, then you can come back and tell me that we still don’t need 64 bits. Until that is the case, anybody who still doesn’t get why 64 bits is a requirement should just shut up rather than make a total fool of himself.

So repeat after me: PAE didn’t ever really fix anything. It was a mistake. It was just a total failure, and the result of hw engineers not understanding software.

So no, PAE does not mean that you can use more than 4GB of RAM. Even before PAE, the practical limit was around 1GB, and PAE didn’t move that post a fraction of an inch!