- Hardware Level Memory management
- Virtual Memory
- Introduction to windows memory management
First, I’d Like to welcome you to the blog since this is -probably- the first post here. Also I have to ask you to take a deep breath because you’re going to have a long day reading one blog and that’s of course if you made it to the end by one day. Buckle up Dorothy!
1- Hardware Level Memory Management
Before I mention any details about windows we’ll have to discuss memory management on hardware level and I am telling you that you will need it so probably you don’t want to skip it unless you’re familiar with it.
1.A Virtual Memory
Intel provides several mechanisms to provide any running software (and I mean a software running over Intel and not over any OS) over it to use virtual memory. Virtual memory in short is a way to provide an interface to physical memory (RAM) using what is called Linear Addresses which translates to physical memory in the end. the reason for using virtual memory is that you can work over a 4GB of addresses even if you’re using 65 MB Memory (Oh Lord CMOS!). anyhow this evolved as well as a perfect way to protect different processes from accessing each others memory and provide much more information about memory thunks. For instance, you can mark a thunk of memory as readable by only kernel level(ring 0) or by any other ring like user level(ring 3) or determine whether a thunk of memory is accessed before or not or mark a thunk of memory as readable or writable..and the list can go on. Anyhow this thunk of memory we’ve been mentioning varies between the virtual memory mechanism a software is using which can be either segmentation or paging or both together.
First of all, if you think segmentation was only used in real mode and old operating systems I have to tell you “Bro, you need some updates!”. There’s a difference between real segmented mode and segmentation over protected mode. In old days segmentation was only used to provide a little more address space on real mode(wasn’t little back then), but I won’t mention much about that old lady, instead i’ll have to ask you to google about the 20bit addresses or Real memory model. Ahh, Now that you googled it Segmentation over protected mode is really different it uses what is called descriptor tables (e.g. GDT/LDT). Without getting too deep, segmentation can basically provide virtual address range that translates to a predefined range of physical memory on a descriptor set in -mostly- GDT such a range doesn’t commit to a standard alignment and can range up to 4GB on x86 , however a virtual memory segment has access permissions (e.g. Read-Write-Execute, ring X privilege). Now you’re thinking, since Segmentation does the job for memory protection why bother making another virtual memory mechanism such as Paging? – Be Patient 😉
Despite the access protection used in segmentation, different running softwares can still view each others memory, and that’s where TSS(Task State Segment) and LDT(Local Descriptor Table) kick in. Windows supports TSS and LDTs under x86 environment which we might inspect on another post…one day!
Paging makes life much easier when it comes to memory management, we can talk about paging all day and not finish so in short Paging is like a unified memory management tool capsule. In segmentation, you had to use TSS to manage running tasks to be able to isolate task memory from other tasks using LDT. Paging however is like GDT+LDT in a structured and more unified manner. To start with an overview, paging is a like having a table with virtual segments and the physical segments they refer to. It consists of three basic elements Page directory, (maybe) Page Table and used page size.
for x86 There’s different page sizes either 4Kbytes or 4Mbytes. In 4Kbyte page size both page directories and page tables are used. In 4 Mbyte page size only page directory is used. Now you’re wondering what is page directories and page tables, you can view them as a chained hashtable with page directory as the hashtable and page table as the chaining list of a hashtable entry. each process has a page directory of its own (called address space) which contains the virtual addresses the process can access and that’s how memory isolation is achieved. and back to page directories.. they are basically an array of elements that has entries of valid page tables called (PDEs) each page table contains the physical address resolution of a virtual address which is called (PTEs). those entries support some kind of memory access protection by providing an 8-bit long flag that can mark the entry as either Readable/writable, belongs to user or kernel, accessed before or not or written or not and all this provides almost all memory protection mechanisms used by segmentation. Most Operating systems however favor paging since they won’t have to suffer much with TSS and write one process structures, not to mention that TSS is not supported in IA64. But of course they still don’t give up segmentation, so both segmentation and paging are used under a paged memory environment. This figure from The Rootkit arsenal shows how memory address is translated under such environment.
2. Introduction to windows memory management
An operating system has to provide multiple memory manager interfaces to allow software either on kernel or user level to allocate memory safely with minimal fragmentation which makes a memory manager the most complex and important component in any OS. However, that happens in most *nix operating system nowadays using the slab allocator. Windows however provides many memory utilization mechanisms under different terminologies and names (e.g. Heap, AWE, Page guards..etc) which we will discuss later on this post. Windows makes a full usage of Intel memory mechanisms, it does support segmentation and different types of paging (32-bit mode, PAE, IA32 and IA64). Thus, paging in windows is heavily used by the memory manager. Most operating systems tend to use both address spaces and segmentation to protected their kernel space from access by user space processes. Provided the fact that user processes have different address spaces (page dirs) which doesn’t include entries of kernel space addresses which makes kernel space inaccessible to them and any reference to such addresses would instantly generate a hardware exception called page fault, Windows uses segmentation to provide another completely define all memory as virtual memory. The first 5 segments widely used on all operating system to isolate kernel memory segment from user segment. and you might have heard or read somewhere that Windows divides memory into two segments. these two segments are segmented in the GDT(we previously mentioned) as four entries(selectors). each segment has two selectors one as code segment(RE) and another as data segment(RW). and each two segments have a privilege level access, so software coming from ring3 cannot accesss memory belongs to ring0 (Of course it cannot under paged environment, but just assume). You can view these selectors if you’re a kernel debugger on bochs by (info gdt) or using KD kernel debugger that comes with windows debugging tools using (dg <offset> <length>).
What matters to us since the rest is easy to figure out, is the segment type and its Pl. you’ll find four segments with two types, Code RE and Data RW which what I’ve shown previously. Pl stands for Privilege Level which includes the ring number the segment belongs to. Windows (as most other OS’s) refers to all memory in its 4 segments, which is an easy way to use paging and still have another protection level using segmentation. I won’t blame you if find that confusing..
We’ve discussed how paging works before, that’s by storing entries of virtual addresses into page tables which translates into physical addresses and the isolation of those two segments happens purely using Paging. Also using a KD debugger we can view a Page directory and page table entries. with only the system process up, using a !process command will view the current process info.The Dir base stands for the page directory physical address of the process, to view its entries we only have to use the PTE command (!pte <Pgdir virtual addr>)
Given the address where the PDE is stored we can resolve the physical address by just dumping the memory we’ll find the Page table address that contains the physical addresses. however you might notice that the contains field has the same value of the DirBase and that’s what is called recursive paging activation apparently, It’s used by windows.
Enough for one day, To be continued..
This was Saad Talaat,
You might have navigated here by a seach. But, my job is to research.