Memory Management 101: Introductionto Memory Management in LinuxChristopher Lameter, Ph.D. firstname.lastname@example.org Jump Trading LLC@qant1
Overview Memory and processesReal/Virtual memory and PagingMachine configurationProcesses use of memoryOvercommitKnobsThere is an advanced MM talk tomorrow called“Flavors of Memory”2
Pages and physical page frame numbers Division of memory into “pages” 1-N bytes become split at pagesize boundaries and becomeM N/page size pages Refer to memory by the PageFrame Number (PFN) and anoffset into the page. Common size is 4k (Intel legacy issues) MMU creates virtual addresses.
Basics of “paging” Process have virtual memory- PFNPage TablesFaults Major Minor Virtual vs physical
Process memory Virtual memory maps tophysical memory View of memory that is distinctfor each process. Pages shared Access control Copy on Write
Swap, Zero pages etc. Swap pageZero pageRead data behaviorWrite data behaviorAnonymous vs file backed pages
Linux Basic memory information/proc/meminfo/sys/devices/system/has lots of more detailedinformation on hardware(processors andmemory)Commands:numactl --hardwarefree, top, dmesgMemTotal:31798552 kBMemFree:25949124 kBMemAvailable: 30823580 kBBuffers:220988 kBCached:4679188 kBSwapCached:0 kBActive:2803000 kBInactive:2336992 kBActive(anon):240776 kBInactive(anon):6432 kBActive(file):2562224 kBInactive(file):2330560 kBUnevictable:0 kBMlocked:0 kBSwapTotal:2097148 kBSwapFree:2097148 kBDirty:48 kBWriteback:0 aim:KernelStack:PageTables:239716 kB195596 kB7396 kB550628 kB443040 kB107588 kB6840 kB11176 kB
Inspecting a process memory use/proc/ pid /status/proc/ pid /*maps(there are other files in /proc/ pid /*with more information about theprocesses)Commands:ps, topName: sshdVmPeak:65772 kBVmSize:65772 kBVmLck:0 kBVmPin:0 kBVmHWM:6008 kBVmRSS:6008 kBRssAnon:1216 kBRssFile:4792 kBRssShmem:0 kBVmData: 1332 kBVmStk:132 kBVmExe:492 kBVmLib:8076 kBVmPTE:168 kBVmSwap:0 kB
User limit (ulimit) max memory sizevirtual memorystack sizeand lots of othercontrols.cl@nuc-kabylake:/proc/6713 ulimit -acore file size(blocks, -c) 0data seg size(kbytes, -d) unlimitedscheduling priority(-e)0file size(blocks, -f)unlimitedpending signals(-i)123132max locked memory(kbytes, -l) 16384max memory size(kbytes, -m) unlimitedopen files(-n)1024pipe size(512 bytes, -p) 8POSIX message queues (bytes, -q) 819200real-time priority(-r)0stack size(kbytes, -s)8192cpu time(seconds, -t) unlimitedmax user processes(-u) 123132virtual memory(kbytes, -v) unlimitedfile locks(-x)unlimited
Overcommit configurationVirtual memory use vs physicalovercommit kbytesovercommit memory0 - overcommit. Guess if mem is available.1 - Overcommit. Never say there is no memory2 - Only allocate according to the ratioovercommit ratiototal swap physical * ratio
Important VM control knobsFound in /proc/sys/vmMore descriptions of these knobs in Kernel sourcecode. linux/Documentation/admin-guideadmin reserve kbytes dirty writeback centisecs min free kbytes numa zonelist orderstat refresh block dump drop caches min slab ratio oom dump tasks swappinesscompact memory extfrag threshold min unmapped ratio oom kill allocating taskuser reserve kbytes compact unevictable allowed hugetlb shm group mmap min addrovercommit kbytes vfs cache pressure dirty background bytes laptop modemmap rnd bits overcommit memory watermark scale factor dirty background ratiolegacy va layout mmap rnd compat bits overcommit ratio zone reclaim mode dirty byteslowmem reserve ratio nr hugepages page-cluster dirty expire centisecs max map countnr hugepages mempolicy panic on oom dirty ratio memory failure early killnr overcommit hugepages percpu pagelist fraction dirtytime expire secondsmemory failure recovery numa stat stat interval
Resources Admin Guide guide/index.html Kernel.org has wikis and documentation(www.kernel.org ) manpages (especially for system callsand coding)
Questions / CommentsYou can reach me at email@example.com or@qant on twitter13
“Simple” Memory Access UMA (Uniform Memory Access) Any access to memory has the same characteristics(performance and latency) The vast major of systems have only UMA. But there is always a processor cache hierarchy– The CPU is fast, memory is slow– Caches exist to avoid accesses to mainmemory Aliasing Coloring Cache Miss Trashing15
NUMA Memory Memory with different accesscharacteristics Memory Affinities depending onwhere a process was started Control NUMA allocs with memorypolicies System Partitioning using Cpusetsand Containers Manual memory migration Automatic memory migration16
Huge Memory Typical memory is handled in chunks of base page size (Intel 4k,IBM PowerX 64K, ARM 64K)Systems support larger memory chunks of memory called Hugepages (Intel 2M)Must be pre configured on boot in order to guarantee that they are availableRequired often for I/O bottlenecks on Intel.4TB requires 1 billion descriptors with 4K pages.Most of this is needed to compensate forarchitectural problems on Intel. Intel processorshave difficulties using modern SSDs and highspeed devices without this.Large contiguous segments (I/O performance)Fragmentation issuesUses files on a special file system that must be explicitly requested bymmap operations from special files.17
An Introduction to Linux memory management. The basics of paging. Understandingbasic hardware memory management and the difference between virtual, physical andswap memory. How do determine hardware installed and how to figure out howprocesses use that memory. How a process uses physical and virtual memory effectively.How to control overcommit and virtual and/or physical memory limits.Basic knobs in Linux to control memory management. System calls for a process tocontrol its memory usage18
An Introduction to Linux memory management. The basics of paging. Understanding basic hardware memory management and the difference between virtual, physical and swap memory. How do determine hardware installed and how to figure out how processes use that memory. How a process uses physical and virtual memory effectively.
Linux in a Nutshell Linux Network Administrator’s Guide Linux Pocket Guide Linux Security Cookbook Linux Server Hacks Linux Server Security Running Linux SELinux Understanding Linux Network Internals Linux Books Resource Center linux.oreilly.comis a complete catalog of O’Reilly’s books on Linux and Unix and related technologies .
Other Linux resources from O’Reilly Related titles Building Embedded Linux Systems Linux Device Drivers Linux in a Nutshell Linux Pocket Guide Running Linux Understanding Linux Network Internals Understanding the Linux Kernel Linux Books Resource Center linu
Perfection PC Perfection PC Inc. Philips Philips Electronics Planar Planar Systems Inc PLEXON Plexon, Inc. Pogo Linux Pogo Linux, Inc. Pogo Linux Altura M2 Pogo Linux, Inc. Pogo Linux Velocity -D50 Pogo Linux, Inc. Pogo Linux Verona 330 Pogo Linux, Inc. Pogo Linux Vor
Yes. Oracle Autonomous Linux, which is based on Oracle Linux, is 100% application binary compatible with IBM's Red Hat Enterprise Linux. This means that applications certified to run on Red Hat Enterprise Linux can run on Oracle Autonomous Linux unmodified. Oracle Linux binaries are provided for patching and updating Red Hat Enterprise Linux
Official Kali Linux Documentation This PDF has been autogenerated on docs.kali.org - Apr 7, 2013 00. Introduction to Kali Linux What is Kali Linux ? Kali Linux is an advanced Penetration Testing and Security Auditing Linux distribution. Kali Linux Features Kali is a complete re-build of BackTrack Linux, adhering completely to Debian development .
2 LXC DOCKER MICHAEL LESSARD A bit of history - Virtualization and containers Chroot (version 7 Unix, 1979) FreeBSD Jails (FreeBSD 4, 2000) Linux vserver (Linux, Oct 2001) Para-virtualization Xen (Linux, 2003) Solaris zones (Solaris 10, 2004) OpenVZ (Linux, 2005) Full virtualization KVM (Linux, 2007) Linux Containers - LXC (Linux 2.6.29 2009)
Memory Management Ideally programmers want memory that is o large o fast o non volatile o and cheap Memory hierarchy o small amount of fast, expensive memory -cache o some medium-speed, medium price main memory o gigabytes of slow, cheap disk storage Memory management tasks o Allocate and de-allocate memory for processes o Keep track of used memory and by whom
Nutrition and Food Science [CODE] SPECIMEN PAPER Assessment Unit A2 1 assessing. 21 Option A: Food Security and Sustainability or Option B: Food Safety and Quality. 22 Option A: Food Security and Sustainability Quality of written communication will be assessed in all questions. Section A Answer the one question in this section. 1 (a) Outline the arguments that could be used to convince .