1
1
I have a program that requires a lot of memory. So much that the malloc fails as the system goes out of memory.
I don't care if the overhead for swapping pages in and out of the disk is too high - I just want the program to run. So, is it possible that by sufficiently increasing the swap partition size the out-of-memory error can be prevented by effectively making more memory is available?
AnkurVj
Posted 2011-09-01T06:21:15.980
Reputation: 971
3
The malloc() function usually allocates virtual memory from the memory heap. Even though you have not specified an operating system nor programming language nor version of runtime library, I'm going to assume that your version of malloc() does not allocate physical memory. Therefore adding more physical memory or increasing swap space may not eliminate the out-of-memory error return you see from malloc().
For a solution, you need to determine if your program is exceeding the amount of (virtual) memory that is allocated, or if the system is actually running out of memory pages. "When Linux Runs Out of Memory" (or this copy) explains how malloc() allocates (virtual) memory, the two ways you might get an out-of-memory condition and how to determine which. Note that availability of memory pages is a complex combination of physical memory size, swap space size, memory usage and process load.
sawdust
Posted 2011-09-01T06:21:15.980
Reputation: 14 697
1
Well, no. It does not increase your memory - it merely means that your OS is able to move things that aren't of immediate use out to free up memory. This will have no difference in terms of performance, but if the software in question isn't actively using memory, your system may be able to swap out RAM to your swap to use for other things.
You may want to try creating and using a swap file, as opposed to a swap partition to see if adding swap works.
You can also try to limit (though it may have consequences) the memory the software uses with ulimit.
Loads of swap is not a replacement for physical memory - you're either going to have to add more memory, or work out why your program needs that much memory, and fix it.
Journeyman Geek
Posted 2011-09-01T06:21:15.980
Reputation: 119 122
What I mean is that by sufficiently increasing swap space, can the out-of-memory error be prevented ? I don't care if the overhead for swapping pages in and out of the disk is too high. I just want the program to run. – AnkurVj – 2011-09-01T06:41:29.143
It should - assuming it dosen't slow down the system to a crawl from all the paging. My recommendation is a VERY large swap file to test, something like 5-8 times the size of your physical memory. – Journeyman Geek – 2011-09-01T06:48:26.430
0
Physical memory is, and only is, the memory chips inside your computer case. The swap file is used when you lack enough physical memory. Increasing the swap file size will help prevent out of memory errors, but I don't recommend it. Having too much swap file available will give your computer an unreasonable sense of ability and probably cause it to slow to a crawl when using most of the page file.
The best solution is to buy more memory chips. It'll make everything run that little bit better.
If you don't mind me asking, what is your application doing that consumes so much memory?
Hand-E-Food
Posted 2011-09-01T06:21:15.980
Reputation: 4 711
I am allocating space for a 60K X 60K matrix – AnkurVj – 2011-09-01T06:41:53.200
So, assuming 32-bit integers, you need 3.4Gb of memory? Ow! I'd suggest making your own swap file system, loading and saving chunks of the matrix as you need them. You can customise it to work in a way that serves you the fastest, unlike the OS's swap file which does what ever it does. – Hand-E-Food – 2011-09-01T23:16:41.053
Okay, I thought that the amount of virtual memory that can be allocated by a program is limited by the total available memory (Physical memory + swap space). i.e. even if the amount of memory allocated to the memory heap maybe larger, malloc may still fail because there is no real memory left to allocate ! – AnkurVj – 2011-09-01T08:22:03.420
On linux, you could use the getrlimit(RLIMIT_AS, p) function to find out the maximum size of a process' total available (virtual) memory. If this limit is exceeded, the malloc() and mmap() functions shall fail with errno set to [ENOMEM]. You might want to also look at the setrlimit() man page. – sawdust – 2011-09-01T08:56:37.363
But even if the limit is not exceeded, can malloc fail because the system has no memory to allocate ? – AnkurVj – 2011-09-01T09:08:24.100
1
Here's something to read about a process running out of memory versus Linux running out of swap space: http://linuxdevcenter.com/pub/a/linux/2006/11/30/linux-out-of-memory.html "The conclusion is that OOM happens for two technical reasons: 1. No more pages are available in the VM. 2. No more user address space is available. 3. Both #1 and #2."
– sawdust – 2011-09-01T09:31:54.637The link is indeed superb .. I'd recommend adding it to the answer – AnkurVj – 2011-09-01T09:53:11.703