0

I have a simple script that I wrote in PHP (which, after reading lots of QAs seems to have less than desirable memory management). It's an extremely small script that loops through configuration files in a folder; compares that to the record in the database, and, if there are differences, it updates the record accordingly.

While the script is less than 20 lines, I am dealing with over 30,000 config files at any given time.

My computer is a an Intel dual core 3.06 Ghz with 8 Gigs of Ram running on Ubuntu 12.04.

When I execute the script CPU climbs to 100% almost immediately and stays there. Using the top command I can see the memory for the PHP process increase consistently until it finally maxes out at 8 gigs and crashes. (I usually get about 3/4 through the script before it crashes, which currently takes about 90 minutes.)

From a hardware perspective, how can I make this process more efficient? If I upgrade to a quadcore, will that cut the time in half it takes to execute the script? If I upgrade to a hexcore, will that cut the time needed by 4? Also, does finishing the script 2x or 4x faster also mean I cut the memory usage, or would I still need to upgrade the RAM?

user658182
  • 135
  • 7
  • 1
    The simple solution is to fix the program. Throwing hardware at a bad program is just a bad idea and really shouldn't be done if at all possible. Your related question at Stack Overflow seems to have at least one suitable answer already. – Michael Hampton Mar 19 '13 at 12:09

2 Answers2

1

Instead of reading the files constantly looking for changes (which is likely causing more IO wait than actual CPU usage) I'd highly recommend using inotify to watch for changes and only react when a change happens:

Docs: http://php.net/manual/en/book.inotify.php

See http://en.wikipedia.org/wiki/Inotify for more background.

Dave Forgac
  • 3,486
  • 7
  • 36
  • 48
0

More CPU just means you would run out of RAM faster. I would have to say your best bet is to correct the memory management in your script. To use that much RAM, I have to think you have a memory leak.

scottm32768
  • 449
  • 3
  • 7
  • It is likely there is either a memory leak because he's not closing files or that the system doesn't have enough free RAM to hold the contents of all of the files in memory. In either case fixing the script is the answer. – Dave Forgac Mar 19 '13 at 04:16
  • @Dave Forgac To prevent the question to being subject to closure, I consciously kept the script out of the question, but would be grateful if you wanted to take a look. Here is a link: http://stackoverflow.com/questions/15470580/using-php-how-can-i-load-1000s-of-configuration-files-into-a-database. I followed the suggestions in the answers so far, unsetting the $obj array; unsetting the file array, etc, and I even used gc_collect_cycles() at the end of the foreach loop. Regardless, those variables are local to that function, should they not be discarded when it the function is complete? – user658182 Mar 19 '13 at 05:05
  • ... in the link in the above comment, what you don't see is the insert function that updates that database. It iterates over the returned object, checks it against the db, and updates if so, else continues onto the next. Admittedly this is a large array, but in the grand scheme of things, 34,000 objects should not be that hard to handle? Also, open to suggestions for finding the memory leak. thanks! – user658182 Mar 19 '13 at 05:09