2

My company has a few servers that we dedicate to running Sphinx Search. All are CentOS 5 with about 48GB of memory and our version of searchd is 1.11. Recently I've been trying to understand why Sphinx uses so little of the available memory for its regular indexes (we don't use RT indexes at the moment). According to the official Sphinx website, all files related to an index except for the .spd and .spp files are stored in RAM. At present this constitutes about 14GG for all our indexes, which can easily fit in RAM. However, when I consult htop for memory usage, it's showing a little less than 1.5GB being used! Interestingly enough, it reports that swap is being used as well, even with vm.swappiness set to 0. If I stop searchd, swap goes down to 0. (It should be noted that the amount of swap used is no greater that the amount of memory being used.) We have it configured so that all our indexes are preopened on startup (preopen_indexes = 1 in our sphinx conf).

I have two questions, given the above information:

  1. What is Sphinx doing with the data in those indexes, if it isn't keeping it in memory?
  2. Is it possible to force Sphinx to keep more data in memory?
altendo
  • 21
  • 1
  • 2

2 Answers2

1

try mlock = 1 and reindex all indexes.

index index1
{
    source      = main1
    path        = /ssd/sphinx/index.main
    charset_type    = utf-8
    docinfo     = extern
    mlock       = 1
}
jim
  • 11
  • 2
0

What is Sphinx doing with the data in those indexes, if it isn't keeping it in memory?

It probably does put them into memory. But the host OS, notices they are not being used much, and so swaps them out.

Would suggest looking at making sure your swappiness is really being applied.

Is it possible to force Sphinx to keep more data in memory?

If you have the memory to spare. Put the indexes on a RAM-Disk :)

And if you really have plenty of memory to spare, turn OFF swap altogether instead. You risk a crash if run out of memory, but it sounds unlikly to happen on that server.

barryhunter
  • 101
  • 2
  • Thanks for the advice. I've gone through the SphinxSearch forums and found much of what you said. Swappiness is indeed set to 0, but will certainly look into just turning swap off. One thing I have also tried is using the mlock option for each index to lock the .spa and .spi files in memory, but I don't see an appreciable increase in memory usage when enabled. It isn't giving me an error when I start searchd. Do you know if the option works properly? I have tried it with 2.0.6 and 2.1.1-dev as well as 1.10 and it seemed to operate the same no matter which version I used. – altendo Feb 13 '13 at 00:40