Enterprise Search has huge resource demands. Everything from CPU to RAM to disk is going to take a beating if you didn't size the search server well, or architect a distributed system properly.
From my perspective (engineer) the load is worst when manipulating the data as it is being indexed, and when the Search systems were not thoroughly tested at load.
There are many competitors in this environment, start looking around for professional services, even a very short term engagement can be very useful in learning what the thought process should be.
What you need will be different on a case by case basis.
How big a server should be sounds like a simple question, however it comes down to what TYPE of data, how many files, what sorts of repositories, how many concurrent queries, etc.