Slow network-home-folder use, many spinning beachballs

Question

Xserve with 10.4.11 connected to Xserve RAID via FC. 208GB of 1.82TB available.

Xserve connects to HP 2848 Gigabit switch

Xserve houses network-home-folders for 14 users.

Clients (mac mini - intel core solo 1.5GHz running 10.4.11)

Symptom: Clients slow to logon to network home folders, slow behavior throughout the workday.

Checked so far: physical networking, dns resolution for server (forwards and backwards). ApplePasswordServer.Server.log (people are authenticating properly). Activity monitor shows no abnormal activity. A brief packet analysis showed no extraneous network traffic.

Tried a simple power-cycle of the switch, clients, and server to no avail.

Answer 1

Here are three things to try:

Check if authentication is your bottleneck

• See how long it takes a user to log in under ssh.

  • Make sure ssh is turned on. [System Preferences -> Sharing -> Remote Login]
  • Open up a terminal and run ssh <username>@localhost
    • Yes, you trust it, and yes, here is the password
    • exit
  • Now, copy the user's password to the pastebin
  • Run time ssh <username>@localhost echo "Done" hit return, and as soon as you are queried for the password, hit ctrl-v to paste, and return
    • This will log the user in over SSH (and do so by authenticating through your server, provided it is a network account)
    • It will run the echo "Done" which should take almost no time to execute, and quit the ssh session
    • it will then display how much time it took. [On my local account, the total was about 1.1 seconds]
  • If the result is large, then authentication is slow

Check if the network file access is slow:

• See how long it takes to transfer files

  • Log in with a local account. [Not strictly necessary, but will help rule out other file accesses slowing you down.]
  • Mount the user's network home folder
  • Make a large file. mkfile 100m 100MB
  • Time how long it takes to make a copy the file from the local computer to the network:
    • time cp 100MB /Volumes/<network-home>
    • or just possibly curl file:/current/path/100MB -o /Volumes/<network-home>/100MB
  • Delete the file and time how long it takes to copy in the other direction
  • [It may not hurt to time how long it takes to create it, too.]

See what other network traffic your computers see:

• consider using iftop to get some feel what the actual transfer speeds are and if the computer is doing other things on the network. [You may want to run it on your server as well].

Answer 2

Are you redirecting the cache folders (and others)?

If not you can find out how from: discussions.apple.com or on AFP548.com
It's built into Leopard - so you don't require a login hook to do this anymore. (Instructions)

Lastly, are most of your users using Mail.app or have a large amount of Mail? I had issues with a couple users (only one had to do have it happening for the symptom to arise) that would bring the server to a near halt for everyone because Spotlight would be opening and closing very small files as fast as it could. A mix between moving to Leopard on the clients and telling Mail.app not to cache anything (mail server is local) fixed this.

Are there any specific users that seem to be "hogging the bandwidth" in terms of the amount of files transferred? I'd recommend looking into making a portable home directory to see if that alleviates issues. There's more setup, but is a decent way of removing users who consume quite a bit of AppleFileServer's time.

Answer 3

I made some more space available on the RAID, and the issue cleared up. I'm not certain that the two are directly related, but I do thank everyone who submitted an answer. They were all really detailed yet clearly written. I'm very impressed!