Authentication timed out on wireless connection

Question

I have been experiencing a connection problem with my wireless network. Occasionally, the wireless connection would be disconnected. On the network manager itself, I would still be able to see my router's SSID as well as those belonging to others. Upon, clicking on my router's SSID, I would be shown an authentication prompt after a while. However, I would not be able to connect, despite having the right password. Curiously, I am still able to surf the internet using the wireless on my handphone.

My computer uses both Fedora and Windows XP. The problem occurs on both OSes. Restarting the network manager or the computer itself doesn't solve the problem. However, restarting the router does allow me to connect again.

The following is being logged in wpa_supplicant.log

Trying to authenticate with 00:02:ef:b7:31:25 (SSID='mywifi' freq=2467 MHz)
Trying to associate with 00:02:ef:b7:31:25 (SSID='mywifi' freq=2467 MHz)
Associated with 00:02:ef:b7:31:25
Authentication with 00:02:ef:b7:31:25 timed out.
CTRL-EVENT-DISCONNECTED bssid=00:00:00:00:00:00 reason=3
CTRL-EVENT-DISCONNECTED bssid=00:00:00:00:00:00 reason=3

I am using Linksys PCI network adaptor and Zyxel gateway router with WPA security. Both are using 802.11g wireless standard. What could be the likely reason causing this behaviour and how do I eliminate it so that I don't have to restart the router every now and then?