Mouse in Virtualbox stops clicking (but moves) unless logout and login

Question

The problem is that I after sometime I can move the mouse but can't click while if I lock or switch the user, it works. Apparently the problem is related to Xorg of the guest machine but do not why it happens. I have this problem in all VM machines.

Environment:

 - Host: Windows 10 Pro
 - Guest 1: ParrotSec (Plasma desktop)
 - Guest 2: Kali (Gnome desktop)
 - Guest 3: Lubuntu
 - Virtualbox 6.0.8

P.S.

• I have Googled it but the solution didn't help.

Edit 1: I could bring it back by restoring the mouse settings (even though apparently nothing was change). But still have no idea why it happens.