Keyboard recomments when switching languages

Question

My Setup: I have USB mouse and keyboards both connected to a KVM Switch which is connected to my laptop and desktop. I have Ubuntu on my desktop and it is this one causing the issue. I also have Windows 10 on the desktop which causes no issues with the KVM.

Behaviour: My HID devices (maybe also other USB devices) reconnects about every 2nd or 3rd time I switch languages. I suspect that what is happening is that in order to switch languages, Ubuntu reconnects the USB HID devices for some reason. During that time, the KVM detects that the desktop stopped being connected (even for a sec) and switches to the next computer (I can confirm it does switch to the laptop). I've tested the following: Disconnecting the keyboard and switching languages with the mouse from the layout menu. Result: Same, KVM switches to laptop.

Disconnected the mouse and switched languages with the keyboard shortcut. Result, same.

Changes the keyboard shortcut to something else and try changing layout then. Result: SAME!

Boot Windows on the desktop and switch languages. Result: No issues. Connecting

Connecting both keyboard and mouse to the desktop directly and switching languages. Result: The KVM still switches to the laptop (as indicated by a green light).

Moving the mouse while switching languages (mouse and keyboard still connected directly to the desktop). Result: KVM switches (obviously) and the mouse experiences a tiny hiccup when switching.

The last test kinda confirms me theory that all HID devices go though a reconnect process. In all the above tests, I had to switch 1-3 times before the reconnect occurred.

I assume Ubuntu is the problem and not the KVM (otherwise it would have happened on Windows too). If I had a keyboard with a PS connector I would have tried that too, I'm curious what would happen.