Great question — answering as the UphillTech team:
I can't tell from your comment what Desktop Environment you are running, but I would guess it is Plasma as Gnome based DE are far less sensitive to hardware issues. I was running into a similar issue, I have a MacBook with the built-in HDMI port going to a DP adapter, and one of the USB-C outputs to DP. Those go to a KVM that my Linux Desktop (Ryzen 5-2600, Nvidia 1660 TI) is also connected to. On the Linux Box side, it has a DP connector going straight to the KVM and the HDMI port going to a DP adapter to the KVM. When I was running Gnome (and several other DE) switching the KVM produce no ill effects. But with Plasma on several distros, when I switched the KVM the result was the HDMI-> DP...
Full answer by Morris Buel: https://unix.stackexchange.com/a/787709 (CC BY-SA 4.0).
Need it done for your business? UphillTech.