As a data point, I use the 3D view every session and over RDP (over wifi which then ends up going over my home's internal 1Gb ethernet or wifi network) as well. The mini-PC that runs my stuff is a Whiskey Lake i5-8265U based system and the RDP clients are anything from Windows PCs and Macs to iPhones and iPads - it works fine on all. The Whiskey Lake i5 system (OnLogic ML100G-51) has only the integrated Intel UHD 620 GPU.

NINA, PHD2, and SharpCap are all clients to APCC and the ASCOM driver while the 3D view is running and nothing bad happens in either app when the view is opened while they are connected.

Perhaps Voyager polls its connected drivers at a more aggressive rate and is sensitive to even a momentary unresponsiveness caused by the 3D view being opened and loaded?