My experience with TCP comes from 30 years of telecommunications and robotics engineering. My primary concerns are much more with Layer-1 of the OSI model (cables, connectors) and Layer-2 (Ethernet frames), not Layer-3 (IP packets) or Layer-4 (TCP/UDP.) 

OSI Layers 1 & 2 are VERY opaque to the average user so consequently they are usually ignored when troubleshooting. I think this is typically a mistake. Sort of like looking for your car keys under a nice streetlight instead of next to your car, where you dropped them. 

PingPlotter is a very graphical, visual troubleshooting tool that has a free version. It is PROFOUNDLY better and more intuitive than using the DOS prompt command line Ping command. PingPlotter also incorporates a very nice, visual, graphical, dynamic traceroute. Download it and try it out. You won't go back to the nasty DOS prompt command line ever again, unless forced to on a strange machine. 

I agree Wireshark is a complicated tool. I already stated that. However I believe that the typical AP mount owner is more qualified than the average person to gain benefit from it, given some time. I would add that starting with PingPlotter instead of Wireshark would be good.

It could be bad to have a firewall or router in between the CP4/5 and the observatory PC. If there is, it might have LAN packet filtering capabilities, which I would disable, if I could.

-Christopher Erickson
Observatory engineer
Waikoloa, HI 96738
www.summitkinetics.com
   

On Thu, May 6, 2021, 9:04 AM Seb@stro <sebastiendore1@...> wrote:

Ethernet is less reliable than USB because Ethernet and TCP use connectionless, multi-point protocols that make any device-to-device communications more vulnerable to disruption by a multitude more things

 

Not sure where you got that from... Communication link reliability has little to do with the underlying protocol granted it is used in the proper context, well managed and used within a properly designed "network" architecture. 

 

Also, TCP protocol IS a connection-based protocol. It is the very reason it is considered "more reliable" that UDP, the latter being a "best-effort" protocol with no handshake and less error-correction mechanisms but typically lower latency. Both have their usecases where they shine. USB and WiFi are no different either. Some are more complex to manage for a "standard" end-user than others, that's all. 

 

The way some manufacturers implement their datalink solution is another key factor. Don't expect Ferrari performance from a Chevy van. And don't drive a Ferrari when you've always driven a Chevy van (at least not without a proper training)...

 

That aside, OP seems to have isolated the problem between the mount and his wired computer. And I agree IP address conflict (connected devices with same address) could be the culprit here given the symptoms. In that case, I would expect communication to re-establish by itself over a few seconds/minutes wait (without powercycling the mount) and then fail again a few seconds/minutes later and re-establish, and so on. A test to verify that would be to run a "perpetual ping" (add "-t" to the usual ping syntax from the command line, e.g. ping X.X.X.X -t, where X.X.X.X is the mount's IP address) and let it run for several minutes. Hit CTRL+C to end the ping.

 

Wireshark isn't a tool for a "standard" end-user. From the OP's posts, I'd say he probably knows a bit about networking and probably already uses it. If not, I would rather suggest to download Advanced IP Scanner (free) or similar, which will help discover every device alive (responsive) and dead (not responsive for a small amount of time) on a network. It will also show the MAC address (which is a unique hardware network identifier) of all devices discovered. Try running a scan while the mount is responsive and take note of the MAC address associated with its IP address. Run it again when it becomes non-responsive and if the tool marks it as "alive" and shows a different MAC address, it means you indeed have a duplicate IP address in your network. The "Name" and "Manufacturer" listed will also help you identify which device is using the same IP address as the mount. If the device is marked "dead" when the mount is non-responsive, then the problem is probably elsewhere.

 

 

Another useful (a bit more advanced) command, if you are familiar with your network IP addressing, is the "tracert" command ("traceroute" in linux) which will essentialy show the path (routing hop) taken by a packet from the computer from which you entered the command to the destination device. Its usage is similar to the ping command, e.g. tracert X.X.X.X, where X is the mount's IP address (wired or wireless). Some routers/firewalls might block this request though and you only get a series of *** + a timeout message, instead of actual routing hop IP addresses, which won't help you much. If you have a "flat" network architechture or only one routing instance, it will only return the destination lP address which won't help you much either (see example below).

 

 

If it goes through however, it will help identify up to which routing network component (switch, router, access point, etc.) communication is achieved properly by returning a series of IP addresses through which packets need to go through to their destination, as well as the round-trip time between hops. Over modern wired ethernet links, expect values below 100ms. Over wireless links, it can go much higher depending on multiple factors, but I'd say below 150-200ms on average would be acceptable (but not particularly good). Over those figure, you possibly have a bottleneck somewhere or a failling network component. BTW that command can even be used over the internet with domain names. Example below is between my computer and google.ca. The last line shows the destination IP address of one of the servers hosting the domain google.ca. Lines 1-8 shows the routing instances every data packet has to come across to reach that google server from my computer.

 

 

 

Note that the fact that the mount is not responsive from either interfaces (Ethernet and WiFi) at the same time is also a clue the problem comes from a common source to both, hence probably not coming from the wireless Access Point.

 

Also worth mentionning, even if you haven't said you are using one, is firewalls (sorry, I'll get a bit technical here). While it is actually often not the firewall root-causing the problem, it can be the one ending the communication by dropping data packets. New generations of firewalls (even home router - WiFi or wired - with firewalls functionnalities) have dynamic adaptative algorithms that "recognize" the type and "behavior" of data traffic that goes through them. They do that to prevent, amongst other things, DoS (denial of service) attacks which consist of an attacker flooding a computer with a massive amount of requests until it crashes by running out of memory.

Now, I've monitored Ethernet/WiFi communications between APCC/APv2 ASCOM drivers and my mount's CP5 (using Wireshark) and based on the amount of connections (not talking about physical hardware connection here, rather software connections at the OSI model layer 4) used, it could well be mis-recognized by some firewall's algorithm as a DoS attack. I'm saying this because it creates a new connection for seemingly every data exchange between the computer and mount, which occurs very often - like every second or so. They possibly implemented this that way for heartbeat or synchronization purposes, but that's only a guess. (That left me scratching my head a bit BTW as there are more memory-efficient ways of accomplishing this). Anyway, thing to note here, is there is nothing you can do about that last part as it's an inherent property of AP's communication between controlling software and mount.

 

But if you are using such a next-gen firewall with that kind of security feature, it could result in similar symptoms to what you are experiencing: communications working for a while and then stopping entirely when packets are dropped. Note here that the firewall is simply doing its job of protecting you. I therefore wouldn't recommend disabling this security feature entirely to solve the problem if that proves to be the case. Rather, I'd try creating a rule to whitelist the mount's IP address in your firewall's configuration in that regard.

Hope this helps as well,

 

Sébastien

On May 6, 2021, at 18:30, alex <groups@...> wrote:

Ok, I opened up the GTOCP4 yesterday and the daughter board seems to be seated fine. I put it back and switched the ethernet cable to a brand new professionally made 15’ cable (ie, I didn’t put the connectors on), and changed the ports on the switch it was plugged into. I rebooted the switch as well in case it was in some weird state. I also hooked up the GTOCP4 to the eagle 2 directly via USB and configured it as the backup port. The primary connection was configured to be the ethernet connection using TCP and a 500ms timeout.

The switch and AP (and all my networking infrastructure) is Ubiquiti UniFi stuff (a prosumer/SOHO brand), so no different brand incompatibility in my network infrastructure. After the initial failures, I configured the router’s DHCP server to assign a fixed IP address assigned to the mount instead of a dynamic one. I’m pretty obsessive about managing my IP address space and am fairly certain there isn’t other devices colliding. The switch is fairly recent, a UniFi US-8-60W and is a fully managed smart switch. I suppose I could configure a separate VLAN for the observatory and put the mount and the eagle 2 on as the only hosts on to make sure there wasn’t interference from other devices on the network, though that seems like overkill.

Last night the ethernet connection failed again but I didn’t notice right away as this time as APCC successfully failed over to the USB connection, so that worked great. I had a perpetual ping repeating once a second the whole night, and showed response times typically between 2 and 9 milliseconds, though occasionally have some 30-50ms ones, and a few 1-2 second ones here and there. Right before the connection failed, the last few pings had 2-7ms ping times, then all the ping requests started timing out. These timeouts have been non-stop from the last 12 hours or so.

While communications was failing, The UniFi controller software didn’t show any abnormal packet loss on the port and I sshed directly into the switch and poked around the internal logs, and didn’t see anything fishy. I tried changing the port the ethernet was plugged into, and power cycling the switch to see if the problem was some bad state the switch was in. Neither woke up the TCP connection. The only thing that fixes it is power cycling the GTOCP4, so to me this seems to be a problem with the state the GTOCP4 is in. If there was some persistent ongoing problem with the network infrastructure, then a reboot of the mount wouldn’t fix the problem.

I’m still mystified as to what’s going on with that ethernet connection. I’m a software engineer and have been programming IP networks professionally over 30 years, and I’ve never seen behavior like this. I haven’t formally done IT/OPs stuff (I program back end web services nowadays), I’ve setup plenty of IP networking equipment over the years.

I could see a transient communications problem starting things off, it wouldn’t explain the inability to ping until the GTOCP4 is power cycled, which magically fixes everything. It’s appearing that some problem occurs, and the GTOCP4 goes into a mode where the network is down and only a power cycle resets it. I’ve never encountered any device with this behavior.

What’s the OS on this thing? It has a reasonably capable ARM processor. Is it running Linux or some embedded OS? Is it possible to SSH into this thing and poke around, check some logs, do something like an ifconfig, netstat, etc ? While the USB connection seems to be working well, I still want to track down what’s going on with the ethernet connection. It’s been my experience that wired ethernet connections are pretty rock solid assuming you avoid problems like long distances or interference with electrical wiring. My cable isn’t near anything like that.

I may try plugging my Mac directly into the same switch and see what Wireshark shows, if anything. It’s been a few years since I’ve messed with it.

Alex

Hi Howard,

Using the supplied serial cable with a USB to serial converter, even the FTDI unit that we sell, will still be limited in its reliability to that of USB.  Chris is referring to a native serial port on a dedicated board inside the computer itself that avoids the Universal Serial Bus.  Our on-board USB port uses the same FTDI chipset, but it has an added advantage over an external USB to serial device.  An external device is powered solely from the bus in the computer.  Our port, however, is also powered so a power drop from the computer won't kill the USB connection because the CP4/5 will keep it alive.