Date   

Re: AP1100/CP4 and NINA

Dale Ghent
 

NINA doesn't necessarily know what is CW up and what is CW down, and it certainly doesn't think in those terms. There is no programmatic way an ASCOM driver can transmit this state to an app.

I'm trying to picture how you're managing to do this conditional flip given what I recall about SGPro's flip options, which aren't fancy at all. Is this a new thing you want to try or is this something you already have working in SGPro?

Either way, you can maybe do this on a per target basis in NINA 1.11's advanced sequencer. For the targets that you start in a CW-up orientation, you can simply omit the Meridian Flip trigger and the mount will sail through the meridian without flipping, simply because the trigger command that manages that is missing.

When I referred to the APCC-enforced limits (be they horizon or meridian), NINA does not know of these limits as, again, there is no programmatic way for an ASCOM driver to describe these to an app such as NINA. If you have the limit set to park or stop tracking, NINA will not have any idea why this is happening and you'll probably end up with a hung sequence. However you can probably recover from this predicament on a multi-target sequence by bounding the halted sequence with a time or a target altitude so that it eventually moves on to the next target which could have an unpark or set tracking rate command to get the mount moving again.

On May 6, 2021, at 13:47, Michael 'Mikey' Mangieri <mjmangieri@xcalrockets.net> wrote:

OK, thanks. I'll check out the 1.11 version and experiment tonight. I use
APCC and APPM.

My normal mode of operation is to slew to the east with CW UP (based on APCC
limits allowing this) and track past meridian with no need to flip. But
when I slew east with CW DOWN the flip needs to occur at the meridian. I
expect NINA to handle this without any additional inputs or configuration
from me.

If I ever need to track past meridian with CW UP I guess that's when NINA
needs to know when to flip? That's the part I would assume you are referring
to when you state that I need to configure NINA to execute flips prior to
those limits being reached?

Mikey

-----Original Message-----
From: main@ap-gto.groups.io <main@ap-gto.groups.io> On Behalf Of Dale Ghent
Sent: Thursday, May 6, 2021 12:54 PM
To: main@ap-gto.groups.io
Subject: Re: [ap-gto] AP1100/CP4 and NINA


Hi Michael, I'm one of the contributors to NINA and have been using my
Mach1GTO+CP3(V2) with it for over 2 years. It works fine. Plenty of other
uses use it with a CP4 and CP5 and that's also fine. It just uses the A-P
ASCOM driver like any other software.

Some general tips - If you have slew/tracking or meridian limits set up in
APCC then of course be sure that you configure NINA to execute flips prior
to those limits being reached. If you use the current in-development version
of NINA, 1.11, you can use the horizon file that APCC saves directly in
NINA's advanced sequencer to trigger actions based on your local horizon,
etc.


On May 6, 2021, at 12:38, Michael 'Mikey' Mangieri
<mjmangieri@xcalrockets.net> wrote:

I recently decided to try N.I.N.A. as a possible replacement for SGP which
I have used for years. Was wondering if the AP community has any issues
with using NINA with respect to meridian flips, CW-UP slews, and such?











Re: Strange guiding errors with AP1200

Allen Gilchrist
 

Rolando,
Thanks for the prompt response.  The 6.4 minutes is close to the 379 seconds I saw.  I was taking 10-second guiding exposures, so the problem could have occurred any time during that interval.  I have the lube kit and instructions, and will follow your advice.  I'll also take a look at the YouTube videos on lubricating and adjusting the gears.  
Allen


Re: Losing Communications with the Mount

Christopher Erickson
 

When I commented about possible comm buffer overflows, I was speaking from the perspective of an embedded (controller, microcontroller) programmer, not as a LAN/WAN administrator or IP wonk. Non-programmers have no access to or awareness of comm buffers and such and don't need to worry about them. Embedded programmers do.

Many types of DoS (Denial of Service) attacks are specifically crafted to exploit various hardware and software buffer limitations and overflows scenarios.

-Christopher Erickson
Observatory Engineer
Summit Kinetics
Waikoloa, Hawaii


On Thu, May 6, 2021 at 3:18 PM Dale Ghent <daleg@...> wrote:


> On May 6, 2021, at 19:50, Christopher Erickson <christopher.k.erickson@...> wrote:
>
>
> Certain kinds of broadcast storms are known to disrupt LAN communications with certain ARM processors via comm buffer overflows, depending on the status of a multitude of various internal variables.

What on earth are you even talking about. Comm buffer overflow? Broadcast storms due to certain ARM processors? This makes -zero- sense.

- Someone who actually works on OS IP stack code





Re: Losing Communications with the Mount

Dale Ghent
 

On May 6, 2021, at 20:16, alex <groups@ranous.com> wrote:

My Eagle 2 computer is a NUC with some specialized hardware to distribute and control power connections, and is somewhat "fragile" to configuration changes. It took me some time to work out some driver issues to get everything to work, so I'm loath to touch it at this point. Besides, I already have wireshark configured on my Mac. Now I just need to remember how to use it.
Well, to get the Mac to even see the packets between the Eagle and CP4, you'll first need to configure the switch to mirror the traffic on the Eagle's and CP4's port on the switch to the port that the Mac is plugged into. I'm sure the unifi controller will let you do this (I use Unifi gear as well, but only for wireless APs, so I don't know what the control pane for their switches looks like.) The Eagle just runs a Windows OS, you can at least try running Wireshark on it to see how it goes before having to go through the rigamarole of setting up port mirroring for a third host.

But whichever way you get it set up, you'll want to get the MAC addresses of the Eagle and the CP4. You can quickly get the CP4's MAC address by looking at the arp cache on the Ealge. You can do this by popping open a Powershell window and running:

arp -a

This will print out the Eagle's view of the network's IP-MAC Address mapping. If you don't see your Eagle in there, ping it first then run the arp command again, and it should then show up. The Eagle's own MAC address won't show up in this output. To get that, you can run the following from a Powershell prompt:

IpConfig /all

Just find the ethernet interface you're interested in and note its MAC address, which is specified in the Physical Address field.

Once you have the CP4 and Eagle's MAC addresses, you can run Wireshark and use the following display filter:

eth.addr==aa-aa-aa-aa-aa-aa and eth.addr==bb-bb-bb-bb-bb-bb

aa-aa-aa... and bb-bb-bb... are the MAC addresses of your Eagle and CP4. It's just a boolean filter, so which order they are specified in doesn't matter. Do note that is a double equal sign, per the norm for boolean notation.

At that point, you start going about your normal work and wait. When/If the freeze-up happens, you can save the capture by Ctrl-A'ing all the packets in the display area and going to File > Export Specified Packets. Save it as the default pcapng format and then someone can do something with that info.

/dale


Re: Spikes in Dec

Sébastien Doré
 

Thank you (again), Brian. 

Will forward my PHD2 logs of the entire last session privately as well as what Howard has pointed me out.

Sébastien


De : main@ap-gto.groups.io <main@ap-gto.groups.io> de la part de Brian Valente <bvalente@...>
Envoyé : 6 mai 2021 17:58
À : main@ap-gto.groups.io <main@ap-gto.groups.io>
Objet : Re: [ap-gto] Spikes in Dec
 
You want to enable DEC compensation here

This adjusts the guiding based on your sky position, which doesn't change (unless you are going to say you have a space telescope)

>>>I realize this is more a question for the PHD forum, (sorry about that) but I thought it could be of general interest here as well if you have the answer at hand.

that's fine - happy to help. Maybe you can shoot me some of your logs direct via email and I can review them. 

I'm not sure I saw what Howard mentioned, but guiding with high quality encoders is a relatively new thing, so anything is possible. lowpass2 was introduced only a few years ago iirc

 
Brian


Re: Losing Communications with the Mount

Dale Ghent
 

On May 6, 2021, at 19:50, Christopher Erickson <christopher.k.erickson@gmail.com> wrote:


Certain kinds of broadcast storms are known to disrupt LAN communications with certain ARM processors via comm buffer overflows, depending on the status of a multitude of various internal variables.
What on earth are you even talking about. Comm buffer overflow? Broadcast storms due to certain ARM processors? This makes -zero- sense.

- Someone who actually works on OS IP stack code


Re: ADATRI on Losmandy HD Tripod

KHursh
 

No, you would need the flat plate that goes on the tripod: LT2APM. You bolt the ADATRI to that


Re: Losing Communications with the Mount

Christopher Erickson
 

(The following is a general comment and not intended as a specific reply to anyone on this thread.)

One important and fundamental fact about troubleshooting Ethernet/WiFi/IP/TCP/UDP networking problems is that anything measured at an OSI Layer higher than the Layer where the problem is originating, will give misleading & potentiality invalid results until problems at the lower Layers are resolved. And all Layers below a particular Layer are invisible to that Layer. All it gets to see is the data coming up from the Layer immediately below it. Troubleshooting tools like Wireshark use special "promiscuous" modes that are available in SOME network interfaces. These special "promiscuous mode" functions allow user apps like Wireshark to look more deeply at the statistics at the lower OSI Levels. Some "promiscuous mode" capable network adapters are more sophisticated and feature-rich than others.

The classic OSI networking model is as follows,

Layer 1 - Physical (cables, connectors, voltages, signals, carriers, etc.)
Layer 2 - Data (Ethernet Frames, Tokens, etc.)
Layer 3 - Network (IP Packets, others)
Layers 4/5 - Transport/Session (TCP, UDP, XTP, etc.)
Layer 6 - Presentation (decryption/encryption, etc.)
Layer 7 - Application (organization & delivery to/from the OS)

For troubleshooting purposes over the years, I have added three more important Layers to the OSI model as well.

Layer 0 - The funding. NOTHING is going to ever work right at any other Layer if this Layer isn't stable and solid.

Layer 8 - THE USER. "Yea, it was definitely a Layer-8 issue! Excessive operator head-space!"

Layer 9 - THE POLITICS. "Ugh. This is a Layer-9 problem and will NEVER get fixed!"

Seriously though, Layer 7 reports the the OS (computer operating system) and the user app interacts with the OS to get the networking services. Layer 7 is NOT the user app. Just FYI.

I hope this helps (and entertains!)

-Christopher Erickson
Observatory engineer
Waikoloa, HI 96738
www.summitkinetics.com
   


On Thu, May 6, 2021, 12:35 PM Seb@stro <sebastiendore1@...> wrote:
Christopher,

As you surely know, much of all Internet communication is TCP-based nowadays and work arguably 365 days/year and almost 24hours a day (like 2 and 3 nines networks). Not sure what 30 years of experience with TCP has to do with that or any communication reliability figure for that matter. 

PingPlotter does indeed look like a valuable tool (you even convinced me to download and try it).

In regards to the old CLI vs GUI debate, I don't think it's the right place to argue on this forum, but I will only add that network command line tools are way less prone to bugs than any GUI tools. Moreover, you don't even have to download, install or update anything, they come built-in with OSes, even in 2021. There must be a good reason for that... My guess is that if you happen to be cutoff from the internet because of a communication problem, you have to rely on something to get back on your feet. Might be a good thing to learn the ropes of using simple command line tools, which anyone here - as you also stated - has the ability to grab. I'd recommend anyone to learn basic command line tools well before trying out any graphical wireshark/pcap/tcpdump analyzer. Well, it seems like you got me started after all. 😉 Sorry about that everybody. At least, now you know where I stand... 

As for your other comments, I'd say that I agree it's good practice to look at the plumbering stuff (lower layers) first, but really solving (like definitively, not "just make it work" for some time until the next software update) most common communication problems beyond those obvious ones also requires looking at least at the TCP/IP layers these days, especially if you don't have an infinite budget to replace everything out of trial and errors and/or infinite amount of time to spend on it (I'm sure most of us would rather spend this precious time under clear skies with their latest AP equipment). 

That is also exactly what PingPlotter does BTW: it compiles and presents statistics from - you stated it - ping and tracert requests which uses a L3 protocol (ICMP) running in the background.


Clear skies,
Sébastien


De : main@ap-gto.groups.io <main@ap-gto.groups.io> de la part de Christopher Erickson <christopher.k.erickson@...>
Envoyé : 6 mai 2021 16:24
À : main@ap-gto.groups.io <main@ap-gto.groups.io>
Objet : Re: [ap-gto] Losing Communications with the Mount
 
My experience with TCP comes from 30 years of telecommunications and robotics engineering. My primary concerns are much more with Layer-1 of the OSI model (cables, connectors) and Layer-2 (Ethernet frames), not Layer-3 (IP packets) or Layer-4 (TCP/UDP.) 

OSI Layers 1 & 2 are VERY opaque to the average user so consequently they are usually ignored when troubleshooting. I think this is typically a mistake. Sort of like looking for your car keys under a nice streetlight instead of next to your car, where you dropped them. 

PingPlotter is a very graphical, visual troubleshooting tool that has a free version. It is PROFOUNDLY better and more intuitive than using the DOS prompt command line Ping command. PingPlotter also incorporates a very nice, visual, graphical, dynamic traceroute. Download it and try it out. You won't go back to the nasty DOS prompt command line ever again, unless forced to on a strange machine. 

I agree Wireshark is a complicated tool. I already stated that. However I believe that the typical AP mount owner is more qualified than the average person to gain benefit from it, given some time. I would add that starting with PingPlotter instead of Wireshark would be good.

It could be bad to have a firewall or router in between the CP4/5 and the observatory PC. If there is, it might have LAN packet filtering capabilities, which I would disable, if I could.

-Christopher Erickson
Observatory engineer
Waikoloa, HI 96738
www.summitkinetics.com
   


Re: Losing Communications with the Mount

Alex
 

On Thu, May 6, 2021 at 04:33 PM, Dale Ghent wrote:
"Interference from other devices"? If you're thinking of interference like it's radio interference, switched ethernet doesn't work that way. VLANs won't save you from any particular layer <=2 issue anyhow. Since you say the pings stop regardless whether it has a DHCP or static address, it seems like a total IP stack wedge of some sort, or the OS is inexplicably downing its interfaces.
By interference, I was talking about some rogue device on my network trying to grab the same IP address.  Putting the observatory on its own private network would eliminate that possibility, though I still think that's overkill.  My gut is telling me that something is going wrong with the mount's IP stack.

It wasn't clear if you tried connecting via USB/serial immediately after this network failure, without rebooting the CP4. If the network craps out but you can still subsequently connect via USB and control the mount, that might help narrow down where things are going wrong.
I configured APCC to use the USB connection as backup if the primary connection failed.  It failed over to the USB network w/out me even noticing. It was the next day I noticed it had switched over, and my repeated pings had been failing non-stop for something like 12 hours when I checked the next day.  So clearly the CP4 wasn't completely hosed, just it's IP stack.

Wireshark does run on Windows. If you're able reproduce this issue with some reliability, you can run that on the box that's connecting to the CP4 via the ASCOM driver and have it run through your normal uses/reproduction paces while Wireshark is running and dumping packets to a pcap file. If the issue resurfaces, the pcap file can be inspected for the packet flow of the entire session.
My Eagle 2 computer is a NUC with some specialized hardware to distribute and control power connections, and is somewhat "fragile" to configuration changes.  It took me some time to work out some driver issues to get everything to work, so I'm loath to touch it at this point.  Besides, I already have wireshark configured on my Mac.  Now I just need to remember how to use it.

Alex


Re: Losing Communications with the Mount

Roland Christen
 

It makes a difference which version of software is in that mount. The version number was not mentioned.

Rolando



-----Original Message-----
From: Christopher Erickson <christopher.k.erickson@...>
To: main@ap-gto.groups.io
Sent: Thu, May 6, 2021 6:50 pm
Subject: Re: [ap-gto] Losing Communications with the Mount

Sounds like you really have it narrowed down to the CP4 firmware running in your controller.

Work close with AP and I am confident they will get it resolved for you.

Certain kinds of broadcast storms are known to disrupt LAN communications with certain ARM processors via comm buffer overflows, depending on the status of a multitude of various internal variables.

-Christopher Erickson
Observatory engineer
Waikoloa, HI 96738
www.summitkinetics.com
   

On Thu, May 6, 2021, 12:33 PM alex <groups@...> wrote:
Oh, I forgot mention that I also tried using the mount's WiFi connection as well, and it also had similar issues, so the ethernet connection doesn't seem to be at fault.

Alex

--
Roland Christen
Astro-Physics


Re: Losing Communications with the Mount

Christopher Erickson
 

Sounds like you really have it narrowed down to the CP4 firmware running in your controller.

Work close with AP and I am confident they will get it resolved for you.

Certain kinds of broadcast storms are known to disrupt LAN communications with certain ARM processors via comm buffer overflows, depending on the status of a multitude of various internal variables.

-Christopher Erickson
Observatory engineer
Waikoloa, HI 96738
www.summitkinetics.com
   


On Thu, May 6, 2021, 12:33 PM alex <groups@...> wrote:
Oh, I forgot mention that I also tried using the mount's WiFi connection as well, and it also had similar issues, so the ethernet connection doesn't seem to be at fault.

Alex


Re: Losing Communications with the Mount

Donald Gaines
 

Thanks Christopher,
I’m going to put a serial port card into my computer and use the serial cable that comes with the mount. Thanks to all of you for all your help. There are some seriously smart people on this forum. 
Thanks again,
Don Gaines


On Thursday, May 6, 2021, Christopher Erickson <christopher.k.erickson@...> wrote:
My experience with TCP comes from 30 years of telecommunications and robotics engineering. My primary concerns are much more with Layer-1 of the OSI model (cables, connectors) and Layer-2 (Ethernet frames), not Layer-3 (IP packets) or Layer-4 (TCP/UDP.) 

OSI Layers 1 & 2 are VERY opaque to the average user so consequently they are usually ignored when troubleshooting. I think this is typically a mistake. Sort of like looking for your car keys under a nice streetlight instead of next to your car, where you dropped them. 

PingPlotter is a very graphical, visual troubleshooting tool that has a free version. It is PROFOUNDLY better and more intuitive than using the DOS prompt command line Ping command. PingPlotter also incorporates a very nice, visual, graphical, dynamic traceroute. Download it and try it out. You won't go back to the nasty DOS prompt command line ever again, unless forced to on a strange machine. 

I agree Wireshark is a complicated tool. I already stated that. However I believe that the typical AP mount owner is more qualified than the average person to gain benefit from it, given some time. I would add that starting with PingPlotter instead of Wireshark would be good.

It could be bad to have a firewall or router in between the CP4/5 and the observatory PC. If there is, it might have LAN packet filtering capabilities, which I would disable, if I could.

-Christopher Erickson
Observatory engineer
Waikoloa, HI 96738
www.summitkinetics.com
   

On Thu, May 6, 2021, 9:04 AM Seb@stro <sebastiendore1@...> wrote:
Ethernet is less reliable than USB because Ethernet and TCP use connectionless, multi-point protocols that make any device-to-device communications more vulnerable to disruption by a multitude more things
 
Not sure where you got that from... Communication link reliability has little to do with the underlying protocol granted it is used in the proper context, well managed and used within a properly designed "network" architecture. 
 
Also, TCP protocol IS a connection-based protocol. It is the very reason it is considered "more reliable" that UDP, the latter being a "best-effort" protocol with no handshake and less error-correction mechanisms but typically lower latency. Both have their usecases where they shine. USB and WiFi are no different either. Some are more complex to manage for a "standard" end-user than others, that's all. 
 
The way some manufacturers implement their datalink solution is another key factor. Don't expect Ferrari performance from a Chevy van. And don't drive a Ferrari when you've always driven a Chevy van (at least not without a proper training)...
 
That aside, OP seems to have isolated the problem between the mount and his wired computer. And I agree IP address conflict (connected devices with same address) could be the culprit here given the symptoms. In that case, I would expect communication to re-establish by itself over a few seconds/minutes wait (without powercycling the mount) and then fail again a few seconds/minutes later and re-establish, and so on. A test to verify that would be to run a "perpetual ping" (add "-t" to the usual ping syntax from the command line, e.g. ping X.X.X.X -t, where X.X.X.X is the mount's IP address) and let it run for several minutes. Hit CTRL+C to end the ping.
 
Wireshark isn't a tool for a "standard" end-user. From the OP's posts, I'd say he probably knows a bit about networking and probably already uses it. If not, I would rather suggest to download Advanced IP Scanner (free) or similar, which will help discover every device alive (responsive) and dead (not responsive for a small amount of time) on a network. It will also show the MAC address (which is a unique hardware network identifier) of all devices discovered. Try running a scan while the mount is responsive and take note of the MAC address associated with its IP address. Run it again when it becomes non-responsive and if the tool marks it as "alive" and shows a different MAC address, it means you indeed have a duplicate IP address in your network. The "Name" and "Manufacturer" listed will also help you identify which device is using the same IP address as the mount. If the device is marked "dead" when the mount is non-responsive, then the problem is probably elsewhere.
 

 
Another useful (a bit more advanced) command, if you are familiar with your network IP addressing, is the "tracert" command ("traceroute" in linux) which will essentialy show the path (routing hop) taken by a packet from the computer from which you entered the command to the destination device. Its usage is similar to the ping command, e.g. tracert X.X.X.X, where X is the mount's IP address (wired or wireless). Some routers/firewalls might block this request though and you only get a series of *** + a timeout message, instead of actual routing hop IP addresses, which won't help you much. If you have a "flat" network architechture or only one routing instance, it will only return the destination lP address which won't help you much either (see example below).
 

 
If it goes through however, it will help identify up to which routing network component (switch, router, access point, etc.) communication is achieved properly by returning a series of IP addresses through which packets need to go through to their destination, as well as the round-trip time between hops. Over modern wired ethernet links, expect values below 100ms. Over wireless links, it can go much higher depending on multiple factors, but I'd say below 150-200ms on average would be acceptable (but not particularly good). Over those figure, you possibly have a bottleneck somewhere or a failling network component. BTW that command can even be used over the internet with domain names. Example below is between my computer and google.ca. The last line shows the destination IP address of one of the servers hosting the domain google.ca. Lines 1-8 shows the routing instances every data packet has to come across to reach that google server from my computer.
 

 
 
Note that the fact that the mount is not responsive from either interfaces (Ethernet and WiFi) at the same time is also a clue the problem comes from a common source to both, hence probably not coming from the wireless Access Point.
 
Also worth mentionning, even if you haven't said you are using one, is firewalls (sorry, I'll get a bit technical here). While it is actually often not the firewall root-causing the problem, it can be the one ending the communication by dropping data packets. New generations of firewalls (even home router - WiFi or wired - with firewalls functionnalities) have dynamic adaptative algorithms that "recognize" the type and "behavior" of data traffic that goes through them. They do that to prevent, amongst other things, DoS (denial of service) attacks which consist of an attacker flooding a computer with a massive amount of requests until it crashes by running out of memory.

Now, I've monitored Ethernet/WiFi communications between APCC/APv2 ASCOM drivers and my mount's CP5 (using Wireshark) and based on the amount of connections (not talking about physical hardware connection here, rather software connections at the OSI model layer 4) used, it could well be mis-recognized by some firewall's algorithm as a DoS attack. I'm saying this because it creates a new connection for seemingly every data exchange between the computer and mount, which occurs very often - like every second or so. They possibly implemented this that way for heartbeat or synchronization purposes, but that's only a guess. (That left me scratching my head a bit BTW as there are more memory-efficient ways of accomplishing this). Anyway, thing to note here, is there is nothing you can do about that last part as it's an inherent p
roperty of AP's communication between controlling software and mount.
 
But if you are using such a next-gen firewall with that kind of security feature, it could result in similar symptoms to what you are experiencing: communications working for a while and then stopping entirely when packets are dropped. Note here that the firewall is simply doing its job of protecting you. I therefore wouldn't recommend disabling this security feature entirely to solve the problem if that proves to be the case. Rather, I'd try creating a rule to whitelist the mount's IP address in your firewall's configuration in that regard.

Hope this helps as well,
 
Sébastien


Re: Losing Communications with the Mount

Dale Ghent
 

"Interference from other devices"? If you're thinking of interference like it's radio interference, switched ethernet doesn't work that way. VLANs won't save you from any particular layer <=2 issue anyhow. Since you say the pings stop regardless whether it has a DHCP or static address, it seems like a total IP stack wedge of some sort, or the OS is inexplicably downing its interfaces.

It wasn't clear if you tried connecting via USB/serial immediately after this network failure, without rebooting the CP4. If the network craps out but you can still subsequently connect via USB and control the mount, that might help narrow down where things are going wrong.

Wireshark does run on Windows. If you're able reproduce this issue with some reliability, you can run that on the box that's connecting to the CP4 via the ASCOM driver and have it run through your normal uses/reproduction paces while Wireshark is running and dumping packets to a pcap file. If the issue resurfaces, the pcap file can be inspected for the packet flow of the entire session.

I would set Wireshark to capture only ethernet frames that involve the CP4's ethernet MAC address. This is so that the pcap file doesn't grow to an unwieldy size and contains only the traffic source/destination that we're interested in. There's no guarantee that this would help elucidate what's going on - we could very well see a normal, healthy session transpire and then the CP4 suddenly stops responding. Then again, we might see something or some abnormal pattern or packet contents that points to something. I can help look at the pcap file if you'd like.

On May 6, 2021, at 18:30, alex <groups@ranous.com> wrote:

Ok, I opened up the GTOCP4 yesterday and the daughter board seems to be seated fine. I put it back and switched the ethernet cable to a brand new professionally made 15’ cable (ie, I didn’t put the connectors on), and changed the ports on the switch it was plugged into. I rebooted the switch as well in case it was in some weird state. I also hooked up the GTOCP4 to the eagle 2 directly via USB and configured it as the backup port. The primary connection was configured to be the ethernet connection using TCP and a 500ms timeout.

The switch and AP (and all my networking infrastructure) is Ubiquiti UniFi stuff (a prosumer/SOHO brand), so no different brand incompatibility in my network infrastructure. After the initial failures, I configured the router’s DHCP server to assign a fixed IP address assigned to the mount instead of a dynamic one. I’m pretty obsessive about managing my IP address space and am fairly certain there isn’t other devices colliding. The switch is fairly recent, a UniFi US-8-60W and is a fully managed smart switch. I suppose I could configure a separate VLAN for the observatory and put the mount and the eagle 2 on as the only hosts on to make sure there wasn’t interference from other devices on the network, though that seems like overkill.

Last night the ethernet connection failed again but I didn’t notice right away as this time as APCC successfully failed over to the USB connection, so that worked great. I had a perpetual ping repeating once a second the whole night, and showed response times typically between 2 and 9 milliseconds, though occasionally have some 30-50ms ones, and a few 1-2 second ones here and there. Right before the connection failed, the last few pings had 2-7ms ping times, then all the ping requests started timing out. These timeouts have been non-stop from the last 12 hours or so.

While communications was failing, The UniFi controller software didn’t show any abnormal packet loss on the port and I sshed directly into the switch and poked around the internal logs, and didn’t see anything fishy. I tried changing the port the ethernet was plugged into, and power cycling the switch to see if the problem was some bad state the switch was in. Neither woke up the TCP connection. The only thing that fixes it is power cycling the GTOCP4, so to me this seems to be a problem with the state the GTOCP4 is in. If there was some persistent ongoing problem with the network infrastructure, then a reboot of the mount wouldn’t fix the problem.

I’m still mystified as to what’s going on with that ethernet connection. I’m a software engineer and have been programming IP networks professionally over 30 years, and I’ve never seen behavior like this. I haven’t formally done IT/OPs stuff (I program back end web services nowadays), I’ve setup plenty of IP networking equipment over the years.

I could see a transient communications problem starting things off, it wouldn’t explain the inability to ping until the GTOCP4 is power cycled, which magically fixes everything. It’s appearing that some problem occurs, and the GTOCP4 goes into a mode where the network is down and only a power cycle resets it. I’ve never encountered any device with this behavior.

What’s the OS on this thing? It has a reasonably capable ARM processor. Is it running Linux or some embedded OS? Is it possible to SSH into this thing and poke around, check some logs, do something like an ifconfig, netstat, etc ? While the USB connection seems to be working well, I still want to track down what’s going on with the ethernet connection. It’s been my experience that wired ethernet connections are pretty rock solid assuming you avoid problems like long distances or interference with electrical wiring. My cable isn’t near anything like that.

I may try plugging my Mac directly into the same switch and see what Wireshark shows, if anything. It’s been a few years since I’ve messed with it.

Alex


Re: Losing Communications with the Mount

Roland Christen
 

I will forward this to our software engineer.

Roland

-----Original Message-----
From: alex <groups@...>
To: main@ap-gto.groups.io
Sent: Thu, May 6, 2021 5:30 pm
Subject: Re: [ap-gto] Losing Communications with the Mount

Ok, I opened up the GTOCP4 yesterday and the daughter board seems to be seated fine.  I put it back and switched the ethernet cable to a brand new professionally made 15’ cable (ie, I didn’t put the connectors on), and changed the ports on the switch it was plugged into.  I rebooted the switch as well in case it was in some weird state.  I also hooked up the GTOCP4 to the eagle 2 directly via USB and configured it as the backup port.  The primary connection was configured to be the ethernet connection using TCP and a 500ms timeout.
 
The switch and AP (and all my networking infrastructure) is Ubiquiti UniFi stuff (a prosumer/SOHO brand), so no different brand incompatibility in my network infrastructure.  After the initial failures, I configured the router’s DHCP server to assign a fixed IP address assigned to the mount instead of a dynamic one.  I’m pretty obsessive about managing my IP address space and am fairly certain there isn’t other devices colliding.  The switch is fairly recent, a UniFi US-8-60W and is a fully managed smart switch.  I suppose I could configure a separate VLAN for the observatory and put the mount and the eagle 2 on as the only hosts on to make sure there wasn’t interference from other devices on the network, though that seems like overkill.
 
Last night the ethernet connection failed again but I didn’t notice right away as this time as APCC successfully failed over to the USB connection, so that worked great.  I had a perpetual ping repeating once a second the whole night, and showed response times typically between 2 and 9 milliseconds, though occasionally have some 30-50ms ones, and a few 1-2 second ones here and there.  Right before the connection failed, the last few pings had 2-7ms ping times, then all the ping requests started timing out. These timeouts have been non-stop from the last 12 hours or so.
 
While communications was failing, The UniFi controller software didn’t show any abnormal packet loss on the port and I sshed directly into the switch and poked around the internal logs, and didn’t see anything fishy.  I tried changing the port the ethernet was plugged into, and power cycling the switch to see if the problem was some bad state the switch was in.  Neither woke up the TCP connection. The only thing that fixes it is power cycling the GTOCP4, so to me this seems to be a problem with the state the GTOCP4 is in.  If there was some persistent ongoing problem with the network infrastructure, then a reboot of the mount wouldn’t fix the problem.
 
I’m still mystified as to what’s going on with that ethernet connection.  I’m a software engineer and have been programming IP networks professionally over 30 years, and I’ve never seen behavior like this. I haven’t formally done IT/OPs stuff (I program back end web services nowadays), I’ve setup plenty of IP networking equipment over the years.
 
I could see a transient communications problem starting things off, it wouldn’t explain the inability to ping until the GTOCP4 is power cycled, which magically fixes everything. It’s appearing that some problem occurs, and the GTOCP4 goes into a mode where the network is down and only a power cycle resets it.  I’ve never encountered any device with this behavior.  
 
What’s the OS on this thing?  It has a reasonably capable ARM processor.  Is it running Linux or some embedded OS?  Is it possible to SSH into this thing and poke around, check some logs, do something like an ifconfig, netstat, etc ?  While the USB connection seems to be working well, I still want to track down what’s going on with the ethernet connection.  It’s been my experience that wired ethernet connections are pretty rock solid assuming you avoid problems like long distances or interference with electrical wiring.  My cable isn’t near anything like that.
 
I may try plugging my Mac directly into the same switch and see what Wireshark shows, if anything.  It’s been a few years since I’ve messed with it.
 
Alex
 

--
Roland Christen
Astro-Physics


Re: Losing Communications with the Mount

Sébastien Doré
 

Christopher,

As you surely know, much of all Internet communication is TCP-based nowadays and work arguably 365 days/year and almost 24hours a day (like 2 and 3 nines networks). Not sure what 30 years of experience with TCP has to do with that or any communication reliability figure for that matter. 

PingPlotter does indeed look like a valuable tool (you even convinced me to download and try it).

In regards to the old CLI vs GUI debate, I don't think it's the right place to argue on this forum, but I will only add that network command line tools are way less prone to bugs than any GUI tools. Moreover, you don't even have to download, install or update anything, they come built-in with OSes, even in 2021. There must be a good reason for that... My guess is that if you happen to be cutoff from the internet because of a communication problem, you have to rely on something to get back on your feet. Might be a good thing to learn the ropes of using simple command line tools, which anyone here - as you also stated - has the ability to grab. I'd recommend anyone to learn basic command line tools well before trying out any graphical wireshark/pcap/tcpdump analyzer. Well, it seems like you got me started after all. 😉 Sorry about that everybody. At least, now you know where I stand... 

As for your other comments, I'd say that I agree it's good practice to look at the plumbering stuff (lower layers) first, but really solving (like definitively, not "just make it work" for some time until the next software update) most common communication problems beyond those obvious ones also requires looking at least at the TCP/IP layers these days, especially if you don't have an infinite budget to replace everything out of trial and errors and/or infinite amount of time to spend on it (I'm sure most of us would rather spend this precious time under clear skies with their latest AP equipment). 

That is also exactly what PingPlotter does BTW: it compiles and presents statistics from - you stated it - ping and tracert requests which uses a L3 protocol (ICMP) running in the background.


Clear skies,
Sébastien


De : main@ap-gto.groups.io <main@ap-gto.groups.io> de la part de Christopher Erickson <christopher.k.erickson@...>
Envoyé : 6 mai 2021 16:24
À : main@ap-gto.groups.io <main@ap-gto.groups.io>
Objet : Re: [ap-gto] Losing Communications with the Mount
 
My experience with TCP comes from 30 years of telecommunications and robotics engineering. My primary concerns are much more with Layer-1 of the OSI model (cables, connectors) and Layer-2 (Ethernet frames), not Layer-3 (IP packets) or Layer-4 (TCP/UDP.) 

OSI Layers 1 & 2 are VERY opaque to the average user so consequently they are usually ignored when troubleshooting. I think this is typically a mistake. Sort of like looking for your car keys under a nice streetlight instead of next to your car, where you dropped them. 

PingPlotter is a very graphical, visual troubleshooting tool that has a free version. It is PROFOUNDLY better and more intuitive than using the DOS prompt command line Ping command. PingPlotter also incorporates a very nice, visual, graphical, dynamic traceroute. Download it and try it out. You won't go back to the nasty DOS prompt command line ever again, unless forced to on a strange machine. 

I agree Wireshark is a complicated tool. I already stated that. However I believe that the typical AP mount owner is more qualified than the average person to gain benefit from it, given some time. I would add that starting with PingPlotter instead of Wireshark would be good.

It could be bad to have a firewall or router in between the CP4/5 and the observatory PC. If there is, it might have LAN packet filtering capabilities, which I would disable, if I could.

-Christopher Erickson
Observatory engineer
Waikoloa, HI 96738
www.summitkinetics.com
   

_._,_._,_


Re: Losing Communications with the Mount

Alex
 

Oh, I forgot mention that I also tried using the mount's WiFi connection as well, and it also had similar issues, so the ethernet connection doesn't seem to be at fault.

Alex


Re: Strange guiding errors with AP1200

Roland Christen
 

The RA worm turns once every 6.4 minutes and so if there was a piece of dirt embedded on the worm teeth, there would be a jump in that time frame. If it repeats forever, then I would suspect a damaged worm. If it happened just once in the same place in the sky I would suspect a damaged tooth on the main worm wheel.

There could also be a piece of dirt embedded in the final spur gear that's attached to the end of the worm. That can be easily cleaned, but that would show up for every 6.4 minute cycle and would not go away.

The fact that it showed up in Dec is probably due to a bad calibration run. Dec axis probably did not move, but the guide software is interpreting part of the RA error as a Dec error.

I would remove the RA gearbox, clean all the grease off both the worm, the main worm wheel and all the transfer gears inside the gearbox. Then re-grease every thing.

Rolando



-----Original Message-----
From: Allen Gilchrist via groups.io <gilchrist.allen@...>
To: main@ap-gto.groups.io
Sent: Thu, May 6, 2021 4:44 pm
Subject: [ap-gto] Strange guiding errors with AP1200

Has anyone else seen anything like this before?  During an imaging session with my AP1200, while guiding with an STi on a 400 mm f.l. 80 mm refractor, I noticed an occasional really large guiding error.  It took a few cycles to bring the guidestar back to the center of the guide window, and then all was OK until it happened again.  I started an autoguider log file, and found that the process repeated every 379 sec.  This suggested some problem in the RA drive system, but there were corresponding spikes in the Dec. error log as well.  In fact these errors were larger than those in RA.  It almost looked like something was binding the RA drive every 379 seconds and then, when the drive slipped free, there was an impact on the Dec axis.  Interestingly, after recording five of these events in the autoguider log, the problem vanished.  I continued observing for about another hour and a half but the problem did not return.  I've attached a couple of plots, one showing the problem, and the second one after the problem went away.  The plot scale is in pixels, and each pixel is 3.82 arcseconds.  Any ideas?

Allen

--
Roland Christen
Astro-Physics


Re: Losing Communications with the Mount

Alex
 

Ok, I opened up the GTOCP4 yesterday and the daughter board seems to be seated fine.  I put it back and switched the ethernet cable to a brand new professionally made 15’ cable (ie, I didn’t put the connectors on), and changed the ports on the switch it was plugged into.  I rebooted the switch as well in case it was in some weird state.  I also hooked up the GTOCP4 to the eagle 2 directly via USB and configured it as the backup port.  The primary connection was configured to be the ethernet connection using TCP and a 500ms timeout.
 
The switch and AP (and all my networking infrastructure) is Ubiquiti UniFi stuff (a prosumer/SOHO brand), so no different brand incompatibility in my network infrastructure.  After the initial failures, I configured the router’s DHCP server to assign a fixed IP address assigned to the mount instead of a dynamic one.  I’m pretty obsessive about managing my IP address space and am fairly certain there isn’t other devices colliding.  The switch is fairly recent, a UniFi US-8-60W and is a fully managed smart switch.  I suppose I could configure a separate VLAN for the observatory and put the mount and the eagle 2 on as the only hosts on to make sure there wasn’t interference from other devices on the network, though that seems like overkill.
 
Last night the ethernet connection failed again but I didn’t notice right away as this time as APCC successfully failed over to the USB connection, so that worked great.  I had a perpetual ping repeating once a second the whole night, and showed response times typically between 2 and 9 milliseconds, though occasionally have some 30-50ms ones, and a few 1-2 second ones here and there.  Right before the connection failed, the last few pings had 2-7ms ping times, then all the ping requests started timing out. These timeouts have been non-stop from the last 12 hours or so.
 
While communications was failing, The UniFi controller software didn’t show any abnormal packet loss on the port and I sshed directly into the switch and poked around the internal logs, and didn’t see anything fishy.  I tried changing the port the ethernet was plugged into, and power cycling the switch to see if the problem was some bad state the switch was in.  Neither woke up the TCP connection. The only thing that fixes it is power cycling the GTOCP4, so to me this seems to be a problem with the state the GTOCP4 is in.  If there was some persistent ongoing problem with the network infrastructure, then a reboot of the mount wouldn’t fix the problem.
 
I’m still mystified as to what’s going on with that ethernet connection.  I’m a software engineer and have been programming IP networks professionally over 30 years, and I’ve never seen behavior like this. I haven’t formally done IT/OPs stuff (I program back end web services nowadays), I’ve setup plenty of IP networking equipment over the years.
 
I could see a transient communications problem starting things off, it wouldn’t explain the inability to ping until the GTOCP4 is power cycled, which magically fixes everything. It’s appearing that some problem occurs, and the GTOCP4 goes into a mode where the network is down and only a power cycle resets it.  I’ve never encountered any device with this behavior.  
 
What’s the OS on this thing?  It has a reasonably capable ARM processor.  Is it running Linux or some embedded OS?  Is it possible to SSH into this thing and poke around, check some logs, do something like an ifconfig, netstat, etc ?  While the USB connection seems to be working well, I still want to track down what’s going on with the ethernet connection.  It’s been my experience that wired ethernet connections are pretty rock solid assuming you avoid problems like long distances or interference with electrical wiring.  My cable isn’t near anything like that.
 
I may try plugging my Mac directly into the same switch and see what Wireshark shows, if anything.  It’s been a few years since I’ve messed with it.
 
Alex
 


Re: Spikes in Dec

Roland Christen
 

Basically the way RA guiding happens is the same as a non-encoder mount, except that the guide software doesn't have to fight a periodic error. So the guide pulses can be sent at a slower cadence. (cadence is not equal to guide star exposure).

Declination is definitely different in an Astro-Physics encoder mount. Reversal backlash is gone, so that a 1 arc sec command in the reverse direction will cause the axis to move 1 arc sec in that reverse direction. A non-encoder mount sometimes needs many 1 arc sec move commands before the Dec axis actually reverses direction. That's because the motor shaft must move a significant rotational distance (number of arc seconds) in reverse before the worm wheel begins to move. It could be 5 arc seconds, or 10 or even more on some mounts. It all depends how tight the gearing is and the clearances in the reduction gears or the amount a belt stretches in a belt drive mount. It's never zero in any mount.

There is a caveat however for the 1 arc sec reversal in an encoder mount. The encoder does not eliminate the actual mechanical delay, or the distance it takes for the reversal to happen. The motor shaft still needs to move 5 or 10 arc seconds at the 1x rate, but the encoder makes that happen automatically after only one move command. It means that any move command in Dec might take a second of time if it is a reversal move, rather than being instant. Therefore the best way to guide in Dec is to let the axis settle for a second or two before taking another guide exposure and sending another guide pulse. Too fast cadence will cause the tracking to be less accurate than it could be. Adding a 2 second delay between guide exposures is a good way to to keep the Dec axis stable and prevent hunting and chasing the seeing.

It would even be better if PHD would wait 1 second after a move command before actually taking the exposure of the guide star so that the axis has a chance to reverse and fully settle. But perhaps that is not in the cards at this time. Maxim DL does insert a small delay of that size after issuing a guide command, and so I have found that it will guide very accurately even at the longest focal lengths.

Rolando



-----Original Message-----
From: Brian Valente <bvalente@...>
To: main@ap-gto.groups.io
Sent: Thu, May 6, 2021 4:58 pm
Subject: Re: [ap-gto] Spikes in Dec

You want to enable DEC compensation here

This adjusts the guiding based on your sky position, which doesn't change (unless you are going to say you have a space telescope)

>>>I realize this is more a question for the PHD forum, (sorry about that) but I thought it could be of general interest here as well if you have the answer at hand.

that's fine - happy to help. Maybe you can shoot me some of your logs direct via email and I can review them. 

I'm not sure I saw what Howard mentioned, but guiding with high quality encoders is a relatively new thing, so anything is possible. lowpass2 was introduced only a few years ago iirc

 
Brian


On Thu, May 6, 2021 at 1:45 PM Seb@stro <sebastiendore1@...> wrote:
Hi Brian,

For the DEC, yes, you recalled right, Lowpass2. For RA, I have Hysterersis selected. 

I see there's also the possibility to select lowpass2 for RA, but I believe I read somewhere not to use it for that axis (possibly in the PHD2 manual). Anyway, I didn't seem to have any guiding problems in RA, so I'd leave it there for now.

Just looking around at other settings, I realize that I have the "Use DEC compensation" checked in the Guiding tab,  Calibration section. Help file says that it usually should stay checked unless in "unsual" cases, like when a mount's controller would apply a compensation automatically. Should I assume that a mount with abs encoders correspond to that definition and uncheck the box? Do you think, this could explain the issue reported by Howard/Roland ?

I realize this is more a question for the PHD forum, (sorry about that) but I thought it could be of general interest here as well if you have the answer at hand.



Sébastien


De : main@ap-gto.groups.io <main@ap-gto.groups.io> de la part de Brian Valente <bvalente@...>
Envoyé : 6 mai 2021 17:01
À : main@ap-gto.groups.io <main@ap-gto.groups.io>
Objet : Re: [ap-gto] Spikes in Dec
 
Seb i seem to recall you using lowpass2 in PHD which is the correct algorithm for high res encoders


--
Brian 



Brian Valente

--
Roland Christen
Astro-Physics


Re: Losing Communications with the Mount

Steve Reilly
 

I’ve used a ton of the Startech Serial cards and they work great. Never had a failure in any of the systems I’ve built. The one I use now has 8 serial cables on the rear but you can get less. The card is a similar to this version

 

-Steve

 

 

From: main@ap-gto.groups.io <main@ap-gto.groups.io> On Behalf Of Donald Gaines
Sent: Thursday, May 6, 2021 5:01 PM
To: main@ap-gto.groups.io
Subject: Re: [ap-gto] Losing Communications with the Mount

 

Hi Howard,

Thanks for the info. I thought I might have add a card with serial ports. I’ll go that route. Thanks for your advice. 

Regards,

Don Gaines

On Thursday, May 6, 2021, Howard Hedlund <howard@...> wrote:

Using the supplied serial cable with a USB to serial converter, even the FTDI unit that we sell, will still be limited in its reliability to that of USB.  Chris is referring to a native serial port on a dedicated board inside the computer itself that avoids the Universal Serial Bus.  Our on-board USB port uses the same FTDI chipset, but it has an added advantage over an external USB to serial device.  An external device is powered solely from the bus in the computer.  Our port, however, is also powered so a power drop from the computer won't kill the USB connection because the CP4/5 will keep it alive.

4741 - 4760 of 83105