[arm-allstar] Echolink, Asterisk & one-way audio issues

Sun Mar 4 21:35:21 EST 2018

Hi Mark,

Thanks for your AllStar and echolink experience feedback.

You and a few others have reported regular crashes when echolink is
enabled AND BEING HEAVILY USED. I manage a couple AllStar/echolink systems
down here and haven't really noticed any problems, but echolink doesn't
get heavily used. Since the failure seems to be a seg-fault in the
chan_echolink thread, I'm hopeful the conditions which generate this crash
won't be too hard to locate/duplicate. I'm actively debugging this now and
am suspecting the cause is sporadic packet loss causing an improperly
handled error condition. We will squash this problem ASAP!

Note that there have been no major chan_echolink changes in over a year
now, so I can't explain why crashes would have gotten worse with recent
software updates??  My suspicion is you may have had a change in your
echolink usage patterns (e.g.: increased usage?) that coincided over the
same timeframe??  ....Anyhow, as I said, I'm looking for a solution.

For a historical footnote, the original ("stock") chan_echolink driver did 
have some crash-me bugs which I also regularly butted heads with back in 
yesteryear (my "ACID" days). Those issues got resolved back around 
August 2016 and were related mostly to echolink nodelist updates going 
haywire.

The one-way audio issue is a different beast. When it was originally
reported, I figured it was due to a bug I introduced; I have VERY heavily
updated the app_rpt code. However, after reviewing several user's systems
where they were experiencing one-way audio, I know it affects AllStar
versions at least as far back as the stock 0.325 version (originally
ohnosec.org SVN release 1512, 10/19/2014)---even old "ACID" systems, too.
So, one-way audio isn't a recently introduced problem.

I originally thought one-way audio was only related to IAX2 connections
between nodes. However, it has recently been reported with echolink, too.
Again, probably indicating an increase in chan_echolink usage. But, this
also means this problem isn't IAX2 specific...So, on a positive note, that
narrows the places to look for this problem!

For me, one-way audio is fleetingly sporadic.  Until a couple weeks ago, I
had NEVER been able to locally duplicate this failure mode.  Then, after a
network hiccup, I had a test node go one-way and I was able to perform a
lot of diagnostics and learned a lot...Now, I've again modified my debug
setup and am again waiting....And, waiting.... We'll get this one fixed,
too. Eventually.

73, David KB4FXC

On Sun, 4 Mar 2018, "Mark Szlaga via arm-allstar" wrote:

> Doug,
> 
> Thank you for your responses.  While I realize Echolink is not the
> preferred protocol, the goal is to move everything where possible to
> Allstar while combining various VoIP protocols in to a single point
> directly connected to the repeater itself.
> 
> Regarding Echolink.  I realize it is a mostly stateless connection, but I
> find it odd that the control channel is still showing as connected, but no
> audio.  Is this a known failure mode with Echolink and Allstar?  We have
> not been seeing it with the previous SVXlink node.  If it is we can work
> around it.  I just need to set expectations for the repeater trustee.
> 
> Regarding Crashing.  This is the error message I am seeing from asterisk.
> Note, this issue is across two different SD cards, and two different
> RasPi3s.  All devices are known good and solid.
> 
> safe_asterisk: line 125: 23632 Segmentation fault
> (core dumped) nice -n $PRIORITY ${ASTBINDIR}/asterisk -f ${CLIARGS}
> ${ASTARGS} &> /dev/${TTY}
> Asterisk ended with exit status 139
> Asterisk exited on signal 11
> Automatically restarting Asterisk.
> mpg123:: no process found
> 
> Upon restart it is fine for a few minutes/hours until it restarts.
> Thankfully every time it restarts and comes back.
> 
> This has gotten worse since the update from the original image installation
> to the latest version:
> RPi2-3 Version 1.5rc4-15 Allstar - February 15, 2018 - KB4FXC, WA3DSP
> 
> Regarding RX Delay.  I need help here.  1.5 seconds is a lifetime in this
> industry.  WAY too long.  Any pointers on things to start investigating?
> 
> Note: during normal local nets I set the RXOnDelay to 10 and RXAudioDelay
> to 7 with no issues.  Works like a champ.
> The ping-pong and pulsing only occurs during connected nets that we are not
> hosting.  This issue is both on Allstar as well as Echolink.
> 
> Thoughts?  Comments/questions?
> 
> Regards,
> Mark - AC8LT
> 
> On Fri, Mar 2, 2018 at 3:29 PM, "Doug Crompton via arm-allstar" <
> arm-allstar at hamvoip.org> wrote:
> 
> > Mark,
> >
> >   I might also add to my previous remarks that Echolink unlike Allstar has
> > no permanent connect mode. If a connection is flaky you are going to lose
> > it at some point and it won't come back unless you reconnect. You should
> > never depend on an Echolink connection. It is what it is.
> >
> >
> > *73 Doug*
> >
> > *WA3DSP*
> >
> > *http://www.crompton.com/hamradio <http://www.crompton.com/hamradio>*
> >
> >
> > On Thu, Mar 1, 2018 at 11:42 PM, "Mark Szlaga via arm-allstar" <
> > arm-allstar at hamvoip.org> wrote:
> >
> > > We are seeing three issues on the WW8GM node (47185) that I have not been
> > > able to find a straight solution for when searching.
> > >
> > > 1) Echolink randomly drops audio carrier entirely.
> > >
> > > While I realize the Echolink module is meant for compatibility, we are
> > > using the module to replace an existing SVXlink node and expand in to
> > > Allstar, with the goal of moving as many nets to utilize Allstar where
> > > possible.
> > >
> > > That said, there is one net held on the ALARA conference node that is
> > still
> > > Echolink that we connect in to.
> > >
> > > Tonight and on previous occasions we have had instances where the audio
> > > drops.  The connection still shows in Allstar and Echolink, I see the
> > node
> > > updating names as the participants transmit, but no audio.
> > >
> > > Every time a drop and reconnect fixes the issue.
> > >
> > > The question is, what causes this?  Is this a "known issue" with the
> > > module, or are there workarounds.
> > >
> > > If this is new, what debugging can I run to capture the necessary
> > > information to troubleshoot?
> > >
> > > 2) Ping-pong, bumping, pulsing...  I'v heard it called a number of
> > > different names, but the result is the same...
> > >
> > > We have had some issues tuning both the Allstar and Echolink sides when
> > > connected to nets.
> > >
> > > NOTE: This issue is ONLY seen during nets with other nodes, not locally
> > or
> > > when we are connected to one node.
> > >
> > > Configuration:
> > > Hardware: Motorola Radius 1225
> > > Software: Raspberry Pi3 running the HamVoIP image, updated to latest last
> > > night.
> > > Audio interface: RB_RIM_RC210 from repeater-builder.com
> > > Carrier config: CoS is set to TSQL and repeater only transmits Tone
> > before
> > > courtesy tone during nets.
> > >
> > > Why this interface, because the node will be connected to the RC210 at
> > the
> > > main repeater site once the bugs are worked out.
> > >
> > > But...  At this point we have RXOnDelay set to 75, with RXAudioDelay set
> > to
> > > 7.  This is enough that we aren't having bumps or bouncing, but 1.5
> > seconds
> > > is a lifetime!
> > >
> > > Any thoughts on things to try to reduce this and get a more manageable
> > > delay timing?
> > >
> > > 3) Last but not least, we do have occasional Asterisk restarts when
> > > connected to Echolink.
> > >
> > > We are seeing Asterisk restarts when connected to Echolink nodes.
> > Allstar
> > > and normal usage does not cause restarts.
> > >
> > > What can I do to the system to log these restarts to see why it is doing
> > > this?  I have been letting a console run with the following
> > > debugging/logging:
> > >
> > > core set verbose 99
> > > core set debug 99
> > > rpt debug level 7
> > > echolink debug level 7
> > >
> > > Unfortunately it seems that when the connection is dropped, no error is
> > > shown on the screen other than the disconnect.
> > >
> > > Thoughts?
> > >
> > > Thanks to the team.  You have made a great product and the GMARC club is
> > > loving it.  I just want to get the last of the gremlins squashed so they
> > > will love this more!
> > >
> > > Regards and 73,
> > > Mark - AC8LT
> > > _______________________________________________
> > >
> > > arm-allstar mailing list
> > > arm-allstar at hamvoip.org
> > > http://lists.hamvoip.org/cgi-bin/mailman/listinfo/arm-allstar
> > >
> > > Visit the BBB and RPi2/3 web page - http://hamvoip.org
> > >
> > _______________________________________________
> >
> > arm-allstar mailing list
> > arm-allstar at hamvoip.org
> > http://lists.hamvoip.org/cgi-bin/mailman/listinfo/arm-allstar
> >
> > Visit the BBB and RPi2/3 web page - http://hamvoip.org
> >
> _______________________________________________
> 
> arm-allstar mailing list
> arm-allstar at hamvoip.org
> http://lists.hamvoip.org/cgi-bin/mailman/listinfo/arm-allstar
> 
> Visit the BBB and RPi2/3 web page - http://hamvoip.org
> 