[arm-allstar] Core Dumps

[Sean McCarthy]

Looking for some assistance figuring this out. System was running
flawlessly for weeks before this started on the 31st at 9 p.m.

Core dumps on the top of the hour consistently with the following messages
in the journal:

Aug 02 21:00:01 FBIARC_HUB crond[3830]: pam_unix(crond:session): session
opened for user root by (uid=0)
Aug 02 21:00:01 FBIARC_HUB crond[3829]: pam_unix(crond:session): session
opened for user root by (uid=0)
Aug 02 21:00:01 FBIARC_HUB CROND[3832]: (root) CMD
(/usr/local/bin/AUTOSKY/AutoSky)
Aug 02 21:00:01 FBIARC_HUB CROND[3831]: (root) CMD ((source
/usr/local/etc/allstar.env ; /usr/bin/nice -19 /usr/bin/perl
/usr/local/sbin/saytime.pl 48092 $NODE1 > /dev/null))
Aug 02 21:00:01 FBIARC_HUB CROND[3830]: (root) CMDOUT (awk: fatal error:
internal error)
Aug 02 21:00:01 FBIARC_HUB systemd[1]: Started Process Core Dump (PID
3860/UID 0).
Aug 02 21:00:01 FBIARC_HUB CROND[3830]: (root) CMDOUT (awk: fatal error:
internal error)
Aug 02 21:00:01 FBIARC_HUB systemd[1]: Started Process Core Dump (PID
3865/UID 0).
Aug 02 21:00:01 FBIARC_HUB CROND[3830]: (root) CMDOUT (awk: fatal error:
internal error)
Aug 02 21:00:01 FBIARC_HUB systemd-coredump[3861]: Process 3859 (awk) of
user 0 dumped core.

                                                   Stack trace of thread
3859:
                                                   #0  0x000000000006aad8
n/a (gawk)
Aug 02 21:00:01 FBIARC_HUB systemd[1]: Started Process Core Dump (PID
3870/UID 0).
Aug 02 21:00:01 FBIARC_HUB systemd-coredump[3866]: Process 3864 (awk) of
user 0 dumped core.

                                                   Stack trace of thread
3864:
                                                   #0  0x000000000006aad8
n/a (gawk)
Aug 02 21:00:01 FBIARC_HUB systemd-coredump[3871]: Process 3869 (awk) of
user 0 dumped core.

                                                   Stack trace of thread
3869:
                                                   #0  0x000000000006aad8
n/a (gawk)
Aug 02 21:00:01 FBIARC_HUB CROND[3829]: (root) CMDOUT (AutoSky Xresult is 0)
Aug 02 21:00:01 FBIARC_HUB CROND[3829]: pam_unix(crond:session): session
closed for user root
Aug 02 21:00:02 FBIARC_HUB CROND[3830]: pam_unix(crond:session): session
closed for user root
Aug 02 21:01:01 FBIARC_HUB CROND[3903]: (root) CMD (run-parts
/etc/cron.hourly)
Aug 02 21:04:01 FBIARC_HUB crond[3934]: pam_unix(crond:session): session
opened for user root by (uid=0)
Aug 02 21:04:01 FBIARC_HUB CROND[3935]: (root) CMD
(/usr/local/bin/AUTOSKY/AutoSky)
Aug 02 21:04:02 FBIARC_HUB CROND[3934]: (root) CMDOUT (AutoSky Xresult is 0)
Aug 02 21:04:02 FBIARC_HUB CROND[3934]: pam_unix(crond:session): session
closed for user root
Aug 02 21:05:15 FBIARC_HUB systemd[1]: Started Process Core Dump (PID
3960/UID 0).
Aug 02 21:05:15 FBIARC_HUB systemd-coredump[3961]: Process 3959 (awk) of
user 33 dumped core.

                                                   Stack trace of thread
3959:
                                                   #0  0x000000000006aad8
n/a (gawk)
Aug 02 21:05:15 FBIARC_HUB systemd[1]: Started Process Core Dump (PID
3986/UID 0).
Aug 02 21:05:15 FBIARC_HUB systemd-coredump[3987]: Process 3985 (awk) of
user 33 dumped core.

                                                   Stack trace of thread
3985:
                                                   #0  0x000000000006aad8
n/a (gawk)
Aug 02 21:08:01 FBIARC_HUB crond[4022]: pam_unix(crond:session): session
opened for user root by (uid=0)
Aug 02 21:08:01 FBIARC_HUB CROND[4023]: (root) CMD
(/usr/local/bin/AUTOSKY/AutoSky)
Aug 02 21:08:01 FBIARC_HUB CROND[4022]: (root) CMDOUT (AutoSky Xresult is 0)
Aug 02 21:08:01 FBIARC_HUB CROND[4022]: pam_unix(crond:session): session
closed for user root
[root at FBIARC_HUB ~]#