Second build sold out. Message will appear here when store is ready for third build ordering.

URL to get current user information [solved by parsing the log file]

edited December 2018 in Problems Now Fixed
I could parse the information on the http://.../admin page to get what I want, but a 'curl http://.../admin' doesn't include the list of channels and users shown by the browser. I'm not enough of a web programer to understand how/if curl and/or wget could get that information so I can parse it out for usage statistics log. Alternative solutions would be welcome.

Comments

  • While my solution is not exactly an answer, I too wanted web access to a log of users. What I did was wrote a script to run 'make users' and pipe the output to a file. Then that file is sent (FTP) to my server. On the server side I wrote an application to parse it and store the data in a SQL database. After that I wrote a simple template to retrieve and display the last n rows from the table.

    I noticed that the 'make users' command only lists the users LEAVING. Therefore I modified the Makefile to also include ARRIVED. Below is some code that might be useful to you or someone else.
    mu_ftp.sh
    #!/bin/sh #Save this file in /root, not /root/Beagle_SDR_GPS or it will be overwritten upon kiwi update! HOST='ftp.local.server' USER='kiwisdr' PASSWD='ftp_passwd' FILE='mu.txt' FILEDIR='/root/' MUDIR='/root/Beagle_SDR_GPS' #When the kiwi is updated, the Makefile is overwritten #if necessary, add the recipe to the Makefile if ! grep -Fq "LEAVING_ARRIVED" $MUDIR/Makefile then sed -i '/LEAVING = /r Makefile_add_njc.txt' $MUDIR/Makefile fi #run make command make -C $MUDIR users3 > $FILEDIR$FILE #FTP to server ftp -n $HOST

    Makefile_add_njc.txt
    LEAVING_ARRIVED = grep -i -e leaving -e arrived | $(LOCAL_IP) users3: -@$(LOGS) | $(LEAVING_ARRIVED) @rm -f /tmp/kiwi.log

    Place both of the above files in /root, not /root/Beagle_SDR_GPS or it will be overwritten upon kiwi update!
    Run mu_ftp.sh as a cron job by running crontab -e and adding a line line this to the bottom of the file, followed by an empty line.
    */5 * * * * /root/mu_ftp.sh
    This runs the script every 5 minutes and should send a mu.txt file to your ftp server.

    Here is the webserver code I used to parse mu.txt and save to the database. It is a coldfusion webserver backend, but it should be relatively easy to modify for something else.


    logFile = ExpandPath(".\") & "mu.txt";
    oFile = Fileopen(logFile,"read");
    arrConns = [];
    regex = '((Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\s*\d?\d)\s*(\d{2}:\d{2}:\d{2}).*?(\d{0,5}\.\d{2})\s*(.Hz)\s*([a-zA-z_-]+)\s*(z\d{1,2})\s*([a-zA-Z0-9-_]{0,10}\s*)("((?!\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}).+)"|\s*)\s*"?(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}|\s*)"?\s*(.+)\s*\((ARRIVED|LEAVING)';

    while(NOT FileisEOF(oFile))
    {
    arrData = [];
    ln = FileReadLine(oFile);
    //f = ln.match(regex);
    f = reFindNoCase(regex,ln,1,true);
    idx = 1;
    for(lng in f.len){
    if(idx NEQ 1 && idx NEQ 3 && idx NEQ 10){ //3: month name, 10: callsign in quotes (regex could be improved)
    //writeOutput("#Mid(ln,f.pos[idx],lng)#
    "); if(lng GT 0){ ArrayAppend(arrData,Mid(ln,f.pos[idx],lng)); }else{ ArrayAppend(arrData,""); } } idx++; } arrayAppend(arrConns,arrData); } IF OBJECT_ID('tempdb..####tempDBkiwi') IS NULL BEGIN CREATE TABLE ####tempDBkiwi( [dt] [datetime], [frequency] [varchar](10) , [freqUnit] [varchar](10), [mode] [varchar](10), [zoom] [varchar](5) , [extension] [varchar](10), [callsign] [varchar](25), [IPAddress] [char](20), [location] [varchar](500), [status] [varchar](20) ) END declare @dt datetime = ; declare @frequency varchar(10) = ; declare @freqUnit varchar(10) = ; declare @mode varchar(10) = ; declare @zoom varchar(5) = ; declare @extension varchar(10) = ; declare @callsign varchar(25) = ; declare @IPAddress char(20) = ; declare @location varchar(500) = ; declare @status varchar(20) = ; INSERT INTO ####tempDBkiwi (dt,frequency,freqUnit,mode,zoom,extension,callsign,IPAddress,location,status) VALUES(@dt,@frequency,@freqUnit,@mode,@zoom,@extension,@callsign,@IPAddress,@location,@status) MERGE logs P USING ####tempDBkiwi S ON P.dt = S.dt AND P.frequency = S.frequency AND P.freqUnit = S.freqUnit AND P.mode = S.mode AND P.zoom = S.zoom AND P.extension = S.extension AND P.callsign = S.callsign AND P.IPAddress = S.IPAddress AND P.location = S.location AND P.status = S.status WHEN NOT MATCHED THEN INSERT (dt,frequency,freqUnit,mode,zoom,extension,callsign,IPAddress,location,status) VALUES (S.dt,S.frequency,S.freqUnit,S.mode,S.zoom,S.extension,S.callsign,S.IPAddress,S.location,S.status) WHEN NOT MATCHED BY SOURCE THEN DELETE; DROP TABLE ####tempDBkiwi


    index.html
    <!doctype html> SELECT TOP #qTop# * FROM logs ORDER BY dt DESC Latest #qTop# Kiwi Users log Time (UTC) Callsign Frequency Extension Zoom Location IP Address Status #DateFormat(dt,"yyyy-mm-dd")# #TimeFormat(dt,"HH:mm:ss")# #callsign# #frequency# #freqUnit# #mode# #extension# #zoom# #location# #IPAddress# #status#

    My logs here: kiwisdr.njctech.com/logs
  • Thanks.
    Wow, that is a *lot* of programming to extract information which the kiwi already knows.
    I'm not familiar with SQL, so it isn't clear to me what criterion you are using to decide when the ARRIVED is not matched by a following LEAVING
    Those lines are extracted from the /var/log/messages file which persists through reboots, so it seems that there could be many stale ARRIVED lines in /var/log/messages
  • Thanks for your reply.

    I'm not doing anything to match up the LEAVING with an ARRIVED. The SQL MERGE only makes sure there's no dupes inserted to the DB. Otherwise it's merely parse of the log. Another thing I didn't find in the logs (at least the make users output) was a listing of all the changes a user made instead of just the ARRIVED and LEAVING. Perhaps that info is somewhere in /var/log/messages? If so it might make sense to parse that instead of the make users output

    I agree it would be better if this was integrated into the kiwi itself, but for me this was the fastest way I know to get what I wanted. My hope is that maybe pieces of this, like the regex or the Makefile mods, might be usable to the next tinkerer...
  • you have stimulated me to find a solution. from the command line of the kiwi:
    root@kiwisdr:~# for channel in 0 1 2 3; do awk -v regex=$channel '/ARRIVED|LEAVING/ && $8 ~ regex' /var/log/messages | tail -1 ; done
    Dec 13 13:19:34 kiwisdr kiwid: 1d:07:14:14.556 0... 0 0.00 kHz am z0 "kiwiwspr_v1.1g" 192.168.158.70 (ARRIVED)
    Dec 16 16:11:08 kiwisdr kiwid: 4d:10:05:48.724 0..3 1 13720.00 kHz amn z6 "104.220.214.86" Woodland, California, USA (LEAVING after 0:00:52)
    Dec 16 14:59:41 kiwisdr kiwid: 4d:08:54:21.086 01.3 2 326.27 kHz cw z6 "109.159.13.119" Runcorn, United Kingdom (LEAVING after 0:04:39)
    Dec 16 13:16:07 kiwisdr kiwid: 4d:07:10:47.324 0123 3 10106.00 kHz cw z0 "87.117.57.31" Sofia, Bulgaria (ARRIVED)
    root@kiwisdr:~#
  • Providing this information via a URL opens up a lot of cans of worms I don't have time to deal with. You can't provide user info accessible to anyone (like /status) because the privacy crowd will show up with torches and pitchforks (we've had enough trouble in the past with the issue of user ip addresses and geolocations). And you can't add a quick and easy authentication scheme (like adding the admin password in cleartext to the URL) without the security people jumping down my throat again.
    cathalferris
  • I agree with your choice for more security.

    And I have found it is easy to use awk to extract the status of each kiwi rx channel. This one liner gets the current status of all the rx channels at KPH:

    rob@Robs-MBP-New:/Users/rob> for kw in 73 74 75 76; do ssh root@10.14.70.$kw "awk -v kiwi=$kw '/ARRIVE|LEAVING/{last[\$8]=\$0} END {for (i=0; i< 4; ++i) print \"kiwi\" kiwi \": \" last[i]}' /var/log/messages"; done 2>&1 | grep "^kiwi"
    kiwi73: Dec 16 17:19:51 kiwisdr kiwid: 4d:18:50:45.274 01.3 0 0.00 kHz am z0 "wsprdaemon_v2.0d1" 10.14.70.84 (ARRIVED)
    kiwi73: Dec 17 04:41:13 kiwisdr kiwid: 5d:06:12:07.172 0123 1 5085.00 kHz am z9 "76.174.24.132" Northridge, California, USA (ARRIVED)
    kiwi73: Dec 17 01:44:15 kiwisdr kiwid: 5d:03:15:08.961 0123 2 8918.00 kHz usb z6 "172.78.224.79" Roanoke, Indiana, USA (ARRIVED)
    ..................

    So you can mark this feature request as 'SOLVED'
  • edited December 2018
    Rob,
    I added a couple of statements to allow for passwords instead of keys and dropped the LEAVING lines. Do you think I have it correct or am I missing something. This syntax is taxing.

    (trusty)ron@localhost:~$ for kw in 107 138 119; do sshpass -p password ssh root@192.168.6.$kw "awk -v kiwi=$kw '/ARRIVE|LEAVING/{last[\$8]=\$0} END {for (i=0; i< 4; ++i) print \"kiwi\" kiwi \": \" last[i]}' /var/log/messages"; done 2>&1 | grep "^kiwi" | grep -v "LEAVING"
    kiwi107: Dec 17 06:29:22 kiwisdr kiwid: 6d:00:59:48.526 0... 0 5085.00 kHz am z7 "192.168.6.1" Weiser, Idaho, USA (ARRIVED)
    kiwi107: Dec 17 06:33:54 kiwisdr kiwid: 6d:01:04:19.696 01.. 1 10105.92 kHz cw z13 "194.79.5.192" Petropavlovsk-kamchatsky, Russian Federation (ARRIVED)
    kiwi107:
    kiwi138: Dec 17 05:52:49 kiwisdr kiwid: 6d:02:03:31.655 0... 0 7188.00 kHz lsb z7 "192.168.6.1" Weiser, Idaho, USA (ARRIVED)
    kiwi138:
    kiwi119: Dec 17 06:29:10 kiwisdr kiwid: 6d:05:59:15.297 0... 0 5085.00 kHz am z7 "192.168.6.1" Weiser, Idaho, USA (ARRIVED)
    kiwi119:
    kiwi119:
    (trusty)ron@localhost:~$

    Ron
    KA7U
  • You have introduced me to a new linux utility. None of my Kiwi's, including those with user and admin passwords, require an ssh password so I haven't needed it. But your syntax appears correct.

    One further enhancement to the script is to change the awk 'for(=0; i< 4; ++i) ...' statement to 'for (i in last) ...' so that all channels are printed of both 4 channel mode and 8 channel mode Kiwis.

    However there is one limitation to this script: If a channel always or never has a listener AND the /var/log/messages file is rotated by linux, then the state of that channel is not printed. An example of that can been seen on kiw73 where my wsprdaemon.sh has been listening to Kiwi73 channel #0 for 22 hours, but the /var/log/messages file is only 8 hours old.

    But in preparing this response I have just noticed that in each printed line the field previous to the rx channel number appears to list the state of all channels at the log line time. So this line reports that on Dec 17 14:24 a listener terminated a session on kiwi73 channel 1 which freed that channel and left channels 0 and 3, while channels 1 and 2 were then free ('0..3'):
    kiwi73: Dec 17 14:24:00 kiwisdr kiwid: 5d:15:54:54.182 0..3 1 14050.00 kHz cw z2 "64.37.29.30" Hugo, Colorado, USA (LEAVING after 0:01:11)
    So a further enhancement of this script would be to look in the archived /var/log/messages* files for the missing channel lines.

    But I'm completely satisfied skipping that enhancement and getting this report:

    pi@Pi-KPH84:~ $ for kw in 62 71 72 73 74 75 76 77 78; do ssh root@10.14.70.$kw "awk -v kiwi=$kw '/ARRIVE|LEAVING/{last[\$8]=\$0} END {for (i in last) print \"kiwi\" kiwi \": \" last[i]}' /var/log/messages"; done 2>&1 | grep "^kiwi"
    kiwi71: Dec 16 17:19:50 kiwisdr kiwid: 15:35:10.807 ..2..... 2 0.00 kHz am z0 "wsprdaemon_v2.0d1" 10.14.70.84 (ARRIVED)
    kiwi72: Dec 17 13:49:23 kiwisdr kiwid: 5d:15:15:18.020 .1234... 0 774.07 kHz amn z7 "Chuck" 24.72.186.54 Kerrville, Texas, USA (LEAVING after 0:44:53)
    kiwi72: Dec 17 04:07:07 kiwisdr kiwid: 5d:05:33:01.522 01234... 1 475.00 kHz am z12 "24.144.206.160" (ARRIVED)
    kiwi72: Dec 16 17:19:57 kiwisdr kiwid: 4d:18:45:51.393 ..2..... 2 0.00 kHz am z0 "wsprdaemon_v2.0d1" 10.14.70.84 (ARRIVED)
    kiwi72: Dec 16 17:19:58 kiwisdr kiwid: 4d:18:45:52.505 ..23.... 3 0.00 kHz am z0 "wsprdaemon_v2.0d1" 10.14.70.84 (ARRIVED)
    kiwi72: Dec 16 17:19:59 kiwisdr kiwid: 4d:18:45:53.645 ..234... 4 0.00 kHz am z0 "wsprdaemon_v2.0d1" 10.14.70.84 (ARRIVED)
    kiwi72: Dec 17 13:18:55 kiwisdr kiwid: 5d:14:44:50.396 01234... 5 474.20 kHz usb z0 "83.247.88.41" Waalre, Netherlands (LEAVING after 0:00:21)
    kiwi72: Dec 17 06:12:51 kiwisdr kiwid: 5d:07:38:46.027 .12345.. 6 1992.00 kHz lsb z0 "99.7.173.236" San Mateo, California, USA (LEAVING after 0:00:49)
    kiwi73: Dec 17 14:24:00 kiwisdr kiwid: 5d:15:54:54.182 0..3 1 14050.00 kHz cw z2 "64.37.29.30" Hugo, Colorado, USA (LEAVING after 0:01:11)
    kiwi73: Dec 17 14:23:02 kiwisdr kiwid: 5d:15:53:55.965 01.3 2 3995.00 kHz usb z8 "68.72.216.124" El Dorado Hills, California, USA (LEAVING after 0:06:39)
    kiwi73: Dec 17 14:18:35 kiwisdr kiwid: 5d:15:49:29.388 0123 3 3545.00 kHz cw z5 "K6CN" 47.158.215.74 Palm Desert, California, USA (ARRIVED)
    kiwi74: Dec 17 11:41:13 kiwisdr kiwid: 5d:13:03:09.171 0... 1 7410.00 kHz usb z9 "65.33.141.35" Bradenton, Florida, USA (LEAVING after 0:01:56)
    kiwi74: Dec 17 11:05:53 kiwisdr kiwid: 5d:12:27:49.118 012. 2 9025.00 kHz usb z9 "C A P MON" 84.221.170.90 Iglesias, Italy (LEAVING after 2:02:47)
    kiwi74: Dec 17 10:28:43 kiwisdr kiwid: 5d:11:50:38.726 0.2. 3 7431.60 kHz usb z9 "174.45.58.213" Durango, Colorado, USA (LEAVING after 2:39:26)
    kiwi75: Dec 17 04:58:22 kiwisdr kiwid: 5d:06:22:08.200 ..2..... 0 3885.26 kHz am z9 "73.162.68.18" San Jose, California, USA (LEAVING after 5:04:30)
    kiwi75: Dec 17 04:33:50 kiwisdr kiwid: 5d:05:57:36.474 0.2..... 1 4425.80 kHz usb z5 "108.74.128.206" Memphis, Tennessee, USA (LEAVING after 0:02:29)
    kiwi75: Dec 16 17:19:53 kiwisdr kiwid: 4d:18:43:39.406 012..... 2 0.00 kHz am z0 "wsprdaemon_v2.0d1" 10.14.70.84 (ARRIVED)
    kiwi75: Dec 17 03:55:24 kiwisdr kiwid: 5d:05:19:09.827 0.2...6. 3 13967.69 kHz am z0 "73.231.152.108" San Francisco, California, USA (LEAVING after 0:05:15)
    kiwi75: Dec 17 03:06:45 kiwisdr kiwid: 5d:04:30:30.783 0123..6. 4 14300.00 kHz usb z0 "216.183.78.70" Sonoita, Arizona, USA (LEAVING after 0:01:34)
    kiwi75: Dec 17 02:59:18 kiwisdr kiwid: 5d:04:23:04.052 0123..6. 5 14300.00 kHz usb z0 "216.183.78.70" Sonoita, Arizona, USA (LEAVING after 0:52:57)
    kiwi75: Dec 17 04:26:37 kiwisdr kiwid: 5d:05:50:23.524 0.2..... 6 6059.03 kHz am z0 "71.237.243.27" Vancouver, Washington, USA (LEAVING after 2:19:59)
    kiwi75: Dec 17 02:29:45 kiwisdr kiwid: 5d:03:53:31.268 0123.56. 7 5085.00 kHz am z0 "36.11.225.121" Tokyo, Japan (LEAVING after 0:07:39)
    kiwi77: Dec 17 09:11:25 kiwisdr kiwid: 1d:07:25:25.781 ..2..... 0 10123.00 kHz usb z8 "99.7.173.236" San Mateo, California, USA (LEAVING after 0:00:55)
    pi@Pi-KPH84:~ $
  • I tried to alias in .bashrc, but it doesn't like ' and if I use ` it seems to load with source ~/.bashrc but the script doesn't run.
    alias recent_users=`for kw in 107 138 119; do sshpass -p password ssh root@192.168.6.$kw "awk -v kiwi=$kw '/ARRIVE|LEAVING/{last[\$8]=\$0} END {for (i in last) print \"kiwi\" kiwi \": \" last[i]}' /var/log/messages"; done 2>&1 | grep "^kiwi"`

    or at least it doesn't print... I'll work on it.
    Ron
    KA7U
  • Be careful making changes to /root/.bashrc New Kiwi releases will overwrite that file (and .profile) if there have been changes made in the release (e.g. I have added some new command alias). Put your changes in /root/.bashrc.local The Kiwi will create this file with some defaults if it's never existed before but then leave it alone forever after that.
    njc
  • It is very difficult to declare my line as an alias. Instead, declare it in a function definition in the .bashrc.local. Functions are executed just like aliases, but don’t suffer from all the problems alias declarations suffer from “ and ‘ characters
  • I settled on a bash script file to run it. The file resides on my local computer so no worries about the KiwiSDR updates affecting it.
    Ron
    KA7U
  • @KA7U could you please post your bash script? Also how would I run this on the kiwi .bashrc.local? I'm not all that familiar with bash.

    Thanks,
    Nick W1NJC
  • You will need to ssh or putty to your Kiwi, or you can add and run this on you Linux PC or Mac
    Add these five lines to the plain text file "/root/.bashrc.local on your Kiwi". Change the "62 71 ... 78" and "10.14.70." to match the IP address(s) of your Kiwis
    The very friendly 'vi' text editor built into the Kiwi can be used to do this. Google for help on using vi.

    function kwu() {
    for kw in 62 71 72 73 74 75 76 77 78; do
    ssh root@10.14.70.$kw "awk '/ARRIVE|LEAVING/{last[\$8]=\$0} END {for (i in last) printf \"kiwi${kw}: %s\n\", last[i] }' /var/log/messages";
    done 2>&1 | grep "^kiwi"
    }

    Then execute "source /root/.bashrc.local"
    Then execute "kwu" from that Kiwi's cmd line to get the list of users.
  • Thank you. Haha yes I know vi, learned it a long time ago but I prefer nano for most things. I appreciate the help.
  • edited December 2018
    @njc
    ron@linux-4cdz:~> cat recent_kiwi
    #! /bin/bash
    #
    #
    for kw in 107 138 119; do sshpass -p password ssh root@192.168.6.$kw "awk -v kiwi=$kw '/ARRIVE|LEAVING/{last[\$8]=\$0} END {for (i in last) print \"kiwi\" kiwi \": \" last[i]}' /var/log/messages"; done 2>&1 | grep "^kiwi"

    ron@linux-4cdz:~>

    Place this file in your user root directory such as:

    ron@linux-4cdz:~/Downloads> cd
    ron@linux-4cdz:~>

    make it executable and then you can call it like this:
    ron@linux-4cdz:~/Downloads> sh ~/recent_kiwi

    from wherever in the directory structures you user might be with his user permissions. Just for knowing it is easy enough to drop all the clients that have left by using it like this:
    ron@linux-4cdz:~> sh ~/recent_kiwi |grep -v LEAVING
    kiwi107: Dec 18 00:37:10 kiwisdr kiwid: 02:39:10.385 01.3 0 7361.83 kHz amn z7 "159.118.203.118" Star, Idaho, USA (ARRIVED)
    kiwi107: Dec 18 00:37:20 kiwisdr kiwid: 02:39:20.212 01.3 1 11282.00 kHz usb z6 "2.126.155.52" Middlesbrough, United Kingdom (ARRIVED)
    kiwi107: Dec 18 00:27:39 kiwisdr kiwid: 02:29:39.219 0123 3 16133.10 kHz usb z11 "192.168.6.1" Weiser, Idaho, USA (ARRIVED)
    ron@linux-4cdz:~>

    This script is designed to run from a linux terminal somewhere on the network. You can run it from a KiwiSDR but you will need to ssh into each kiwiSDR from the KiwiSDR you are planning to run it from first to establish the ~/.ssh/known_hosts file. You will also need to install sshpass if you are dealing in ssh passwords. Once you have initialized one of the KiwiSDRs or all of them you might have, if you wish, then you can run this bash script file from the kiwiSDR you have already logged into.
    Example from a KiwiSDR:
    root@kiwisdr:~# ./recent_kiwi
    kiwi107: Dec 18 00:44:26 kiwisdr kiwid: 02:46:26.914 .1.3 0 3930.00 kHz lsb z9 "159.118.203.118" Star, Idaho, USA (LEAVING after 0:07:27)
    kiwi107: Dec 18 00:37:20 kiwisdr kiwid: 02:39:20.212 01.3 1 11282.00 kHz usb z6 "2.126.155.52" Middlesbrough, United Kingdom (ARRIVED)
    kiwi107: Dec 18 00:31:55 kiwisdr kiwid: 02:33:55.652 ..23 2 7139.00 kHz iq z7 "K7GLD" 67.215.45.53 John Day, Oregon, USA (LEAVING after 0:25:30)
    kiwi107: Dec 18 00:27:39 kiwisdr kiwid: 02:29:39.219 0123 3 16133.10 kHz usb z11 "192.168.6.1" Weiser, Idaho, USA (ARRIVED)
    kiwi138: Dec 17 23:15:34 kiwisdr kiwid: 6d:19:26:17.085 .1.. 0 7870.00 kHz usb z0 "98.218.114.249" Greenbelt, Maryland, USA (LEAVING after 0:00:52)
    kiwi138: Dec 18 00:15:17 kiwisdr kiwid: 6d:20:25:59.241 .... 1 14006.91 kHz cw z13 "194.79.5.192" Petropavlovsk-Kamchatsky, Russia (LEAVING after 2:00:03)
    kiwi138: Dec 17 02:44:15 kiwisdr kiwid: 5d:22:54:58.173 012. 2 3920.00 kHz lsb z7 "W7LPN" 184.170.180.142 Weiser, Idaho, USA (LEAVING after 0:00:59)
    root@kiwisdr:~#

    If the script does not query all the IP addresses in your list, an extra character may have slipped in with the cut and paste operation from this blog. If that happens, type it in the editor and double check the syntax.
    Ron
    KA7U
  • @KA7U
    Ron,
    This script is not working for me. When I run it I get only four older log entries, all LEAVING. If I look at a tail of /var/log/messages I can see that there are more recent entries in there. My script file:

    #! /bin/bash
    #
    #
    for kw in 103; do ssh root@192.168.68.$kw "awk -v kiwi=$kw '/ARRIVED|LEAVING/{last[\$8]=\$0} END {for (i in last) print \"kiwi\" kiwi \": \" last[i]}' /var/log/messages"; done 2>&1 | grep "^kiwi"


    Here's the output I get:

    Debian GNU/Linux 8

    BeagleBoard.org Debian Image 2016-05-13

    Support/FAQ: http://elinux.org/Beagleboard:BeagleBoneBlack_Debian

    default username:password is [debian:temppwd]

    Dec 17 12:44:17 kiwisdr kiwid: 3d:10:08:38.206 ........ 0 1906.00 kHz lsb z9 "192.168.68.4" Millbury, Massachusetts, USA (LEAVING after 0:02:01)
    Dec 17 12:16:19 kiwisdr kiwid: 3d:09:40:39.883 .1...... 1 3838.00 kHz lsb z11 "W2FBS" 8.41.69.20 Webster, New York, USA (LEAVING after 1:23:52)
    Dec 16 06:27:31 kiwisdr kiwid: 2d:03:51:51.839 ....4... 2 3995.00 kHz lsb z0 "47.201.106.161" Zephyrhills, Florida, USA (LEAVING after 1:15:38)
    Dec 16 07:13:45 kiwisdr kiwid: 2d:04:38:06.130 ....4... 4 3995.00 kHz lsb z0 "K1YW" 96.252.62.227 Hudson, Massachusetts, USA (LEAVING after 2:00:23)

    And here's a tail of the log:


    root@kiwisdr:~# tail /var/log/messages
    Dec 20 11:18:28 kiwisdr kiwid: 18:06:21.869 012..... 0 3838.00 kHz lsb z10 "208.103.69.223" Waterford, Connecticut, USA (LEAVING after 0:24:57)
    Dec 20 11:26:50 kiwisdr kiwid: 18:14:44.765 .1...... 2 257.80 kHz cw z0 "89.103.207.243" Usti Nad Labem, Czech Republic (LEAVING after 0:18:29)
    Dec 20 12:36:21 kiwisdr kiwid: 19:24:15.026 .1...... 1 3834.00 kHz lsb z11 "W2FBS" 8.41.69.20 Webster, New York, USA (LEAVING after 1:48:56)
    Dec 20 12:44:10 kiwisdr kiwid: 19:32:03.970 0....... 0 14097.00 kHz lsb z0 "94.9.210.245" Wolverhampton, United Kingdom (ARRIVED)
    Dec 20 12:44:46 kiwisdr kiwid: 19:32:40.489 ........ 0 14097.00 kHz lsb z13 "94.9.210.245" Wolverhampton, United Kingdom (LEAVING after 0:00:47)
    Dec 20 13:02:31 kiwisdr kiwid: 19:50:25.646 ........ ca_pause 16378 ca_pause_old 10 ca_shift=4 code_creep=-14 code_period_ms=1 code_period_samples=16368
    Dec 20 14:10:17 kiwisdr kiwid: 20:58:11.081 0....... 0 3838.30 kHz lsb z12 "w1haf" 208.103.69.223 Waterford, Connecticut, USA (ARRIVED)
    Dec 20 14:18:54 kiwisdr kiwid: 21:06:48.326 ........ 0 3941.00 kHz lsb z4 "w1haf" 208.103.69.223 Waterford, Connecticut, USA (LEAVING after 0:08:48)
    Dec 20 14:51:41 kiwisdr kiwid: 21:39:35.071 0....... 0 18115.00 kHz cw z9 "98.144.144.160" Racine, Wisconsin, USA (ARRIVED)
    Dec 20 14:51:52 kiwisdr kiwid: 21:39:45.992 ........ 0 18115.00 kHz cw z10 "98.144.144.160" Racine, Wisconsin, USA (LEAVING after 0:00:21)


    How does the {last[\$8]=\$0} work? Does it create an array which you're iterating in the for loop?
  • edited December 2018
    I copied your line to t.sh on my Kiwi and modified the addresses and all works fine for me.
    Did you supply your own BB or modify the OS? Your output suggests the awk program is not running and the output of the awk program is not being piped to the 'grep "^kiwi"'.
    The awk program fills a awk array "last[*]" (indexed by the rx channel # field $8) with lines which contain ARRIVED or LEAVING. When awk reaches the END of the file it iterates through that array and prints out the last saved lines.
    ===============

    pi@Pi-KPH84:~/wsprdaemon $ cat t.sh
    for kw in 73 74; do ssh root@10.14.70.$kw "awk -v kiwi=$kw '/ARRIVED|LEAVING/{last[\$8]=\$0} END {for (i in last) print \"kiwi\" kiwi \": \" last[i]}' /var/log/messages"; done 2>&1 | grep "^kiwi"
    pi@Pi-KPH84:~/wsprdaemon $ bash t.sh
    kiwi73: Dec 20 15:09:34 kiwisdr kiwid: 8d:16:40:28.767 0.2. 0 0.00 kHz am z0 "wsprdaemon_v2.0e" 10.14.70.84 (ARRIVED)
    kiwi73: Dec 20 15:34:08 kiwisdr kiwid: 8d:17:05:02.619 0123 1 10238.41 kHz cw z10 "216.160.190.188" Evergreen, Colorado, USA (ARRIVED)
    kiwi73: Dec 19 11:28:58 kiwisdr kiwid: 7d:12:59:52.472 012. 2 7527.00 kHz usb z5 "r" 68.134.187.44 Glen Burnie, Maryland, USA (ARRIVED)
    kiwi73: Dec 20 15:22:55 kiwisdr kiwid: 8d:16:53:48.887 0123 3 4981.71 kHz cw z10 "K6TOP" 47.143.93.176 Gilroy, California, USA (ARRIVED)
    kiwi74: Dec 20 15:09:36 kiwisdr kiwid: 8d:16:31:31.414 01.. 0 0.00 kHz am z0 "wsprdaemon_v2.0e" 10.14.70.84 (ARRIVED)
    kiwi74: Dec 20 15:04:22 kiwisdr kiwid: 8d:16:26:17.640 01.. 1 3796.00 kHz lsb z7 "78.20.125.115" Geraardsbergen, Belgium (ARRIVED)
    kiwi74: Dec 20 15:22:09 kiwisdr kiwid: 8d:16:44:04.432 012. 2 5250.00 kHz usb z10 "G.MARCONI" 84.221.170.90 Selargius, Italy (ARRIVED)
    kiwi74: Dec 20 15:30:11 kiwisdr kiwid: 8d:16:52:06.465 012. 3 8743.00 kHz usb z5 "Ex FNWH" 92.155.55.219 Avesnelles, France (LEAVING after 0:04:29)
    pi@Pi-KPH84:~/wsprdaemon $

    =======
    I mosty run the script from my Raspberry Pi where I define it as a function in my .bashrc. This version simplifies the awk program so fewer \"s are required:

    function kwu() {
    for kw in 62 71 72 73 74 75 76 77 78; do
    ssh root@10.14.70.$kw "awk '/ARRIVE|LEAVING/{last[\$8]=\$0} END {for (i in last) printf \"kiwi${kw}: %s\n\", last[i] }' /var/log/messages";
    done 2>&1 | grep "^kiwi"
    }
  • I just tried running it from another machine (raspi) and I get mostly the wrong output, less the GNU Debian BeagleBone stuff at the top. There's some newer entries but still an old one, all LEAVING, and only 4 total.
    I also tried the latest version of gawk with no change. I didn't change anything on the OS.
    As I study the output, is it possibly returning one entry (the latest) for each channel (or whatever the 0,1,2,4 figures are)? That's what it looks like.

    Also, do you know what the "action 17" suspended is about?

    root@kiwisdr:~# tail /var/log/messages -n 100

    ...

    Dec 20 10:53:41 kiwisdr kiwid: 17:41:35.507 01...... 0 3841.55 kHz lsb z10 "208.103.69.223" Waterford, Connecticut, USA (ARRIVED)
    Dec 20 11:08:32 kiwisdr kiwid: 17:56:26.118 012..... 2 366.80 kHz cw z0 "89.103.207.243" Usti Nad Labem, Czech Republic (ARRIVED)
    Dec 20 11:18:28 kiwisdr kiwid: 18:06:21.869 012..... 0 3838.00 kHz lsb z10 "208.103.69.223" Waterford, Connecticut, USA (LEAVING after 0:24:57)
    Dec 20 11:26:50 kiwisdr kiwid: 18:14:44.765 .1...... 2 257.80 kHz cw z0 "89.103.207.243" Usti Nad Labem, Czech Republic (LEAVING after 0:18:29)
    Dec 20 12:36:21 kiwisdr kiwid: 19:24:15.026 .1...... 1 3834.00 kHz lsb z11 "W2FBS" 8.41.69.20 Webster, New York, USA (LEAVING after 1:48:56)
    Dec 20 12:44:10 kiwisdr kiwid: 19:32:03.970 0....... 0 14097.00 kHz lsb z0 "94.9.210.245" Wolverhampton, United Kingdom (ARRIVED)
    Dec 20 12:44:46 kiwisdr kiwid: 19:32:40.489 ........ 0 14097.00 kHz lsb z13 "94.9.210.245" Wolverhampton, United Kingdom (LEAVING after 0:00:47)
    Dec 20 13:02:31 kiwisdr kiwid: 19:50:25.646 ........ ca_pause 16378 ca_pause_old 10 ca_shift=4 code_creep=-14 code_period_ms=1 code_period_samples=16368
    Dec 20 14:10:17 kiwisdr kiwid: 20:58:11.081 0....... 0 3838.30 kHz lsb z12 "w1haf" 208.103.69.223 Waterford, Connecticut, USA (ARRIVED)
    Dec 20 14:18:54 kiwisdr kiwid: 21:06:48.326 ........ 0 3941.00 kHz lsb z4 "w1haf" 208.103.69.223 Waterford, Connecticut, USA (LEAVING after 0:08:48)
    Dec 20 14:51:41 kiwisdr kiwid: 21:39:35.071 0....... 0 18115.00 kHz cw z9 "98.144.144.160" Racine, Wisconsin, USA (ARRIVED)
    Dec 20 14:51:52 kiwisdr kiwid: 21:39:45.992 ........ 0 18115.00 kHz cw z10 "98.144.144.160" Racine, Wisconsin, USA (LEAVING after 0:00:21)
    Dec 20 15:35:06 kiwisdr rsyslogd-2007: action 'action 17' suspended, next retry is Thu Dec 20 15:35:36 2018 [try http://www.rsyslog.com/e/2007 ]
    Dec 20 15:40:01 kiwisdr rsyslogd-2007: action 'action 17' suspended, next retry is Thu Dec 20 15:40:31 2018 [try http://www.rsyslog.com/e/2007 ]
    Dec 20 15:41:22 kiwisdr kiwid: 22:29:15.961 0....... 0 18140.00 kHz usb z6 "164.163.159.33" Juazeiro Do Norte, Brazil (ARRIVED)
    Dec 20 15:41:22 kiwisdr rsyslogd-2007: action 'action 17' suspended, next retry is Thu Dec 20 15:41:52 2018 [try http://www.rsyslog.com/e/2007 ]
    Dec 20 15:42:09 kiwisdr kiwid: 22:30:03.232 0....... 0 18104.60 kHz usb z6 "164.163.159.33" Juazeiro Do Norte, Brazil (LEAVING after 0:01:00)
    Dec 20 15:42:09 kiwisdr rsyslogd-2007: action 'action 17' suspended, next retry is Thu Dec 20 15:42:39 2018 [try http://www.rsyslog.com/e/2007 ]
    Dec 20 15:45:01 kiwisdr rsyslogd-2007: action 'action 17' suspended, next retry is Thu Dec 20 15:45:31 2018 [try http://www.rsyslog.com/e/2007 ]
    Dec 20 15:50:01 kiwisdr rsyslogd-2007: action 'action 17' suspended, next retry is Thu Dec 20 15:50:31 2018 [try http://www.rsyslog.com/e/2007 ]
    Dec 20 15:50:57 kiwisdr rsyslogd-2007: action 'action 17' suspended, next retry is Thu Dec 20 15:51:27 2018 [try http://www.rsyslog.com/e/2007 ]
    Dec 20 15:55:01 kiwisdr rsyslogd-2007: action 'action 17' suspended, next retry is Thu Dec 20 15:55:31 2018 [try http://www.rsyslog.com/e/2007 ]
    Dec 20 16:00:01 kiwisdr rsyslogd-2007: action 'action 17' suspended, next retry is Thu Dec 20 16:00:31 2018 [try http://www.rsyslog.com/e/2007 ]
    Dec 20 16:05:01 kiwisdr rsyslogd-2007: action 'action 17' suspended, next retry is Thu Dec 20 16:05:31 2018 [try http://www.rsyslog.com/e/2007 ]
    Dec 20 16:05:36 kiwisdr rsyslogd-2007: action 'action 17' suspended, next retry is Thu Dec 20 16:06:36 2018 [try http://www.rsyslog.com/e/2007 ]
    Dec 20 16:09:38 kiwisdr rsyslogd-2007: action 'action 17' suspended, next retry is Thu Dec 20 16:10:38 2018 [try http://www.rsyslog.com/e/2007 ]
    Dec 20 16:11:12 kiwisdr rsyslogd-2007: action 'action 17' suspended, next retry is Thu Dec 20 16:12:12 2018 [try http://www.rsyslog.com/e/2007 ]
    Dec 20 16:12:43 kiwisdr rsyslogd-2007: action 'action 17' suspended, next retry is Thu Dec 20 16:13:43 2018 [try http://www.rsyslog.com/e/2007 ]
    Dec 20 16:14:49 kiwisdr rsyslogd-2007: action 'action 17' suspended, next retry is Thu Dec 20 16:15:49 2018 [try http://www.rsyslog.com/e/2007 ]
    Dec 20 16:16:16 kiwisdr rsyslogd-2007: action 'action 17' suspended, next retry is Thu Dec 20 16:17:16 2018 [try http://www.rsyslog.com/e/2007 ]
    Dec 20 16:20:01 kiwisdr rsyslogd-2007: action 'action 17' suspended, next retry is Thu Dec 20 16:21:01 2018 [try http://www.rsyslog.com/e/2007 ]


    pi-star@pi-star(rw):kiwi$ sh ./recent_kiwi

    kiwi103: Dec 20 15:42:09 kiwisdr kiwid: 22:30:03.232 0....... 0 18104.60 kHz usb z6 "164.163.159.33" Juazeiro Do Norte, Brazil (LEAVING after 0:01:00)
    kiwi103: Dec 20 12:36:21 kiwisdr kiwid: 19:24:15.026 .1...... 1 3834.00 kHz lsb z11 "W2FBS" 8.41.69.20 Webster, New York, USA (LEAVING after 1:48:56)
    kiwi103: Dec 20 11:26:50 kiwisdr kiwid: 18:14:44.765 .1...... 2 257.80 kHz cw z0 "89.103.207.243" Usti Nad Labem, Czech Republic (LEAVING after 0:18:29)
    kiwi103: Dec 16 07:13:45 kiwisdr kiwid: 2d:04:38:06.130 ....4... 4 3995.00 kHz lsb z0 "K1YW" 96.252.62.227 Hudson, Massachusetts, USA (LEAVING after 2:00:23)
  • The grep at the end of my line filters out all lines except those which start with "kiwi". If other lines appear on your output, then the pipe '|' isn;t working. the ssh command produces a lot of stderr output which is supposed to be filtered by that grep.
  • The grep is working. That stuff with the syslogd is from the tail of /var/log/messages, which I included to show what's there and some of what I think the script should be returning. The last block is my output from the script.
  • the script only outputs the most recent ARRIVED or LEAVING line for each rx channel. It appears that is what you are getting, I have no idea of the meaning of the syslogd lines.
  • Then it's working as intended. Thanks.
  • @njc
    Nick,
    I'm sorry about being late getting back to this. Busy social life here. Hi Hi

    I think this will work for you:
    #! /bin/bash
    #
    for kw in 107 138 119;
    do sshpass -p password ssh -q root@192.168.6.$kw cat $kw /var/log/messages |grep -E 'ARRIVED|LEAVING' | tail -8
    done

    This should give the last 8 arrived and leaving entries in the /var/log/messages file and a header for each listing giving the last 3 digits of the IP address. I haven't figured out how to get nice formatting for the IP header, so for now I'm settling for an error message following the 3 digits of the IP. An example output follows:

    root@kiwisdr:~# ./recent_kiwi
    cat: 107: No such file or directory
    Dec 21 04:59:01 kiwisdr kiwid: 1d:02:25:34.983 01.. 1 7203.90 kHz lsb z8 "k8anm" 75.167.83.66 Phoenix, Arizona, USA (ARRIVED)
    Dec 21 05:10:00 kiwisdr kiwid: 1d:02:36:33.500 01.. 1 7166.80 kHz lsb z8 "k8anm" 75.167.83.66 Phoenix, Arizona, USA (LEAVING after 0:11:09)
    Dec 21 05:10:30 kiwisdr kiwid: 1d:02:37:03.770 01.. 1 7196.96 kHz lsb z8 "k8anm" 75.167.83.66 Phoenix, Arizona, USA (ARRIVED)
    Dec 21 05:26:55 kiwisdr kiwid: 1d:02:53:29.291 0... 1 7029.97 kHz cw z10 "k8anm" 75.167.83.66 Phoenix, Arizona, USA (LEAVING after 0:16:36)
    Dec 21 05:46:53 kiwisdr kiwid: 1d:03:13:26.766 01.. 1 5371.50 kHz usb z9 "174.87.208.120" Reno, Nevada, USA (ARRIVED)
    Dec 21 06:13:51 kiwisdr kiwid: 1d:03:40:24.938 012. 2 670.00 kHz am z0 "73.181.135.242" Seattle, Washington, USA (ARRIVED)
    Dec 21 06:16:48 kiwisdr kiwid: 1d:03:43:21.498 0.2. 1 5371.50 kHz usb z9 "174.87.208.120" Reno, Nevada, USA (LEAVING after 0:30:06)
    Dec 21 06:29:58 kiwisdr kiwid: 1d:03:56:31.666 0... 2 3786.91 kHz lsb z7 cw_decoder "73.181.135.242" Seattle, Washington, USA (LEAVING after 0:16:17)
    cat: 138: No such file or directory
    Dec 21 04:02:32 kiwisdr kiwid: 1d:01:32:43.652 .... 0 16.00 kHz amn z0 "103.231.91.66" Auckland, New Zealand (LEAVING after 0:26:22)
    Dec 21 04:04:14 kiwisdr kiwid: 1d:01:34:24.910 0... 0 3907.97 kHz lsb z11 "73.198.60.179" Egg Harbor, New Jersey, USA (ARRIVED)
    Dec 21 04:04:38 kiwisdr kiwid: 1d:01:34:48.913 0... 0 3922.05 kHz lsb z8 "73.198.60.179" Egg Harbor, New Jersey, USA (LEAVING after 0:00:35)
    Dec 21 04:05:16 kiwisdr kiwid: 1d:01:35:27.162 0... 0 3902.37 kHz lsb z8 "73.198.60.179" Egg Harbor, New Jersey, USA (ARRIVED)
    Dec 21 04:11:09 kiwisdr kiwid: 1d:01:41:19.915 01.. 1 3940.00 kHz lsb z8 "192.168.6.1" Weiser, Idaho, USA (ARRIVED)
    Dec 21 04:16:28 kiwisdr kiwid: 1d:01:46:39.747 01.. 0 5830.00 kHz amn z6 navtex "73.198.60.179" Egg Harbor, New Jersey, USA (LEAVING after 0:11:23)
    Dec 21 05:52:26 kiwisdr kiwid: 1d:03:22:36.968 01.. 0 7200.00 kHz lsb z0 "50.251.11.145" Las Cruces, New Mexico, USA (ARRIVED)
    Dec 21 06:03:58 kiwisdr kiwid: 1d:03:34:09.109 .1.. 0 7160.00 kHz lsb z0 "KC5AM" 50.251.11.145 Las Cruces, New Mexico, USA (LEAVING after 0:11:43)
    cat: 119: No such file or directory
    Dec 20 15:56:17 kiwisdr kiwid: 13:29:20.646 .... 0 15720.00 kHz amn z0 "192.168.6.1" Weiser, Idaho, USA (LEAVING after 0:07:10)
    Dec 21 00:16:56 kiwisdr kiwid: 21:49:59.883 0... 0 1906.57 kHz lsb z7 "ve1ej" 72.38.6.154 Welland, Canada (ARRIVED)
    Dec 21 00:17:21 kiwisdr kiwid: 21:50:25.353 .... 0 3750.00 kHz lsb z5 "ve1ej" 72.38.6.154 Welland, Canada (LEAVING after 0:00:36)
    Dec 21 03:44:36 kiwisdr kiwid: 1d:01:17:40.515 0... 0 3750.00 kHz lsb z5 "ve1ej" 72.38.6.154 Welland, Canada (ARRIVED)
    Dec 21 04:11:28 kiwisdr kiwid: 1d:01:44:31.923 01.. 1 3940.00 kHz lsb z8 "192.168.6.1" Weiser, Idaho, USA (ARRIVED)
    Dec 21 04:14:15 kiwisdr kiwid: 1d:01:47:18.901 .1.. 0 1914.00 kHz lsb z7 "ve1ej" 72.38.6.154 Welland, Canada (LEAVING after 0:29:49)
    Dec 21 04:18:52 kiwisdr kiwid: 1d:01:51:55.862 01.. 0 1914.00 kHz lsb z7 "ve1ej" 72.38.6.154 Welland, Canada (ARRIVED)
    Dec 21 05:10:59 kiwisdr kiwid: 1d:02:44:03.151 01.. 0 1914.00 kHz lsb z7 "ve1ej" 72.38.6.154 Welland, Canada (LEAVING after 0:52:19)
    root@kiwisdr:~#

    So for whatever it is worth... If you are a bash programmer, perhaps you would add the magic words to make this look better.
    Ron
    KA7U
  • do echo "=== $kw ===" ; sshpass -p password ssh -q root@192.168.6.$kw grep -E 'ARRIVED|LEAVING' /var/log/messages | tail -8
  • Thanks, guys. I think I have it all working the way I want to now.
  • @rrobinet
    That is much nicer formatting than using a cat error message!
    Ron
    KA7U
  • edited January 2020
    Thanks to John's new '/users' URL, I have been able to refine my bash script to list the Kiwis and their users and have attached it to this as a .txt file
    Here is an example of its output when run at KPH which looks cleaner if I could use a fixed width font:
    pi@KPH-Pi4b-85:~ $ ./kiwi-stats.sh
          KIWI_2:0              ''   559970  am 0:11:19       47.232.165.120 Burbank California USA
          KIWI_3:0  'POETTO BEACH' 12748100 usb 2:18:46         37.77.112.52 TorreAnnunziata Italy
          KIWI_3:1              ''  3808000 lsn 5:07:24       192.34.131.166 Burnsville NorthCarolina USA
          KIWI_4:0              ''  6715000 usb 6:09:18         37.77.112.52 Italy
          KIWI_4:1              ''  4997000 usb 6:09:07         37.77.112.52 TorreAnnunziata Italy
          KIWI_4:2              '' 11175000 usb 0:29:10         67.182.68.69
          KIWI_4:3              ''  7034010  cw 0:14:34        50.39.102.200 Portland Oregon USA
          KIWI_5:1              ''  7055000  cw 4:33:00        108.23.95.233 Bellflower California USA
          KIWI_5:2              ''  7055000  cw 3:09:51        108.23.95.233 Bellflower California USA
          KIWI_6:0              ''  7046900  cw 4:18:07        108.23.95.233 Bellflower California USA
          KIWI_7:0         'w4hbk' 24999090 usb 3:12:09        172.3.127.249 GulfBreeze Florida USA
    Found 7 Kiwis configured for 40 total rx channels, 11 total listeners, 14 wsprdaemon channels
    pi@KPH-Pi4b-85:~ $
    


    Attachments:
    https://forum.kiwisdr.com/uploads/Uploader/be/7b301d271d3250a5593a4e9165f0d5.txt
  • jksjks
    edited January 2020
    To get fixed-width font select text block and apply "code" option from paragraph-symbol pulldown (6th from left). I have done this for your post.
Sign In or Register to comment.