Monitoring Enterprise Wifi Using a Raspberry Pi 2
by Thomas Spiegelman, Operations Engineer for Amplify

Wireless sucks.  There I said it.  As a network engineer who has dealt primarily with switching and routing, wireless is just a pain.  I wanted a simpler, cheaper way to diagnose inconsistent wireless problems.  Oh hey, there is a Raspberry Pi 2 now?!
 
With the Raspberry Pi 2, I created a simple, inexpensive setup to collect data on a wireless network.  I tacked on a TP Link WDN3200 wireless adaptor (this adaptor has 5ghz), and a 32GB MicroSD Card. I left 8 of these Pis in different locations within the wireless network.  This setup cost under $650. You’d have a hard time finding a single enterprise WAP for that price anymore!
 
The plan

What do I want to test on the network?  Let’s set some goals for testing:

1. Test network for packet loss
2. Record RSSI and Noise level
3. Check Authentication.  WPA2 / WPA2 Enterprise handshake
4. Track Association issues
5. Record Wireless Drops / Reauths
6. Log all data to centralized server

My setup

Raspberry Pi 2 running Raspian, TP Link WDN3200, 32GB SD card, and a standalone rsyslog server.   I am already connected to my wireless network with all of the network info defined in /etc/wpa_supplicant/wpa_supplicant.conf.  I’ve also configured the hostnames on all of the Pis to their respective serial number so that I didn’t have 8 Raspberry Pis using the same hostname of “raspberrypi”:

sethostname=$(cat /proc/cpuinfo|grep erial | awk '{print $3}')

sudo sed -i 's/raspberrypi/$sethostname/g' /etc/hosts

sudo sed -i 's/raspberrypi/$sethostname/g' /etc/hostname

Now the tests!

The first test is the ping test for goal number 1.  I didn’t want to just have a constant stream of ICMP messages, so I’ve decided that I would do a 3 ping burst every 15 seconds to the gateway and log the data.  As long as I am logging these pings every 15 seconds I can kill 2 birds with one stone and add the RSSI and Noise data from iwconfig to the same log messages for goal number 2.  Here is my code (named pingthings):

#!/bin/bash
#script for pinging the dg every 15 seconds and logging that data and the signal quality data

#grabbing the wifi signal from iwconfig and formatting it to be logged
wisignal=$(iwconfig wlan0 | grep -i quality | sed -e 's/^[ \t]*//;s/[ \t]*$//;s/\/70/\/70,/')

#Grabbing the default gateway IP address to ping
pdefaultgw=$(ip route show | grep default | awk '{print $3}')

#sending to the log file that the acript is starting and logging the initial signal strength
echo "pingthings, Interface Started, $wisignal" 2>&1 | logger &
while true
do
  #checking wlan0 to see if it has a valid ip address assigned, if it does then start pinging
  ifconfig wlan0 | awk '/inet addr/{print substr($2,6)}' | grep -q -Eo '([0-9]*\.){3}[0-9]*'
  if [ $? -ne 0 ] 
  then
    echo "pingthings, wlan0 not fully running yet" 2>&1 | logger &
    sleep 10
  else
    srcaddy=$(ifconfig wlan0 | grep inet\ addr: | awk '{print $2}')
    pingrta=()
    pcount=0
    #do 3 pings and log the round trip time to a table
    for i in {1..3}
    do 
      pingrt=$(ping -c 1 $pdefaultgw | grep -o "time\=.*")
      if [ $? -ne 0 ] 
      then
        pcount=$(( $pcount + 1 ))
        pingrta+=('NULL')
      else 
        pingrt=$(echo $pingrt | sed 's/time=//' | sed 's/ms//')
        pingrta+=($pingrt)
      fi
    done


    #now we are going to check how many pings were successful then send data to the log according to the number of successful pings
    wisignal=$(iwconfig wlan0 | grep -i quality | sed -e 's/^[ \t]*//;s/$[ ]*//;s/\/70/\/70,/')
    if [ $pcount -eq 0 ] 
    then
      echo "$(date +%s), $srcaddy, $wisignal, pingthings pass, RTT: ${pingrta[*]}" 2>&1 | logger &
    elif [[ $pcount -gt 0 && $pcount -lt 3 ]]
    then
      echo "$(date +%s), $srcaddy, $wisignal, pingthings $pcount out of 3 FAIL, RTT: ${pingrta[*]}" 2>&1 | logger &
    elif [ $pcount -eq 3 ] 
    then
      echo "$(date +%s), $wisignal, pingthings 3 out of 3 FAIL" 2>&1 | logger &
      pdefaultgw=$(ip route show | grep default | awk '{print $3}')
    else 
      echo "$(date +%s), $srcaddy, pingthings error" 2>&1 | logger &
    fi
    sleep 20
  fi
done

I made this file executable and placed it in /usr/local/bin/pingthings.

Next, I wanted to take care of goals 3 and 4 by bouncing the interface every 15 minutes.  I figured if there are reauth problems or association errors (as long as the WAP doesn’t cache the client) then wpa_supplicant would see the problems and log the data.  I also wanted to stop the pings from the script above while the interface was restarting.  To accomplish this I’ve just put all of the following info in an init.d script (named wlanmonitor): 

#! /bin/sh
#

### BEGIN INIT INFO
# Provides:          pingthings
# Required-Start:    $remote_fs $syslog
# Required-Stop:     $remote_fs $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Start daemon at boot time
# Description:       Enable service provided by daemon.
### END INIT INFO
# Some things that run always
touch /var/lock/wifizzles

# Carry out specific functions when asked to by the system
case "$1" in
  start)
    echo "Starting script pingthings "
    pingthings &
    ;;
  stop)
    killall pingthings
    echo "pingthings, Interface Being Stopped" 2>&1 | logger &
    ifdown wlan0
    sleep 15
    echo "bringing it back up"
    ifup wlan0
    ;;
  reload) 
    /etc/init.d/wlanmonitor stop
    /etc/init.d/wlanmonitor start
    ;;
  *)
    echo "Usage: /etc/init.d/pingthings {start|stop}"
    exit 1
    ;;
esac

exit 0

I then set that to a cron job for every 15 min:

SHELL=/bin/bash

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

*/15 * * * * root bash -c "/etc/init.d/wlanmonitor reload"

Now goal 5 will come with logging all the info.  Almost everything wifi related on the Raspberry Pi by default will be managed from wpa_supplicant.  Logs from wpa_supplicant will not only show me the activity from the interface bouncing (wlanmonitor script), but I will also see if any other wifi activity occurs.  I also want to log this to a centralized rsyslog server that I am going to setup (below).  The logging is easily done by editing /etc/rsyslog.conf and adding the following lines directly under where you see ### RULES ###:

:msg, contains, "wlan0"           @@10.60.1.5:514
if $msg contains 'wlan0' then /var/log/wpa_sup.log
& ~

:programname, contains, "wpa_supplicant"           @@10.60.1.5:514
if $programname contains 'wpa_supplicant' then /var/log/wpa_sup.log
& ~

:programname, contains, "wpa_action"           @@10.60.1.5:514
if $programname contains 'wpa_action' then /var/log/wpa_sup.log
& ~

:msg, contains, "pingthings"           @@10.60.1.5:514
if $msg contains 'pingthings' then /var/log/latency.log
& ~

@@ means it uses TCP logging instead of UDP logging which is essential since we are constantly bouncing the interface.  10.60.1.5 is the IP I am reserving for my rsyslog server.

Finally, I need a centralized rsyslog server to receive all the data from the raspberry Pis.  I would recommend doing some Googling on how to setup your server and configure client settings for rsyslog.  You might need to do some tweaking that’s specific to your network to ensure that your clients cache the logs correctly when the interface is bouncing. You will also need to make sure that you set your server up to receive tcp.   Here is my template that I’ve used on the rsyslog server to receive the files:

$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$template PiLogging, "/var/log/%HOSTNAME%.log"
if $fromhost-ip startswith '10.' then ?PiLogging
& stop

This simply states, any logging coming from a 10 net ip address, send it to /var/log/(hostnameofclient).

This gives me Raspberry Pi 2s that I can simply plug into a usb port (they work great taped to a LCD TV, in the TV’s USB port) which auto connects to a wifi network and sends me logs to a centralized log server with very beneficial data.  It is amazing how easily you can diagnose problems with this information.  Packet loss, auth problems, coverage problems, it’s all logged!

Some future steps for me in this project:

1. The old Raspberry Pi models were not capable of running Chef, it was just too taxing on the Pi.  Now with the quad core Raspberry Pi 2s I am hoping that I can run Chef.  This means I should be able to add to a Chef server (or git, pull and run Chef Solo) the wifi information of a network that I want to test and have the Raspberry Pi pull SSID / password / 802.1x creds through a Chef run, having it place that information automatically in its wpa_supplicant.conf file.

2.   Setup OpenVPN to have the Raspi connect back over a VPN tunnel to the rsyslog server.  This will mean I can have Pis on multiple LANs and still have them “phone home” and log all information to the same rsyslog server.

I hope this is helpful and happy network monitoring!!!   Thanks to my colleagues Ryan Walsh and Robert Lahnemann for the help on this project.