Simple thoughts

Managing your users is good. But the work that comes with creating user accounts upfront is tedious and boring. At one point we got lost in trying to hook up each system manually to our WiFi. So we decided to outsource this to our users so that they have to register their devices by themselves.

Luckily pfSense proves to be extremely flexible, so with a custom portal page and some additional scripts we are able to get the important information we need. We ask the user to provide

• User
• Email address (to get in touch with the user) and
• Accept our Acceptable use policy.

From there the system automatically detects

• MAC address (for device identification)
• Initial IP during registration (the subnet from where the users connects tells us roughly from which geographical area the devices is connected to)
• Date of registration
• Hostname

Every time a new devices is connected to the network, the system redirects the very first HTTP request to our custom portal page. After entering the required fields and hitting 'Register' the system automatically detects MAC address and system hostname and creates a new user in our FreeRADIUS installation. If successful, the user is granted access to the network. If the system connects again, it uses RADIUS MAC authentication to see whther the user is already registered. If yes, then access is permitted. If not, the user is redirected to the portal page again.

(Note: Using solely the MAC address makes the system vulnerable to spoofing attacks. However our users typically don't have this knowledge. At least not yet.)

During this self-signup, the user is only granted access with restricted traffic limits (we make use of pfSense' WISPr-Bandwidth-Max-Down and WISPr-Bandwidth-Max-Up capability with low initial values). From there the admins can promote the system to higher traffic caps if found eligible.

The whole configuration/deployment process is a little bit more complex. If someone wants to dive deep into it, make sure to check out our project page.

Using pfSense with the built-in FreeRADIUS can give you quite a lot of information; they are just not always visible through the Web UI.

For instance if Radius logging is turned on you can keep track of all Captive Portal sessions by accessing the log files. This is particular useful when the users have the ability to create their own accounts on the fly through a custom portal page based on their MAC addresses. But you might want to clean old and unused accounts once in a while.

In order to spot accounts that have been inactive for some time, you need to know who connected when for the last time. With shell access, simply copy and invoke this script. This compiles a CSV list of all users ever registered with their MAC addresses together with the last time they have been connected through the Captive Portal. Import the CSV into Excel, filter and sort by the last connected column to see which accounts are ready for removal.

The result looks like this:

MAC;TYPE;NAME;EMAIL;HOSTNAME;INITIAL_IP;REGISTERED;LAST_SEEN 
904ce5a76256;apzu;Public_IT;abc@pih.org;APZU-30;2011-09-15;20120128 
904ce5a76249;apzu;Public_IT;abc@pih.org;APZU-33;2011-09-16;20111019 
001217dfc379;priv;user1;abc@pih.org;APZU-DFA08FCD6D;2011-09-16;20120128 
001a7367b7ff;priv;user2abc@pih.org;;2011-09-18;20120127

(Note that part of the above dump has some custom information as it is tight to they way we use pfSense with a self-register capability for new users. But it should be straight forward to customize this.)

It would be simple to do this automatically (e.g. delete every account not connected in the last 3 months), but as I have some VIP users that I don't want to clear, I just do this once in a while manually. Additionally it could be run as a cronjob every now and then and I guess you could automatically publish it through a web page or mail it to someone. Let me know if you did this, then I can steal it from you ,)

It seems troublesome to send email especially through gmail accounts from *nix systems. Using pfSense as our Captive Portal box running on top of FreeBSD is no exception. So that's what we did to get pfSense sending us emails through shell scripts.

I've tried a couple of things, but eventually sticked to a perl module (including the BSD packages mailx and msmtp), but they all didn't work in one way or another. I came pretty far, but at the end figured out that TLS/SSL support is not build-in. And compiling the packages on the pfSense box seemed not advisable. After all it is a firewall.

To install it, simply invoke this from the shell:

pkg_add -r p5-Net-SMTP-TLS

In case you have a sligthly outdated pfSense installation like I do and this command fails, you might need to tune the package repository a little bit.

setenv PACKAGESITE ftp://ftp-archive.freebsd.org/pub/FreeBSD-Archive/old-releases/i386/8.1-RELEASE/packages/Latest/

Afterwards use this script as an example on how to send mails through the perl module.

Don't forget to put your gmail password in a file called send_gmail_config.txt (just the password, nothing else) and well protect it.

As I always forget them, I just put them down right now right here (they were sitting for too long in my Drafts folder). And I might update them as I go.

Replace multiple chars

sed "s/=ABC=//g"

Grep for multiple words

I know, I know. This shouldn't be here, but I kept on forgetting it...

egrep "FULL|bytes|speedup" snapback.log

Calculate size of directories

du --max-depth=1 -h .

Not sure what I did with this, but it seems pretty important...

NUMBER=$[ ( $RANDOM % 100 ) + 1 ] 
cat servers | while read line; do ./collect_authorized_keys.sh $line done; done 
awk -F":" '{ print $1 }' /etc/passwd 
cat tt | while read line; do export NUMBER=$[$NUMBER + $line]; echo $NUMBER total size > s1; done;

Replacing newlines

My first thought about replacing newlines in files would have been sed, but it is not. tr is the way to go:

cat file | tr 'n' ',' 

To not only replace line endings, but also text before or after them, give this a try:

sed -n '1h;2,$H;${g;s/n/,/g;p}' <file> 

Simulate system load

dd if=/dev/zero bs=100M | gzip | gzip -d | gzip | gzip -d | gzip | gzip -d > /dev/null & 

Howto kill all child processes of a (bash) process

Killing a single process is easy; killing multiple processes also. But only if you know all of the PIDs. If you need to kill a process including all children (sub-processes), try this one:

ps -o pid= --ppid $PID_TO_KILL | xargs kill 

(Note that there are a gazillion solutions out there, but they all seem to be a bit over-complicated. Or I'm missing something..)

Check if a process is running

kill -0 pid 2>/dev/null 
echo $? 

In my current role as the 'IT guy' for Partners In Health I'm also managing the whole IT including our networks in Malawi (note: It is only a rumor that 'IT guy' translates to everything that has a power plug...). We not only provide Internet access to our employees, but also for the government including the local Ministry of Health. As our project has grown a lot in the last few years, so did the numbers of computers that are connected to the network.

Currently we are have connect 20+ access points and with this roughly ~100 devices are connected to the network every day (designed partially on top of this). And all of them squeeze through our tiny satellite link. We came a long way with adding network management tools and traffic shaping to manage the scarce bandwidth better, but at the end we also depend on the fairness of the users: If someone (or his/her system) is misbehaving, it impacts everyone else. With this it is crucial to know who is using the network and how: Welcome to the world of pfSense.

Throughout the upcoming months some of the important lessons learned and findings are shared here. This will include topics like

• Deployment
• Email notifications
• Traffic limits
• Active Users & Accounting
• Bandwidth Management / Traffic shaping
• Captive Portal self registration with custom portal page
• Basic HTTP filtering based on time of the day
• Monitoring

With all this I guess we may run one of the biggest (if not _the_ biggest) freely available public hotspot in Malawi. I like my work in the low-resource settings...

I'm by no means an expert in WiFi network planning and installation, but over the past year I have collected some knowledge and best guesses on how things work in terms of performance. Here is an open call for everyone to correct my views and not so obvious elements that impact the speed of your wireless network.

Which 802.x standards are relevant? - 11a on 5 GHz; almost unused - 11b on 2.4 GHz; nominal up to 11 Mbit/s, effectively max 600 Kbyte/sec - 11g on 2.4 GHz; up to 54 Mbit/s, effectively max 3 Mbyte/sec - 11n on 2.4 GHz and 5 GHz; some devices only follow the draft spec and not the final one (impact?)

Which standards has the best coverage/signal range? Unclear. In general 5 GHz signals have a reduced signal range due to its higher frequency, but as 11n sends with higher power, the standard claims that the range is extended compared to 11b/11g.

What are sources for Non-WiFi interference? Technically everything in the license-free 2.4 GHz and 5 GHz frequency bands like Bluetooth, Microwaves, Babyphones, Wireless Video Transmission adapters, Amateur radio, Walkie-Talkies.

Is WiFi using a shared medium? Yes, kind of similar to a BNC or "hub-ed" ethernet every participant needs to share the same medium. Therefore the more traffic is seen on the medium or the more clients are using it, the more likely are collisions and therefore re-transmissions. Minimizing shared access to this shared medium is one element of improving WiFi network performance. In other words wireless is a shared medium, meaning that all clients and neighboring APs compete for the same limited bandwidth, in addition, each client’s speed varies depending on the protocol it is running (802.11 a/b/g/n) and the signal strength, interference and noise it is experiencing.

Which channels for 11b/11g are available? In the US 1 - 11; in Europe 1 - 13; in Japan 1 - 14 (14 only for 11b, note that channel 14 is not linear therefore channel 10 and 14 are completely non-overlapping). The availability of different channels will depend on the device and maybe the country setting. So channels 12+ are not guaranteed to be fully functional. However it might be worthwhile to configure backbone links to this unusual channels (even though it might be against local law). This leaves 4 channels (instead of the typical 1, 6, 11), namely 1, 6, 10, and 14 as (almost) non-overlapping channels.

Airtime consumption / utilization A wireless frame transmitted to or from a client connected at a low data rate may utilize 10 milliseconds of airtime, whereas it may take only 100 microseconds for a client connected at a high data rate. Even though the high speed client could have sent 100 frames in the time the slow client takes to send one frame, the fast client still has to compete fairly for the airtime on a frame by frame basis, so it spends most of its time sitting idly waiting for the slow client to finish so it can have another chance to transmit. Unfortunately this means that a single low speed client can slow down all of the other clients on the WLAN.

What is the impact of client connected with poor signal strength? As explained with the airtime consumption slow clients consume more airtime to transfer a given amount of data, leaving less airtime for other clients, decreasing network capacity and significantly degrading the performance of all clients on the network.

What is a poor signal and how to detect it?

RSSI: over -50 perfect, below -85 unusable

Noise: from 0 bad over 90 very good to -120 none

SNR: higher is better 25 min requred for full 54mbps link 40 dB Excellent signal strength 25-40 dB Very good signal s 15-25 dB Low signal 10-15 dB Very low signal 5-10 dB No signal strength

For the Voice you need a minimum -25 dB

In OS X option-mouseclick on WiFi icon to see stats of connected network. Open Performance windows of Wireless Diagnostics to track signal strength over time.

What is the Basic Rate? Depending on client device and signal strength the AP and client are handshaking about the maximum available performance. The farther away a client is from the AP (read the weaker the signal), the lower is the Basic Rate. 802.11g supports rates of 6, 9, 12, 18, 24, 36, 48, 54 in addition to the 802.11b rates of 1, 2, 5.5 and 11Mbps. These can be seen e.g. with inSSIDer. A client can fall back to lowest basic rate to save energy or to increase link stability.

Will one client with a low basic rate slow down other clients? Depends. If the client is not busy than there shouldn't be a big impact. But if the far away client is busy transmitting and receiving many connection drops and re-transmissions, the whole shared medium air should be congested. Additionally the latency should be higher, so that errors are detected later. In these cases even closer clients with a high Basic rate should suffer.

What is the maximum number of concurrent connected clients to one AP? Numbers seem to vary quite a bit. There are some Enterprise-devices which limit the number of connected clients to 10. Consumer APs seem to be more open, but the performance will suffer as all of them are talking on the same channel. Looks like above 20 is unreasonable.

What happens if clients with different 802.11 standards are mixed on one AP? If 11b clients are connected to a 11b/11g AP point or 11g clients to a 11g/11n AP it will decrease the performance. Statements vary from "just a little bit for everyone" to "bigger performance penalty" due to the compatibility mode that one AP is using for these mixed cases. It seems like if possible the mixed mode should be avoided, e.g. if still legacy 1b clients are present (most likely the case in our environments), than a dedicated AP 11b only could overcome this.

What is the best alignment of external antennas for indoor APs? Sometimes APs have up to 3 external antennas that can be aligned in different angles. Facing them in all three room axes is best to increase signal strength and to reduce interference. If there are less than 3 (up to none) antennas, then they will be molded inside of the case. Knowing its position might help with mounting or placing such an AP.

Can one Wifi client bring down an AP? To a good degree: yes. If the client is misbehaving (accidentally though wrong configs or drivers or purposely e.g. by trying to break into encryption) it can flood the shared medium air with packets. This way there will be many collision with other clients and therefore packet losses and re-transmissions. This way one client can monopolize the whole network segment.

What impact has a client-to-client communication? As every communication goes through the AP, the sending client first needs to send the data, which is then repeated by the AP for the receiving client. Therefore the same message goes over the air twice and leaving only half of the available throughput.

Hiding SSIDs Hiding SSIDs will not improve security as special WiFi clients (I refer to them as promiscuous WiFi stacks, but I'm not sure whether this is technical true) are still able to sniff transmissions within hidden SSID and extract the SSID from there. But hiding SSIDs might get rid of the beacon announcements at the basic rate. Additionally if this is used for backbone links they don't indicate the availability of an AP to ordinary users if there is no for them. On the other side a hidden SSID will not be detected by survey tools like inSSIDer and therefore make the channel planning less obvious: If your neighbor doesn't see your SSID and channel, he might be not aware of a collision.

How to test real-world WiFi speed? iperf daemon running on a server and iperf client on a wireless client measures the "real-world" IP performance.

Mixing encryption Some AP offer multiple encryption methods (like WPA2/AES and WPA2/TKIP). Even though it is supposed to work seamlessly there seem to be sometimes performance problems in the wild; especially with 11n networks. So the rule of thumb should be to not allow mixed encryption. As TKIP needs to be seen as almost broken anyways, switching to WPA2/TKIP for at least 11g networks is a good advice.

Which features offer "enterprisey" components? - Centrally managed through a WiFi controller - VoIP support through QoS - Roaming between multiple AP (internal roaming); helpful especially for mobile VoIP phones - Load Balancing - Ability to run AP with reduced power. This can be used to place the APs a lot closer together (higher density) while still avoiding radio interference among access points, resulting in more consistent coverage

What is Client Isolation (sometimes called AP isolation)? When activated an AP is blocking all client-to-client communication in his wireless network segment (is done on layer 2 through MAC addresses). Even though it seems common understanding that this doesn't have any impact on performance, it might not be the whole truth. Client-to-client communication is always happening with half of the max speed as it always goes through the AP: First the client needs to send out his message, the AP picks it up and re-distributes it in the wireless segment. Thus the same message goes over the air twice. Now for general broadcasts and Windows shares this might happen more often than expected and for this case AP isolation should block and therefore eliminate unnecessary traffic resulting in higher performance. Unsure: Will this help against tools like Firesheep in unprotected networks?

Which tools are available to troubleshoot WiFi problems? - inSSIDer - GPS device - Others: NetStumpler, EarthStumpler, KnsGem, Kismet - External scanners (necessary if Non-WiFi interference should be detected)

What are Wireless repeaters or Wireless Distribution systems (WDS)? Although the details vary both technologies can be used to increase the coverage of a WiFi area. In a nutshell they pick up a WiFi signal from another AP and re-distributes it into its area. Usually they operate (they have to?) with the same settings for channel and encryption and could also offer a roaming capability. The drawback is that it reduces the maximum speed significantly as the signal will be re-send as many times as there are repeaters or WDS with the same settings (must be so, but I could only find reference to reduce the speed once by 50%). So they can be used to easily extend the coverage, but need to be carefully chosen to not introduce more bottlenecks.

Are multicasts in WiFi settings different than in wired settings? From a high point of view, no. But as an AP needs to make sure that every connected client is able to receive the multicast (same for broadcasts?), it will send out multicast packets with the lowest basic rate configured for the AP. And here the airtime consumption is (much) higher (leaving lesser room for other packets) and the available maximum throughput is capped at the basic rate (making HD video broadcasting impossible).

What is a "hidden node"? Two Wireless clients can be connected from the opposite ends of the coverage area to an AP point. Both see the AP, but are not able to see each other. This is one of the reasons why every traffic always goes through the AP.

What impact has the Beacon interval? Every AP announces its presence and configurational details in the interval of the beacon. Typically every minute a broadcast is send with the lowest basic rate so that clients can pick up the details. For static backbone link there is no need to constantly announces their details so increasing the interval will reduce the number of broadcast. Unsure if turning out the SSID broadcast also eliminates the beacon at all.

What is the ACK Timeout (Sensitivity Range)? (todo)

Conclusions Keep WiFi signal locally wherever possible Don't give client access to wireless backbone links; separate backbone from enduser connections

--

May 11th, 2011 at 17:48 | #1Reply | Quote

What impact has the Polarization?

Seems like usually V-Pol (vertical polarization) is used. Although I could find some talk about H-Pol (horizontal polarization) for high density areas. For Point-to-Point you can use both as long as both endpoints are configured the same way. For a forest area between the links, H-Pol may work slightly better.

--

May 16th, 2011 at 08:33 | #2Reply | Quote

Order of WiFi signals to connect to

Looks like most Operating Systems treat the latest configured WiFi network with the highest priority. This means that it tries to connect to WiFi networks in reverse order of it’s first configuration; not necessarily to the ones with best signals strength/coverage. When having a few high power/long distance antennas and multiple local Access Point could lead to the point where even though a local Access Point with good signal is available, the system tries to connect to the low speed (due to above reasons), weak signal long distance antennas. Align the order to the ones with the most local signal first.

--

July 24th, 2011 at 11:10 | #3Reply | Quote

Interference

Wi-Fi 11b uses Direct Sequence Spread Spectrum (DSSS) (http://en.wikipedia.org/wiki/Direct-sequence_spread_spectrum). Its carrier does not hop or change frequency and remains centered on one channel that is 22 MHz-wide. While there is room for 11 overlapping channels in this 83 MHz-wide band, there is only room for three non-overlapping channels. Thus there can be no more than three different Wi-Fi networks operating in close proximity to one another.”

WiFi 11g uses OFDM (http://en.wikipedia.org/wiki/OFDM) for 54 Mbps and DSSS for 11 Mbps and is designed to reduce interference.

Question: If 11g falls back to speed 11 Mpbs and below, does it also move back to DSSS?

--

July 24th, 2011 at 11:11 | #4Reply | Quote

Bluetooth Interference

Bluetooth use FHSS (frequency-hopping spread spectrum) http://en.wikipedia.org/wiki/Frequency-hopping_spread_spectrum as an AFH (adaptive frequency hopping) technology to ensure that anti-interference robustness in the 2.4 Hz band. All the 79.1 channel with each 1 Mhz hopping 1,600 times per second.

Ever had to track down a client devices in an area covered by many (unmanaged) WiFi Access Points?

If tools like kismet/kismac are not working for you to track computers (e.g. because of unsupported hardware, crashes, bad wifi antenna on your laptop, then just build it yourself.

All you need is a flashable consumer-level wifi access point (like a Linksys WRT54) and flash it with dd-wrt. This can put the router in monitor mode and together with the addon wi-viz you get an overview over all wireless activity.

And in case you want it mobile, simply put a bunch of batteries, e.g. 8 AA batteries with 1.5V each in a row and connect it to the router. Finally connect your laptop with an Ethernet cable, stuff everything in a little bag and walk around. If you know the MAC address of the devices your are looking for (laptop, handheld,...) just see very you have the strongest signal and walk in that direction. And voilà, you are able to geographically localize every WiFi device. Regardless of it's connection to a specific access point. Welcome wiFi Catcher. Welcome Jack Bauer.

Sometimes I prefer to have a private (read secure and non-observable) web connection - being a developer and admin makes you a bit more paranoid...

So how can you establish a connection that besides from being non-observable may also bypass potential content filters or firewall rules? Of course with a simple SSH tunnel: ssh -D 8080 -C -q -N user@server   (-f to put it in background?)

Now simply configure your browser to use the SOCKS proxy running on your localhost at port 8080 and off you go.

The drawback is, that you need a Unix server outside to connect to. But who has not such a system somewhere? And even if not by now, maybe just go in the clouds.

And for those unlucky guys running Windows and PuTTY: even you could do that.

Update: It seems like SSH can even be misused to tunnel Remote Desktop connections. This might do the trick:

sudo ssh -D 8180 -p 8999 @  -L 127.0.0.1:3333::3389

Sometimes a process should just run for a maximum amount of time. A nightly long network transfer, a backup, or a statistical report shouldn't accidentally run until next business hours. A watchdog timer which kills a process once a certain time has passed by is needed. As usual in Shell programming multiple ways are possible, all of them have certain drawbacks. The simplest one would be just to call the long running process in the background and capture its PID. But then it is not that easy to capture the return value of it. Here is my shot with an additional at job:

#!/bin/sh
MY_PID=$$
TIMEOUT=1 # in minutes

# Install at job as watchdog to remove long running process
WATCHDOG_CMDFILE=/tmp/`basename $0`-$MY_PID
echo "# watchdogfile script" > $WATCHDOG_CMDFILE
echo "kill -0 `echo $MY_PID` 2>/dev/null" >> $WATCHDOG_CMDFILE
echo "if [ $? -eq 0 ]; then" >> $WATCHDOG_CMDFILE
echo "  ps -o pid= --ppid `echo $MY_PID` | xargs kill" >> $WATCHDOG_CMDFILE
echo "  echo "long running process aborted because it ran too long"" >> $WATCHDOG_CMDFILE
echo "fi" >> $WATCHDOG_CMDFILE
echo "rm -f `echo $WATCHDOG_CMDFILE`" >> $WATCHDOG_CMDFILE
at -f $WATCHDOG_CMDFILE now + $TIMEOUT min

# Start my very sophisticated long running task
sleep 3600 # 1 hour
RET=$?

# do whatever you normally do after the long running process finishes
echo $RET

(Note that at sends out emails with the stdout/stderr. If you have another notification method to indicate an aborted job, ensure that nothing is printed to std & stderr.)

OK. Not really "Programming stuff" anymore. But still important:

Don't allow remote root logins at all, even with SSH

Why not? Because:

• User name of root is known, therefore the account is vulnerable for brute-force attacks
• Working as root should be an explicit switch and not the default policy. Just like being aware of switching hats.
• Bad for auditing, if multiple users have root access.

But now you say: "I don't care as I use SSH for logins".

• Depending on the auth method, password is still transfered over the wire.

• But using public key auth instead of passwords might be even worse. Still you have to trust all(!) clients that the private key is stored safely. Read:

• "Good enough" passphrase.

• There is no way to tell from the public key (which is the only thing known by the server), if the private key has a passphrase at all.

• Trust the client system (that it is not compromised)

• Auto lock of the client system must be enabled after a few minutes of inactivity.

• Sensible use of background daemons like ssh-agent or Pageant(Putty for Windows) on client systems necessary. What if the users start the keyring app, enters his passphrase and never shuts down his system. And now imagine a laptop running out-in-the-wild without any local password protection having an open private key in its memory!

What to do? Dunno. Maybe:

• Use "ordinary" user accounts
• SSH with either public key or passwd auth (depending on your decisions reg the previous points)
• Enforce sudo (better) or su (less better) to gain temporarily root privileges

Anything else???

Are you using SSH for your secure maintenance of your servers? - Sure.

Do you copy files with SCP between hosts? - Most likely.

Is there a need to transfer big files over slow and unreliable network connections (*)? - Could be.

Can SCP resume a download after the connection crashed? - No.

So why not simply use rsync over SSH for your file transfer. A minor drawback is, that unless you set up a rsync daemon (not appropriate for my case) you have to call rsync manually. Sadly rsync doesn't offer something like "automatic retry in case of a connection failure". (**)

Good for us, because now it's tool time again; a single bash script does the trick:

#!/bin/sh 
# reliable file transfer

# try rsync for x times 
I=0 
MAX_RESTARTS=5 
LAST_EXIT_CODE=1 
while [ $I -le $MAX_RESTARTS ] 
do 
	I=$(( $I + 1 )) 
	echo $I. start of rsync 
	rsync -av --partial --progress -e "ssh" x-ian@x-ian.net:~/MY_BIG_FILE . 
	LAST_EXIT_CODE=$? 
	if [ $LAST_EXIT_CODE -eq 0 ]; then 
		break 
	fi 
done

# check if successful 
if [ $LAST_EXIT_CODE -ne 0 ]; then 
	echo rsync failed for $I times. giving up. 
else 
	echo rsync successful after $I times. 
fi 

Ah, just a sidenote as I always forget the syntax: If you need to remote execute a command via SSH with variables from your local shell, take this:

CMD="test -e M\_BIG\_FILE || cp MY\_BIG\_FILE \`hostname -s\`-MY\_BIG\_FILE" 
ssh x-ian@x-ian.net $CMD 

--

(*) If you only have an unstable satellite link, even 150 MB are way too big. (**) Make sure that you actually test over the network; using rsync with source and destination files on the same system deactivates the delta-calculation algorithm.