Yes I know I still haven't done part two of that mail server post. I'll get it done soon, I promise.

While chatting on IRC, someone mentioned that they were having a problem with a process going mental and creating a bunch of file descriptors in linux, eventually hitting the "max FD limit" linux has. They couldn't figure out which process it was and they couldn't find a program that would list a count of how many FDs a process has open. A few minutes later I'd thrown together this bash one-liner for him. I'm posting it here just in case someone else might find it useful.

echo "$(for pid in $(ls -a /proc|egrep '^([0-9])*$'|sort -n 2>/dev/null); do if [ -e /proc/$pid/fd ]; then FHC=$(ls -l /proc/$pid/fd|wc -l); if [ $FHC -gt 0 ]; then PNAME="$(cat /proc/$pid/comm)"; echo "$FHC files opened by $pid ($PNAME)"; fi; fi; done)"|sort -r -n|head -n4

To explain: It loops through every file/folder in /proc that is a process ID, then checks that there's a file descriptor folder. Then it gets a count of all the FDs that process currently holds, gets the process name and outputs how many file descriptors that process has open, as well as the process name. This is then reverse-sorted and cut down to only the four processes with the most FDs open.

Okay yes I know I said the second part of my mail server post was incoming, it still is, but for now I'd like to break out and complain about something again.

Virgin Media recently announced a new 'Premiere' tariff for Virgin Mobile. This tariff gives you unlimited calls to landlines, 2500 minutes to mobiles, unlimited texts and unlimited data, for as low as £21 for existing Virgin Media customers.

As soon as I read "all-you-can-eat data", I was skeptical. Very few mobile operators offer truly unlimited data (and I'm with Three UK, who are one of maybe two operators here that offer proper unlimited data - their policies state that customers have an effective data cap of 1TB per month). So I did what anyone else does, and I looked up their policies (which you can find here), which state that:

Unlimited Mobile internet for daily use by Pay Monthly customers: We'll monitor how much mobile internet you use each month so that we can protect the network we use for all of our customers. If we consider your use to be excessive, we won't charge you any more, but we may restrict your access to the mobile web depending on how often and how excessive we think your usage is. As a rule of thumb, we are likely to consider any usage over 1GB per month to be excessive. Unlimited use is within the UK and is for your personal, non-commercial use only. It doesn't include making internet phone or video calls, peer to peer file sharing, using your phone as a modem, or while you are abroad.

Virgin Mobile are basically stating that they'll either restrict or throttle you if you use more than 1GB of data a month. That's hardly unlimited.

Then you've got the unlimited texts and landline minutes. But Maff, you may cry. How can Virgin screw up unlimited texts and landline calls? Quite easy. From the same document on virgin's website:

Unlimited texts: Unlimited texts are subject to a fair use allowance of 3000 texts per month. If your usage exceeds this amount then we reserve the right to charge you for the excessive element of your usage at the text rate for other mobile networks for your tariff outlined in our Tariff Table. Unlimited use is within the UK and is for your personal, non-commercial use only. It doesnt include texts to shortcode services, group text, or picture messages and any of these uses will be charged at the text rate for other mobile networks for your tariff outlined in our Tariff Table.

Unlimited landline minutes: Unlimited landline minutes are subject to a fair use allowance of 9000 minutes per quarter (3 months). If your usage exceeds this amount then we reserve the right to charge you for the excessive element of your usage at the rate for calls to landlines for your tariff outlined in our Tariff Table. Unlimited use is only for UK originating calls from the eligible Virgin Media phone to UK landlines (01,02 and 03 numbers). All other call types will be charged at the rates indicated in the Tariff Table and are not included in the allowance of minutes. Unlimited minutes are for your personal, non-commercial use only.

Right there. "Unlimited texts" actually means "3000 texts per month and we'll probably charge you standard rates if you go over that". "Unlimited landline minutes" actually means "9000 minutes per quarter year, and we'll probably charge you standard rates if you go over that".

Now it's fair to assume that most people won't use more than 3000 minutes or texts per month, but please, don't call that unlimited. It's misleading. As for Virgin's so-called unlimited data, I really don't understand how it can be called "unlimited data" when Virgin considers usage exceeding 1GB per month "excessive".

I really have nothing against Virgin, they were one of the first operators I was with (The other was BT CELLNET, now o2), and I'm sure their 3G network is great (I last used Virgin's network several years before 3G was even a thing in the UK so I have no idea, but the 2G network they had was good and coverage was always great), but it just rubs me up the wrong way when an ISP or mobile network provider claims that a service is "unlimited" - I did the same thing with BT when I discovered their top-tier "Unlimited 8mbps" broadband plan was actually subject to a fair use cap of 100GB. We didn't find this out until BT emailed us saying we'd hit 80GB that month and that we'd be billed for any usage past 100GB.

I'm happy to change this if I just found the wrong policies page on Virgin's website. If I pointed out the wrong policies and Virgin's "Premiere" tariff is subject to different policies, please point it out and I'll happily reflect that in this post.

So a few months ago, I moved my primary mail hosting to my own VPS. Over the months since then, I've been tweaking and adding to my mail system, and I figure it'd help both myself and others if I documented what I've done, so I'll start with a list of all the software I use.

Main Software

• Gentoo - My VPS runs Gentoo. I personally prefer it over other distros, as it's both lighter and less screwed up.
• postfix - I was originally going to use Exim, but found it strangely difficult to configure, plus postfix seems to universally have lower transaction times.
• dovecot 2 - I'm switching away from Gmail, so obviously the things I would've missed most would be things like push email and mail filters. Dovecot supports IMAP IDLE, and has sieve/managesieve, so it was easy to port my Gmail filters over.
• saslauthd - While dovecot has its own SASL authentication, I prefer to use this when authenticating over SMTP. EDIT: I have since switched from saslauthd to Dovecot 2 for SASL authentication. Dovecot works well enough for it that I questioned why I actually needed saslauthd.
• Mutt - Mutt is my primary MUA. I'll also be discussing configuration changes I made to Mutt, and ways I made it work more like Gmail.

Extra Software

• SpamAssassin - This should be obvious. Does a great deal to cut down on spam. Plus, with a filter set up with dovecot's sieve, I have a spam folder like before. EDIT: When I first published this post, I had only just set up Postgrey and had no idea what kind of impact it would have on incoming spam/junk mail. It had a massive impact – I haven't received a single spam email since. Spamassassin and Amavisd may actually be unnecessary when using Postgrey (Unless of course you plan to host mail for others, or if you receive a *lot* of spam).
• Postgrey - This does a fantastic job of cutting down on spam.
• Amavisd - Somewhat necessary for making postfix work with SpamAssassin, but also makes it easy to offload the antispam part of the mail system to another server. Amavisd can also be used for integrating antivirus systems into your mail scanning process, but I don't need that.
• OpenDKIM - Used for signing outgoing mail with my DKIM key, and for validating incoming signed mail. This does a good job of ensuring that mail sent from my domain is actually coming from one of my servers.
• policyd-spf - Originally, I used pypolicyd-spf, but it quite literally breaks every time there's an update to python, it's since been replaced with this perl equivalent which has never had any issues. This uses SPF to validate incoming mail, and ensure that the sending server is actually authorised to send mail for the given domain.
• fail2ban - This isn't strictly part of the process I go through when setting mail up, but fail2ban helps cut down load a lot when bots are trying (and failing) to use a server as an unauthenticated relay.

In the near future, I'll write a second post detailing how I linked all this together, including config excerpts, but in this post I'm just discussing the software I used, as well as why I use each package. I'll also leave you with a list of extremely good tips.

• Get an SSL certificate. This makes setting secure mail up a lot easier, especially if you plan to send or receive mail remotely with stuff like IMAP.
• If you do get an SSL certificate, disable or firewall unencrypted mail ports. Obviously leave port 25 in place, but if you're sending or receiving mail remotely, disable the unencrypted IMAP/POP3 ports (143 for IMAP and 110 for POP3), and set your MTA up to only accept submission mail through 465.
• Set up SPF records for your domain appropriately. SPF does a good job of telling other mail systems who is or is not allowed to send mail for your domain.
• Generate a DKIM key, and add it to your domain's DNS. As with SPF, DKIM (DomainKeys Identified Mail) does a fantastic job of indicating to other mail systems whether an email is actually legitimate or not.
• Use blacklists. There's a large number of DNS-based blacklists which indicate whether a given IP address is known for sending spam or for attempting to compromise servers. This can go a long way in preventing spam.
• Report any spam you receive. Reporting received spam to places like SpamCop not only reduces the chance of you receiving similar spam in the future, but it helps others too. It helps identify servers that send spam (Contributing to blacklists), helps identify possible domains used for spam (again, contributing to blacklists), and can contribute to the accuracy of antispam systems like SpamAssassin.
• Monitor your services extensively. This is definitely a big one. It's not easy to monitor your server by looking at logs, and often unless you've got systems set up to email you when anything out of the ordinary happens, you just plain don't know what's happening with your server. Packages like Monitorix (disclaimer: I'm the package maintainer for monitorix on Gentoo), do a fantastic job of showing you at-a-glance whether anything abnormal is happening, so it's easy to see if and when your mail server is rejecting mail. This can also be great for indicating when you've misconfigured something.
• Use external monitoring services. Services like MXToolbox have free accounts, and you can use them to set up checks so that you get an email if your server's IP is on any IP blacklists. Services like Pingdom are also great for monitoring both uptime and external availability.
• Make sure your forward and reverse DNS match and that your reverse DNS is your primary mail domain (or what your server actually identifies itself as). This is definitely a good way of ensuring your mail isn't identified as spam.

One final thing to note, the guide and so on will discuss my current mail setup. This means it assumes you'll be using sockets for things like Postgrey. Please read everything carefully before making configuration changes to your own mail setup, as what works for me may not work for you.

That's all for now, but I'll be adding to this list, and writing a second post documenting how I actually set my mail system up, very soon. EDIT: Disregard that. Second part of this post will arrive eventually but I am tremendously lazy.

Since I figure it's bad form to have a blog and /only/ use it for ranting, here's a somewhat useful post. I've been with [a][1] [number][2] [of][3] [different][4] [hosts][5] over the past year or two, and I figure it'd be useful for others to know why I like or don't like them.

[SimplexWebs][6]

SimplexWebs has been in the hosting business for quite a while now, and do enterprise webhosting, online radio hosting and domains, as well as VPS (Xen, powered by OnApp) hosting. I was lucky enough to grab one of their limited birthday sale servers, which gave you 256MB RAM, decent CPU speed, 20GB disk and 100GB monthly bandwidth, for £25 a year.

This server's basically been my primary server/main workhorse - despite initially purchasing it to run a VPN server, and has held up ridiculously well. Uptime has been fantastic - my server's only been down a few times over the last half a year or so, including migrations between their old SolusVM platform to their new OnApp platform, and including the recent downtime when they moved datacentres.

Server and network speed are really good. I think this is one of the fastest servers I've ever used, and that includes the brief time I was with Linode. Support is fantastic, their team is always quick to respond and very helpful, and you definitely get the feeling that they actually care about their customers. Overall, they're one of the best hosts I've ever been with, and I don't feel I'll ever need to move to another host.

Here's some benchmarks:

Network speed:

~ wget cachefly.cachefly.net/100mb.test -O /dev/null
--2012-04-28 02:34:32--  http://cachefly.cachefly.net/100mb.test
Resolving cachefly.cachefly.net... 140.99.94.175
Connecting to cachefly.cachefly.net|140.99.94.175|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 104857600 (100M) [application/octet-stream]
Saving to: `/dev/null'

100%[======>] 104,857,600 11.8M/s   in 8.7s

2012-04-28 02:34:41 (11.5 MB/s) - `/dev/null' saved [104857600/104857600]

Disk I/O:

~ dd if=/dev/zero of=/tmp/disktest bs=64k count=16k
16384+0 records in
16384+0 records out
1073741824 bytes (1.1 GB) copied, 4.52632 s, 237 MB/s

[Bhost][7]

Bhost are a UK-centric VPS provider, primarily dealing with OpenVZ servers although they've recently launched Xen PV plans. I should mention their servers are some of the cheapest you can get, the cheapest OpenVZ server being £4.70 (before VAT) and including 512MB RAM and 512MB burst RAM.

Their support is pretty great, considering they're a budget host, and the servers themselves are quite speedy. I do have some issues with the OS templates they have available - their Arch images are broken due to the host kernel being too old, although I'd say that's an Arch issue, and their Gentoo image is too old to actually use. Debian works fine though, so that's what I'm using. Overall, network and server speed is really good. Uptime is fantastic, in the entire time I've been with them (a number of months now), my server's almost never been down. Definitely one of the most reliable hosts I've been with.

Here's some benchmarks, since I still have a server with them: Network speed:

# wget cachefly.cachefly.net/100mb.test -O /dev/null
--2012-04-28 02:30:40--  http://cachefly.cachefly.net/100mb.test
Resolving cachefly.cachefly.net... 205.234.175.175
Connecting to cachefly.cachefly.net|205.234.175.175|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 104857600 (100M) [application/octet-stream]
Saving to: `/dev/null'

100%[======>] 104,857,600 10.2M/s   in 9.8s

2012-04-28 02:30:49 (10.2 MB/s) - `/dev/null' saved [104857600/104857600]

So there's been some ruckus lately because, following the launch of Google Drive, people took to the internet to compare the Terms & Conditions of Google Drive, to that of Dropbox and SkyDrive. The main point seems to be that people see Google Drive's T&Cs as being too unrestrictive. Case in point: https://twitter.com/#!/jmacdonald/s...

What I have a big problem with, is that people don't seem to have properly read the relevant portion of Google's terms and conditions. The full text of this portion is as follows:

Some of our Services allow you to submit content. You retain ownership of any intellectual property rights that you hold in that content. In short, what belongs to you stays yours.

When you upload or otherwise submit content to our Services, you give Google (and those we work with) a worldwide license to use, host, store, reproduce, modify, create derivative works (such as those resulting from translations, adaptations or other changes we make so that your content works better with our Services), communicate, publish, publicly perform, publicly display and distribute such content. The rights you grant in this license are for the limited purpose of operating, promoting, and improving our Services, and to develop new ones. This license continues even if you stop using our Services (for example, for a business listing you have added to Google Maps). Some Services may offer you ways to access and remove content that has been provided to that Service. Also, in some of our Services, there are terms or settings that narrow the scope of our use of the content submitted in those Services. Make sure you have the necessary rights to grant us this license for any content that you submit to our Services.

What this essentially states is "You retain full rights to all content you submit to a google service. When you do this, you grant us and our partners the right to store, copy, modify (This is for things like converting document formats), create derivative works (This is things like thumbnails, scaled/rotated/edits that you perform to photos in Picasa's photo editor), communicate (Transmit over the internet), publish (Display on a blog post, for instance), publicly display (Displaying things on the internet is a public display) and distribute the content (Transmitting your data through various google services). This license you grant us, only allows us to use the agreed rights when operating our services, promoting our services (Remember, this is not a google drive specific Terms & Conditions, this is for all of google's services, so this can include things like reviews on Google Maps), improving our services, and creating new services."

The main issue that people seem to have is they don't read the line that states Google can only use the agreed rights when operating/improving/promoting their services, or when creating new services. It also states that there are "terms or settings" which further narrow down what Google is actually allowed to do with your data, and chances are there is, or soon will be, a supplimentary Terms & Conditions page specific to Google Drive.

The jist of this is, you retain full rights to all content you put on google drive. Google is only allowed to do things like convert your files between data formats (Word Document to RTF, for instance, or JPG to PNG), or transfer your data between their servers. The license includes stuff like that because you clicking "Convert my document to Word please" does not constitute you giving google the right to actually perform that action, they have to have the rights in the license to avoid any legal issues. Google does not, contrary to what everyone seems to believe, reserve the right to take your holiday photos from Google Drive and use it as the background on the main google home page.

Calm down, people. It's not nearly as bad as you claim.

There are times when linux frustrates me. Not with issues specific to one distro, but software packages in general which are written for linux.

My prime example here is Watch. In Gentoo, it's included in the procps package. I recently was confused to find that my ident daemon, which I keep running because I'm an avid IRC user, was being flooded with traffic near constantly. Netstat told me it was because of two IRC servers I ran. So I logged in, checked netstat there, and sure enough, it was them. But I had no idea what process on the servers was actually creating the connections.

I assumed it would be the ircd itself, but I wanted proof before investigating further. No problem, I thought, I'll just run watch --differences=cumulative -n 0.1 'lsof +M -i4|grep auth', which according to watch's manpage, would show what's changed in a command's output, rather than clearing the screen and displaying the output every .1 seconds. It did do this, in a way, however, because the program creating the connection to my ident server only kept that connection for a fraction of a second, the output vanished, and thanks to the unhelpful way that watch handles output which only shows up once, all I got was some white blocks showing that there had at one point been text there.

My solution? Throw together a bash one-liner which looped infinitely until the offending program was identified: while true; do UNR=$(lsof -M -i4|grep auth); if [ -n "$UNR" ]; then echo "$UNR"; break; fi; done

This did eventually work, and it turned out to be a runaway process on my personal box constantly creating connections to both IRC servers.

A while ago I installed this script which was inspired by this xkcd comic. I keep forgetting it's installed, but that itself makes for some hilarious double-take moments when bowsing.

I run a number of different servers from a number of different providers. I also run servers for friends. In this post, I'll be discussing one server in particular, a friend's server that's primarily used for IRC, web hosting and minecraft. This server runs Arch.

Now I'm going to start by saying I don't always make fantastic decisions and I'm not always known for making sure everything's perfect before rebooting a server. I have, in the past, screwed servers up. But this case was a little different.

Yesterday I log into the server to update a firewall rule, and discover that, because I've never used the nat table in iptables before on that box, the module was never loaded. Of course, the box has fantastic uptime and hasn't been rebooted in over 160 days. Now I'll stop for a second to mention how Arch handles kernel upgrades. When the kernel's upgraded, the previous kernel is left on the system, and all kernels older than that are removed completely. This includes the currently running kernel, and all modules. And this box hadn't been rebooted since kernel 3 was released. I have the box set up to be as zero-maintenance as possible. Emails whenever anything happens, cronjobs taking care of updates and removal of old packages from the cache, scripts to reboot any services that have crashed, gone down or have stopped responding. But as I discovered during a routine check a week prior, the Arch box hadn't been upgrading any of its packages due to a recent change to the filesystem package. I manually started the update process, it informed me that it couldn't upgrade the system because of a file conflict (as the news article mentioned). No problem, I force-installed the update to filesystem and then upgraded the rest of the system as usual.

Now fast-forward back to yesterday, I'm telling my friend I need to reboot his server to apply the firewall rule. He gives the okay, and I reboot the server. Emails flood in saying that various services are down and that the server's offline, as always. But then a few minutes pass, there's no email that everything's back online again. I ping the server, nothing. Log into the server host's control panel, it's listed as online, so I VNC into it to see what the issue is. It can't find the harddrive and has thrown itself into a recovery terminal. What.

I figure a kernel change has messed with the partitions, so I boot the server into my usual recovery system - the gentoo live CD. Nothing seems out of place with grub or the fstab, so I look at the next culprit - the config file for mkinitcpio. It's blank.

Somehow, pacman disregarded the usual rules about protecting config files against being overwritten and messed with the config file, so the initramfs Arch needs in order to boot up properly was completely broken. No problem, I'll just chroot in -- OH WAIT. The Gentoo LiveCD runs kernel 2.6.31, and arch refuses to do /absolutely anything/ unless the kernel is new enough (In this case it had to be 2.6.32 or newer). The server host isn't exactly good, and doesn't provide any recent install media. Cue ten minutes of me googling for the kernel versions of everything in the media list, eventually settling for a Fedora 13 disk that had a recent-enough kernel. I get chrooted in, fix the mkinitcpio config and start generating the images. It complains that /dev isn't mounted. Okay. I look at the source for mkinitcpio, discover it's trying to access /dev/fd, which somehow doesn't exist. I symlink it over from /proc/self/fd and start it again. Everything seems to work, so I reboot.

This time, it recognises the hard drive, but the partition device names have changed. It's now seeing the disk as /dev/xvda. Bizarre, but has happened before. I boot the gentoo livecd again, edit grub's menu.lst and fstab, reboot back into Arch. It boots! But doesn't have a network connection. By this point, I'm close to pulling out my hair. I google around and find that because I included a bunch of xen drivers that mkinitcpio forgot before, everything's working a little differently. Specifically, it now works closer with Xen, and is trying to unplug everything at boot. No problem, I'll put xen-emul-unplug=none in the kernel boot line. Reboot the server and the harddrive device name has changed again. I boot into gentoo, change menu.lst and fstab to use /dev/sda again, and reboot.

Finally, the server's booted and has a network connection. And this is why I no longer install arch on any of my boxen. I can't trust it to reboot without throwing a hissy fit and killing itself.

TL;DR Arch package issue sends me on an hour-long crusade to make a box boot again.

So for the millionth time, I'm starting a blog. Hopefully, this time I'll actually have stuff to write about. As a bit of an introduction, I'm Maff. I'm a computer science student, sysadmin and server support drone from Scotland. I'm also a bit of a designer, a developer and a security dork. I also love videogames, music and a whole host of other things.

I'm starting this blog in part because I want somewhere to document my pursuits (software or otherwise), but also as a place to post about the funny things that happen in real life, and as a place to rant. I may also post about the software I write, from time to time - I'm a semi-active developer over at github, and I'm the gentoo package maintainer for Monitorix.