At my current job, I take care of all things systems. That’s infrastructure, security, networking, in-house desktop support, and automation. That’s a lot, I know, but someone has to do it. Nonetheless, it’s a great job. I’ve had a lot of jobs in my life, and this one is definitely top three. That being said, there are a bunch of a-holes in China making my life very hard right now by constantly running DDOS (dynamic denial of service) attacks against our IP address space. They particularly like to target our Joomla and WordPress sites. I’ve been fighting with them off and on for a few weeks now, but this morning, they launched the mother of all attacks against us. This was a geographically distributed, multi-datacenter DDOS attack. They were using in excess of 100 IPs in separate subnets scattered across at least five countries. So how did I fix this? Let’s get into it.

My predecessor at my current company used a platform called check_mk to monitor our network.  Unfortunately, check_mk has a feature that populates based on network discovery and can be very chatty. check_mk is also very convoluted as it’s built on top of Icinga, which is built on top of Nagios.  When making changes, there were layers and layers of configuration files you had to dig through, at least, in the check_mk instance my predecessor had bequeathed me.  Needless to say, I was not a fan and it wasn’t very efficient.  I understand why they forked to create Icinga.  At the time, Nagios was stagnant.  Since then, I feel like the Nagios camp has progressed significantly.  I also understand why they forked Icinga to check_mk, but it’s not for me.   For the granularity I want in monitoring, check_mk would be more work intensive than Nagios.

I’ve been a Linux Systems Administrator for more than two decades, and I’m intimately aware of what’s necessary to run Linux in the enterprise. I knew after the viral epidemic that is systemd, that the Linux ecosystem was inherently broken, and we needed a change. At that point, I sought out a group of folks identifying themselves as the VUA, or Veteran Unix Administrators. This group is vehemently opposed to systemd and began a fork of Debian without systemd, now known as Devuan (pronounced dev one.) Devuan is still in it’s infancy, only one major release under it’s belt, but it’s a massively capable server platform. My company is now running solely on Devuan Linux for all of our LAMP stack applications. We are even using it for virtualization using vserver. On the desktop side of things, it’s coming along quite nicely as well, but there are definitely some growing pains. My company recently purchased a Dell XPS 15 (9560, 2017) for me, and it came with Ubuntu. I figured it was time to “put my money where my mouth is,” and try to install Devuan on this very new hardware that uses an NVME disk and boots via UEFI. This is a journal of that experience. I’m doing most of this from memory, so there may be some minor typos and/or mistakes. Feel free to reach out if something doesn’t seem correct.

As a veteran Unix/Linux Systems Engineer, I get the question everyday, “What is the best Linux distro?” First, I can tell you, there is no best Linux distro; each of them has their strengths and their weaknesses. Secondly, that is the wrong question. The question you should be asking: What is the right Linux distro? I can tell you that there is no right answer here.

The Wall Street Journal published an article by Rachel Shannon-Solomon stating that DevOps is not ready for the enterprise. You can, and should, see that article here. In my opinion, this entire article is a plethora of ridiculous statements. Not only is DevOps built for the enterprise, it’s ready now; enter DevOps 101: CAMS.