This is the fourth profile in a series on Linux Foundation system administrators leading up to SysAdmin Day. Do you have a super-hero sysadmin you’d like to recognize? Send your nomination to This e-mail address is being protected from spambots. You need JavaScript enabled to view it
by July 25 and enter them to win a free ticket to LinuxCon and CloudOpen North America taking place in Chicago August 20-22, 2014. (See the full contest announcement for more details.)
Linux.com: How long have you been a sys admin?
I set up my very first Linux system in 1998. It was a combination web server and mail server running on an old beige tower and served the non-profit where I worked at the time extremely well. I didn’t know enough about systems administration at the time to keep sendmail patched, so six months later the system was also serving “warez.” Needless to say, that catapulted IT security to the forefront of my interests.
When did you start at the Linux Foundation and how did you get the job?
I started in November 2011. I was working at McGill University InfoSec at the time, and was also active with Fedora Project — which is how my name showed up on the list of candidates. The Linux Foundation was looking for a systems administrator with a strong background in IT security — who would also be a good fit for a decentralized team of passionate open-source advocates. I’m extremely glad I was a good fit for the position, as I can’t imagine receiving as much satisfaction from any other job.
What do you do for the Linux Foundation? What’s your specialty?
I’m part of the Collaborative Projects IT team, which is responsible for providing IT hosting for projects like kernel.org, codeaurora.org, opendaylight.org, allseenalliance.org, and a few others. The requirements are different for each project, so our specialty is working with the developers involved in each one of them and finding solutions that would satisfy their needs. We provide a full development stack, from git repository hosting to continuous integration and release distribution. We try not to specialize in any one thing, but we
certainly have areas where each member of the team has higher expertise than others. I am usually the go-to resource for SELinux policies and other process confinement questions.
Will you describe a typical day at work for you?
We are very client-oriented, so we try to give priority to incoming developer requests. We try to automate as much as we can, but a lot of things still require admin involvement, especially with regards to continuous integration (CI) systems. Besides that, there’s always documentation to be written or updated, or some system refactoring to do. We’ve added a number of very large projects in a very short period of time throughout the last 18 months, so some of the decisions we’ve made along the way require rethinking in order to allow us to scale better. Unfortunately, once you have production systems in place, refactoring requires very careful planning — otherwise you risk downtime that would interfere with project timelines.
What’s your favorite part of the job/ thing to do and why?
Every sysadmin’s guilty pleasure is writing a tool that would automate away a menial task. I call it “guilty pleasure” because sometimes the total amount of effort spent on writing such tools goes vastly beyond what it would take to continue doing things manually (obligatory xkcd link: http://xkcd.com/1205/). However, sysadmins hate being distracted by small requests (we have “the zone,” too, just like programmers: http://heeris.id.au/2013/this-is-why-you-shouldnt-interrupt-a-programmer/), which is how we justify our efforts. Thankfully, we don’t have to justify them to management — only to fellow members of the sysadmin team.
What is your nightmare scenario? How have you prepared for it?
Well, if we’re prepared for it, then it’s no longer the nightmare scenario. We try to introduce lots of redundancy into the infrastructure to assure that there are as few single points of failure as possible, but at some point you have to do risk-benefit analysis and draw the line past which such redundancy becomes prohibitively expensive either in terms of money or effort. For example, if there’s a major natural calamity on the North American West coast, we’ll definitely be impacted, since most of our infrastructure is in Portland, Oregon.
However, we will be able to recover without any data loss due to off-site backups in Montreal. A more realistic nightmare scenario would be a systems failure while the sysadmins responsible for that particular part of the infrastructure are not reachable, which is why we try to keep ample documentation. Oh, and the rule is that we don’t all fly to Linux Foundation events on the same days, so as not to all be in the air at the same time.
What is your favorite sysadmin tool and how do you use it?
I don’t really have a favorite tool — not any more than a plumber would have a favorite wrench (yes, I think plumbers are a great analogy for sysadmins). I do have a preferred set of tools that I use every day, but I expect it’s similar to what most other sysadmins keep in their proverbial tool belt.
What’s your favorite story about working at the Linux Foundation?
The thought that we are in a position to tell Linus what he can and cannot do on the Linux systems we manage makes me giggle every time – because it’s so wonderfully absurd. However, I would say my favorite bit is how many companies are so eager to help you out if they find out that you are working for the Linux Foundation. We rely on the goodness of a great number of donors to run our infrastructure — most of whom volunteered their services without any prompting on our part. My thanks go out to Google, Rackspace, ISC, Intel, Vexxhost, Yubico — among many others — for helping us with hardware and hosting. And, of course, to all the generous members of the Linux Foundation. You make it possible for us to run infrastructure where Linux development itself happens.
What do you do for fun, in your spare time?
I’m a (human) language nerd, so my favorite hobby is learning how to read books over and over again in different languages. It’s like crossword puzzles on a nightmare level. I can read fluently in four languages (English, Russian, French, and Spanish.) When I don’t have spare brain cells left, though, I enjoy walking, biking, and being generally silly with my family.
Read more about Linux Foundation System Administrators:
Linux Foundation SysAdmin Michael Halstead’s IT Career Started at Age 15
Linux Foundation SysAdmin Andy Grimberg Loves New Tech and Snowboarding
To Linux Foundation SysAdmin Ryan Day, Elegance is the Best Tool