New year, new job.

January 18, 2012 Leave a comment

So it’s a new year and this is the obligatory ‘Beginning of the year’ post.  It’s been put off for a while because well I’ve been a bit busier than normal.  The main reason for that is that I’m a few days into a new job.  I’ve left AirDat for a sysadmin position at Glencoe Software.

Read more…

Categories: Ramblings Tags: , ,

The 7 Deadly Sysadmin Sins

December 6, 2011 Leave a comment
We’ve all been there, needing to fix something and not having the time, energy, etc to fix it the right way.  Sometimes this is the easiest way to avoid a massive rabbit hole.  But you know it’s wrong you do it and keep going but part of you dies inside.  The 7 Deadly Sysadmin Sins.
  1. chmod 777
  2. chmod 4755 $file
  3. setenforce 0
  4. echo ” |passwd –stdin root
  5. service iptables stop
  6. echo ‘reboot’ > /etc/cron.daily/
  7. curl http://randomwebsite/ | bash

The last one bugs the crap out of me when good software developers assume this is a valid way to install software (outside of your personal machine).

Running a puppetmaster in ec2

November 30, 2011 1 comment

I struggled with this for a few days before figuring it out, so I’ll post it here in hopes it saves someone a few minutes.  When you install puppet and start the puppetmaster (webrick or rack-enabled) it generates a ssl cert for that machine and also generates a CA that you will use to sign all of your clients.

Recent versions of puppet do not add subjectAltNames to the server certificate when it’s generated by the puppetmaster process.  This means that if you do not use the same name as your masters hostname to connect to puppet you will get a lovely cert mismatch.  I posted a question on serverfault about this (here).  It looks like the common practice for EC2 in particular is to use a uuid as the certname for each puppet client.  This avoids name collisions and problems with hostnames changing everytime the instance is rebooted.  It’s a little harder to keep track of since they aren’t very easy to remember, so caveat emptor.

Read more…

Categories: Sysadminery Tags: , , , , , ,

The psychology of server naming

November 15, 2011 Leave a comment

found on -- thanksOne of the more hotly debated topics among sysadmins is what to name servers.  Some people use this as an outlet for their creativity or pop culture references.  Servers named after Lord of the Rings characters, super heros, greek mythology abound.  There’s a strong push from those of us who have moved past the ‘clever’ phase of our careers to name machines in logical consistent manners.  web0X, db0Y, rackXpduY, all get bandied around and are debated with often the same fervor as Vim vs Emacs (vim for the record).  The truth, sadly is that all the good names are taken ™©®.  The grizzled veterans who’ve done time on a VAX will exclaim, “This naming scheme is crap, lets just use IPs, they are immutable.”  Well for one they are wrong, ips are not immutable.  Take a look at EC2, have fun with that.   Second humans are bad at remembering numbers, 10 digits is the longest number most people can retain (why phone numbers are that length) and usually not for very long.

Names also provide a very important psychological edge for our poor meat brains.  Names allow us to recall information in a similar way that a key allows you to recall information from a database.  A message from  your alert system saying ‘Alert is DOWN!’ would (in theory) trigger something in your memory.  Gandalf is a wizard, that’s the master DNS server!  This key isn’t as good as a more meaningful name but it’s a key none the less.  I prefer names which are functional and overload information into the rest of the domain. tells me very quickly this is a proxy server, it’s one of a multinode cluster, likely load balanced, and is located in Atlanta.  All of this allows me to asses the situation at hand faster.  All of the pertinent details should be written down in a wiki, or some other document source, but the naming gives me a fast way to access that without having to go look it up. is DOWN only tells me something is broken, not how important, how impacting or anything about it.  Maybe that’s a dev box or 1 node in a 40 node cluster, but I don’t know that (unless I just memorize it which stresses the meat brain) until I look it up.

Consistency is the key, I don’t in general like ‘clever’ names not because they are unprofessional or silly, but because they only mean something to the person who came up with it.  I know why I named the database ‘pearl’ (bonus points to anyone that guesses), but my other team members might not and likely that someone coming behind me wouldn’t either.  I’m a huge fan of code names and clever names for software / service names / etc just not machine names.  Here are some of the conventions I use.

Multinode clusters are numbered 2 digit starting at 01.

10 servers in a web cluster, web01 – web10.  Using 2 digit precision gives you 99 machines before you end up changing field sizes.

Short hostnames are the most common functional purpose.

Sometimes it’s ok to call it a server and put more information into the sub domain.  ‘Web’ in general sucks, it’s too generic and means very little, what does it ‘do’.

If you think they’ll be more than one, name it 01.

Don’t use a sequential numbering system for unrelated things.

If you have two webservers that serve different content/services/etc don’t name them web01 / web02.  This creates a logical grouping of those two machines which are not actually tied together from a service standpoint.  I’ve heard of shared filesystems being named fs01, fs02, fs03, fs04, etc.  They aren’t related other than that they are all shared filesystems, why are you grouping them into something that looks like a cluster.  People assume that 1 is related to 2 to 3 to 4.  Put some thought into it and give it a name based on what it does or what’s important about it.

Use A / B notation for duality relationships.

I name my netapp filers: filer01a / filer01b.  They are both addressable services but provide failover for each other.  There will never be a ‘c’ since netapp doesn’t support wheel based failover.  They are a matched set, so they are named as such.  A vs B gives less cardinality than 1 vs 2 and that’s a good thing.

Use subdomains in a consistent manner to produce a lightweight hierarchy of information. lets me denote physical location, security context (internal), content type (www), and functional purpose (proxy) all in one name.  Granted this assumes a high degree of machine / service separation and may not work for everything, but you can use that name to store quickly accessible information.

Order is important, remember that.

In english we read left to right.  Information is ordered in that direction as well.  Put the thing you care about most (or quickest) to the left and less immediate information flows to the right).  tells me it’s a database (important!), it’s part of a cluster (less important than being a db but still relevant), HR database (eeek will I get paid?!), and finally location which may not matter (alt is a backup site, I can deal with that later).  Order frames your response into the correct context. tells me this machine is a database (important), in Atlanta (wait that’s the dr site I might not care right away), it’s HR (wait we don’t have a primary b/c it died last week), and that it’s a clusternode.  Is this better or worse?  Depends on the context, order is important.

The crux of the whole point is that names are useful things, humans name things not because they want to be clever but because it’s an effective way to partition information about something without having to memorize it all.  It comes down to the difference between knowing something and memorizing it.  You design a convention and stick to that convention until it doesn’t work, then you redefine that convention.  The convention saves you time but only if everyone ‘gets’ the convention or it can be easily explained.  If your convention is a complicated scheme involving lollipop guild chairmen’s you are requiring the audience to have immediate intrinsic knowledge of Mid 1930’s Judy Garland films, which is the same as asking them to look it up.

The horrible state of ruby in a production environment.

November 5, 2011 6 comments

As a long time perl guy I was attracted by ruby.  It’s very perl like method chaining is extremely useful and intuitive.  I like ruby, but the state of  ruby applications in a production environment is horrible.  There are plenty of really good tools out there for ruby developers.  Vagrant, sahara, bundler, capistrano, etc the list goes on.  RVM and rbenv are two really good alternatives for maintaining your development environment in a sane manner.  We are in the stone ages when it comes time to go to production.  Distro support for ruby is shaky at best.  Most places are still running centos/rhel5 which leaves them with ruby 1.8.5 or if lucky ruby 1.8.7.  If you’ve upgraded to a rhel6-ish you’re fortunate enough to get wait… ruby 1.8.7.  I’m not as familiar with debian but I’m fairly certain it’s 1.8.  Ubuntu has an available 1.9.1 package but that’s officially a beta version, plus most indications is that it’s extremely buggy.  As of writing ruby is on 1.9.3.  When it comes to rubygems the situation is even worse.  Most ‘best practices’ recommend managing everything with gems.  This introduces a world of pain especially when you start installing things that are based on ruby but provide shell level commands (rackup, unicorn, etc).  Now you’ve got two package managers trying to determine the state of a system, but one of them only knows about one part of it.  It’s a mess. Read more…

puppet and vim

March 30, 2011 2 comments

I’ve been using vi/vim for nearly my entire professional life, and most of my computer life as well. I gave emacs an honest try for a couple of days a few years ago but just couldn’t grok the shortcuts and make it feel natural. Recently I overhauled my setup on my laptop and in specific tuned to to what I generally spend a lot of development time on… puppet.
Read more…

oh hey, look! New blog…

February 2, 2011 1 comment

I make no promises, I probably won’t update this iteration any more frequently than I did my old blog.  But i am going to try, whatever that’s worth.  Why the move?  A couple of things.  Mainly I got really tired of having to upgrade wordpress every time I wanted to use it.  So I’m lazy and will let do it for me.  This also frees up my slicehost for some other things I want to do with it, hopefully the results of that will end up here.  My tiny slicehost was getting hammered by bing and google whenever they did indexing.  Mysql would freak out apache would refuse to keep up and things would spiral.  It was annoying.  If you read this regularly (all 2 of you) you saw a ‘shoo’ page up for a while, this was me seeing if the spider/mysql interaction was the culprit of my slicehost panicking about once a week.  Turns out it was, or it’s plausible enough to be a reason.  I never really liked running mysql just for a tiny blog on a little server anyway.  So here we are.  I’m actually going to try to blog more, now that my job is new and exciting and I’m doing some interesting things rather than trying to figure out the best way to gouge my eyes out during a meeting I hopefully will have some interesting (at least to me) things to talk about.

If I’ve done something horrible to the rss feed or anything let me know in the comments.  I’ll likely leave the redirect around for a while, I doubt that I’ll shell out for the domain premium, especially considering want’s you to move registration rather than just an A record…. Hello? No.

The new address is hopefully I’ll do more with it.

Categories: Ramblings Tags: , , ,