1. chmod 777
2. chmod 4755 $file
3. setenforce 0
4. echo ” |passwd –stdin root
5. service iptables stop
6. echo ‘reboot’ > /etc/cron.daily/fix-hanging-db.sh
7. curl http://randomwebsite/foo.sh | bash

The last one bugs the crap out of me when good software developers assume this is a valid way to install software (outside of your personal machine).

Recent versions of puppet do not add subjectAltNames to the server certificate when it’s generated by the puppetmaster process.  This means that if you do not use the same name as your masters hostname to connect to puppet you will get a lovely cert mismatch.  I posted a question on serverfault about this (here).  It looks like the common practice for EC2 in particular is to use a uuid as the certname for each puppet client.  This avoids name collisions and problems with hostnames changing everytime the instance is rebooted.  It’s a little harder to keep track of since they aren’t very easy to remember, so caveat emptor.

First get puppet installed.  I tend to use gems (even though I despise them) since they update much faster upstream than anything else.  Do NOT run puppet or start the puppetmaster.

Generate a uuid or pick some string/name/moniker that’s going to be uniq and consistent.  (uuidgen to get a uuid)

Setup a basic puppet.conf, rpm installs will do this for you, gem installs you are on your own.

[main]
logdir = /var/log/puppet
rundir = /var/run/puppet
vardir = /var/lib/puppet
ssldir = $vardir/ssl
pluginsync = true
server = puppet
environment = production
certname = ENTER_UUID_HERE
dns_alt_names = puppet
report = true

You can also manually generate the cert with the following.
puppet cert generate --dns_alt_names puppet ENTER_UUID_HERE
Your master cert will have the subjectAltNames field now, This is all addressed in http://projects.puppetlabs.com/issues/10739 and should be fixed in the next puppet release.

The concept of using the uuid for ec2 type instances is sound and will prevent ssl headaches in the future.

Names also provide a very important psychological edge for our poor meat brains.  Names allow us to recall information in a similar way that a key allows you to recall information from a database.  A message from  your alert system saying ‘Alert gandalf.example.com is DOWN!’ would (in theory) trigger something in your memory.  Gandalf is a wizard, that’s the master DNS server!  This key isn’t as good as a more meaningful name but it’s a key none the less.  I prefer names which are functional and overload information into the rest of the domain.  proxy01.atl.example.com tells me very quickly this is a proxy server, it’s one of a multinode cluster, likely load balanced, and is located in Atlanta.  All of this allows me to asses the situation at hand faster.  All of the pertinent details should be written down in a wiki, or some other document source, but the naming gives me a fast way to access that without having to go look it up.  172.14.2.1 is DOWN only tells me something is broken, not how important, how impacting or anything about it.  Maybe that’s a dev box or 1 node in a 40 node cluster, but I don’t know that (unless I just memorize it which stresses the meat brain) until I look it up.

Consistency is the key, I don’t in general like ‘clever’ names not because they are unprofessional or silly, but because they only mean something to the person who came up with it.  I know why I named the database ‘pearl’ (bonus points to anyone that guesses), but my other team members might not and likely that someone coming behind me wouldn’t either.  I’m a huge fan of code names and clever names for software / service names / etc just not machine names.  Here are some of the conventions I use.

Multinode clusters are numbered 2 digit starting at 01.

10 servers in a web cluster, web01 – web10.  Using 2 digit precision gives you 99 machines before you end up changing field sizes.

Short hostnames are the most common functional purpose.

Sometimes it’s ok to call it a server and put more information into the sub domain.  ’Web’ in general sucks, it’s too generic and means very little, what does it ‘do’.

If you think they’ll be more than one, name it 01.

Don’t use a sequential numbering system for unrelated things.

If you have two webservers that serve different content/services/etc don’t name them web01 / web02.  This creates a logical grouping of those two machines which are not actually tied together from a service standpoint.  I’ve heard of shared filesystems being named fs01, fs02, fs03, fs04, etc.  They aren’t related other than that they are all shared filesystems, why are you grouping them into something that looks like a cluster.  People assume that 1 is related to 2 to 3 to 4.  Put some thought into it and give it a name based on what it does or what’s important about it.

Use A / B notation for duality relationships.

I name my netapp filers: filer01a / filer01b.  They are both addressable services but provide failover for each other.  There will never be a ‘c’ since netapp doesn’t support wheel based failover.  They are a matched set, so they are named as such.  A vs B gives less cardinality than 1 vs 2 and that’s a good thing.

Use subdomains in a consistent manner to produce a lightweight hierarchy of information.

proxy01.www.internal.nyc.example.com lets me denote physical location, security context (internal), content type (www), and functional purpose (proxy) all in one name.  Granted this assumes a high degree of machine / service separation and may not work for everything, but you can use that name to store quickly accessible information.

Order is important, remember that.

In english we read left to right.  Information is ordered in that direction as well.  Put the thing you care about most (or quickest) to the left and less immediate information flows to the right).  database01.hr.alt  tells me it’s a database (important!), it’s part of a cluster (less important than being a db but still relevant), HR database (eeek will I get paid?!), and finally location which may not matter (alt is a backup site, I can deal with that later).  Order frames your response into the correct context.  database.atl.hr.clusternode1 tells me this machine is a database (important), in Atlanta (wait that’s the dr site I might not care right away), it’s HR (wait we don’t have a primary b/c it died last week), and that it’s a clusternode.  Is this better or worse?  Depends on the context, order is important.

The crux of the whole point is that names are useful things, humans name things not because they want to be clever but because it’s an effective way to partition information about something without having to memorize it all.  It comes down to the difference between knowing something and memorizing it.  You design a convention and stick to that convention until it doesn’t work, then you redefine that convention.  The convention saves you time but only if everyone ‘gets’ the convention or it can be easily explained.  If your convention is a complicated scheme involving lollipop guild chairmen’s you are requiring the audience to have immediate intrinsic knowledge of Mid 1930′s Judy Garland films, which is the same as asking them to look it up.

Several of the things I played with were adding a lun to a machine and getting it to show up without rebooting.  Translating docs gleaned from the web to my configuration was a bit tough at the beginning because we have a highly redundant fabric.  That meaning we have 2 HBAs in each host each with 2 fiber paths.  What this means is that when I get luns to show up I see them 4 times for each lun.  Apparently most people that write about their SAN experiences do it with a single path to their storage device through the fabric.   I also went through the rigamarole of removing a lun from a host (again without rebooting).  All in all it was pretty clean, a series of echo’s to the /sys subsystem, not nearly as ugly as adding and removing actual scsi devices.  It was also completely non-disruptive to other luns and overall performance.

There’s been quite a bit of debate among the other SAs at work about how we should handle luns at the host level.  Originally the thought was to add LVM on top of the LUN (which with multipath is kinda a bear) then create the filesystem on top of LVM.  The thought was originally that this would enable us to grow and shrink as needed and give us a similiar flexibility to the NFS volumes we are so used to dealing with.  Turns out shrinking is still iffy.  I’ve tried it twice now and had catestrophic failures both times.  The filer seems to handle it fine, but the host just flat out fails to see it as a valid filesystem once the lun gets smaller.

With multipath configured correctly what we see with an fdisk -l is 5 new ‘disks’.  sdX – sd{X+4} and dm-X.  So depending on how many existing scsi devices (including other luns) we have sde,f,g,h and dm-0 (assuing a, b, c already existing and no other san luns).  What’s a little confusing is that each of these devices is the same disk, you don’t want to use the sdX devices for anything (unless it’s a onetime operation) in case you lose a path.  So you do everything to the dm-X device created by multipath.  The other confusing thing is that while these are ‘disks’ they also are not.  You can create partitions on them but you don’t really need to, so it kinda confuses your brain in what you are used to dealing with.

So the original plan with LVM was to create a partition consisting of the entire disk, add it as a PV, create a volume group, then a LV on the volume group of the whole size.  It struck several of us that this really was overkill.  Where LVM shines is when you have lots of descrete storage objects and you want to group them all together.  Logically this ‘thing’ is a single lun where all the physical abstraction is already done (with about 4 levels of abstraction in the case of NetApp).  The other alternative, which I ended up doing for this particular implementation, was to just create a filesystem right on dm-0.  I didn’t create a partition, didn’t do LVM, just mkfs.ext3 /dev/dm-0.  Worked like a charm, no wasted space, very simple.

There is a gotcha though.  Multipath has the annoying habit of renaming the multi-disk device (dm-X) when the host reboots and it encounters additional luns.  So if you add a lun to a machine that already has one then reboot it’s possible, nigh on likely, that they will swap dm-1 and dm-0 to the opposite of what you expect.  This is pretty annoying from a mounting standpoint.  This is one potential winning point for LVM, since the LVM data is written to the disk itself you can have a consistent name which to use in fstab etc.  But all that overhead just for a consistent name?  Am I really getting anything else out of LVM in this scenerio?

Enter ext2/3 labels.  Most SAs I know don’t like labels because if you do things like label a disk ‘/’ and try to put it in another machine for recover purposes, you probably won’t get the disk you expect (it’ll depend on bus order).  However labels give us a way to consistently name a dm device regardless of what multipath wants to call it.  This also lets me give meaningful symbolic names to SAN disks that may move hosts (oracle volumes is their current use, so there are 2, 1 for primary and 1 for standby).  So I use e2label /dev/dm-0 FOO to label my san disk.  Then in fstab I use LABEL=FOO. An interesting side effect is that df output shows the uuid of the disk rather than it’s multipath name, but other than that it seems to work.

Next I need so spend a bunch of time with a non-critical volume and figure out all the ins and outs of growing and (maybe) shrinking the fileystem.  All of the above work was done on a RHEL5 system (64bit), my feeling is that all bets are off when it comes to RHEL4 and LVM might be a very real hard requirement.  I also wonder if multipath is the right way to go.  Would it be possible to use LVM to create a fault tolerant storage device?

This was my second trip to LISA, I went last year in Dallas.  This year was definatly better.  It being in San Diego meant not only the the weather was fantastic, but there was also lots to do outside of the conference.  I think that particular locale made it easier for people on the west coast to come.  For those that haven’t been, LISA consists on basically 2 ‘tracks’, which you can mix and match.  The technical sessions, which are later in the week and the training sessions, which run the entire week long (up to 2 per day).  This year I took a full week of training.

All in all the training is good.  There’s a bit of a trick to it.  You have to take classes on things that you don’t know anything about because they tend not to be very advanced.  Also from year to year the training doesn’t seem to improve much, many of the instructors are using their same slides from the previous year (or further back).  The Technical sessions are more like 2 hour lectures on a particular topic.  They are not geared toward training but more toward presenting.  I’ve never found them that interesting.

By far where LISA wins out is what’s called the ‘Hallway Track’.  This is basically the hobnobbing in the halls with other SAs and going out to dinner and all the conversations that occur as a result of that.  Each year I’ve found that amazingly useful.  Most SAs operate in a bit of a vacum, they may have a small (or large) team, but there’s a heirarchy there.  If they are in a solo shop they don’t usually have someone to bounce ideas off of.  Even if they do company size, budget, etc limit the focus.  The people you meet with at LISA cover all sorts of areas from the large financial sector, to academia, to startups.  It’s those different perspectives that are really valuable and why I go back each year.

I am however disappointed that LISA has been shrinking a bit, they need to improve their training and try to attract different groups of SAs.  I think alot of small shop SAs don’t come to LISA because it has this air of being only for really large installs.  If you are serious about your craft as an SA you hope to one day be in a large install, either by growing the one you have or moving into one.  I think it’s important for small SAs to attend a conference like this to learn about techniques they could implement before they get too big.

Another point of disappoitment is with the vendors.  As an employee for what easily can be described as a vendor we don’t have a presence at LISA (except as attendees).  There were probably only 25 vendors this year and most of them were in the SAN space.  SUN was there, but they didn’t really seem into promoting their product.  FreeBSD was there as well, but they are kinda preaching to the choir with their userbase if you ask me.  Fedora had no presence there, Red Hat had nothing.  Where were all the security product vendors?  Where were the groupware vendors?  Splunk always shows up and does a good job.  In fact I started using Splunk last year after talking with their sales guys at LISA ’07. Alot of people at LISA are either buying or making the technical recommendation and indirectly buying the products that their companies will use.  Reductive Labs personally has LISA to thank for at least one contract.  I wouldn’t have bought a support contract with them if I hadn’t met Luke and had some really good discussions with him.

The after hours stuff at LISA is also excellent.  They have BoFs (Birds of a Feather) sessions about a wide range of topics.  Anyone can run a BoF, they are meant to be informal roundtable discussions about a particular topic.  They often stray off the technical, there was a hockey bof last year, and their’s usually a semi-secret scotch bof every year as well.

Next year LISA will be in Baltimore.  I will likely go because it’s so close and I get alot out if each year.  Even when I don’t learn alot of training, being around a bunch of other SAs and spending a week discussing methods and other aspects of SA always invigorates me and gets me all fired up for when I return to work.  It’s kinda like a career reset.  When your job has you beaten down and you’re burned out sometimes a vacation helps release that pent of stress.  Sometimes however getting away from work but staying technical and learning about other ways of doing your job can be an even bigger boost.

This is such a horrible approach.  It almost immediately leads to spaghetti and data bleed.  Nowhere else in computer science that I know of is this approach even attempted.  Almost everyone tries to come up with some way to replicate this type of functionality into their configuration management system.  They think that this will save them time and allow them to generalize their configs in a sane way.  What it really results in is a mess of what comes from where and when you need to make changes where a particular stub should go.  Almost immediately you’ve created confusion, and created an inheritance that is arbitrary at best.

I think the problem actually stems from a top down approach to config management.  People seem to think that consolitation occurs from the top down, when in reality it comes from the bottom up.

The way I approach this problem is to come up with a way to model your changes as individual configs that are additive from the bottom up.  Apache is actually a very good example of this.  If you look at the add on components, (mod_perl, mod_python, squirrelmail, etc) they add in their own configs that add up to a global configuration for a single apache server.

Take sudoers for example.  People always want to have some concatenation scheme for sudoers so that they can have global configs that everyone inherits then a role (say webserver) config that gets picked up, then host specfic configs that allow for one off host configurations.  A better approach is to be able to add single config records for each thing you need and build a total config, rather than start from the top and work down, start at the bottom and build up.

In the scope of puppet, this is done rather nicely with a type.  Types allow you to add a single record in the appropriate place and not rely on any type of inheritance to determine where and if a configuration lands on a given server.  In the example above I would have a sudo type that I would put at each class level to only add the sudo config for that class.  I don’t have to concern myself with what level of inheritance I’m at (global or otherwise)  or what order things are in because I’m just inserting a configuration record rather than a trying to manage a whole section of a file.