Why did I leave?  There are a myriad of reasons, most best discussed over beer or scotch.  I liked AirDat and it was what I needed at the time.  I learned a lot about HPC and weather forecasting and worked with some really cool, really smart people.  Ultimately I got to the point where I needed to have some different challenges and be more involved in some of the decisions that were being made.   I wish those guys the best I really hope they succeed, they have some killer technology that could prove to be really useful if they can get everything to fall into place.

At Glencoe I will be their sysadmin supporting not only the corporate endeavors for our products but also supporting and working on the OME and it’s other open source projects.  It’s exciting to get back into the open source world, as well as dive into the informatics world and the challenges that lie within.

Another big change with the move to Glencoe is that I’m now working from home full-time.  I’ve converted part of my house to a home office and the family and I are trying to adjust to this as well as me working closer to UK/GMT time.  It’s something I’ve been wanting to do for some time and I’ve finally got the opportunity and plan to make the most of it.

1. chmod 777
2. chmod 4755 $file
3. setenforce 0
4. echo ” |passwd –stdin root
5. service iptables stop
6. echo ‘reboot’ > /etc/cron.daily/fix-hanging-db.sh
7. curl http://randomwebsite/foo.sh | bash

The last one bugs the crap out of me when good software developers assume this is a valid way to install software (outside of your personal machine).

Recent versions of puppet do not add subjectAltNames to the server certificate when it’s generated by the puppetmaster process.  This means that if you do not use the same name as your masters hostname to connect to puppet you will get a lovely cert mismatch.  I posted a question on serverfault about this (here).  It looks like the common practice for EC2 in particular is to use a uuid as the certname for each puppet client.  This avoids name collisions and problems with hostnames changing everytime the instance is rebooted.  It’s a little harder to keep track of since they aren’t very easy to remember, so caveat emptor.

First get puppet installed.  I tend to use gems (even though I despise them) since they update much faster upstream than anything else.  Do NOT run puppet or start the puppetmaster.

Generate a uuid or pick some string/name/moniker that’s going to be uniq and consistent.  (uuidgen to get a uuid)

Setup a basic puppet.conf, rpm installs will do this for you, gem installs you are on your own.

[main]
logdir = /var/log/puppet
rundir = /var/run/puppet
vardir = /var/lib/puppet
ssldir = $vardir/ssl
pluginsync = true
server = puppet
environment = production
certname = ENTER_UUID_HERE
dns_alt_names = puppet
report = true

You can also manually generate the cert with the following.
puppet cert generate --dns_alt_names puppet ENTER_UUID_HERE
Your master cert will have the subjectAltNames field now, This is all addressed in http://projects.puppetlabs.com/issues/10739 and should be fixed in the next puppet release.

The concept of using the uuid for ec2 type instances is sound and will prevent ssl headaches in the future.

Names also provide a very important psychological edge for our poor meat brains.  Names allow us to recall information in a similar way that a key allows you to recall information from a database.  A message from  your alert system saying ‘Alert gandalf.example.com is DOWN!’ would (in theory) trigger something in your memory.  Gandalf is a wizard, that’s the master DNS server!  This key isn’t as good as a more meaningful name but it’s a key none the less.  I prefer names which are functional and overload information into the rest of the domain.  proxy01.atl.example.com tells me very quickly this is a proxy server, it’s one of a multinode cluster, likely load balanced, and is located in Atlanta.  All of this allows me to asses the situation at hand faster.  All of the pertinent details should be written down in a wiki, or some other document source, but the naming gives me a fast way to access that without having to go look it up.  172.14.2.1 is DOWN only tells me something is broken, not how important, how impacting or anything about it.  Maybe that’s a dev box or 1 node in a 40 node cluster, but I don’t know that (unless I just memorize it which stresses the meat brain) until I look it up.

Consistency is the key, I don’t in general like ‘clever’ names not because they are unprofessional or silly, but because they only mean something to the person who came up with it.  I know why I named the database ‘pearl’ (bonus points to anyone that guesses), but my other team members might not and likely that someone coming behind me wouldn’t either.  I’m a huge fan of code names and clever names for software / service names / etc just not machine names.  Here are some of the conventions I use.

Multinode clusters are numbered 2 digit starting at 01.

10 servers in a web cluster, web01 – web10.  Using 2 digit precision gives you 99 machines before you end up changing field sizes.

Short hostnames are the most common functional purpose.

Sometimes it’s ok to call it a server and put more information into the sub domain.  ’Web’ in general sucks, it’s too generic and means very little, what does it ‘do’.

If you think they’ll be more than one, name it 01.

Don’t use a sequential numbering system for unrelated things.

If you have two webservers that serve different content/services/etc don’t name them web01 / web02.  This creates a logical grouping of those two machines which are not actually tied together from a service standpoint.  I’ve heard of shared filesystems being named fs01, fs02, fs03, fs04, etc.  They aren’t related other than that they are all shared filesystems, why are you grouping them into something that looks like a cluster.  People assume that 1 is related to 2 to 3 to 4.  Put some thought into it and give it a name based on what it does or what’s important about it.

Use A / B notation for duality relationships.

I name my netapp filers: filer01a / filer01b.  They are both addressable services but provide failover for each other.  There will never be a ‘c’ since netapp doesn’t support wheel based failover.  They are a matched set, so they are named as such.  A vs B gives less cardinality than 1 vs 2 and that’s a good thing.

Use subdomains in a consistent manner to produce a lightweight hierarchy of information.

proxy01.www.internal.nyc.example.com lets me denote physical location, security context (internal), content type (www), and functional purpose (proxy) all in one name.  Granted this assumes a high degree of machine / service separation and may not work for everything, but you can use that name to store quickly accessible information.

Order is important, remember that.

In english we read left to right.  Information is ordered in that direction as well.  Put the thing you care about most (or quickest) to the left and less immediate information flows to the right).  database01.hr.alt  tells me it’s a database (important!), it’s part of a cluster (less important than being a db but still relevant), HR database (eeek will I get paid?!), and finally location which may not matter (alt is a backup site, I can deal with that later).  Order frames your response into the correct context.  database.atl.hr.clusternode1 tells me this machine is a database (important), in Atlanta (wait that’s the dr site I might not care right away), it’s HR (wait we don’t have a primary b/c it died last week), and that it’s a clusternode.  Is this better or worse?  Depends on the context, order is important.

The crux of the whole point is that names are useful things, humans name things not because they want to be clever but because it’s an effective way to partition information about something without having to memorize it all.  It comes down to the difference between knowing something and memorizing it.  You design a convention and stick to that convention until it doesn’t work, then you redefine that convention.  The convention saves you time but only if everyone ‘gets’ the convention or it can be easily explained.  If your convention is a complicated scheme involving lollipop guild chairmen’s you are requiring the audience to have immediate intrinsic knowledge of Mid 1930′s Judy Garland films, which is the same as asking them to look it up.

Ruby development in general is moving very fast, most developers are working on 1.9 and likely 1.9.3.  Trying to stay on 1.8 means you may be stuck with libraries/gems/tools that don’t work as well as their 1.9 counterparts.  In addition you have to put up with constant complaints from the development side about why they have to run unfrozen caveman ruby and why can’t they use the latest tool/gem/whizzbang they want.  So now you roll up your sleeves and decide to upgrade your system ruby.  The ‘recommended’ method for updating ruby is installing from source.  I love installing from source, I also love mowing my grass.  They are both mindless pursuits that I start and autonomously go through until I complete them.  There’s no shame in compiling and I don’t shy away from it, but it has its place.  That place is not on a production server.  It doesn’t scale, it introduces error and inconsistencies and isn’t reproducible.  I don’t want any of my production servers to even have a compiler on them if I can help it.  The next best alternative is rvm.  I love the concept of rvm (though not necessarily the implementation, I prefer rbenv there).  But if you tell me to use rvm on my production servers I’m going to laugh at you and ask you to come back when you have your big boy developer pants on.  Now I’m not only building ruby on every machine I have, I’m doing fancy shell tricks to determine my execution environment.  I’m also at the mercy of random files littered in directories for what libraries I can see, what binaries I’m going to use, etc.  How is this anything but a recipe for a long night with cold pizza and a bottle of scotch that ends with updating my resume?  RVM works great when it’s your environment on your laptop, but in my infrastructure it’s just not a viable solution.

What about packages you say.  Excellent question.  I’m a fan of packages, rpm in specific, but I have no objections to debs either (solaris sit down over there, you don’t have packages, you have tarballs there’s a difference).  I’m comfortable building packages, I’ll bust out a specfile to deploy 5 or so bash scripts cause it’s the right thing to do.  If your OS uses packages us it, as much as you can.  Not using your native package management system is like jumping out the second story window because you didn’t want to dent the carpet on the stairs.  Ruby packaging is ugly at least from the rpm side.  The 1.8 specfile won’t cleanly rebuild 1.9 and when you do get it to work, there are all sort of library issues abi compatibility problems and a host of other things.  Not all of this is ruby’s fault.  A fair amount of it lies with people distributing rpms of other things that use ruby but not following the correct dependency management techniques.  At the end of the day that doesn’t matter, building upgraded packages for ruby is non-trivial task.  It’s also a rabbit hole.  Rebuild ruby, well there’s 6 packages there (though in a single spec) as someone decided it was awesome to break each binary out into its own package.  Now rebuild rubygems.  Using anything OS level that depends on ruby, chances are good that spec writer locked the version to 1.8 so now you’re rebuilding that as well.  Don’t forget the random ruby C library extensions as well (ruby-mysql, ruby-shadow, etc).  At a certain point you wonder if it wouldn’t be easier to just maintain your own custom distro or pull your eyelids off with plyers.

Here’s where the real evilness starts to creep in.  You’ve now spent hours, days, weeks trying to work through technical debt and build a standard and repeatable environment to support development and are ready to pull your hair out.  The thought starts to creep in ‘Maybe it would be easier to run rvm, or build it on each box, how much extra work could it be’.  People don’t build crappy infrastructures intentionally, they make one small compromise after another until they are neck-deep in debt they have no idea how to pay off.

At this point the python folks are grinning from ear to ear (eggs and pip and the trouble they represent are another topic for discussion) because they’ve kinda moved past this.  For one python development benefits from not being as ‘trendy’ as ruby/rails and the like.  They also benefit from the fact that redhat engineering, as well as ubuntu engineering are pretty heavy python shops.  They have a vested interest to make sure that python doesn’t suck at the OS level.

So where does that leave us?

1. Build from source / use rvm (there’s no difference between the two other than shell magic) YUCK!
2. Package everything into rpms/debs (a ton of extra work, weird corner case breakage)
3. Something else

Here’s my idea on something else, until the state of distro support of ruby is saner.

• package ruby into its own location (/opt/ruby/$version or something that makes you happy)
• use bundle pack and bundle install –deployment

Repackaging a language it’s own prefix is not my favorite plan but till I can sanely update ruby OS wide it’ll have to do.  You still are going to need to do PATH tricks or edit the shebangs on every script in order for things to work right, but this way we’re not installing ruby 17 times b/c we run 17 apps on a single machine.  I do not install gems here, with 1 exception.  I install bundler.  If you’re working with ruby or supporting ruby and you are not using bundler then you need to be shot (here yeller, here boy… that’s a good dog).

Once you start using bundler, use bundle pack.  This tells bundler to install gem dependencies in the vendor/cache directory of your application.  You should then put Gemfile.lock under revision control (or include it in your deployment packaging).  This will enable you to run bundle install –deployment on your production environment.  The –deployment flag tells bundler to avoid running gem install and use the vendor/cache directory.  This keeps all of your application gems ‘inside’ your application.  Which means you avoid messes with wrongly ‘activated’ gems, accidental version upgrades etc.  You do need to remember to run any commands under bundle exec (like bundle exec rake db:migrate) else you’ll start running into subtle errors, but the trade-off in sanity is worth it.

Hopefully as ruby adoption continues this state of affairs will get better.  Better OS packages are a start.  Standards around deployment like bundler are a must.  I’d really like to see a gem->rpm/deb integration rather than continued fractured directions but I’m not sure that’s much on anyone else’s radar.  It all (almost) makes me pine for the days of installing everything with CPAN.

First plugin I use is pathogen, it’s a must and the only plugin that actually resides in the normal places in vim’s rc directory. Pathogen allows you to define a bundle directory (~/.vim/bundles by default) and will load plugins in whole from that tree. That allows each plugin to have it’s own tree and you don’t get file collisions etc. It also makes it super easy to enable or disable a plugin.

I won’t go into super detail on all the vim’ness that I’ve setup most of it is from this great article. I’m not a pure developer so some of the things I don’t use but the plugins he recommends are spot on.

One of the core tenets in puppet is to not duplicate work, it even gets enforced to a certain degree with not being able to duplicate resources. When you combine this the the recommended module structure and correct use of the autoloader and you will find yourself creating a lot of manifest files. There are two things I use that help with this. One is a template plugin, the other is snipmate. The template plugin comes into play whenever I create a new file, I have a skeleton directory that contains stub for different filetypes. When I edit a new file it reads in that template, interpolates a few tokens and inserts it into my new file buffer. I always work on modules from the top of the module (makes moving easier and helps with the git side of things I think). So in my ssh module when I do:

$ vi manifests/server.pp

I get a file that looks like:

This file didn't exist prior to editing.

If you notice the header comment is mostly filled out, the class declaration is already set and a few other things. How does this work? Well the skeleton file has tokens in it that I have defined (called tags, which is a bit of an overload of the term). That tag can be a simple string or you can use vi’s scripting language to a certain extent to figure things out. In the case of the class declaration I know that I almost always work in the top dir of a module so I know what module I’m in from that. Then the autloader in puppet has very specific rules how how filenames translate to classes so I take advantage of that with a little substitute. The template plugin I use is eteSkeleton, I liked how relatively simple it was, however I did have to fork it and fix a few things with it. The original source is here, my fork is here and my specific tags file is here to see how I did the class logic.

I also use snipmate to create definitions, language constructs etc within manifests. Snipmate lets you type if and get a completed if statement that you can tab through the various elements of. Those familiar with TextMate will recognize it instantly. My snippets file for puppet is here, it was originally done by R.I. Pienaar but I modified it to match my personal syntax style etc.

Syntax highlighting is pretty simple, there’s a puppet.vim that’s included in the puppet source tree, it defines the groups and objects and how to highlight them, then you can control the colors through vim’s colorschemes. I don’t change my colors from the default, I just tweak my terminal settings to brighten up some of the colors (dark blue on black specifically).

ftdetect/puppet.vim handles file detection of manifest files, there’s a lot of things that are keyed off of set filetype=puppet, this just matches *.pp. There’s not really a case where I’m editing a manifest and it’s not a .pp so we don’t have to read lines or anything to determine filetype, if we did that would go here. ftplugin/puppet.vim contains any filetype specific settings I want for puppet.

kp=pi looks a little cryptic. kp is shorthand for kewordprg (or keyword program). This defines the program that vim will run when you press ‘K’ (note the capital) in normal mode. In stock vim if you try this on a word it will run man (cword is vim-speak for word under cursor). pi is actually shipped as part of puppet (at least it’s in the gem install) as an analog to ri (ruby documentation). It acts exactly the same as puppet describe. This means that in a manifest I can position the cursor on say an ‘exec’ resource that I’m writing and hit ‘K’ and it’ll shell out and run pi exec (puppet describe exec) and give me all the resource documentation for that version of puppet (assuming I’m on a box that has puppet installed). This is great when I can’t remember the exact attribute on a resource or similar. My only complaint is that pi doesn’t page when it returns a lot of text (see pi file) and it’s annoying to have to page up to start reading (I run 100% of the time in screen and have my pageup key mapped to copymode). But wait I have a solution. Vim always runs &kp so I can’t really set kp to something with a pipe and pager in it. If you try: :set kp="pi | less -F" you will end up running: pi exec |less -F execas kp always appends . So in my .bashrc I define the following:

function pi() {
  command pi "$@" | less -F
}

This defines a local bash function called pi which calls pi with whatever arguments were passed and pipes it to less -F (-F only paging if there’s more than 1 screen of text). The command construct in bash prevents a loop between the function and the actual command. It also allows the path to be used rather than hardcoding /usr/bin/pi which would have the same net effect. It’s a tad more portable. Now in vim when I cursor over a resource and hit ‘K’ I get nice puppet documentation without having to switch windows or anything. Be aware that since it’s a locally defined function you have to set set shellcmdflag=-ic in your .vimrc else when vim shells out the function won’t be defined. I could also make a shell command in my path somewhere and use that as that adding the ‘i’ flag to the shell causes it to bg vim in certain circumstances as it’s an interactive shell now.

The surround plugin is great for throwing quotes or curlys around things, since puppet uses two different syntaxes for variables when they are quoted vs bare, I use that alot to wrap a variable in curlys. The normal mapping (see surround’s documentation) to do that is: ysiw}. Cryptic and hard to remember? Yup. Basically it’s ‘ys’ (you surround); ‘iw’ (inner word, :h text-objects will change your life), ‘}’. The surround plugin uses the left brace to surround with spaces and the right brace to surround without. I map this to } to make it easy.

NERD_Commenter (a very awesome plugin to do filetype/language specific comments) uses a dictionary for filetypes to define additional comment characters.

let g:NERDCustomDelimiters = {
      \ 'puppet': { 'left': '#', 'leftAlt': '/*', 'rightAlt': '*/' }
      \ }

This defines the comment character (and alternate comment character) for puppet filetypes. Using NERD_Commenter you can switch between shell style and c-style comments. Do things like visually select a block and comment it one swoop, and uncomment sections really easily.

Puppet resources use fat comma’s to specify parameters. The style guide (and I tend to agree) states that all co-located fat commas should be aligned based on the longest parameter in the list. Fixing that is tedious and annoying. This is where the ‘Align’ plugin comes in handy. Align can do a million things to reformat text, but simply for this case I only use it for fat commas. I want to select a block of text with visual select mode (greatest thing in vim ever btw).
Then run :Align =>. This will align everything on those delimiters. I have this mapped to = in my .vimrc. I don’t use tabs so there might be additional settings to get it to align with tabs instead of spaces, but I think it honors expandtab and smarttab as needed.

I have ctags and taglist setup to integrate with puppet, but I’m finding myself not really using it much. I tend to think in the autoloaders terms and just open a new window/etc to what I need to look up rather than use vim to jump back and forth. If I could get it setup such that when I cursor over a statement like:
include ssh::serverand hit ] (which causes a taglookup); it would then take me to where class ssh::server was defined I would probably use it more, but since ‘::’ isn’t a part of a word boundary that doesn’t work. I got the ctags configuration from Nasrat and it works great, I just haven’t really figured out how to integrate it into my coding-workflow. If anyone has any more practical enhancements or advice leave them in the comments.

Syntax checking is a must. I’ve written a couple of shell scripts to wrap around puppet --parseonly to check a whole module, do erb checking etc etc. Then I found syntastic. Syntastic is simply put… fan-fucking-tastic. It supports doing syntax checking based on filetype so you can change things per language. Vim has always had the ‘make’ and ‘makeprg’ settings but syntastic wraps that up in a nice package and gives you a way to extend it. It also provides a function you can put in your statusline to alert you when you open (or save) a file that the file has syntax errors (and where they are).

A quick aside about the statusline. I used to hate it, this ugly white line at the bottom that broke up the visual and flow of the text and really didn’t contain anything useful. Then I discovered how to change it and fill it with all sorts of useful information. All of my statusline settings (with the exception of setting laststatus=2 in .vimrc) are found in statusline.vim.

The code found in syntax_checkers/puppet.vim defines the proper makeprg and does the other syntastic setup. The hard part is getting the errorformat set correctly. Vim often times bails back home to it’s ‘C’ roots and this is one of those times. errorformat (:h errorformat for in depth information) is a pattern that will be matched against the output of the makeprg (in this case puppet apply –parseonly … ). The difficulty is it’s not a regular expression, it’s a scanf expression. scanf is an older c routine for matching single or multiline text and it’s a bit cryptic and confusing. Give me a regex anyday, you combine that with how vim needs things escaped and it was a chore. But it’s done now and hopefully I’ve saved you some time. Puppets parse-only option it’s terribly smart. Once it finds a single error it will stop parsing. I talked to Luke about this and it’s a limitation of the DSL in ruby. The parser can’t really continue once it finds and error so it doesn’t have any way to report on all the errors. The net result is that if you sit down and bang out a manifest really fast then syntax check it, you’ll get the first error, fix it, save see the second, etc etc etc. Still nicer than having to quit, run puppet, edit, fix, and repeat. Be aware that forcing a syntax check each time the buffer is opened does slow vim down a bit (well not really vim, it’s just waiting on puppet), so if you notice it and it bothers you (it doesn’t me) look at syntastic documentation on how to disable it and only turn it on when you want it. I’m toying around with seeing if I can get syntastic to check against running puppet in noop mode to catch things like duplicate definitions and other things that parser doesn’t deal with, but there’s a lot to ignore and filter through and noop mode generates a lot of output since it’s not really changing anything. I’m also not sure if it’s all that useful at that phase, I generally do a lot of sandbox and vm testing anyway which I would still have to do. I’m sure if someone wanted to write a specific parser for manifests that was smarter than parseonly but friendlier than noop there would be beer and scotches all around, but I tend to think most of us are more interested in writing actual manifests than something to check manifests.

That’s all of my puppet specific vim setup, my entire vim tree is online at http://github.com/stick/vimfiles and is mostly documented as to what various things do. All the credit for the various plugins goes to their authors, the vim community is really good, just spend a bit getting to know vim before you jump in.

Happy puppeteering!

If I’ve done something horrible to the rss feed or anything let me know in the comments.  I’ll likely leave the redirect around for a while, I doubt that I’ll shell out for the domain premium, especially considering wordpress.com want’s you to move registration rather than just an A record…. Hello? No.

The new address is http://onastick.wordpress.com hopefully I’ll do more with it.

Wow, this is a hard email to write, one that ultimately I knew someday
I’d write but never really expecting when I would write it.

After 7 and a half years I’m leaving Red Hat.  It was a tough decision
and one that hasn’t been taken lightly.

As with most endings it’s bitter sweet.  I’m excited for my new
opportunity and a what it holds.  I’m sad to be leaving a really great
company, probably more than just a company but truly a culture.  I’m
most sad about leaving many friends and colleagues.  There are a lot of
relationships built over the better part of a decade and while they are
not torn asunder they are changed.

What once was RHN has been long gone, but those that were a part of it
remain, some still at Red Hat in different roles, others moved on to
other things.  RHN was almost a culture within a culture.  That time,
and the relationships that developed out of it are some of the high
points of the last 7 years.

IT never gets enough credit.  There are a lot of really smart, really
passionate people in IT and yet they are vilified at worst and
overlooked at best in most cases.  Next time you congratulate a team on
a project well done or a milestone achieved, remember that in all likely
hood not a bit of it would have been possible if IT hadn’t silently been
doing their job.

I guess there’s a certain amount of irony that despite 7.5 years, 2
departments and numerous managers I’ve never posted to memo-list till
now.  Not that I don’t believe in it.  In fact quite the opposite, it
serves a very important purpose and despite it’s annoyances (and ability
to distract us and fill up our mailboxes), it underlines one of the core
tenets, openness.

So after all these years and with a heavy heart I take my leave,
wishing everyone the best.  What Red Hat does and more importantly
*how* they do it is what makes all the difference in this place.
Without ‘The Red Hat Way’ this place is just a company like any other.
I’ve seen that ‘way’ change over the years and we’ve survived and
thrived as a result.  Hopefully that trend will continue, but it takes
everyone, from the top down to embrace what makes this place special in
order to maintain it.

I hope that everyone, who wishes, keeps in touch.  My last day isn’t
until 12/17 so I’ll be around the building till then.  I’ll still be in
the RDU area (where I’m headed is no secret but better discussed over
lunch or a cold beer).  I’ll still be on freenode, and I can always be
found at stick (at) miscellaneous (dot) net.

First Course

T – PAN-FRIED GULF SHRIMP & CHORIZO SAUSAGE EMPANADA – roasted cumin N.C. sweet potatoes, pear& fennel salsa, dried cranberries, scallion dough, roasted chipotle & black bean jus

C – SMOKED SEAFOOD SALAD – pickled succotash, butternut squash, corn, snow peas, mustard greens & baby romaine, molasses & juniper vinaigrette

Second Course (no pictures )

T – CAJUN PAN-ROASTED CERTIFIED ANGUS BEEF RIBEYE – capellini pasta & pancetta ham casserole, sweet garlic-braised escarole, ratatouille, sweet potato purée, chipotle & mango butter

C – CAJUN-SEASONED DIVER SEA SCALLOPS – pan-fried green tomato & smoked gouda cheese stack, spinach, crispy pork belly, wasabi sweet potatoes, celery root, veal & white truffle jus

Dessert

T – Empire Chocolate Soufflé – classic chocolate soufflé baked to order, topped with powdered sugar, served with chantilly cream

C – Cheese Tray – Drunken Goat, Morbier, Smoked Gouda

I picked up the whole album based solely on this song, the one most people have heard “Kiss With A Fist” is fun and poppy, I haven’t given the whole album a good solid listen yet to make a judgement on the rest but so far Cosmic Love completely carries it.