Home > Ramblings, Sysadminery, technology > The horrible state of ruby in a production environment.

The horrible state of ruby in a production environment.

As a long time perl guy I was attracted by ruby.  It’s very perl like method chaining is extremely useful and intuitive.  I like ruby, but the state of  ruby applications in a production environment is horrible.  There are plenty of really good tools out there for ruby developers.  Vagrant, sahara, bundler, capistrano, etc the list goes on.  RVM and rbenv are two really good alternatives for maintaining your development environment in a sane manner.  We are in the stone ages when it comes time to go to production.  Distro support for ruby is shaky at best.  Most places are still running centos/rhel5 which leaves them with ruby 1.8.5 or if lucky ruby 1.8.7.  If you’ve upgraded to a rhel6-ish you’re fortunate enough to get wait… ruby 1.8.7.  I’m not as familiar with debian but I’m fairly certain it’s 1.8.  Ubuntu has an available 1.9.1 package but that’s officially a beta version, plus most indications is that it’s extremely buggy.  As of writing ruby is on 1.9.3.  When it comes to rubygems the situation is even worse.  Most ‘best practices’ recommend managing everything with gems.  This introduces a world of pain especially when you start installing things that are based on ruby but provide shell level commands (rackup, unicorn, etc).  Now you’ve got two package managers trying to determine the state of a system, but one of them only knows about one part of it.  It’s a mess.

Ruby development in general is moving very fast, most developers are working on 1.9 and likely 1.9.3.  Trying to stay on 1.8 means you may be stuck with libraries/gems/tools that don’t work as well as their 1.9 counterparts.  In addition you have to put up with constant complaints from the development side about why they have to run unfrozen caveman ruby and why can’t they use the latest tool/gem/whizzbang they want.  So now you roll up your sleeves and decide to upgrade your system ruby.  The ‘recommended’ method for updating ruby is installing from source.  I love installing from source, I also love mowing my grass.  They are both mindless pursuits that I start and autonomously go through until I complete them.  There’s no shame in compiling and I don’t shy away from it, but it has its place.  That place is not on a production server.  It doesn’t scale, it introduces error and inconsistencies and isn’t reproducible.  I don’t want any of my production servers to even have a compiler on them if I can help it.  The next best alternative is rvm.  I love the concept of rvm (though not necessarily the implementation, I prefer rbenv there).  But if you tell me to use rvm on my production servers I’m going to laugh at you and ask you to come back when you have your big boy developer pants on.  Now I’m not only building ruby on every machine I have, I’m doing fancy shell tricks to determine my execution environment.  I’m also at the mercy of random files littered in directories for what libraries I can see, what binaries I’m going to use, etc.  How is this anything but a recipe for a long night with cold pizza and a bottle of scotch that ends with updating my resume?  RVM works great when it’s your environment on your laptop, but in my infrastructure it’s just not a viable solution.

What about packages you say.  Excellent question.  I’m a fan of packages, rpm in specific, but I have no objections to debs either (solaris sit down over there, you don’t have packages, you have tarballs there’s a difference).  I’m comfortable building packages, I’ll bust out a specfile to deploy 5 or so bash scripts cause it’s the right thing to do.  If your OS uses packages us it, as much as you can.  Not using your native package management system is like jumping out the second story window because you didn’t want to dent the carpet on the stairs.  Ruby packaging is ugly at least from the rpm side.  The 1.8 specfile won’t cleanly rebuild 1.9 and when you do get it to work, there are all sort of library issues abi compatibility problems and a host of other things.  Not all of this is ruby’s fault.  A fair amount of it lies with people distributing rpms of other things that use ruby but not following the correct dependency management techniques.  At the end of the day that doesn’t matter, building upgraded packages for ruby is non-trivial task.  It’s also a rabbit hole.  Rebuild ruby, well there’s 6 packages there (though in a single spec) as someone decided it was awesome to break each binary out into its own package.  Now rebuild rubygems.  Using anything OS level that depends on ruby, chances are good that spec writer locked the version to 1.8 so now you’re rebuilding that as well.  Don’t forget the random ruby C library extensions as well (ruby-mysql, ruby-shadow, etc).  At a certain point you wonder if it wouldn’t be easier to just maintain your own custom distro or pull your eyelids off with plyers.

Here’s where the real evilness starts to creep in.  You’ve now spent hours, days, weeks trying to work through technical debt and build a standard and repeatable environment to support development and are ready to pull your hair out.  The thought starts to creep in ‘Maybe it would be easier to run rvm, or build it on each box, how much extra work could it be’.  People don’t build crappy infrastructures intentionally, they make one small compromise after another until they are neck-deep in debt they have no idea how to pay off.

At this point the python folks are grinning from ear to ear (eggs and pip and the trouble they represent are another topic for discussion) because they’ve kinda moved past this.  For one python development benefits from not being as ‘trendy’ as ruby/rails and the like.  They also benefit from the fact that redhat engineering, as well as ubuntu engineering are pretty heavy python shops.  They have a vested interest to make sure that python doesn’t suck at the OS level.

So where does that leave us?

  1. Build from source / use rvm (there’s no difference between the two other than shell magic) YUCK!
  2. Package everything into rpms/debs (a ton of extra work, weird corner case breakage)
  3. Something else

Here’s my idea on something else, until the state of distro support of ruby is saner.

  • package ruby into its own location (/opt/ruby/$version or something that makes you happy)
  • use bundle pack and bundle install –deployment

Repackaging a language it’s own prefix is not my favorite plan but till I can sanely update ruby OS wide it’ll have to do.  You still are going to need to do PATH tricks or edit the shebangs on every script in order for things to work right, but this way we’re not installing ruby 17 times b/c we run 17 apps on a single machine.  I do not install gems here, with 1 exception.  I install bundler.  If you’re working with ruby or supporting ruby and you are not using bundler then you need to be shot (here yeller, here boy… that’s a good dog).

Once you start using bundler, use bundle pack.  This tells bundler to install gem dependencies in the vendor/cache directory of your application.  You should then put Gemfile.lock under revision control (or include it in your deployment packaging).  This will enable you to run bundle install –deployment on your production environment.  The –deployment flag tells bundler to avoid running gem install and use the vendor/cache directory.  This keeps all of your application gems ‘inside’ your application.  Which means you avoid messes with wrongly ‘activated’ gems, accidental version upgrades etc.  You do need to remember to run any commands under bundle exec (like bundle exec rake db:migrate) else you’ll start running into subtle errors, but the trade-off in sanity is worth it.

Hopefully as ruby adoption continues this state of affairs will get better.  Better OS packages are a start.  Standards around deployment like bundler are a must.  I’d really like to see a gem->rpm/deb integration rather than continued fractured directions but I’m not sure that’s much on anyone else’s radar.  It all (almost) makes me pine for the days of installing everything with CPAN.

  1. krisnarengga
    November 5, 2011 at 2:09 AM

    Is’t mean ruby is worst ?

  2. November 5, 2011 at 5:03 AM

    or 4. go JRuby.
    Deployment of each app with all the deps it needs is a *big* plus for me; yes you pay a big of memory but the performance boost is often worth it. If you need native code it might be an issue, but compiling native extensions is a massive pain in any case.

  3. November 5, 2011 at 6:21 AM

    Hi. It really looks like a mess, doesn’t it? 🙂

    Two things come to mind, though.

    1st. GitHub > CPAN or similar hubs.

    Every project on GitHub seems to attract enough developers to keep things changing at a fast pace to the detriment of steady maintenance. Plus, new projects being developed all the time, sometimes doing something similar but in a different way. Give it users and suddenly the new piece is to replace some older one to most users.

    That’s to say that maintenance is taking a back-seat to aggressive development.

    2nd. Linux distros take forever to package up new versions of programming tools. Ruby has been affected by that slowness for a while. It’s true that moving from Ruby 1.8 to 1.9 hasn’t helped with matters. The promise for Ruby 2.0 is that it will keep backwards compatibility with 1.9, if that helps. 🙂

    Ruby 1.9 brought many changes to the language and that also put the alternative Ruby implementations under pressure as they tried to cope with being great 1.8 implementations and still move forward. In some ways it has been as painful as Python moving from 2.x to 3.x.

    The issues affecting Ruby have also been affecting other programming communities. Stability hasn’t been the norm.

    Good luck to us.


  4. November 5, 2011 at 7:11 AM

    The best “something else” is to use JRuby.

    With JRuby, you can package up your application along with a specific version of every library you use into a single file, and only distribute that file. The target systems need nothing more than a JDK installed, which is easily available on any Linux flavor and preinstalled on many.

    I sympathize, but using JRuby really does make this all go away.

  5. gonyea@gmail.com
    June 18, 2012 at 3:12 PM

    Hey, you don’t leave your name anywhere :(. I’m packaging up rbenv as an RPM and it’s working out quite well. I generally agree with this blergh post about the state of Ruby.

  6. Dharma
    July 6, 2012 at 10:51 AM

    good insights, with a touch of sarcasm/satire that helps illustrate the plight of setting up a prod env – it’s now July 2012 and I still can’t locate an adequate solution to putting ruby 1.9.2/3 on RHEL6… time to setup a pizza debit account and train my liver for the onslaught…

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: