eightbitraptor

Running Spinach in Emacs

February 13 2012

There are quite a few specific plugins and modes for doing Ruby development in Emacs and some of them are quite mature. Cucumber.el and Rspec.el for instance allow fine grained control of all of your acceptance features and unit tests from inside Emacs.

I use Spinach for Acceptance testing, rather than Cucumber and I was looking for a simple way of being able to verify my acceptance suite without context switching away from Emacs.

There doesn't seem to be a Spinach mode as yet so as a quick workaround I've borrowed some functionality from Rspec mode and written a quick function to run all your features.

It looks like this:

(defun run-spinach-for-project ()
    (interactive)
    (rspec-from-project-root
        (compile "bundle exec spinach"))
    (rspec-end-of-buffer-target-window rspec-compilation-buffer-name))

to make this work, you'll need to grab this snippet and load it from somewhere inside your .emacs.d and run it with M-x run-spinach-for-project, or bind it to a keyboard shortcut

(global-set-key (kbd "C-c s") 'run-spinach-for-project)

You'll need rspec.el installed to be able to use this, as I use it's functions for sniffing the project root directory and auto-scrolling the compilation buffer.

You can check out this and the rest of my Emacs configuration on Github

Testing and TDD - A small demonstration

November 10 2011

A couple of weeks ago I gave a short talk about testing with rspec to the rest of the development team at On The Beach as part of our 20% time initiative.

I primarily spoke about some of my Rspec best practices around testing your models and controllers, How and when to mock, and some thoughts on Object creation libraries like Factory Girls and Machinist.

It mostly carried on from a talk fellow developer Tom (@seenmyfate) gave first at a previous 20% time, and then again at MagRails about how to speed up your test suite. Hence there's a lot of references in here to what came before.

I've put the slides on heroku for your viewing pleasure, although it's doubtful how much you'll get out of them without context.

I also recorded a short screencast of mmy attempt at the code demonstration. Although please bear in mind, this was recorderd spontaneously over lunch today so it's pretty slow as I actually have to think about stuff. Also I couldn't be arsed to edit this in any way, there's no sound and you get all of my typos and mistakes, and my innefficient usage of Vim will probably drive you to drink.

Please, any comments, suggestions, Vim tips then get in touch with me. Twitter, email, IRC, anything.

Now that I've got all the snivelling out of the way, I really hope this is useful to someone.

The command I type at 0:15 seconds in is this:

    :map ;t :!rspec --no-color %<cr>

If you're not familiar with the syntax for Vim shortcuts, this is me telling Vim that when I press semi colon then t in normal mode I want it to run the current buffer (%) through the command rspec --no-color This is so I can really quickly run my tests and see the output inside Vim. The --no-color switch is required in GUI versions of Vim to avoid printing shell color codes throughout your output.

I'd recommend watching the non-HD version, the text is still readable and the framerate is much better than Vimeo's HD encoding.

SSH tab-completion in Bash

November 07 2011

Dealing with ZSH fanbois appears to be an occupational hazard in my office. Their latest pitchfork to rattle was how zsh autocompletes hostnames that are configured in the .ssh/config file and tired as I was of their mocking whenever I had to rely on my bash history for a hostname, I decided to do something about it.

Looking at man bash and reading a little about the builtins pointed me in the direction of complete. The bash manpage says this about complete:

complete (TAB)
  Attempt  to perform completion on the text before point.

And continuing a little further down, underneath the heading Programmable Completion is a wall of text on how to use it and the arguments it can take.

Long story, short: you can use complete a little bit like this

complete -o default -W <word_list> <command_to_autocomplete>

So with a little bit of Awk magic I built a script to find all the hosts defined in my config file, build a word list out of them and then tell bash to autocomplete the ssh command based on my wordlist. The script looks like this:

#!/bin/sh
# ~/.bash/ssh_autocomplete.sh
SSH_COMPLETE=( $(awk '/^Host [^\*]/ {print $2}' ~/.ssh/config) )
complete -o default -W "${SSH_COMPLETE[*]}" ssh

I sourced this file from my bashrc and all is well. So how does this work? Firstly we're using Awk and a regular expression to match all of the Host line in my .ssh/config and then print the second chunk of the line, bearing in mind Awk splits on whitespace. The SSH_COMPLETE variable is being defined with braces around it which is bash's way of defining an array.

Next thing we're doing is telling complete to tab complete our ssh command from the dumped out contents of our SSH_COMPLETE array

And the final thing to do is marvel at the awesome, and watch the fanboys eat their words as I tab out ssh hostnames like sysadmin during downtime.

Bash for life!

We are tired of writing crap!

October 27 2011

To begin, a quote:

Why is there a software craftsmanship movement? What motivated it? What drives it now? One thing; and one thing only.

We are tired of writing crap.

That's it. The fat lady sang. Good nite Gracy. Over and out.

We're tired of writing crap. We are tired of embarrassing ourselves and our employers by delivering lousy software. We have had enough of telling our customers to reboot at midnight. We don't want bug lists that are a thousand pages long. We don't want code that grows more tangled and corrupt with every passing day. We're tired of doing a bad job. We want to start doing a good job.

That's ... what ... this ... is ... about. Nothing else.

What we are not doing:

We are not putting code at the center of everything.

We are not turning inward and ignoring the business and the customer.

We are not inspecting our navels.

We are not offering cheap certifications.

We are not forgetting that our job is to delight our customers.

What we will not do anymore:

We will not make messes in order to meet a schedule.

We will not accept the stupid old lie about cleaning things up later.

We will not believe the claim that quick means dirty.

We will not accept the option to do it wrong.

We will not allow anyone to force us to behave unprofessionally.

What we will do from now on:

We will meet our schedules by knowing that the only way to go fast is to go well.

We will delight our customers by writing the best code we can.

We will honor our employers by creating the best designs we can.

We will honor our team by testing everything that can be tested.

We will be humble enough to write those tests first.

We will practice so that we become better at our craft.

We will remember what our grandmothers and grandfathers told us:

Anything worth doing is worth doing well.

Slow and steady wins the race.

Measure twice cut once.

Practice, Practice, Practice.

This is a repost from Uncle Bob Martin's blog of a message in response to a lot of the confusion and acrimony directed at the ideas behind Software Craftmanship.

The primary message: We are tired of writing crap is one I feel very strongly about, and I'm convinced that the fast pace of agile development isn't mutually exclusive with the notion of producing well tested, well thought out and well designed software.

As Uncle Bob mirrors completely my beliefs about how we should be developing software and articulates them far better than I could, please head on over to his blog and read the article in full.

On Pairing

June 20 2011

Over the last couple of years I've had the good fortune to work with several companies that practice "agile" development, from quite strict Scrum based approaches, through Kanban all the way to some ad-hoc "pick the bits that work" methods, but one of the things that has been stable throughout was the emphasis on pair programming and the belief that it improves code quality, results in better programmer productivity and introduces fewer bugs into production code.

I want to take this opportunity to put some of my thoughts on paper regarding pair programming, when I believe it's necessary, why it can help and most importantly, why it's not a silver bullet.

Why Pair?

Pair programming has a lot of benefits, especially as part of an agile development approach. It can mean less bugs slip through into production. When two developers are working on a particular task you can pick up on mistakes that have been made that you potentially wouldn't have spotted on your own. With two people dedicated to solving a problem you'll probably end up spending less time heading down routes that you may have been tempted to follow but ultimately, may have been fruitless.

It also helps maintain focus. You have an ally in your attempts to solve a problem or build a feature. If you are prone to procrastination whilst working alone, this can be a lifesaver – you'll bounce ideas off of each other, keeping you motivated and always striving to write the best code you can.

One of the major benefits I've found personally whilst pairing has always been when working with people who are more experienced. There is in my opinion no substitute for the knowledge osmosis that can come with pairing, even for a relatively short period of time, with someone who has a better handle on the problem domain than you, or is simply a more experienced developer than you. It's hardcore on the job training, and worth it's weight in gold.

This benefit also extends to being able to discuss the problem at length, with someone who is just as emotionally involved as you, but might think differently about solving it. After all, there's no substitute for a rubber duck that can talk back right?.

Pairing with someone can also help us to bring our heads down out of the clouds. There's nothing quite like someone grabbing you and pulling you back down to earth when you venture out into architecture astronaut territory, you may be in love with your fantastically elegant solution, but sometimes simplest is best and it can help to have that reiterated.

Pair Programming is not a panacea

The problem with pair programming is that it's very developer specific. What works for one pair cannot work for everyone, that's all part of the joy of people being different! It's important to be able to work in an environment that's productive for both parties and allows everyone to just sit down and write awesome code.

You will need to make allowances, and you will need to accept that. Not everyone is going to be able to work the same way you work. Not everyone will love your awesome new blank keyboard and not everyone's going to be as productive as you are in Vim. The sooner you accept this the better. Keep multiple text editors installed, or better still, learn to use multiple editors so you can sit down with anyone and immediately be productive.

Pair programming, when done wrong can harm productivity just as much as doing it right can benefit it. If you're constantly fighting with your pair over who's approach is best, or if you can't see the screen properly, or if you just can't use someone elses tools it can be downright frustrating.

Techniques for pairing

Wikipedia defines pairing as the driver/navigator approach with one person typing code, and one person reviewing it as it is typed. This approach has a couple of benefits. Firstly, your driver can concentrate on the small things, the down and dirty implementation details of the method or function she is working on right now, whereas your navigator can think in more high level terms of how this particular piece of code is going to fit into the architecture, the things it needs to return and the edge cases it might have during it's use.

This is an awesome way of developing, providing that you keep in mind a few points

Turn off twitter: Seriously, it's distracting, and downright insulting to whoever you're with. If you're navigating you'll also end up losing your driver, who will wander off aimlessly without focus, unsure of where to go from the method that he's just finished.
Talk lots: Neither of you are mind readers, you can't expect anyone to know how your brain works, where you envisage the architecture heading or why in fact you think that this awesome function should return an array rather than a hash.
Don't be embarrased: No question is stupid, don't just assume that the other guy knows better than you. Neither of you will learn anything and you'll run the risk of heading off in completely the wrong direction. And similarly…
Don't be a dick: No question is stupid, just because the other guy doesn't know that you're awesome implementation is O(1) efficient doesn't make him an idiot. Just explain what you are doing and that way you'll both stay on the same page and you get the warm fuzzies at being able to teach someone something.
Pomodoro is shit: It's designed for lone developers to focus their efforts whilst ignoring distractions and disruptions. Forcing a pair to context switch every 20 minutes is a surefire way to make sure no one can concentrate properly on anything.
Mirror your screens: You should have 2 screens and they should be mirrored I can't stress this enough. Forcing me to peer over your shoulder at 11pt font will make me stop giving a shit about anything we're doing. I'm not killing my eyes so you can keep an email client open on a second screen (see my first point).

In my experience the best method of paring doesn't involve such formal definitions as driver and navigator, and it has no allocated time after which you should switch roles. It just involves one machine (a Mac Pro in my case), 2 screens (mirrored of course), 2 keyboards, 2 mice, and 2 developers. You can see everything that the other person is doing, if you're talking as you should be, you'll know exactly why they're doing what they're doing, and best of all, if you think it's shit you can just start typing over them!

This may sound mental, especially at the beginning of a session. You will end up with absolute chaos as you both try and talk and type over each other. But this is the beauty of it, it forces you to listen to the other person and integrate their ideas into what you're coding, it allows one person to be on a roll and write a ton, whilst still giving you the confidence of knowing that you can interrupt them at any point.

At it's best it can make pairing a very organic process, switching between 2 developers almost seamlessly, each one knowing where the other was going and where to pick up where they left off. It's these situations where you really will notice the gain in productivity, and you'll leave work at the end of the day feeling like you've got so much more done, and you'll probably make some really close friends to boot.

Conclusions

Pair programming can be an extremely productive way to work, it can be very rewarding for all concerned. However it can just as easily fuck up your concentration, code quality and can make you despise the people you work with, and that's not even taking into account some of the other big issues, like having to syncronise lunches/breaks, and trying to make your co-workers use your custom bash profile. The most important points I am trying to get across in this post are:

Find what works for you and go for it. Feel free to ignore anything I've just said. If I'm talking shit and you've found the perfect way for you then that's great, run with it and I'm really pleased it works.
Don't let management force you to pair in a certain way. If it's not going to work then no amount of shoehorning is going to make it.
Don't knock it until you've tried it. Try everything until you find what works best. Honestly, it's the only way to be sure.

Special mention goes to designers (@cgovias I'm looking at you). Designers need love too. Pair with your designers, show them how quickly you can get their psd's into the browser and they'll bear you in mind when they're working as well. Frontend stuff doesn't need to be painful, it just helps if the left hand knows what the right is doing! Plus you'll learn a metric fuckton of Photoshop shortcuts! *

Relaunch! Part 2: Building the basic architecture

May 21 2011

In this entry I want to talk about where this site is hosted and exactly how to go about setting up the Virtual Instances and getting everything ready and talking to each other so that we can deploy and configure the app.

Setup

Firstly you're going to need a few things…

Ruby installed (wait, you do want to deploy a Rails app right?)
An SSH keypair that you want to use (I'll assume ~/.ssh/id_dsa for now )
A Brightbox cloud account
The Brightbox CLI tools

These last 2 are the kickers! To request a Brightbox cloud beta account you should go here and be nice: Apply for a Brightbox cloud account.

There are 2 ways to install the Brightbox utilities depending on whether you're on Ubuntu or not. If you are, then:

sudo apt-get install bbcloud

and if you're not:

gem install bbcloud

The awesome folks at Brightbox have already written documentation on how to use the CLI apps, suffice to say it's good enough that I'm going to save myself a few paragraphs and just link to it so you should go and read that and come back when you're familiar with how the tools work.

Architecture

Back now? Good. So the next step is to plan out our architecture and to make sure that all of our boxes can talk to each other. The basic architecture for this site is shown in the diagram below:

Ignoring my terrible Illustrator skills and focusing on the content you can see I have 3 (at the moment) servers, 2 main application boxes talking to a single backend database. Clearly this isn't going to serve your needs for a large traffic site, but as an education excercise and running a low traffic blog it'll do just fine.

Box 1: (Kumiko)

This is where our main Varnish instance sits, requests come into this box and hit Varnish which is running on port 80.

Sitting behind Varnish is an Apache/Passenger instance listening on port 8008 for any Rails request to come in.
Box 2: (Yasuna)

This box just has an Apache/Passenger instance running on port 8008 which will service Rails requests that come into it.
Box 3: (Lain)

This is my database box, it's running a MySQL server instance and that's it. We'll configure MySQL to be greedy about the amount of memory it can use on this machine so we can maximise the query cache size and tweak the system for maximum responsiveness.

And that's it. So the next thing you'll want to do using your new found Brightbox cloud CLI skills, is go and build yourself 3 machines. You can call them what you want, I just use anime characters for my server names because that's just how I roll.

I'm going to assume you're using a recent Ubuntu image for these machines, if you didn't then you'll have to adjust the rest of these tutorials relative to your system.

Database Configuration

The first thing I want to do is get the database up and running so ssh onto your shiny new (and very bare) ubuntu box and run

sudo apt-get install mysql-server

That will, fairly obviously, install your MySQL server and all the dependancies required to make it work. Follow the instructions and for petes sake make sure you set a decent and secure MySQL root password

The next step is to configure MySQL - You can grab my config file from here, it's very very close to the example small server setup that comes bundles in the examples directory of the MySQL distribution.

The basic gist is, we want to optimize the size of the query cache, and various buffer sizes for the amount of memory we can realistically allocate on the free Brightbox cloud beta, and tune concurrency so that we can service multiple queries at a time. We are also telling mysql where it's data directory, log files et al are to live.

If you want to know more about how much more we can wring from MySQL (and there is a lot) I thoroughly recommend adding the MySQL performance blog to your feed reader.

Finally we need to start the MySQL service by running

sudo service mysql start

And that's all we need to do on this box for now. Later on we will need to create users and databases, but there's little point doing that until we have a machine that needs to talk to the database.

Application Server 1

Right now jump on to one of your other boxes so we can start setting up Apache and Passenger.

You're going to need a lot more software for these ones, so before you do anything make sure your apt repositories are up to date.

sudo apt-get update && sudo apt-get upgrade

And from then on it's basically just a case of installing stuff. Now it's a good idea in the interest of repeatability to set up some kind of configuration management system so we can have a safe, repeatable, idempotent way of managing our server profiles: Something like Puppet would be perfect.

However, pragmatically, that's a lot of overhead, so here's a bash script that does kind of the same thing

Matt's super duper brightbox cloud server setup script of DOOM

Obligatory Warning: This is hacked up based on my bash history and it does alter files as root. It's basic and hacky. Read it, understand what it does before you run it. It's not my fault if it kills your cat.

That script, in a nutshell does the following:

Installs Ruby.
Builds Rubygems (from source).
Installs Bundler.
Installs development libraries.
Installs Apache.
Installs Passenger.
Builds the mod_rails Apache module.
Edits your apache2.conf to load and configure the passenger module.

Providing that has all happened successfully and no errors happened, we end this post having 3 seperate VM's running, 2 clear web/application servers and one database server.

This also means that I've already deviated from the rough planning list I set out in the introductory post having merged parts 2 and 3 into one.

The next post is going to be about how to get our Rails applications configured with Capistrano so we can deploy them easily and repeatably to our cloud hosting.

So far there's been no mention of Varnish, never fear, it is coming. We're going to get the individual apps responding on their own IP's before whacking Varnish in front and kicking away the stool, so to speak.

I'll concentrate on Varnish in 1 or 2 posts time. *

Relaunch! Part 1: The rewrite

May 10 2011

Welcome to the new look version of eightbitraptor.com!

Before you all sigh quietly into your coffee and skip this post just be aware that this isn't your usual hand wringing apology for not blogging in forever.

This is the first of a multi part serious of posts about the re-architecture of the site from the ground up, with a much needed Rails 3 rewrite along the way.

Out with the old.

The old site is available on github, it was written using Sinatra and hosted using a free Heroku account. It used a flat file store for its data which allowed me to write posts in my editor of choice and simply drop them in place.

I was quite partial to this approach up until the point where I tried to add more features, the codebase grew and grew and I realised I was slowly implementing most of Rails ActionView!

Adding more varied types of data was also hassle, as my fragile document system couldn't really handle anything that wasn't a simple blog post, I needed a real data store.

Also there were no tests, which was a massive faux-pas when you're being paid to be a professional Rails developer.

So armed with the latest Rails 3 and a a decent text editor I set to work

In with the new.

There isn't really much to say about the codebase behind this app. It's a basic Rails 3 MySQL backed web app, with a frontend written using HTML5, CSS3 and jQuery (all the latest buzzwords obviously). The source is on github and it's built using a variety of interesting projects. My favourite of which is Rails Admin which introspects your models to provide basic CRUD functionality.

Which means that my codebase consists predominantly of front end code, and what little logic there is in my models is well tested using Rspec.

So, what next?

How is he going to spin this out for more than one post? I hear you ask. Basically the most interesting part of this whole excersise is that I am using the new Brightbox cloud beta for my hosting.

Eightbitraptor lives behind a Varnish instance configured on one of my nodes. Varnish acts as my caching layer and my load balancing layer passing requests back to 2 seperate Apache/Passenger installs running on their own instances.

Both of these instances are talking to a shared MySQL database instance.

I want to talk about how to set up this kind of arrangement (and a few other asides) all on the Brightbox cloud servers running Ubuntu 10.10.

As a rough guide the next few parts will be as follows:

Part 1: This introduction
Part 2: Getting the Brightbox servers up and running
Part 3: Installing your database and Rails stack
Part 4: Deploying your app using Capistrano
Part 5: Using Varnish as a caching layer

This structure isn't set in stone, more articles may appear and there this architecture isn't going to stay stable so more tweaks will be documented in the future.

I am however sorry about not posting since last year, hopefully this will make up for it

Returning from /dev/fort

September 14 2010

A couple of weeks ago the Reevoo team, along with James and Norm of /dev/fort and super awesome creative guy Gavin O'Carroll set of to a remote location somewhere in the UK for the second commercial /dev/fort. It was ace.

We spent a tiring and productive week in the farmland of Dorset with no internet connection, no distractions and no management telling us what to do and we built something that we thought would be genuinely useful to Reevoo's users and would hopefully hammer home the message of how awesome we are :)

My experiences of /dev/fort were all positive. We ate loads of nice food (special mention to Norms Toad in the Hole and George's fish pie), prepared and cooked by the group, we drank loads of great local beer

and of course, we programmed, a lot.

a couple of things that working in an environment like /dev/fort really made stand out for me were that a) it's amazing how much you can learn when working undistracted with a group of like minded people and b) even if you think it's not, the internet is a massive distraction.

This was painfully obvious as soon as I got back to work the following week, and returned straight away to keeping my twitter client open and both my gmail accounts open. The first half a day was embarrassing, I'd find myself checking RSS/Twitter/Email about every 10 minutes. And of course the quality of code I wrote was pretty rubbish.

My modus operandi now is basically: Close Everything. I still check my email periodically, and twitter, but not having these things open or a notification icon blinking at you from somewhere on your screen means that when you get in the zone nothing is going to pull you out of it, well except that guy throwing Nerf darts at you periodically but at least you get punch that guy in the face!

We had 6 devs, 3 designers, a usability guy, 2 of Reevoo's category guru's, the /dev/fort guys, no idea what we wanted to build at the beginning of the week and no constraints and after a lot of discussion, some furious deliberation and a lot of hacking we made this:

http://justbuythisone.com

I'm very proud of it and the part I played in it, and am very glad to have been involved in a /dev/fort. so cheers to James and Norm, buy them a beer the next time you see them.

Thanks to Norm for the photos, you can see the rest on his flickr page

Android Development without Eclipse

July 27 2010

I started having a bash at some Android programming recently, nothing amazing, just flexing the old Java muscles again (for the first time since university, back when you had to avoid the Velociraptors to get to lectures), and it's actually pretty fun.

What's not fun however is the behemoth that is Eclipse. I swear, before I got fed up and threw it away, I spent longer bashing my head at Eclipse than I actually spent setting up the toolchain, writing some code and getting it on my phone.

Whilst the "Android developer website is amazingly helpful":http://developer.android.com I thought it might help to have a quick reference to get started. And the most important step is to "download the sdk from here":http://developer.android.com/sdk/index.html, unzip this to a directory where it isn't going to get in your way, I use @~/code/android-sdk@ and add @~/code/android-sdk/tools@ to your path.

The next important thing is to install the SDK components for whatever android version you're using and create an Android Virtual Device to use in the emulator. This is all covered in "the Android install guides":http://developer.android.com/sdk/adding-components.html and when your done you should be good to go at creating a project.

The basic workflow is this:

get the android target version
```
android list target
```
this will return a list of stuff depending on how many platform components you have installed, what's interesting is the line that starts with "id: " because that gives you the @-t@ paramater for use in the next step
Create your project
```
android create project -t 1 -p  -k  -a 
```
Write some codes! then build it with
```
ant debug
```
this will create your .apk that we'll install to the phone
to get the code on the device we first need to start a device, run the android tool and fire up the avd that you created during the install step, this will start the emulator. Once the emulator has started (this takes about 2-3 minutes on my MacBook) you should be able to do this
```
~ > adb devices
List of devices attached
emulator-5554  device
```
This means the Android bridge is successfully talking to your emulator.
The last step is to actually push your package to the emulator, this is acheived by
```
adb install /path/to/bin/package.apk
```
And you should be able to use your app on the emulator, Yay.

Note: If you are reinstalling a new version of your app you'll need to add the @-r@ flag to @android install@ otherwise it will whinge that the app apready exists.

The steps to getting your app on an actual phone are pretty much the same however:

Make sure you have USB debugging turned on on your device and plug in the phone (duh!)
Run @adb usb@ to restart the bridge listening on the USB interface.
Verify your device is showing up with @adb devices@ and install as above

and breath a breath of fresh air at being able to use $EDITOR to make cool things again.

RSS Awesomeness, Creating feeds with Sinatra & Builder

July 26 2010

So I was reading "the Design Monkey blog":http://designmonkey.blogspot.com this evening and suddenly a vast sense of inadequacy welled up inside me. Was it because I had such vastly inferior design skills? Or maybe far less comical writing ability?

Hell No.

The design monkey provides that most awesome of things (as I'm sure do many others), an RSS feed of his eloquent prose, so not to be outdone and armed with the best of tools, I set out creating one with Ruby, builder and Sinatra

This turned out to take about 10 minutes and be amazingly less impressive than it sounds but here goes anyway. First you should create a builder template for your RSS feed. Mine looks like this:

  xml.instruct! :xml, :version => "1.0"
  xml.rss :version => "2.0" do
    xml.channel do
      xml.title "eightbitraptor"
      xml.description "The personal blog of developer, music lover and recovering sysadmin Matt House"
      xml.link "http://eightbitraptor.com/posts"

      for post in locals[:posts]
        xml.item do
          xml.title post.title
          xml.description to_html(post.body)
          xml.pubDate pretty_date(post.published)
          xml.link post_url(post)
        end
      end
    end
  end

This is saved in my views folder as @feed.builder@ so that in my main Application I can define my route like so:

  get '/feed.xml' do
    builder :feed, :locals => { :posts => Post.all }
  end

Where a Post class does all the magical awesome (but is essentially just a way of grouping together titles and bodies as referenced in the builder template).

And that is pretty much all you need to do to generate functional rss feeds with Sinatra. If you want to dig deeper, especially into what my mystical Post object is, then the code is on github as per usual.

Get the source

Running MPD on OSX

July 21 2010

After finally getting fed up and giving iTunes the boot, I got round to making MPD work on my Mac. and unfortunately, apt-get install it ain't!

First step is to actually get hold of and install mpd, if you're using sensible and using homebrew that's as easy as:

brew install mpd

Which will pull in all of the required dependancies and compile them all for you. Then comes the mpd config file. This is all pretty standard stuff, you can adapt from the standard and massively verbose example included with the mpd sources. Mine lives at /usr/local/Cellar/mpd/0.15.9/share/doc/mpd/mpdconf.example. The stuff you need to care about is:

music_directory
playlist_directory
log_file
db_file
pid_file

mixer_type "software"

Make sure these paths are all writeable by the user that you intend to run mpd as. In my case, I run mpd as the mpd user, and I made the mpd user and my normal user account members of group mpd.

What this amounts to is a music and playlist directory that the mpd user can read from and that I can add songs to. If you run a multi user system it's probably a good idea to put this somewhere outside of your home dir.

A special point regarding the mixer_type line: I have found this necessary when running on Snow Leopard to avoid mplayer crashing hard when trying to skip playing tracks, but as is normal with these things YMMV.

Once this has been set up you should be able to start mpd with

mpd --create-db

and watch it chug away for a while depending on how much music you have.

Client

I use the excellent Theremin, which is an OSX native MPD client and does the job admirably. If that's not your style there are an excellent array of decent mpd clients out there.

Last.fm

Last fm Scrobbling is acheived by the use of the lastfmsubmitd daemon, and it's built in client lastmp. It's dead easy to set up. Clone the sources from Github and follow the instructions in the INSTALL file. The client scrobbler lives inside the contrib folder of the checkout.

I installed lastfmsubmitd to /usr/local/bin and created it's config file, and then simply copied the contrib/lastmp script to /usr/local/bin.

One gotcha if you're not familiar with running Python stuff (I'm not) is that lastmp will bail out complaining it can't import libmpdclient2. this is easily fixed with:

easy_install py-libmpdclient2

which will ramble on about installing eggs, I guess these are pythons equivalent of gems.

Both of these daemons apparently need to be running to actually make scrobbling happen so I normally wrap these up in @/usr/local/bin/music_starter@, which looks like

  #! /bin/sh
  /usr/local/bin/mpd && \
  /usr/local/bin/lastfmsubmitd && \
  /usr/local/bin/lastmp

Tying it all together

You can start the whole kit and caboodle on boot by creating the following plist file and adding it to launchctl:





    Disabled
    
    Label
    com.eightbitraptor.mpd
    Program
  /usr/local/bin/music_starter

I put this in /Library/LaunchDaemons/com.eightbitraptor.mpd.plist. Add it to launchd and start it like this:

sudo launchctl load -w /Library/LaunchDaemons/com.eightbitraptor.mpd.plist
sudo launchctl start com.eightbitraptor.mpd.plist

And Job done! Now you too can get rid of stinking iTunes. Now all that's left is to find something comparable to "mp3tagedit":

Configuring rubygems

August 26 2009

After spending ages meticulously installing gems with

matthouse [~] > sudo gem install some-random-gem --no-ri --no-rdoc

I finally got round to trying to find a better way. It turns out that rubygems has it's own config file, which is cool. It's located at ~/.gemrc and is a simple yaml file. My gemrc now looks like this:

:update_sources: true
:sources: 
- http://gems.rubyforge.org/
- http://gems.github.com
:backtrace: false
:verbose: true
gem: --no-ri --no-rdoc

You can find out more about the options by running

matthouse [~] > gem help environment

The options in my config file work as follows

Sources: kind of speaks for itself
backtrace: whether gem will show you stack traces on install errors
update_sources: this tells gem to automatically update it's own sources when you install a gem
verbose: this defines how much information gem will spit out to stdout
gem: these are the important ones, the flags that are passed to gem on the command line

Defining a cron job that happens on machine reboot

August 20 2009

I have an app that uses the Sphinx search engine and it's partner Rails plugin thinking-sphinx. This is a neat little search engine that can be embedded into Rails projects.

The only downside is that you need to start the server before the app will run properly. You start thinking sphinx by running

rake ts:start

I started writing a full init script for this until I realised that writing a massive init script that I can manage with chkconfig (our servers are Redhat Enterprise) is slightly overkill for one freaking rake task.

After some research I found out that our venerable friend cron supports running tasks on reboot, you simply need to define your job with @reboot instead of the standard time information.

Nothing is as enlightening as an example so here goes nothing

matthouse@sakura $ crontab -l
@reboot echo "cron is ace" >> /tmp/somefile

to read more check @man 5 crontab@ and read all the way to the bottom

Managing Ubuntu instances with Puppet

July 28 2009

My machines at home can be pretty varied. I like to test out distributions and tend to just format my main dev box and try out new and exciting things pretty frequently. But one thing I get really sick of is the time spent making my environment personal, getting things like commonly used software installed, and my dot files in the correct places.

I needed a way of managing my infrastructure that doesn't involve me sitting down for a couple of hours after every install grabbing packages, compiling software etc.

And that's where Puppet comes in :)

h4. What is Puppet

Puppet is a configuration management framework. It's kind of a logical step forward from Cfengine. It is written in Ruby, by Luke Kanies of Reductive Labs and basically consists of 3 things:

A domain specific language, for declaring configurations
A client and server application for distributing your configurations, and
A library for interpreting and applying the configuration

Puppet makes deploying identical environments across one or multiple machines one hell of a lot easier than it would be normally, it also lets you track changes and keep specific configurations in one central repository so that you can see at a glance, what state your machines are in and what package versions they have installed etc.

It also deals with abstracting the operating system layer so that your puppet recipes can be OS agnostic, the packages are handled automatically your default package manager.

Why Puppet? What about the alternatives

There are a couple of alternatives to Puppet. Cfengine, which Puppet is meant to be a replacement for, owing to Luke Kanies experience at writing custom Cfengine plugins. The Puppet Wiki has a very good article on why Puppet is not Cfengine and how it compares, which makes very interesting reading.

The other main alternative is Chef. Chef is also written in Ruby and is definitely worth keeping an eye on. It tries to alleviate some of the main issues in Puppet, mainly that instead of a custom DSL the recipes are written in Ruby, which lets you do something like this:

node[:gems].each do |gem|
  gem_package gem[:name] do
    version gem[:version]
    source gem[:source]
    action :install
  end
end

where gems is just a massive hash of ruby gems, versions and sources. Currently in Puppet, installing a named gem with a specific version requires a seperate package declaration for each package, which isn't very DRY and can clutter up your recipes a bit.

At the moment Chef is in early stages and seems to be a bit of a pain to use for anything other than Ubuntu so I prefer to concentrate on using Puppet, at least until Chef gets a bit more mature.

h4. Installing Puppet and setting up the Puppet Master

Puppet is pretty easy to set up, and on a blank install of Ubuntu the process is pretty much as follows:

Install Ruby, and Rubygems
Install Puppet
Write/Pull your recipes into /etc/puppet/
Start the Puppetmaster daemon and create a certificate for your client
Run Puppet.

I prefer to use a custom rubygems install rather than the provided package. This is due to Debian's lack of support for Rubygems because they have apt and don't like having multiple package providers.

I disagree with this because Rubygems, like CPAN or Python Eggs, is a very specific package manager for a very specific purpose, but this is a topic for another post/rant! Normally I'd compile and install Rubygems using the excellent checkinstall program, but seeing as how I like to automate my system, I've included a deb with my Puppet repository. So we can install it directly from there.

So lets install ruby, clone the Puppet repository, install Rubygems and then install Puppet.

sudo apt-get install ruby git-core
cd /etc/ && git clone git://github.com/shadowaspect/puppet.git
sudo dpkg -i /etc/puppet/files/deb/rubygems1.8-1.3.5_i386.deb
sudo gem install puppet

Now we need to look at starting the server with the correct certificate name, my recipes assume that your puppetmaster host is called puppet, so make sure that it is configured in your hosts file and start up the puppetmaster with:

sudo puppetmasterd --certname puppet

So now we have a repository of Puppet recipes and our puppetmaster running you can run puppet on the machine with

sudo puppetd --test --verbose

This will fail the first time due to an invalid client certificate, to rectify this you need to have the puppetmaster sign the client cert, you can do this by running (on the master)

sudo puppetca -l # for a list of unsigned certs
sudo puppetca -s  # from the above list

Writing Recipes

Puppet pulls it's configuration from the files located in /etc/puppet. It basically assumes a couple of things:

that you have a site.pp file containing node definitions for each machine/class of machine on the network
you have a puppet.conf file detailing some information about the actual puppet process itself.

Puppet has several useful constructs available for writing recipes, stuff like classes and basic inheritance, and also defines for creating your own custom functions, you can see examples of this by checking out my github repository, but a basic node definition would look like this:

node mattsmachine{
  package{ "apt":
    ensure => installed,
  }
  file{"/etc/apt/sources.list":
    ensure => present,
    owner => root,
    require => Package["apt"]
  }
}

This noddy little example just makes sure that apt is installed and that the sources.list is present on the machine. You can do so much more awesome stuff than this, for which I advise you to check out the excellent tutorials on the Puppet site, and have a browse around my custom Puppet repository. Get the source

Where next?

Obviously the initial install of Puppet (installing Ruby and Git by hand, compiling rubygems etc) can be pretty time consuming in itself. At work we use RedHat, and so we have created a custom kickstart file that completely automates the process of installing the operating system and performing the first puppet run. All you have to do is boot the machine from the network, point the installer at the kickstart file and go and grab a coffee while it installs the OS (from a network share), configures the network, installs ruby and puppet, locates the puppetmaster and runs puppet to install the rest of the system depending on the role it's destined to perform.

This is exceptionally neat, and I think there are definitely options out there to make something like this work with Ubuntu/Debian either via the Debian preseed files, or tools like Fai, so expect more posts on the topic in the future.

Multiple SSHD servers on ArchLinux and OpenID Delegation

July 12 2009

A reasonably productive sunday morning has led to me having a couple of interesting things to talk about :) The first is running multiple ssh daemons on the same machine, and the second id about OpenID delegation, so that you can identify yourself on OpenID enabled sites with your own personal URL, rather than the URL of your OpenID provider.

Multiple SSH Servers and why you should bother?

The use case I found for running multiple SSH servers is one of security through obscurity. For instance I wanted to have my server accessible from other machines on my local network using standard password authentication so that I can log in and manage stuff remotely with relatively few restrictions. To this end I set up the standard SSH server listening on port 22.

This works great internally and because port 22 is blocked on my firewall I don't have to worry (too much) about people outside of the network.

But what if I want to access from outside my network, for instance I want to ssh in from work and grab some music that I left at home? I don't really want to open port 22 and similarly I don't really want to restrict the internal users too much. So the best solution is to run a seperate sshd listening on a higher port with restricted security settings.

This is actually pretty easy to get set up. But there are a couple of gotchas and these instructions pertain to Arch Linux, but the basic principles are the same for any distro.

Copy the contents of /etc/ssh to /etc/ssh-external and remove the specific host keys in the new directory.
Edit the new sshd_config, and customise it to your liking making sure to pay attention to the following settings:

Port 55225 # A new higher port to listen on
PidFile /var/run/sshd-external.pid # Make sure that this new instance uses a seperate pid
PubkeyAuthentication yes # enable Public Key Auth
PasswordAuthentication no # and don't let users use passwords
AllowUsers extern # Allow access only to the extern user

Copy (or symlink) the sshd binary to a new location, eg. sshd-external. This is not strictly necessary however it makes tracking connections easier as the processes will show up seperately in ps and top etc.
copy the file /etc/rc.d/sshd to a new location eg. /etc/rc.d/sshd-external
Edit the new rc file to point to your new sshd binary and new host_key locations.

There is also a reference to include the file /etc/conf.d/sshd which defines the $SSHD_ARGS variable. You'll need to redefine the include in your new ssh rc script so that it pulls in a different file. I suggest /etc/conf.d/sshd-external. Once this file is defined we can use it to set our $SSHD_ARGS variable. This will need to point to our new config file and host key explicitly to stop the sshd-external daemon using the default values, as these are already in use by our standard sshd; This can be done like so:

#
# Parameters to be passed to sshd
#
SSHD_ARGS="-f /etc/ssh-external/sshd_config -h /etc/ssh-external/ssh_host_rsa_key"

Once this is all set up you should then be able to login from anywhere. Providing you have remembered to take your private key with you and that the user you configured for access has your public key in his ~/.ssh/authorized_keys file, and that you have remembered to open the correct port on your router/firewall ;)

There are a couple of extra things that you can do to improve security in addition to the standard key based auth above, one of which is to restrict your external user to a restrictive shell (ie. /bin/bash --restrictive) or a chroot jail, I leave this as an excercise for the reader.

Delegating OpenID Authentication to your personal website.

This is going to be a much shorter section that the previous one! I love OpenID but one thing I didn't like is having to sign in using the URL to an openID provider like Flickr or LiveJournal. If I'm signing in using my identity then why the hell can't I use my own URL?

It turns out that you can, and most OpenID providers will play nice and give you instructions on how to set it up. Basically it just involves some new information in the head of your websites markup. That extra info looks a lot like this:

Obviously replace username with your actual OpenID providers username!

This small bit of markup now allows me to sign in to openID enabled sites with theshadowaspect.com as my identity.

Debugging Ruby using -rdebug

July 09 2009

After a particularly annoying hour or two spent tracking down a problem in one of our apps recently, I spent my time wishing that Ruby had a decent debugger! Well it turns out that it does, and it's pretty nifty.

Debuggers can be amazingly helpful when a particular bit of code is borking and you're not sure why, and they save the effort of modifying your code to output debug symbols and variables all the time. You can run the debugger on a script like so:

[08:08]sakura mtmp $ ruby -rdebug testscript.rb 
Debug.rb
Emacs support available.
/opt/local/lib/ruby/vendor_ruby/1.8/ubygems.rb:10:require 'rubygems'
(rdb:1)

This also works with Rails projects, simply run the debugger against script/server.

First and foremost, use the h key from the debug console for help

You can set breakpoints using the syntax below. Breakpoints are lines at which the program execution will pause so you can poke around the environment.

(rdb:1) b testscript.rb:10
Set breakpoint 1 at testscript.rb 10
(rdb:1) b testscript.rb:20
Set breakpoint 2 at testscript.rb:20
(rdb:1)

This is the kind of output you'd see when you hit a breakpoint: we can see the line at which the execution stopped (we defined a breakpoint above as being on testscript.rb line 20), we can also see 5 lines of context each side of the Breakpoint, from here we can step through the code or just continue the exectution as normal.

(rdb:1) c
Breakpoint 2, toplevel at testscript.rb:20
testscript.rb:20:some_object = DoNothing.new(1)
(rdb:1) l
[15, 24] in testscript.rb
   15    def has_been_run?
   16      return @times_run
   17    end
   18  end
   19  
=> 20  some_object = DoNothing.new(1)
   21  5.times do
   22    some_object.run
   23  end
   24  
(rdb:1)

You can grab information about variables that are currently in scope and what their values are, the code below grabs all the instance variables belonging to the object some_object. Certain commands can also be used to grab the values of all globals and all constants that are defined in your program.

(rdb:1) v i some_object
  @do_nothing_for => 1
  @times_run => 0

This is just a basic introduction to the Ruby debugger, there is a ton of stuff that you can do with it, such as thread manipulation: stopping and starting the threads of your program at will, setting watchpoints and catchpoints, and loads more.

Always Learning! Enumerable#inject and Hash#merge in Ruby

May 14 2009

I learned something cool today. While discussing neat one liners in Ruby, someone suggested a potential challenge, it went something like this:

Given an array of numbers, how would you convert that into a hash where the key is the number and the value is the number of times that it appears in the array. so for example you start with the array

[1,1,2,3,2,4,5,3,3,3,1,1,2,6,5,5,5,5]

And what you want to end up with is something like this:

{5=>5, 6=>1, 1=>4, 2=>3, 3=>4, 4=>1}

So how would you do it? The solution I came up with after a couple of minutes was this:

h = Hash.new(0)
[1,1,2,3,2,4,5,3,3,3,1,1,2,6,5,5,5,5].each { |i| h[i] += 1 }
puts h

This is a fairly simple solution, first you create a new hash and initialise every value with 0, then we iterate over each value of the array, passing the value to a block which finds the hash entry with the correct key and increments its value (this is acheivable because we initialise our hash with a Fixnum as a default value). I thought this was pretty neat but it turns out, as with most things in Ruby, that there is a much cooler way :)

It looks like this:

p [1,1,2,3,2,4,5,3,3,3,1,1,2,6,5,5,5,5].inject(Hash.new(0)){ |memo,i| memo.merge({i => memo[i] += 1}) }

So how is this working?

The first thing we need to look at is Enumerable#inject. This takes an argument and a block. Inject then calls the block once for every element in the Collection we are calling it for, in this case this is our array of values. The argument we give to the inject function will be yielded as the first argument to the block (memo), the second argument in the block(i) is the value in our collection we are working on.

An important thing to note with inject is that the return value of the block is passed into the block on the subsequent call, replacing the old value of our first block argument (memo in this case).

This lets us do something funky inside the block using the Hash#merge function. merge takes a hash as an argument and merges it's values into the hash that it's called on. A couple of examples to illustrate this (in irb):

irb(main):007:0> {"first" => 2}.merge({"second" => 2})
=> {"second"=>2, "first"=>2}
irb(main):001:0> {"first" => 2}.merge({"first" => 4})
=> {"first"=>4}

So what is our merge function doing? It's passing in an empty hash initialised with zeros, and for every entry in the array it's merging in a new hash that just contains the incremented value for that particular key.

The p at the beginning is just a shorthand way of saying puts, so that our final hash is printed to the terminal

Clever huh?

Rails Development with Ubuntu

March 31 2009

Having recently purchased myself a new Dell Mini 9, I got a chance to set up a development environment for rails apps so that I can hack on the train to and from work. It's not particularly hard but I thought I'd document it for reference and hopefully someone will find this useful.

First of all a note or 2 about the hardware, the machine is a 1.6GHz Intel Atom chip. A single core, cpu that supports hyper-threading (it shows up in /proc/cpuinfo as 2 seperate cpu's but you can only control scaling on cpu0). It has a 4Gb SSD and 1Gb of RAM. This minimal spec turns out to be more than enough for a full Rails/mysql/mongrel development stack if you're careful with a few defaults.

First of all I removed all unnecesary packages using a combination of apt-get and synaptic. apt-get remove is useful if you know what you want to kill (such as openoffice.org) but I find synaptic really useful for browsing through stuff you're not sure about, libraries for example, and killing anything that doesn't sound useful.

The next step is to install ruby, rubygems and a few useful dependancies. This is covered in detail elsewhere on the web so I'll be brief, basically just:

sudo apt-get install ruby1.8 irb1.8 rdoc1.8 ri1.8 \
             readline build-essential libopenssl-ruby1.8
sudo ln -s /usr/bin/ruby /usr/bin/ruby1.8 #repeat for irb,ri and rdoc
wget http://www.rubygems.org/rubygems/rubygems-1.3.1.tar.gz
tar zxvf rubygems-1.3.1.tar.gz && cd rubygems-1.3.1
sudo ruby setup.rb

If you'd like a nice deb package that you can track with apt, you can replace the last step with:

sudo apt-get install checkinstall
checkinstall 'sudo ruby setup.rb'

Checkinstall is an awesome piece of software, it's so awesome it deserves it's own post so I'll skip over the details, check it out.

After setting up this step you should now have a fully functioning rubygems installation, test it by issuing gem -v

Next step is to get mysql installed and configured. I did this using apt-get, because it's easy:

sudo apt-get install mysql-server mysql-client libmysql-dev

Be aware that you need the mysql development headers in order to compile the mysql gem, which we will install next, alongside adding Github to our gems sources, installing rails and a bunch of other useful stuff.

sudo gem sources -a http://gems.github.com
sudo gem install rails mysql sqlite3-ruby capistrano hoe hpricot webby RedCloth rake

And that is pretty much all there is to it. you can now run your favourite editor (in my case gedit) and start writing Ruby.

There are a couple of Dell Mini specific issues that I had to do before I found this machine to be a truly comfortable development environment though.

Remove the bottom Gnome taskbar, and make the fonts and theme stuff as small as possible, 1024x600 is a pretty tight screen resolution for web stuff, so it's good to make the most of it. I use a combination of 8 pt Liberation Sans and Mono for my desktop/coding fonts, at 96dpi with subpixel smoothing, and slight hinting. Clearlooks Compact GTK theme, Metabox for Metacity and the GNOME-Brave icon set.
Slow the keyboard repeat rate down. The keyboard on the mini 9 is small, it's reasonable easy to get used to but it is exceptionally sensitive by default to repeated keypresses, which are hard to avoid on a keyboard this small. I turned my repeat rate up to it's longest setting in Gnome Keyboard preferences and the problem has pretty much gone away.
Turn off the trackpad while typing. This gets very annoying very quickly, the trackpad is pretty sensitive and due to the size of the whole unit, you can very easily get in the way of yourself while typing, especially for longer stints (like this blog post) thankfully the default synaptic driver handles this by default, you just have to enable the daemon. To do this either:
- run syndaemon -d -t in a terminal to make it active per session, or
- add an entry to Gnome -> Preferences -> Sessions to make it run on login.

That's pretty much everything I've done so far. If I find anything else that works really well I'll update this post. Also any comments/suggestions then email them to me :)

Fixing font rendering in Arch/Gnome

January 28 2009

If you use Arch linux you may have noticed that the default font rendering in Gnome is a bit lacking, especially if you don't have Xfce installed as well, this appears to be because Xfce installs gsfonts as one of it's dependancies yet Gnome doesn't. So the first to do is install it.

$ pacman -S gsfonts

The next thing I like to do is to install the liberation font set, and the microsoft web fonts. These are available using pacman and can be installed as follows:

$ pacman -S ttf-liberation ttf-ms-fonts

Then I like to install the versions of freetype, libxft and cairo with the cleartype patches applied. These are available from AUR, so grab them and install them. If you have yaourt you can simply do

yaourt -S cairo-cleartype freetype2-cleartype libxft-cleartype

Then install the packages that it creates (usually in /tmp/yaourt somehwhere). NB: Make sure that if you have installed any of the non cleartype versions of these packages using pacman that you remove them before installing the new versions. If you don't have yaourt installed for whatever reason you can grab the tarballs directly from the AUR repo and build the packages like this:

$ tar zxvf package.src.tar.gz
$ cd package
$ makepkg
$ pacman -U package-i686.pkg.tar.gz

The remaining step is to configure your font preferences in the Gnome Appearance dialog, my preferred settings are:

Resolution 96dpi
Subpixel (LCD) smoothing
Slight Hinting
RGB Sub-pixel order

Now you should be able to restart X and enjoy some much better looking fonts