I upgraded my Mac recently, and moved from a laptop with two internal drives – an SSD and an HDD – to a single, large SSD. I wanted to note down the aspects of this transition I’d have like to have known beforehand, and also want to know for

My old computer placed various core user folders – Documents, Music, etc – onto the HDD, whilst storing the System and Applications on the SSD. I symlinked the User folders into my home directory, and OSX was none the wiser.

As far as backing up goes, I used to use the excellent SuperDuper. When I moved to my symlinked setup, I went to Time Machine, which can back up multiple drives to a single Time Machine volume. I told it to back up both the SSD and HDD; it backed up the symlinks themselves from the SSD, and the directories they referred to from the HDD.

When I transfer data to a new Mac with Migration Assistant, I do so from a backup or bootable clone of the original computer – never from the computer itself. I chose to restore from Time Machine. This will only restore from a bootable drive – so I was only given the option to restore from my SSD, which contained Applications and my Desktop, but not my Documents. This unnerved me a bit at the time.

However, this is because the Migration Assistant only offers to restore from bootable (system) drives. Once I’d restored, I booted up the new system, mounted my Time Machine drive, and dragged everything over from the “Latest” directory symlink inside the HDD’s backup folder; I soon had my documents and other files back in the original locations they should have been.

So that’s the main note I wanted to make: restoring from that kind of setup works entirely fine, but you musn’t panic when your HDD isn’t offered as a volume to restore from.

I’m always impressed with Migration Assistant – it holds onto my system preferences, my eccentric Unix setup, my MySQL and Postgres databases, everything like that. What’s worth remembering is the stuff it doesn’t:

  • I use Caps Lock as my Control key, and vice versa. (Blame Vim). I had to re-specify this preference, and frequently, it would get overwritten on wake-from-sleep. A reboot (which presumably repaired permissions fixed this).
  • For a while, I noticed git wasn’t working. This was because it was not installed through homebrew, but directly into /usr/local/bin – and that was no longer in my PATH. Adding it back to my PATH re-enabled it. (All my homebrew binaries were working fine).
  • My /etc/hosts had been overwritten back to default; I need to copy that across manually. As had all my Apache config (/etc/apache2/httpd.conf), my vhosts config file (/etc/apache2/extra/httpd-vhost.conf), and /etc/php.ini . Nothing we can’t recover quickly, but it did mean the usual dance of setting up vhosts for sites that already existed, and pointing php.ini at the correct MySQL socket file (/tmp/mysql.sock on OSX).

I think that’s it – everything else transferred entirely seamlessly, and there was nothing to fear about my unusual setup – you just have to make sure you’ve been backing up correctly, and everything will work.

I’m working on a hardware project at the moment that’s more complex than a basic microcontroller-build that I could have implemented with an Arduino. So I’ve been using a Raspberry Pi as the centre of the project, and writing my code in the high-level languages I’m most proficient in. (In this case, Ruby).

As the project nears completion, it’s really important that the device doesn’t manifest as a computer in any way: it’s just a physical object. To that end, I’d like all the software inside the miniature computer to run at startup without any manual intervention.

My Pi is running Raspbian – the Pi-focused version of Debian – so, fortunately, there’s a tool easily available to do that for us: upstart.

We’ll write an upstart configuration for our script, which will turn our script into an upstart service, which we can then start at login.

First, let’s install upstart:

sudo apt-get install upstart

This will issue some warnings, because – on my install – it was replacing the traditional init.d setup. Don’t worry; everything will continue to work.

Once you’ve installed upstart, reboot your Pi, either from the command line or with a power-cycle.

Let’s now make an upstart config file. Here’s a very basic one:

Put this code into /etc/init/myscript.conf. You should then be able to run the script by typing sudo start myscript, and kill it with sudo stop myscript. And, of course, you’ll discover it’ll start automatically on startup.

That’s a very simple example, with no dependencies. But that won’t work for the script I’d like to use. In this example, I’m running a Ruby script (using the Pi Piper library) that blinks an LED attached to GPIO Pin 17. That script needs to be run as root to get access to the GPIO pins, and it needs to reference the directory’s Gemfile. Upstart scripts are run as root, so that’s not an issue – but we need to set up the environment correctly. That’s not so hard:

As you can see, we just have to export the BUNDLE_GEMFILE variable so that Bundler will know where the Gemfile is located.

Also, you’re going to have to make sure that all references to files in your code are done with absolute paths. That wasn’t a problem with the simple shell script example, but becomes an issue with more real-world type code – especially the program I’m ultimately running, which has various includes, dependencies, and data files to load. An obvious place you’ll run into this with Ruby is when setting up the $LOAD_PATH.

Rather than starting my Ruby script

$LOAD_PATH << 'lib'

I had to do this:

$LOAD_PATH << File.expand_path(File.dirname(__FILE__)) + '/lib'

And similarly, any other references to file loading will need to be absolute – and, ideally, derived using tricks like File.dirname(__FILE__) rather than through hardcoded paths.

Anyhow: it took me a while to piece together, but now that I have, it felt worth writing down – because I’ve now got reasonable complex computation-backed hardware working in an entirely headless enclosure – and one that’s resilient to power-cycling.

Ars Technica has a short article for HyperCard’s 25th Birthday.

I’m not sure I quite buy the notion of HyperCard as proto-web-browser. But I totally buy Atkinson’s original goals with it:

“Simply put, HyperCard is a software erector set that lets non-programmers put together interactive information”

It was not the first thing I wrote software in – that honour goes to GW-BASIC, I think – but it was the first tool I made something useful and unprovoked in. I was eight or nine when I discovered it at school. It made it possible to realise what was in my head, not what was in a book.

And it was the first thing that made designing the visual interactions of software easy for me. Software isn’t just arithmetic and lines of code – it’s something people use. HyperCard made sure that the visual end of software was usually the first part of a stack you made, not the last. (I was always disappointed that Visual Basic looked like it did this, but it didn’t quite live up to expectations).

Look at XCode now, with its integrated Interface Builder; that’s one of the many legacies of HyperCard. It showed the average computer user (not the average programmer) that interaction and interface was important to great computing experiences, and gave them the tools to poke around.

It is a tiny percentage of the reason I do what I do now, but a memorable one.

A new year, and a new toy to begin it.

This all began when Tom started tweeting the prose from the back of a chocolate box.

Tt tweets

One look at that and, having gagged a little on the truly purple prose, there was only one obvious continuation: a machine to churn out chocolate descriptions infinitely.

Which was as good a time as any to play with Markov chains. Wikipedia will explain in more detail, but if you’ve never encountered them, a very rough explanation is: Markov chains are systems that model what the next item in a list will be based on the previous ones. The more previous items you have, the better it can predict the next thing.

They’re often used in toy text generators. You give them source text to seed them, randomly pick a word from the source text, and then start choosing what should come next. What’s nice about this is with nothing other than a piece of maths, and a tight corpus, we can produce things that usually read like English without having to teach a computer something as complex as grammar. Of course, sometimes you get grammatical-yet-nonsensical English out, but that’s hardly in a problem in our case.

So I took the full prose from the back of Tom’s chocolates (Thornton’s Premium selection, for reference), some Markov text-generation code from an illuminating installment of Rubyquiz, and fiddled for a bit.

A short piece of work later and I had Markov Chocolates.

Markov

Roughly once every four hours (but it varies), you’ll get a fresh, tasty new Markov Chocolate in your Twitter feed. It’s another of my daft toys, but it still makes me chuckle. I’m thinking of expanding the corpus soon, and I hear the Markov coroporation are keen to branch out into new product lines. For now, you can get your chocolate fix here.

Drivetime Spotify App

04 December 2011

Drivetime

Music Hack Day London this weekend. Tom, Blaine and I ended up making a thing which didn’t demo perfectly, called Drivetime.

It solves a common problem: how do I listen to the same classic pop and rock anthems on a Friday afternoon as people in other studios? The answer is with the Drivetime app.

Drivetime is a Spotify app. You drag a playlist into it to start broadcasting – you set your station name with a simple text field. Then, anyone loading the app can click on your name and hear what you’re playing. Not just the current track – it’ll start the track for others at exactly the point you’re currently at, and it’ll advance through the playlist when a track finishes. At any point, you can stop listening, and choose another song to listen to.

That’s it, really: a simple broadcast/listen system, all built into Spotify as a native application.

Drivetime Screengrab

The front-end is just HTML/CSS/Javascript, which is how Spotify apps are built; the communication with the client is via socket.io, on top of node.js on the server. Tom and Blaine wrote the back-end; I wrote a bunch of the front-end, which Blaine promptly made much better, and did all the hot pink neon. We all played a lot of Tina Turner in the hacking rooms, and we went home to bed betimes on the Saturday night.

All the code’s in github. The Spotify app only runs (at the moment) if you’re a Spotify developer. We might see if we can get it a bit further and release itspot, so people can listen to one another’s classic rock selections.

It was a fun hack, even if my explanation of it was incoherent, and part of a great weekend – so many superb hacks, large and small, on display.

I was all ready to get really worked up about this post from Wieden + Kennedy, on “why we’re not hiring creative technologists any more; we’re hiring coders”.

Then I went and read it, and basically agreed with it all very fundamentally. In a nutshell: it’s not resisting the name, or the approach; it’s resisting the idea that it’s something you can pick up quickly on a course to teach you to become one. Which is, like so many similar things, nonsense; I hadn’t realised we’d hit that point on the sliding scale. When Igor says

“Only hire people to work at the crossover of creative and technology if they have strong, practical, current coding skills.”

I say: of course; why would you do otherwise? I thought that’s what that job title meant. I seriously didn’t realise that was happening, and I’m very concerned it does. This is worth taking very seriously: if you want people to think through software, they need to be able to make that software. Not wave their hands around and have ideas about technology. We think with our hands, be we artists, designers, developers, or writers. Having another layer of people to “have ideas” is not what you need. Ideas are free.

I still use that particular title to describe myself a lot, simply because I’m not best at being your average Joe Developer. I can; I have been; it’s just not my sweet spot. I’ve made reasonable scale projects that work well; I understand how to go from a fragile prototype and turn it into solid reality, and what making things work under load looks like: and yet the bit I’m good at, the bit I care about, is the going-from-nothing-to-something-working. How will you know what a thing is until you’ve held it in your hand? How fast can you change it as you learn from it? When’s it best to step away from vim and go back to pen and paper? That’s me.

So: I totally agree with Igor about the fact that whatever you call that role, it has to have solid, actual coding chops. Not a smattering of Processing here and some weak PHP there: actual, full-on, end-to-end skills. Code that’s live in the world.

But: the most interesting thing in the article wasn’t even the stuff about Creative Technology. It was about what it means to be an agency – or, being honest, a company – that wants to engage with technology through staff members like these.

While you don’t need to become an engineering company, you face some of their challenges. You need to understand, accept and embrace some of the nuts and bolts of software development, and take on board the work dedicated shops are doing on its processes. You need such a strong streak of code running through the atmosphere that coders want to come to you, and everyone else gets code spilling over them.

This is so true. You can’t just slap technologists or developers into a company to become a technology company. Technology has its own heartbeat, its own demands. You have to begin to wrestle with the processes of an engineering company, of an attitude that leads to better work. You have to learn how it’s going to shape your culture – by which I mean, how you want it to. You get to choose; you get to control these things. It will change it, that’s for certain, but you get to hae some control over it. And similarly: you have to resist it just enough to stop becoming nothing but a software house; to retain the “creative” streak you were trying to hang onto when you started hiring for that job title.

The article it ends in this nugget:

this is hard, and it’ll take time. It’s not just procedural, but cultural, so a big part of doing it comes down to who you hire and how you let them do their thing. But that’s exactly the point. That’s why it’s most important, way before you get all that fixed, and as the first major step on that road: just don’t hire “creative technologists” who aren’t strong coders.

Yep. That’s his real point: the headline is attention-grabbing, but here’s the meat, and the most important line here is this is hard and it’ll take time.

It’s a cracking post. It’s all true. I’m going to stick to my guns and say I’m a technologist, of some kind: what I am best at is not one thing, but a mish-mash of things, and I’m better for the diversity of them. But I’ll also stick my head up and say yes, at the end of the day, if you want the Whole Thing Just Made: I will do that. I can do that. That’s why I get to use the T-word.

My only other advice for filling these positions: you don’t just need people who can do these things; you need people who can’t not do these things. Their instinct when faced with problems ought to be “let’s see what works; let’s check the assumptions we’re making are true by Just Doing It.” It’s not about jumping the gun: it’s recognising when you need to feel something, rather than guess something. And you don’t want to have to train that: you want people who just have to know for themselves.

So yeah, if you’re in one of those places that isn’t a software company, but you increasingly need to be a software company because, as Igor says, we have to be – that’s what the modern world now looks like – then it’s a really, really sharp piece of writing. I went in sceptical, but really: it was telling me what I already believed, and confirming it, and that’s a good thing, because it’s a message that needs to be written, not just assumed we all know. Good stuff.

The Setup

06 August 2011

A while back, I was interviewed for The Setup, and that interview is now up on the site. In it, I talk about what I use to get stuff (vaguely) done. Almost none of it will be surprising if you know me and what I do, but worth linking to nontheless.

If you’re working in Ruby, you’re probably using Bundler. And if you’re using bundler, you’ll probably know that typing bundle install foo will install your bundle to a directory called foo.

Of course, the problem is that Bundler remembers this configuration, and if you now run bundle install, you’ll install your bundle to… foo.

This is annoying. It’s especially annoying if you never meant to install to foo, and that was just a typo.

So: if you want to reset bundler to installing to the default location – which is your system’s current gem folder – you’re going to spend up a good hour messing around on Google looking for a plain English solution.

Can you guess who did this, and who this article is written for? That’s right, it’s me in the past!

Your solution: just run bundle install --system. That’ll install your bundle to the default system location – and continue to do so in the future. Problem solved.

(As usual, when I write about how to do something technically, it’s because I couldn’t find the answer. That’s all.)

So, here’s a thing I’m making.

My Nikon D90 can be triggered by the cheap ML-L3 IR remote. It costs about £15. You point it at the camera, push the button, and it takes a picture.

This remote works with everything from the D90 down (so, towards the D3100/D40 end of the line).

What these cameras don’t have built in, however, is an intervalometer: a timer that will make the camera take a picture every n seconds or minutes. (The D300 and up (and, I believe, the new D7000) have a built-in intervalometer.)

I thought it might be interesting to build one. The project had a few criteria:

  • It couldn’t be hard-wired to a computer. It had to be a stand-alone, battery-powered device
  • It had to have a half-decent way of controlling it; ideally, not just stabbing at buttons.
  • I wanted it to have a 16×2 LCD screen, mainly because I wanted to both design for that constraint, and work out how to control said screen
  • Ideally, it wouldn’t require taking apart an ML-L3 remote to build.

Here’s where we are:

End-to-end: it works. Note that I said “making” earlier, though: it’s still not finished, because it’s not packaged. And whilst packaging is difficult, I think that’s what’ll make it feel finished for me: a black box I can easily take into the field.

You turn it on, rotate an encoder to set a time, and click the encoder in to arm it. Hold the encoder to disarm. The time varies from 1 second to 15 minutes – after 90 seconds, it increases in minute chunks. (15 minutes is the maximum time the D90 will stay on before powering down).

Most people ask me why it says “SAFE” and “ARM”. Well, it sounds a bit threatening, but I genuinely believed that OFF and ON were inaccurate, in that the device is “on” if the screen’s on. So I was just referring to the state of the timer. And something that fitted into four characters would work well with the layout of the screen I’d chosen.

How it works

There’s very little componentry here, but each section of the Intervalometer was a neat little thing to learn on its own.

First, the IR trigger itself. Nikon’s IR remote is relatively simple: a button, some circuitry, and an IR LED. Pushing the button doesn’t just light up the IR LED; it fires a very short “coded” burst of light, so that the only thing it’ll trigger will be a Nikon camera.

Fortunately for our needs, there’s an Arduino library called NikonIRControl, which emulates that coded burst in software – so a single command will send the appropriate burst to a digital output pin. That’s our IR trigger sorted, and all we’ve had to buy is an IR led. Which feels better – and cheaper – than just soldering two wires to a Nikon remote.

The screen is a 16×2 LCD, with a serial “backpack” pre-attached. That means I can just send serial data to it over a single wire, which again, keeps the number of wires from the Arduino down. I’m using the NewSoftSerial library to talk to it, which makes life easy.

The main controller is a quadrature rotary encoder with a push-switch in it. The switch is easily read, like any momentary push-switch on an Arduino. The encoder is a little trickier, because it’s encoded. In the end, I read it off an interrupt, using code from this page – and then smoothed it out a bit by making it only read every other click.

Finally, there’s just the case of the timer. Timers are a bit more fiddly than I’d have liked. You can’t just use delay, because that delays all code on the chip. I tried doing various things including counting milliseconds, but in the end, relied on the TimedAction library, which works well enough, and does a similar thing without my broken code.

Once each piece was in place and working, it was just a case of pulling it all together. The code – which is available on Github – is broken down into a series of files, pretty much one for each section of the project. I found this much easier to manage than the tyranny of One Big File.

For piecing the hardware together, I built a simple “shield” out of a piece of Veroboard. I got a lot of laughs when I said I was using veroboard, but it worked very well for me. With some headers soldered in, it was quite easy to line it up with the Arduino – making it easily changed, but also swappable. The usual electronics-debugging issues aside, this went fairly smoothly, and it only took a battery pack to give me a portable – if fragile – working intervalometer.

What’s next? Obviously, packaging it up – something sturdy and black, with an obvious power switch and that big knob. I was considering moving it to an Arduino Mini, for size reasons, but am not sure I can face more electronics debugging. Similarly, I’m not sure I’ll build a dedicated PCB or anything like that, yet.

But: if I can get this lot into a box, that’ll be good. Also: I should take some timelapses with it.

So whilst it’s not what I would call “finished”, it is an end-to-end demo – and that feels like good enough to share with the world.

(And, of course: if you’d like to use – or build on – my code, you’re more than welcome to.)

Something on my mind grapes

26 September 2010

Silly 30-minute weekend project: this led to this. It usually raises a smile, but every now and then (as above) it’s solid gold.

(Also, I found a neat pattern for decoupling all the OAuth keys so that it’s much easier to distribute/opensource the sourecode, which I’ll probably do at some point).