NPR & New-sweek get Shuffled.

I heard one of the most irritating news/interest bits I ever have on NPR (WHYY, 'round these parts) earlier today. On the way to work, I think.

A Newsweek editor (the famous one, but I've forgotten his name and, really, it's not relevant) was commenting on the randomness of his new iPod Shuffle, or the apparent not-quite-randomness of it. He believed that that his Shuffle had a prediliction for Steely Dan, as various songs by that band came up more frequently than other artists on it, which came up significantly less frequently than he thought that They Should.

Questing for the reason, he went a-interviewing. First Steve Fucking Jobs, who assured him that the iPod Shuffle did not have its own musical preferences. Second, a couple of (thankfully) unnamed mathematicians and cryptologists (no doubt, self-declared), who told him that he was making things up and that the human brain sought out patterns in randomness. The problem here is that THESE PEOPLE ARE ALL MORONS.

No, the iPod Shuffle probably isn't even close to having a cryptographically secure level of random, and there's no reason it needs to (nor would you want to wait for the puny processor in the thing, heavy on the music-decoding bits, light on the hard-integer-math bits, to cough and choke its way through a cryptographically secure PRNG anyway)... but that's not even the point. The point is that there are two broad-side-of-a-barn-obvious explanations for this:

1. There's more Steely Dan than other music in his iPod Shuffle. Therefore, there's a greater stastical probability of getting Steely Dan.
   
2. All of these portable music players, and most in-general-purpose-OS music players, don't even pretend to do real random, because the (time) cost of doing so would grow exponentially linearally (see below) with the number of tracks being randomized. Instead, they panel by something (by Artist name, by Album name, by Track name, by--as does my Rio Karma--internal hash representation within DB-not-FS storage of a given track), getting to a panel of some number reasonably large (> 64) to be inclusive of related stuff and reasonably small (< 1024) to avoid being totally tiresome and also not take too long with the un-sort algorithm (I think I've seen this done as the opposite of a Bubble Sort, which is un-sane... have we not learned our qsort lesson yet?), then they may bother interleaving panels, but probably don't.

And, you know what? It's better this way. As I mentioned, the Karma (seems to, I don't have source) panel by its internal hash (based on track length, id3-or-whatever-else tag, and the POSIX time of addition to the player for uniqueness, as near as I can tell), but I happened to dump my music onto it as albums, or directories of artists with subdirectories of albums. So I'll often hear one track by an artist and be reminded of another (not necessarily the next track, not necessarily even off the same album) during a long drive (I've been doing quite a few lately)... and not have to go searching for that track, calm in the knowledge (from common-sense observation of my music player's behavior) that that track would pop up soon enough. And, sure enough, it does.

What's more: this is actually a selling point in the advertising for the iPod Shuffle. They laud the way that it may seem to find interesting correlations in your music. If that's not just total marketing BS, it might be because they're looking at the "type of music" portion of the id3/whatever AAC uses tag as part of its paneling, and actively intermixing your trip-hop or Intelligent Dance Music (yeah... wuh? Also, greetz, 'dolph) with other like-labeled music... a practice some, who've actually seen the bastardization everybody in the fucking world but me brings on the "style" slot, would argue is woefully ill-informed, but that's neither here nor there if they do it right in iTunes, because they'll still get lucky plenty of the time.

Paneling is not a new concept, though I didn't really see the benefit of it for less-than-pseudo-randomizing until I got the Karma. Remember $FORMER_EMPLOYER (more on them in a bit, actually)? It's frequent that they want a subset of the vaguely 350 mil. individuals in the US (or the somewhat fewer than that households), and wanted to do some operation (often, merging new addresses in, and removing redundancies--people don't like getting junk mail to begin with, they really hate getting the same junk mail repeatedly) on that whole group. But this is a huge operation, one which you'd really like to split up and not run on that big Sun, with its fast I/O and slow CPU, but on those cheap Intel-driven machines with their crappy I/O but fast CPUs, so you want a way to split out the work and feed it to four or five of those machines. How do you do it? Well, often by first-digit-of-zip-code. Sure, some zips (9xxxx comes to mind) are heavier than others, but it's Close Enough most of the time.

People, computer science is really so very Not Hard. You need to understand a couple of Very Simple first principles, and you need to possess the capability to Reason. It's just not asking for much. Computers are not wacky, unpredictable, and scary. Unless broken (or obfuscated, thanks much Microsoft), they function along predictable, logical paths. Please to not be trying to read some sort of innate emotion into them. There's No There There.