How To Load Bundled Images In A Prism Webapp

One of the features in the Prism Webapp Bundle for Google Wave is a toaster pop-up notification of unread waves using the window.platform.showNotification() method.  The third parameter is named aImageURI, and is described by the nsIPlatformGlue IDL as, “The URI of an image to use in alert. Can be null for no image.”

Which is great, except … what URI scheme and path does one use?  Every example I could find always passed null for the image, so after giving up on the web I joined the Prism mailing list and posted a question.  The first response was to use the inline data scheme, with a base64-encoded image.  It was ugly, but it worked.

A later response, however, yielded the proper way:

resource://webapp/path/to/image.png

A Prism Webapp Bundle For Google Wave

Though Prism and Google Wave go great together simply creating a web app from the Prism Firefox add-on, Prism supports some script extensions that allow for more desktop-like integration of apps running inside it.  For example, you can call the window.platform.showNotification() method to cause a little toaster pop-up with the number of unread Waves.

I’ve created a webapp bundle that does just that.  Unfortunately, such bundles at present only work with the stand-alone version of Prism.  The Firefox add-on is really a better way to run Prism, but if you’re using it you’ll need to do a little manual mucking in your webapp profile to use this bundle.

Stand-Alone Bundle

So, if you just want the bundle, here you go.  Note that I haven’t really tested it on the stand-alone version, so please let me know if something is broken.

Hack Your Webapp

As I said, if you’re using the add-on version, you’ll need to do a little manual hacking.  After you create the webapp, as described in my earlier post, open up Explorer and navigate to your Prism webapp bundle cache.  On Windows, this is in %APPDATA%\WebApps (something like C:\Users\Brian\AppData\Roaming\WebApps); on Linux, it is ~/.webapps.  You should see your Google Wave webapp in that directory.  Add the webapp.js script to that directory, and also add in images/google-wave-52×32.png.  Now you should get a toaster pop-up and task bar notification when there are new waves.

It would be nice if Google were to add a <link rel=”webapp”> to Wave, referencing an appropriate bundle.  If anybody there sees this and cares to use my code as a crude starting point, I am releasing this code under an MIT license.

Google Wave and Prism: A Match Made In Heaven

I received a Wave invite from Tim this morning. (Thanks, Tim!)  I’m still not sure of Wave’s usefulness as a tool, although I had quite a positive experience doing a little collaborative feedback and editing.  However, after about five minutes of using it, I was sure of one thing:

This thing screams for its own window.

That’s where Prism comes in.  Prism allows web applications to be run in a separate browser process, complete with a separate profile, their own window, and a unique taskbar icon.  For long-lived applications like a calendar or a chat tool, this is far more useful and stable than opening yet-another tab.  Furthermore, I like to read web pages in a tall window (roughly the same proportions as an 8.5×11 piece of paper), but I prefer my communications tools in a wide window.  Prism let’s me easily size Wave however I’d like.

After you install the add-on and restart Firefox, just navigate to Wave, click on the Tools menu in Firefox, and click Convert Website To Application.  You’ll want to cut out the cruft from the end of the URL, leaving just https://wave.google.com/wave/.  And it’s usually helpful to leave the status bar in place.  If you’d prefer to have wave in the system tray, you can check that box here, too.

You’ll also want to pick a different icon – especially if you’re on Windows 7.  The default favicon.ico that Prism auto-downloads is very small, and scales up really poorly.  Here’s a 256×256 one that I used as a PNG and as an ICO.  It looks great in my task bar.

I also use Prism with Google Calendar and Toodledo, and love it.  And I’m about thiiiis close to pulling Google Docs into it as well.

How To Make Java Ignore IPv6

Sure, IPv6 is going to save us all from the apocalypse, defeat communism, cure the swine flu, and bake you the most delicious brownies you’ve ever tasted.  Someday.  But in the meantime, for real people trying to do real work, it’s a fucking nuisance.

As more systems have started shipping with the technology, little compatibility issues continue to crop up.  One of the more recurrent problems I’ve encountered is incompatibilities between Java and IPv6 on Linux – specifically Ubuntu.  Up until recently, it was quite easy to eliminate the problem by merely blacklisting the appropriate kernel modules, thusly:

# echo 'blacklist net-pf-10' >> /etc/modprobe.d/blacklist.conf # echo 'blacklist ipv6' >> /etc/modprobe.d/blacklist.conf

However, as of Ubuntu 9.04 (Jaunty), IPv6 support is no longer a module – it’s hard-compiled into the shipping kernels.  No big deal, though, because there’s a system control variable that allows you to remove IPv6 support from the kernel.

# echo 1 > /proc/sys/net/ipv6/conf/all/disable_ipv6

Except that doesn’t work.  It seems there was a kernel bug where that setting was just plain broken.  And it hasn’t been shipped with the normal Ubuntu kernels yet.  So, what is one to do, short of re-compiling their own kernel?

Here is a copy-paste from an IM exchange I had with Java earlier:

# Java has entered the chat.

Java: Hey dude, what’s up?

Ardvaark: hey, i’m having a problem getting you to listen to an ipv4 socket when ipv6 is installed on my ubuntu box

Java: Yeah! I totally support IPv6 now! You didn’t even have to do anything because I abstract you from the OS details! Isn’t that great?!

Ardvaark: awesome, i guess, except that it doesn’t work.

Ardvaark: i really need you to just listen on ipv4, because the tivo just doesn’t like galleon on ipv6

Ardvaark: so sit the hell down, shut the hell up, and use ipv4

Ardvaark: pretty please

Java: Okay, geez, no need to get all pissy about it.

Ardvaark: and while you’re at it, could you please stop using like half a gig RAM just for a silly hello world program?

Java: Don’t push your luck.

# Java has left the chat.

And now that we’re back in reality, the magic word is -Djava.net.preferIPv4Stack=true.

Hadoop World 2009

I had the privilege of attending Hadoop World 2009 on Friday.  It was amazing to meet, listen to, and pick the brains of so many smart people.  The quantity of good work being done on this project is simply stunning, but it is equally stunning how much farther there remains to go.  Some interesting points for me include:

Yahoo’s Enormous Clusters

Eric Baldeschwieler from Yahoo gave an impressive talk about what they’re doing with Hadoop.  Yahoo is running clusters at a simply amazing scale.  They have several different clusters, totally some 86 PB of disk space, but their largest is a 4000-node cluster with 16 PB of disk, 64 TB of RAM, and 32,000 CPU cores.  One of the most compelling points they made was that Yahoo’s experiences prove that Hadoop really does scale as designed.  If you start with a small grid now, you can be sure that it will scale up – way up.

Eric made it clear that Yahoo uses Hadoop because it so vastly improves the productivity of their engineers.  He noted that, though the hardware is commodity, the grid isn’t necessarily a cheaper solution; however, it easily pays for itself through the increased turnaround on problems.  In the old days, it was difficult for engineers to try out new ideas, but now you can try out a Big Data idea in a few hours, and see how it goes.

A great example is the search suggestion on the front page.  Using Hadoop, they cut the time to generate the search suggestions on the front page from 26 days to 20 minutes.  Wow!  For the icing on the cake, the code was converted from C++ to Python, and development time went from 2-3 weeks to 2-3 days.

HDFS For Archiving

HDFS hasn’t been used much as an archival system yet, especially not with the time horizons of someplace like my employer.  When I asked him about it, Eric told me that the oldest data on Yahoo’s clusters is not much more than a year old.  Ironically, they tend to be concerned more about removing data from the servers due to legal mandates and privacy requirements, rather than keeping it around for a Very Long Time.  But he sees the need to hold some data for longer periods coming soon, and has promised he’ll be thinking about it.

Facebook, though, is already making moves in this area.  They currently “back up” their production HDFS grid using Hive replication to a secondary grid, but they are working on (or already have – it wasn’t quite clear how far along this all was) an “archival cluster” solution.  A daemon would scan for least-recently used files and opportunistically move them to a cluster built with more storage-heavy nodes, leaving a symlink stub in place of the file.  When a request for that stub file comes in, the daemon intercepts it and begins pulling the data back off the archive grid.  This is quite similar to how SAM-QFS works today.  I had a chance to speak with with Dhruba Borthakur for a bit afterwards, and he had some interesting ideas about modifying the HDFS block scheduler to make it friendly for something like MAID.

Jaesun Han from NexR gave a talk on Terapot, a system for long-term storage and discovery of emails due to legal requirements and litigation.  I asked him about whether they were relying on HDFS as their primary storage mechanism, or if they “backed up” to some external solution.  He laughed, and said that they weren’t using one now, but would probably get some sort of tape solution in the near future.  He also said that he believed HDFS was quite capable of being the sole archival solution, and I believe he was implying that it was fear from the legal and/or management folks that was driving a “back up” solution.  At this point, the Cloudera CTO noted that both Yahoo and Facebook had no “back up” solution for HDFS, except for other HDFS clusters.  It certainly seems like at least a couple multi-million dollar companies are willing to put their data where their mouth is on the reliability of HDFS.

What’s Coming

There is a tremendous sense that Hadoop has really matured in the last year or so.  But it’s also been noted that the APIs are still thrashing a bit, and it’s still awfully Java-centric.  Now that the underlying core is pretty solid, it seems like a lot of the work is moving towards making your Hadoop grid accessible to the rest of the company – not just the uber-geek Java coders.

Doug Cutting talked about how they’re working on building some solid, future-proof APIs for 0.21.  Included in this is switching the RPC format to Avro, which is intended to solve some of the underlying issues with Thrift and Protocol Buffers while opening up the RPC and data format to a broader class of languages.  It’s worth noting that Avro and JSON are pretty easily transcoded to one another.  Also, they’ll finally be putting some serious thought into a real authentication and authorization scheme.  Yahoo (I think) mentioned Kerberos – let’s hope we get some OpenID up in that joint, too.

There is a sudden push towards making Hadoop accessible via various UIs.  Cloudera introduced their Hadoop Desktop, Karmasphere gave a whirlwind tour of their Netbeans-based IDE, and IBM was showing off a spreadsheet metaphor on top of Hadoop called M2 (I can’t find any good links for it).  I hadn’t thought about that before, and it seemed so simple it was brilliant; Doug Cutting mentioned the idea, too, so it has some cachet.

Final Thoughts

It is worth noting that Facebook seems to be driving a lot of the really cool backend stuff, and people are noticing.  That’s not to say other organizations aren’t doing cool things, but during the opening presentations, Facebook got all the questions.  I mean, Dhruba recently posted a patch adding error-correcting codes on top of HDFS.  How cool is that?!

Death of My Kurobox

After serving faithfully for over three years, and a year-and-a-half after getting a hard drive upgrade, my Kurobox Lulu died this past week.  I suspect a power surge caused a stroke.

You see, I went to turn on the printer, and it started cursing at me with long beeps and flashing lights.  Google told me that I should try unplugging it, holding down some keys, and then plugging it back in.  Unfortunately, because I didn’t have the plugs labeled, I accidentally unplugged Lulu.  She wouldn’t start up when I turned her back on.

Connecting the net console to the bootloader, I discovered that it would hang shortly after decompressing the kernel.  Worse, it failed the exact same way when trying to boot into EM mode.  I could poke around on the hard drive, thanks to U-Boot’s features, so I was left to conclude that the motherboard had been fried – probably the memory.  The coincidence with the printer’s death leads me to believe it was a power surge.

So now I had to decide with what to replace the venerable Lulu.  Despite her minuscule memory and pathetic processor, she had served admirably in several roles, including a Galleon TiVo server, a Samba file server, a print server, a scan server, running a TwitterBot in IRC, and regularly syncing data to Amazon S3.  Though three years had passed, the Kurobox was still selling new for $150, or more if you got their fancy ARM version.  Skeptical that they were still worth that much, I started shopping around.

So, what can you buy today for the $150 you could spend on a Kurobox?  How about a 1.6 GHz Atom 230 CPU/motherboard combo ($64.99), 2GB of Kingston memory ($35.49), and a Mini ITX Computer Case with 200w Power Supply ($39.99) in which to house it all.  So for the exact same price as the Kuro, you can get a modern, mainstream CPU and a generous quantity of memory.  I was prepared to spend more – not having to deal with the headaches of U-Boot and a PPC Gentoo build would have been worth an extra hundred easy – but at the same price it was just a no-brainer.

So I ordered it all, and it should all be arriving this week.  Hopefully I’ll be able to get new Lulu on her feet quickly; Hedda and I are both really missing her.

Interpreting the Imperial Network

A guest post by ferrix.

Windows Installer: As anyone who has done .msi development knows, you will never find a more wretched hive of scum and villainy.

Visual Studio 2005 valiantly tries to make things easier by offering “Setup and Deployment” projects.  This thing magically binds together the build outputs of other projects and burps out a plausible .msi file.  Hoora……waitaminute, something’s not quite right here.  Yeah.. it turns out that if you want anything but the barest minimum of shoving files and registry keys onto target machines, you’re going to have to do some post-processing, son.  Fortunately, Microsoft provides a handy COM API for torturing the .msi SQL database until it agrees to do your bidding.

What?  Oh, sorry…you didn’t know that an .msi was basically a demented relational database crammed into a file?  Congratulations, now you can share my nightmares.

But I come here not to complain about the .msi file format, nor Visual Studio.  The main course of today’s rant will be the installer engine itself, msiexec.  Specifically Windows Installer 4, which led me on a merry chase today.  I accidentally missed a dependency for one of my custom actions, and got the following lovely error:

The installer has encountered an unexpected error installing this package. This may indicate a problem with this package. The error code is 2869

I’ve been hacking on these beasties for a couple years, this was not my first dance with 2869.  In fact, the internet is filled with stories about it.  This isn’t actually an error about what went wrong during the install, it’s an error about [what went wrong when the installer was trying to tell you [what went wrong during the install] ].  This is what we call a masking error, meaning “Your installer is so broken, I can’t even tell you about it properly”.  A detailed install log offers up:

DEBUG: Error 2869:  The dialog ErrorDialog has the error style bit set, but is not an error dialog

Most forum threads about this error are from hapless end users trying to get their program download to work, and vendors supplying fixed versions.  Everyone addresses the root cause, i.e. the error actually being thrown first, that eventually leads to the 2869.  Often it has to do with impersonation problems.  Well and good, but I already knew what was wrong with my custom action.  What I wanted, and couldn’t find anywhere, was someone who understood why the error reporting mechanism itself was failing.  (Spoilers: eventually found several right answers.  It’s easier to find them in retrospect once I knew what was wrong.)

What could be wrong with the ErrorDialog?  This guy came right off the truck from Visual Studio, and my tweaks never touched it.  Nevertheless, I spent about an hour poring over the documentation, trying to find any possible detail that was different between my .msi and the spec.  It all checked out totally fine.  But no matter what I tried, not an error dialog.  Not an error dialog.  My kingdom for an error dialog!

It’s such a pointed error, you see, and there are so many subtle requirements, I thought I must be missing something.  Was it the phase of the moon or something about the feng shui orientation of my laptop?  In casting about for a solution I happened upon this note.  It talked about adding an entry to the Error table, which is advice I hadn’t seen before:

In order to see the actual error, open the MSI with ORCA and add the following entry to the “Error” table.

1001 | “Error[1]:  [2]“

My logs never showed a 1001 error code, and a missing entry in the error table doesn’t have any relevance to the properties of the error dialog being correct.  And yet, and yet…  The page referred to 2869.  With nothing to lose, I tried adding the entry.  As if by magic, the error reporting immediately began working just fine.  Total changes needed to the error dialog: zero.  Total time wasted on this: one afternoon.

What happened?  In this case not only is the 2869 masking the underlying error, but the windows installer engine itself was lying about the nature of the masking error, and as a side effect of the problem, hiding the real error code (1001) to boot!  Why 2869 and not something like, “So listen, I see there’s no format string in the Error table for #1001… so regrettably I must now poop myself.”

I can totally imagine how it went down.  The developer needs to implement a new error formatting behavior in version 4, but in the event that the .msi has broken error handling, he has to tell the user about it somehow.  Adding a brand new error code would require changes up and down the source tree.  It’s almost deadline already, and hey look: this error 2869 is pretty close, it’s about error dialogs not working.  Surely anyone who gets that error will quickly understand what was meant.  One line of code, and under the wire home free.

In addition to venting, my goal in writing up this harrowing experience is to provide more link love for pages that show how to fix Windows Installer error 2869 when the log says your error dialog “has the error style bit set, but is not an error dialog” but really it is, and it’s a phony error hiding error 1001 instead.

These aren’t the errors you’re looking for.  Move along.

Chronicles of Windows 7 Part 2: Multi-Monitor Support

One of the awful, terrible, no-good parts of Vista was its multi-monitor support.  My set up is a laptop mounted in a docking station with a second monitor.  My mobile nature means I frequently dock and undock the laptop, requiring the operating system to detect how many screens there are, switch the primary display to and from the external monitor, and move all the windows back and forth, possibly repositioning them because of the difference in resolutions. Adding to the chaos is that I frequently sleep and wake the computer, sometimes docking it or undocking it while asleep, or shortly after waking.

Vista failed at this horribly.  Often it would fail to detect the new display, requiring me to hit the external-monitor toggle key several times.  On occasion it would detect the new display, but then drive the new monitor at the wrong resolution.  Sometimes, after the first full moon on the second Tuesday of a month where the average temperature had been between fifty-four and sixty-eight degrees Fahrenheit (I think), it would forget which monitor was supposed to be the primary, and I would be forced to re-tweak the settings for three days in a row – and then all would suddenly be fine again.

I won’t even mention what would happen if I put the laptop to sleep, undocked it, and woke it up later!  But to add insult to injury, the churnings of the graphics subsystem would cause flickers and re-sizes for five to ten seconds while it was figuring out whether things would work or not.  The entire experience was nothing short of maddening!

I am pleased to report, however, that Windows 7 has been flawless.  It never fails to correctly identify the monitors present, nor does it ever forget which display is supposed to be primary.  The resolutions are always correct, and it makes the appropriate adjustments quickly and with no flicker.  It even handles with aplomb every combination of sleeping, waking, docking, and undocking I can think of!

In short, it does exactly what it should.  Finally!

Twitter and Compression of Meaning

An interesting and often-mocked technical limitation of Twitter is the 140-character limit.  The semantics of the buzzword “microblog” aside, I notice in my own Tweets that the strict requirement for very short messages has improved my writing.

(Let me note, before going any further, that I am not an abbreviations kind of guy. I virtually always write complete words and sentences using proper grammar, syntax, and punctuation.  Though I do use emoticons from time to time, I limit myself to a standard subset of smiley-, frowny-, and winky-faces, with the occasional raised or angry eyebrow thrown in – basically anything I can make with a colon, a dash, greater-than, less-than, and the parenthesis.  I cringe when reading things like, “c u 2nite” or “lol” or “<_<”.  Perhaps that makes me an angry old man; if so, that is a mantle I will wear.)

When tweeting large thoughts, I find myself editing the message to fit within the allotted space while maintaining the same meaning.  The need to transmit the same semantics in a smaller space requires wielding more powerful language: stronger verbs, more-nuanced adjectives and adverbs, and better-placed punctuation and pronouns.  Squeezing the same content into less space is the very definition of data compression; so, in a very real sense, this is data compression for people.  What once might have rambled on for an entire paragraph now takes one or two concise sentences.

Users get more bang for their buck when reading a tweet.  Coupled with the instantaneous and ubiquitous accessibility of these meaning-laden tweets, it’s obvious why Twitter has taken off.

Like most compression, though, there is a tradeoff between space and time.  Though I have not measured, I have a strong sense that the ratio of time-editing-per-character is higher on Twitter than on any other medium I use.  On the receiving side, the message might take longer to comprehend, especially if the vocabulary is unfamiliar to the recipient.

And some messages, like some data, will simply never fit into the allotted space.  That’s why I still have a blog.

Chronicles of Windows 7 Part 1: Qualcomm Gobi 3G Modem and VMWare NAT

So I went ahead and installed Windows 7 RC 1.  The process is remarkably smooth, and the OS is nicely polished.  The new task bar is a long-overdue change, formerly difficult or esoteric system tasks are now simple and obvious, and the Libraries paradigm in Explorer has pleasantly surprised me.

But that’s not to say there aren’t some niggling issues.  This is a new release – nay, a pre-release – of the most popular operating system in the world.  There are bound to be some compatibility problems.  What is truly amazing is how well things work right out-of-the box.

As I use the OS day-to-day, I’ll post some updates about real-life surprises and tribulations.  Here are my first two.

Qualcomm Gobi 3G Modem

Winodws 7 recognized almost every single piece of hardware on my HP Elitebook 8530w, including the silly fingerprint reader and the webcam I never use.  The one thing it didn’t already have drivers for was the built-in Qualcomm Gobi un2400 modem 3G.  What’s worse, the Vista drivers from HP’s support site don’t install, either.

Fortunately, some amazingly enterprising soul figured out the problems, and was not only able to divine how to install the drivers, but then even wrote a schnasty little program to force-feed the Gobi modem its appropriate firmware.  Major kudos!  Unfortunately for me, it still doesn’t work.  There’s some magic incantation that isn’t being done quite right for my AT&T setup, so I’ll have to wait until the drivers get updated.  Hopefully that’ll be soon – paying for a data plan I’m not using is rather annoying.

But, really, given how esoteric and fragile these 3G modems are, it’s not that surprising something bjorked their spaghetti-like functioning.  (Did you read the “More About The Firmware” section at that link?!)

VMWare NAT Failure

The only other true problem I’ve had is with VMWare Workstation 6.5.  It works like a charm, except that NAT routing fails to work correctly.  Interestingly, the guests can ping out, but other connections fail.  It’s a known issue, though, and will certainly be fixed soon.  And the work-around is simple enough: Just use bridging instead.