Life and code.
RSS icon Home icon
  • Interpreting the Imperial Network

    Posted on August 2nd, 2009 ferrix No comments

    A guest post by ferrix.

    Windows Installer: As anyone who has done .msi development knows, you will never find a more wretched hive of scum and villainy.

    Visual Studio 2005 valiantly tries to make things easier by offering “Setup and Deployment” projects.  This thing magically binds together the build outputs of other projects and burps out a plausible .msi file.  Hoora……waitaminute, something’s not quite right here.  Yeah.. it turns out that if you want anything but the barest minimum of shoving files and registry keys onto target machines, you’re going to have to do some post-processing, son.  Fortunately, Microsoft provides a handy COM API for torturing the .msi SQL database until it agrees to do your bidding.

    What?  Oh, sorry…you didn’t know that an .msi was basically a demented relational database crammed into a file?  Congratulations, now you can share my nightmares.

    But I come here not to complain about the .msi file format, nor Visual Studio.  The main course of today’s rant will be the installer engine itself, msiexec.  Specifically Windows Installer 4, which led me on a merry chase today.  I accidentally missed a dependency for one of my custom actions, and got the following lovely error:

    The installer has encountered an unexpected error installing this package. This may indicate a problem with this package. The error code is 2869

    I’ve been hacking on these beasties for a couple years, this was not my first dance with 2869.  In fact, the internet is filled with stories about it.  This isn’t actually an error about what went wrong during the install, it’s an error about [what went wrong when the installer was trying to tell you [what went wrong during the install] ].  This is what we call a masking error, meaning “Your installer is so broken, I can’t even tell you about it properly”.  A detailed install log offers up:

    DEBUG: Error 2869:  The dialog ErrorDialog has the error style bit set, but is not an error dialog

    Most forum threads about this error are from hapless end users trying to get their program download to work, and vendors supplying fixed versions.  Everyone addresses the root cause, i.e. the error actually being thrown first, that eventually leads to the 2869.  Often it has to do with impersonation problems.  Well and good, but I already knew what was wrong with my custom action.  What I wanted, and couldn’t find anywhere, was someone who understood why the error reporting mechanism itself was failing.  (Spoilers: eventually found several right answers.  It’s easier to find them in retrospect once I knew what was wrong.)

    What could be wrong with the ErrorDialog?  This guy came right off the truck from Visual Studio, and my tweaks never touched it.  Nevertheless, I spent about an hour poring over the documentation, trying to find any possible detail that was different between my .msi and the spec.  It all checked out totally fine.  But no matter what I tried, not an error dialog.  Not an error dialog.  My kingdom for an error dialog!

    It’s such a pointed error, you see, and there are so many subtle requirements, I thought I must be missing something.  Was it the phase of the moon or something about the feng shui orientation of my laptop?  In casting about for a solution I happened upon this note.  It talked about adding an entry to the Error table, which is advice I hadn’t seen before:

    In order to see the actual error, open the MSI with ORCA and add the following entry to the “Error” table.

    1001 | “Error[1]:  [2]”

    My logs never showed a 1001 error code, and a missing entry in the error table doesn’t have any relevance to the properties of the error dialog being correct.  And yet, and yet…  The page referred to 2869.  With nothing to lose, I tried adding the entry.  As if by magic, the error reporting immediately began working just fine.  Total changes needed to the error dialog: zero.  Total time wasted on this: one afternoon.

    What happened?  In this case not only is the 2869 masking the underlying error, but the windows installer engine itself was lying about the nature of the masking error, and as a side effect of the problem, hiding the real error code (1001) to boot!  Why 2869 and not something like, “So listen, I see there’s no format string in the Error table for #1001… so regrettably I must now poop myself.”

    I can totally imagine how it went down.  The developer needs to implement a new error formatting behavior in version 4, but in the event that the .msi has broken error handling, he has to tell the user about it somehow.  Adding a brand new error code would require changes up and down the source tree.  It’s almost deadline already, and hey look: this error 2869 is pretty close, it’s about error dialogs not working.  Surely anyone who gets that error will quickly understand what was meant.  One line of code, and under the wire home free.

    In addition to venting, my goal in writing up this harrowing experience is to provide more link love for pages that show how to fix Windows Installer error 2869 when the log says your error dialog “has the error style bit set, but is not an error dialog” but really it is, and it’s a phony error hiding error 1001 instead.

    These aren’t the errors you’re looking for.  Move along.

  • Chronicles of Windows 7 Part 2: Multi-Monitor Support

    Posted on July 14th, 2009 Brian No comments

    My Desk with Docked LaptopOne of the awful, terrible, no-good parts of Vista was its multi-monitor support.  My set up is a laptop mounted in a docking station with a second monitor.  My mobile nature means I frequently dock and undock the laptop, requiring the operating system to detect how many screens there are, switch the primary display to and from the external monitor, and move all the windows back and forth, possibly repositioning them because of the difference in resolutions. Adding to the chaos is that I frequently sleep and wake the computer, sometimes docking it or undocking it while asleep, or shortly after waking.

    Vista failed at this horribly.  Often it would fail to detect the new display, requiring me to hit the external-monitor toggle key several times.  On occasion it would detect the new display, but then drive the new monitor at the wrong resolution.  Sometimes, after the first full moon on the second Tuesday of a month where the average temperature had been between fifty-four and sixty-eight degrees Fahrenheit (I think), it would forget which monitor was supposed to be the primary, and I would be forced to re-tweak the settings for three days in a row – and then all would suddenly be fine again.

    I won’t even mention what would happen if I put the laptop to sleep, undocked it, and woke it up later!  But to add insult to injury, the churnings of the graphics subsystem would cause flickers and re-sizes for five to ten seconds while it was figuring out whether things would work or not.  The entire experience was nothing short of maddening!

    I am pleased to report, however, that Windows 7 has been flawless.  It never fails to correctly identify the monitors present, nor does it ever forget which display is supposed to be primary.  The resolutions are always correct, and it makes the appropriate adjustments quickly and with no flicker.  It even handles with aplomb every combination of sleeping, waking, docking, and undocking I can think of!

    In short, it does exactly what it should.  Finally!

  • Twitter and Compression of Meaning

    Posted on June 10th, 2009 Brian No comments

    An interesting and often-mocked technical limitation of Twitter is the 140-character limit.  The semantics of the buzzword “microblog” aside, I notice in my own Tweets that the strict requirement for very short messages has improved my writing.

    (Let me note, before going any further, that I am not an abbreviations kind of guy. I virtually always write complete words and sentences using proper grammar, syntax, and punctuation.  Though I do use emoticons from time to time, I limit myself to a standard subset of smiley-, frowny-, and winky-faces, with the occasional raised or angry eyebrow thrown in – basically anything I can make with a colon, a dash, greater-than, less-than, and the parenthesis.  I cringe when reading things like, “c u 2nite” or “lol” or “<_<”.  Perhaps that makes me an angry old man; if so, that is a mantle I will wear.)

    When tweeting large thoughts, I find myself editing the message to fit within the allotted space while maintaining the same meaning.  The need to transmit the same semantics in a smaller space requires wielding more powerful language: stronger verbs, more-nuanced adjectives and adverbs, and better-placed punctuation and pronouns.  Squeezing the same content into less space is the very definition of data compression; so, in a very real sense, this is data compression for people.  What once might have rambled on for an entire paragraph now takes one or two concise sentences.

    Users get more bang for their buck when reading a tweet.  Coupled with the instantaneous and ubiquitous accessibility of these meaning-laden tweets, it’s obvious why Twitter has taken off.

    Like most compression, though, there is a tradeoff between space and time.  Though I have not measured, I have a strong sense that the ratio of time-editing-per-character is higher on Twitter than on any other medium I use.  On the receiving side, the message might take longer to comprehend, especially if the vocabulary is unfamiliar to the recipient.

    And some messages, like some data, will simply never fit into the allotted space.  That’s why I still have a blog.

  • Chronicles of Windows 7 Part 1: Qualcomm Gobi 3G Modem and VMWare NAT

    Posted on May 28th, 2009 Brian No comments

    So I went ahead and installed Windows 7 RC 1.  The process is remarkably smooth, and the OS is nicely polished.  The new task bar is a long-overdue change, formerly difficult or esoteric system tasks are now simple and obvious, and the Libraries paradigm in Explorer has pleasantly surprised me.

    But that’s not to say there aren’t some niggling issues.  This is a new release – nay, a pre-release – of the most popular operating system in the world.  There are bound to be some compatibility problems.  What is truly amazing is how well things work right out-of-the box.

    As I use the OS day-to-day, I’ll post some updates about real-life surprises and tribulations.  Here are my first two.

    Qualcomm Gobi 3G Modem

    Winodws 7 recognized almost every single piece of hardware on my HP Elitebook 8530w, including the silly fingerprint reader and the webcam I never use.  The one thing it didn’t already have drivers for was the built-in Qualcomm Gobi un2400 modem 3G.  What’s worse, the Vista drivers from HP’s support site don’t install, either.

    Fortunately, some amazingly enterprising soul figured out the problems, and was not only able to divine how to install the drivers, but then even wrote a schnasty little program to force-feed the Gobi modem its appropriate firmware.  Major kudos!  Unfortunately for me, it still doesn’t work.  There’s some magic incantation that isn’t being done quite right for my AT&T setup, so I’ll have to wait until the drivers get updated.  Hopefully that’ll be soon – paying for a data plan I’m not using is rather annoying.

    But, really, given how esoteric and fragile these 3G modems are, it’s not that surprising something bjorked their spaghetti-like functioning.  (Did you read the “More About The Firmware” section at that link?!)

    VMWare NAT Failure

    The only other true problem I’ve had is with VMWare Workstation 6.5.  It works like a charm, except that NAT routing fails to work correctly.  Interestingly, the guests can ping out, but other connections fail.  It’s a known issue, though, and will certainly be fixed soon.  And the work-around is simple enough: Just use bridging instead.

  • Multi-Threading with VFS

    Posted on May 14th, 2009 Brian No comments

    One of the new features in the BagIt Library will be multi-threading CPU-intensive bag processing operations, such as bag creation and verification.  Modern processors are all multi-core, but because the current version of the BagIt Library is not utilizing those cores, bag operations take longer than they should.  The new version of BIL should create and verify bags significantly faster than the old version.  Of course, as we add CPUs, we shift the bottleneck to the hard disk and IO bus, but it’s an improvement nonetheless.

    Writing proper multi-threaded code is a tricky proposition, though.  Threading is a notorious minefield of subtle errors and difficult-to-reproduce bugs.  When we turned on multi-threading in our tests, we ran into some interesting issues with the Apache Commons VFS library we use to keep track of file locations.  It turns out that VFS is not really designed to be thread-safe.  Some recent list traffic seems to indicate that this might be fixed sometime in the future, but it’s certainly not the case now.

    Now, we don’t want to lose VFS – it’s a huge boon.  Its support for various serialization formats and virtual files makes modeling serialized and holey bags a lot easier.  So we had to figure out how to make VFS work cleanly across multiple threads.

    The FileSystemManager is the root of one’s access to the VFS API.  It does a lot of caching internally, and the child objects coming from its methods often hold links back to each other via the FileSystemManager.  If you can isolate a FileSystemManager object per-thread, then you should be good to go.

    The usual way of obtaining a VFS is through the VFS.getManager() method,which returns a singleton FileSystemManager object.  Our solution was to replace the singleton call with a ThreadLocal variable, with the initialValue() method overloaded to create and initialize a new StandardFileSystemManager.  The code for that looks like this.

    private static final ThreadLocal fileSystemManager = new ThreadLocal() { @Override protected FileSystemManager initialValue() { StandardFileSystemManager mgr = new StandardFileSystemManager(); mgr.setLogger(LogFactory.getLog(VFS.class)); try { mgr.init(); } catch (FileSystemException e) { log.fatal("Could not initialize thread-local FileSystemManager.", e); } return mgr; } };

    The downside is that we lose the internal VFS caching that the manager does (although it still caches inside of a thread).  But that’s a small price to pay for it working.

  • Human Beings Are Big-Endian

    Posted on May 11th, 2009 Brian No comments

    I always have trouble remembering the difference between big-endian and little-endian.  The names don’t make any sense, so it ends up being a mere definition – and I have trouble with arbitrary definitions.  In the past, after figuring it out, I have noted to myself that human beings are big-endian as a memory-aid.  That is, we put our most-significant digits on the left.  And that’s great to remember, except then I forgot which endianness we were.

    I guess I need a memory-aid for my memory-aid.

    So this is a note to my future self: Human beings are big-endian.  Well, at least English-speaking, Arabic-numeral-using, base-ten-counting human beings who assume that linear memory addresses increase as you go from left-to-right.  Those assumptions seem good enough for me, though.

  • Funny Smelling Code – Endlessly Propagating Parameters

    Posted on May 8th, 2009 Brian No comments

    We’re currently working on a new version of the BagIt Library: adding some new functionality, making some bug fixes, and refactoring the interfaces pretty heavily.  If you happen to be one of the people currently using the programmatic interface, the next version will likely break your code.  Sorry about that.

    The BagIt spec is pretty clear about what makes a bag valid or complete, and it might seem a no-brainer to strictly implement validation based on the spec.  Unfortunately, the real-world is not so simple.  For example, the spec is unambiguous about the required existence of the bagit.txt, but we have real bags on-disk (from before the spec existed) that lack the bag declaration and yet need to be processed.  As another example, hidden files are not mentioned at all by the spec, and the current version of the code treats them in an unspecified manner.  On Windows, when the bag being validated has been checked out from Subversion, the hidden .svn folders cause unit tests to fail all over the place.

    It seems an easy enough feature to add some flags to make the bag processing a bit more lenient.  In fact, the checkValid() method already had an overloaded version which took a boolean indicating whether or not to tolerate a missing bagit.txt.  I began by creating an enum which contained two flags (TOLERATE_MISSING_DECLARATION and IGNORE_HIDDEN_FILES), and began retrofitting the enum in place of the boolean.

    And then I got a whiff.

    I found that, internally, the various validation methods call one another, passing the same parameters over and over.  Additionally, the validation methods weren’t using any privileged internal information during processing – only public methods were being called.

    I called Justin this morning to discuss refactoring the validation operations using a Strategy pattern.  This would allow us to:

    1. Encapsulate the parameters to the algorithm, making the code easier to read and maintain.  No more long lists of parameters passed from function call to function call.
    2. Vary the algorithm used for processing based on the needs of the caller.
    3. Re-use standard algorithm components (either through aggregation or inheritance), simplifying one-off cases.

    He had also come to the same conclusion, although driven by a different parameter set.  It’s a good sign you’re headed in the right direction when two developers independently hacking on the code come up with the same solution to the same problem.

  • Useful PDF ImageMagick Recipes

    Posted on May 6th, 2009 Brian No comments

    It turns out that ImageMagick is really quite good at reading, writing, re-arranging, and otherwise mucking with PDFs.  Unfortunately, you need to know the proper incantation, which can take much trial and error to figure out.  So, for my own future reference:

    Split A PDF Into Parts

    $ convert -quality 100 -density 300x300 multipage.pdf single%d.jpg

    The quality parameter is the quality of the written JPEGs, and the density is the DPI (in this case, 300 DPI in both X and Y).

    Join JPEG Parts Into A PDF

    $ convert -adjoin file*.jpg doc.pdf

    Rotate a PDF

    $ convert -rotate 270 -density 300x300 -compress lzw in.pdf out.pdf

    This assumes a TIFF-backed PDF. The density parameter is important because otherwise ImageMagick down-samples the image (for some reason). Adding in the compression option helps keep the overall size of the PDF smaller, with no loss in quality.

    Now, if I can just figure out how to make future me remember to look here…

  • Yet Another Vista Anecdote

    Posted on March 24th, 2008 Brian No comments

    I upgraded to Vista SP1 this weekend. There really is no reason not to try, especially since the version I have has a disk-imaging backup tool. So I could lobotomize my machine and re-insert the old image if something went horribly wrong. Heck, anybody with a Linux boot disc and dd could do the same. Everything went smoothly, unlike Thud, who seems to constantly have problems with the thing.

    Dog crapping on a Windows Vista logoPersonally, I don’t find Vista to be the steaming pile of crap that is so commonly reported on Slashdot and the like – well, any more steaming than a Microsoft operating system usually is. I mean, sure there have been some minor hiccups with certain driver upgrades, but it’s never blue-screened on me. I’ve even had some updates go awry, but it recovered nicely when I did a hard power-cycle. Sure, it’s not a break-through tower of innovation like OS X, but it’s a relatively solid next-version operating system.

    I don’t even mind the UAC security pop-ups that are so widely panned. Actually, I can’t see how they would be at all annoying unless your daily job involves opening and closing administrative and system maintenance tools all day. And in that case: What, exactly, do you do for a living? Though playing Mother-May-I with my computer is neither the best nor final move in the security game, it is a reasonable response to certain classes of maliciousness and mistakes that are quite common in the current landscape. I appreciate UAC in the same way I appreciate needing to sudo when using Linux.

    Vista Logo with flowersThis is my first foray into this argument, and I’m pulling out my toe after this brief dip into the chilly waters of the Vista-sucks sub-genre. (Being baited just isn’t as fun as it used to be.) It seems to me that most of the people howling about how much Vista sucks are just the usual slash-trolls, squirming in pleasure at the orgy of “Vista Sucks; Mac/Linux/BeOS/Whatever Rules” articles and posts, just because they can. That’s cool; whatever gets your rocks off. But all-in-all, Vista is pretty good. Sure, there are problems with it, but they’re being fixed as fast as a megalithic monopoly can move. And how quickly we’ve forgotten that the same rhetoric was spewed about the now-venerable Windows XP when it was released. It’s the same people who swore they’d stick with Windows 98 until 2098.

    So, to summarize: Vista is OK. Service Pack 1 installed with no problems. And my mission of lowering the signal-to-noise ratio in the internets, even if ever-so-slightly, has been successful. Back to work for me.

    “Scoop Your Poop” created from makes a great desktop; icons to the left by Rob!, used under a CC-BY 2.0 license. “Vista of Flowers” created from Carlsbad Flower Field March 2007-DSC_3575 by akaporn, used under a CC-BY 2.0 license. Vista logo (a trademark of Microsoft) taken from the web, and used fairly in good faith for parody and description.

  • No Backup On Battery in Vista

    Posted on March 19th, 2008 Brian No comments

    With the release of Vista SP1, I decided it would be a good time to back up my computer – in case something went horribly wrong. Better safe than sorry, or somesuch. But when I tried to run a backup this morning, I was rebuffed with this inscrutable message.

    The backup application could not start due to an internal error: You cannot create a backup or change backup settings when your computer is running on battery power.  Attach a power cord and try again. (0x8100000B) Please check your system configuration and try again.

    It got it right – I was running on battery power. But, really? This is the best you can do? I mean, I’m sure there’s some really good Raymond Chen reason for why I can’t or shouldn’t or Microsoft-knows-best reason I haven’t thought of as to why this is. I can come up with several on my own, in my head.

    At least tell me why I can’t start a backup on battery power. As it is, I feel like I need to jump through a hoop for no reason at all.