The Dumping Ground

Blame for the Metro Crash

The Metrorail system is deeply inculcated into the fabric of transportation in the DC Metro Area.  Everyone uses it, at least once or twice.  It is a testament to the fantastic success a rapid transit rail system can be, especially one spanning several different conflicting jurisdictions and built during an era when the construction of public transportation usually gave way to highways, interchanges, and parking lots.  The tight integration into our everyday lives is what makes Monday’s crash so disturbing for so many.

And now the blame starts circulating.  The results of the National Transportation Safety Board investigation will not likely be known for more than a year, but the root cause of the problem is obvious to anyone with a sense of the Metrorail’s history: The system has been underfunded for decades, robbing funds from necessary capital improvement and deferring maintenance in order to simply keep operating.  Just this past march, we played the same game again.

The local jurisdictions have been wringing their hands over dedicated funding.  Metro must beg, borrow, and steal to keep the trains running, and this accident is the direct result.  Mayor Fenty recognizes this fact, and took some of the blame on Good Morning America today.

The Federal government has also recognized both the lack of funding and that a significant portion of the Federal Government takes the Metro to work.  It has offered up $150 million per year to Metro if DC, Maryland, and Virginia all agreed to pony up $50 million each.  The offer has been on the table for years, and each jurisdiction has had its hand in stalling.  Virginia was the lone hold-out for a couple of years, but DC is currently to blame for the current delay.

As shameful as the current lack of funding is, such a massive cash infusion should never have been necessary.  Local jurisdictions failed to fund the system correctly in the first place.  This accident rests squarely on the shoulders of every politician elected to the Virginia General Assembly, the Maryland General Assembly, and the DC Council in the past thirty years.  Everyone knew this was coming, and they failed to act; blood is on their hands.  Greater Greater Washington sums it up nicely:

In the past, WMATA has followed some NTSB recommendations and not followed others. Two recommendations which they did not successfully complete include the installation of data recorders on all railcars and full retirement or reinforcement of the 1000 Series Railcars. They are currently taking a lot of heat for this, but in reality, they have had little choice in the matter.

The 1000 Series makes up about one-third of the Metro Fleet. Removing them from the tracks would mean major cutbacks in rail service. They’re already scheduled for retirement when replaced by the new 7000 Series in a few years. And while data recorders would have made the NTSB investigation easier, it would probably have not prevented this crash. Perhaps this tragedy will serve as a wakeup call to everyone in the process. Metro is underfunded, and has been for years. Deferred maintenance is taking its toll, and is keeping railcars in service longer than they should be. Everyone, from the local jurisdictions to the federal government should be willing to fund upgrades, especially considering that lives are at stake.

In the meantime, if you’re suddenly afraid of taking the Metro: Don’t be. Driving a car is still orders-of-magnitude more dangerous than transit. Two fatal crashes in over thirty years is a damn good record, and you’re a fool to fear the Metro more than your car.

Twitter and Compression of Meaning

An interesting and often-mocked technical limitation of Twitter is the 140-character limit.  The semantics of the buzzword “microblog” aside, I notice in my own Tweets that the strict requirement for very short messages has improved my writing.

(Let me note, before going any further, that I am not an abbreviations kind of guy. I virtually always write complete words and sentences using proper grammar, syntax, and punctuation.  Though I do use emoticons from time to time, I limit myself to a standard subset of smiley-, frowny-, and winky-faces, with the occasional raised or angry eyebrow thrown in – basically anything I can make with a colon, a dash, greater-than, less-than, and the parenthesis.  I cringe when reading things like, “c u 2nite” or “lol” or “<_<”.  Perhaps that makes me an angry old man; if so, that is a mantle I will wear.)

When tweeting large thoughts, I find myself editing the message to fit within the allotted space while maintaining the same meaning.  The need to transmit the same semantics in a smaller space requires wielding more powerful language: stronger verbs, more-nuanced adjectives and adverbs, and better-placed punctuation and pronouns.  Squeezing the same content into less space is the very definition of data compression; so, in a very real sense, this is data compression for people.  What once might have rambled on for an entire paragraph now takes one or two concise sentences.

Users get more bang for their buck when reading a tweet.  Coupled with the instantaneous and ubiquitous accessibility of these meaning-laden tweets, it’s obvious why Twitter has taken off.

Like most compression, though, there is a tradeoff between space and time.  Though I have not measured, I have a strong sense that the ratio of time-editing-per-character is higher on Twitter than on any other medium I use.  On the receiving side, the message might take longer to comprehend, especially if the vocabulary is unfamiliar to the recipient.

And some messages, like some data, will simply never fit into the allotted space.  That’s why I still have a blog.

DC Alerts Does It Again

I am really not intending to make this blog a running commentary on DC Alerts, but sometimes my hand just gets forced.  Though my last post on the issue was a bit tongue in cheek, my first post was leveled a serious criticism: DC Alerts needs to better train its operators about both when to send an alert and what to say in an alert.

A line of thunderstorms rolled through DC this morning, and upon waking I discovered an alert in my inbox.  Though the timing and purpose is fine, the content is – shall we say – questionable.  (Highlight mine.)

Subect: Alert DC – Severe Weather Watch

NWS issued a Severe Thunder Storm WATCH for the District from 05:55 06/09/09 to 06:45 06/09/09 . Please add_protective_actions_here .

This is just embarrassing.  Stupidity like this lessens the impact and usefulness of the system, and threatens the safety of the community.  The problem is entirely a human one, and it needs to be fixed.

DC Alerts Is Ready For the Zombie Apocalypse

I was pretty harsh on DC Alerts in my last post.  And I stand by that.  But say what I might about DC Alerts, I do have to give them props for one thing: They are ready for the Zombie Apocalypse.

If you’ve ever gotten an alert where a person might have been injured, the phrase “conscious and breathing” is often used to describe their state. It’s a curious turn of phrase, and makes one imagine the other possible permutations.  A nice table will be helpful here.

As you can see, DC Alerts really has their bases covered!  Hopefully, with their help, I’ll increase my odds of surviving.

In Case Of Zombies by Drunken Monkey used under a CC-BY-NC license.

DC Alerts Needs To Take Its Role Seriously

The idea behind Alert DC’s text-messaging/email alert system is a good one: Broadcast information about emergency situations to interested and/or affected parties using a low-latency, high-value medium.  The technical implementation is slick, too, allowing you to pick the type of alerts you receive (traffic, weather, police alerts, etc.); and permitting you to limit the alerts you receive to geographical areas of interest to you (neighborhood, schools, by address, etc.).

Unfortunately, though the concept and implementation of DC Alerts is pretty nice, the people using it have turned the system into a joke.  Let’s dissect a recent example:

Subject: Weather Report Update

Weather report update: Light to moderate rain will fall across the area through tomorrow morning.  The thunderstorms are well to the South of the District. NWS states that the heaviest rain will be in the area towards day break. Expect rainfall amounts through to the morning will range from one to two inches. WASA will have additional crews checking catch basins until the morning. DPW has provided 45 sandbags to the resident. DDOT have no problems to report.

Let’s take this point-by-point:

• It’s raining, and there’s severe weather, but not here.
• The water company is doing its job, and has crews out cleaning catch basins.
• Somebody got some sandbags.
• There’s nothing wrong with the streets.

Let’s see if I can summarize this in a Twitter-esque 140 characters or less:

Alert! Everything’s OK! (Except for this one random dude someplace, and we got him some sandbags.)

When you put it that way, it’s suddenly very clear that this does not qualify as an alert-worthy event.  Nor do most of the other so-called alerts we get.  To add insult to injury, the messages we receive from DC Alerts are often riddled with spelling errors, grammar mistakes, ridiculous descriptions, and geography snafus.

The DC Alerts system has great potential to become a powerful emergency-communications tool for the hyper-connected world we live in.  But it will never be taken seriously until the human beings using it dramatically ramp up both the quality and the relevance of the messages.

Chronicles of Windows 7 Part 1: Qualcomm Gobi 3G Modem and VMWare NAT

So I went ahead and installed Windows 7 RC 1.  The process is remarkably smooth, and the OS is nicely polished.  The new task bar is a long-overdue change, formerly difficult or esoteric system tasks are now simple and obvious, and the Libraries paradigm in Explorer has pleasantly surprised me.

But that’s not to say there aren’t some niggling issues.  This is a new release – nay, a pre-release – of the most popular operating system in the world.  There are bound to be some compatibility problems.  What is truly amazing is how well things work right out-of-the box.

As I use the OS day-to-day, I’ll post some updates about real-life surprises and tribulations.  Here are my first two.

Qualcomm Gobi 3G Modem

Winodws 7 recognized almost every single piece of hardware on my HP Elitebook 8530w, including the silly fingerprint reader and the webcam I never use.  The one thing it didn’t already have drivers for was the built-in Qualcomm Gobi un2400 modem 3G.  What’s worse, the Vista drivers from HP’s support site don’t install, either.

Fortunately, some amazingly enterprising soul figured out the problems, and was not only able to divine how to install the drivers, but then even wrote a schnasty little program to force-feed the Gobi modem its appropriate firmware.  Major kudos!  Unfortunately for me, it still doesn’t work.  There’s some magic incantation that isn’t being done quite right for my AT&T setup, so I’ll have to wait until the drivers get updated.  Hopefully that’ll be soon – paying for a data plan I’m not using is rather annoying.

But, really, given how esoteric and fragile these 3G modems are, it’s not that surprising something bjorked their spaghetti-like functioning.  (Did you read the “More About The Firmware” section at that link?!)

VMWare NAT Failure

The only other true problem I’ve had is with VMWare Workstation 6.5.  It works like a charm, except that NAT routing fails to work correctly.  Interestingly, the guests can ping out, but other connections fail.  It’s a known issue, though, and will certainly be fixed soon.  And the work-around is simple enough: Just use bridging instead.

Confession of a Credit Card Deadbeat

The roiling economy has uncomfortably squeezed the profits of various huge financial mega-corporations.  As the bottom-of-the-barrel customers are no longer able to honor their obligations, the fatty underbelly of fees and interest which the poorest consumers have struggled to pay is suddenly looking a little lean.  To make back their profits, the credit card companies are looking to eliminate the cash-back and frequent flier miles, reduce or rescind interest grace periods, and reinstate annual fees.

There’s no way this could backfire.

An amusingly twisted word in the credit card parlance is “deadbeats”.  Counter-intuitively, these companies use that term to describe their very best customers.  Do you pay your card off in full every month?  Do you rarely, if ever, accrue interest or late fees?  Do you regularly cash in your frequent flier miles and cash-back bonus?  Yes?  Well, since you don’t make much money for them, they don’t like you.  They tolerate you, but you’re gaming the system, getting a free ride.  You’re a deadbeat.

It’s nice to know what they really think of you.

Don’t weep, though.  They still make plenty of money from customers like me.  Every time we use our cards, the merchant is charged hefty fees for privilege of accepting our card of choice.  We don’t pay that directly, but we pay it indirectly through higher prices in restaurants, shops, and online.

But it’s not enough, so they’re coming after the good customers.  Ironically, they’re coming after the customers who need them least. I admit it: I am a credit card deadbeat.  And if I suddenly have to pay an annual fee or lose my grace period, what incentive do I have just to not carry cash?  I will happily abandon the credit card companies, tossing their aggressive advertising, obnoxious phone calls, and invasive behavior tracking right along with their annual fees and interest rates.  And good riddance!

And, assuming I’m not the only one happy to return to legal tender, then the credit card companies are sowing the seeds of their own doom.  As their best customers jump ship, their balance sheets will be left a ghetto of poor credit customers.  As losses mount from those who can no longer pay, the ratio of good assets to bad will finally topple the once-mighty giants of consumer finance.

Stack of Credit Cards modified from the original Too Much Credit by Andres Rueda under a CC Attribution 2.0 Generic license.

Multi-Threading with VFS

One of the new features in the BagIt Library will be multi-threading CPU-intensive bag processing operations, such as bag creation and verification.  Modern processors are all multi-core, but because the current version of the BagIt Library is not utilizing those cores, bag operations take longer than they should.  The new version of BIL should create and verify bags significantly faster than the old version.  Of course, as we add CPUs, we shift the bottleneck to the hard disk and IO bus, but it’s an improvement nonetheless.

Writing proper multi-threaded code is a tricky proposition, though.  Threading is a notorious minefield of subtle errors and difficult-to-reproduce bugs.  When we turned on multi-threading in our tests, we ran into some interesting issues with the Apache Commons VFS library we use to keep track of file locations.  It turns out that VFS is not really designed to be thread-safe.  Some recent list traffic seems to indicate that this might be fixed sometime in the future, but it’s certainly not the case now.

Now, we don’t want to lose VFS – it’s a huge boon.  Its support for various serialization formats and virtual files makes modeling serialized and holey bags a lot easier.  So we had to figure out how to make VFS work cleanly across multiple threads.

The FileSystemManager is the root of one’s access to the VFS API.  It does a lot of caching internally, and the child objects coming from its methods often hold links back to each other via the FileSystemManager.  If you can isolate a FileSystemManager object per-thread, then you should be good to go.

The usual way of obtaining a VFS is through the VFS.getManager() method,which returns a singleton FileSystemManager object.  Our solution was to replace the singleton call with a ThreadLocal variable, with the initialValue() method overloaded to create and initialize a new StandardFileSystemManager.  The code for that looks like this.

private static final ThreadLocal fileSystemManager = new ThreadLocal() { @Override protected FileSystemManager initialValue() { StandardFileSystemManager mgr = new StandardFileSystemManager(); mgr.setLogger(LogFactory.getLog(VFS.class)); try { mgr.init(); } catch (FileSystemException e) { log.fatal("Could not initialize thread-local FileSystemManager.", e); } return mgr; } };

The downside is that we lose the internal VFS caching that the manager does (although it still caches inside of a thread).  But that’s a small price to pay for it working.

Human Beings Are Big-Endian

I always have trouble remembering the difference between big-endian and little-endian.  The names don’t make any sense, so it ends up being a mere definition – and I have trouble with arbitrary definitions.  In the past, after figuring it out, I have noted to myself that human beings are big-endian as a memory-aid.  That is, we put our most-significant digits on the left.  And that’s great to remember, except then I forgot which endianness we were.

I guess I need a memory-aid for my memory-aid.

So this is a note to my future self: Human beings are big-endian.  Well, at least English-speaking, Arabic-numeral-using, base-ten-counting human beings who assume that linear memory addresses increase as you go from left-to-right.  Those assumptions seem good enough for me, though.

Funny Smelling Code – Endlessly Propagating Parameters

We’re currently working on a new version of the BagIt Library: adding some new functionality, making some bug fixes, and refactoring the interfaces pretty heavily.  If you happen to be one of the people currently using the programmatic interface, the next version will likely break your code.  Sorry about that.

The BagIt spec is pretty clear about what makes a bag valid or complete, and it might seem a no-brainer to strictly implement validation based on the spec.  Unfortunately, the real-world is not so simple.  For example, the spec is unambiguous about the required existence of the bagit.txt, but we have real bags on-disk (from before the spec existed) that lack the bag declaration and yet need to be processed.  As another example, hidden files are not mentioned at all by the spec, and the current version of the code treats them in an unspecified manner.  On Windows, when the bag being validated has been checked out from Subversion, the hidden .svn folders cause unit tests to fail all over the place.

It seems an easy enough feature to add some flags to make the bag processing a bit more lenient.  In fact, the checkValid() method already had an overloaded version which took a boolean indicating whether or not to tolerate a missing bagit.txt.  I began by creating an enum which contained two flags (TOLERATE_MISSING_DECLARATION and IGNORE_HIDDEN_FILES), and began retrofitting the enum in place of the boolean.

And then I got a whiff.

I found that, internally, the various validation methods call one another, passing the same parameters over and over.  Additionally, the validation methods weren’t using any privileged internal information during processing – only public methods were being called.

I called Justin this morning to discuss refactoring the validation operations using a Strategy pattern.  This would allow us to:

1. Encapsulate the parameters to the algorithm, making the code easier to read and maintain.  No more long lists of parameters passed from function call to function call.
2. Vary the algorithm used for processing based on the needs of the caller.
3. Re-use standard algorithm components (either through aggregation or inheritance), simplifying one-off cases.

He had also come to the same conclusion, although driven by a different parameter set.  It’s a good sign you’re headed in the right direction when two developers independently hacking on the code come up with the same solution to the same problem.