The Dumping Ground

In the little free time that I have, I have been messing around with writing a .NET program to help me with the large amount of photo metadata editing I want to do on all the photos I've been uploading to my Flickr photostream.

It's been fun doing some .NET again. It's easy to forget how nice of a language C# is especially with all of the fancy new features 2.0 brings. Last night, I came up with a fun little method of using iterators.

Here's the scenario: I need to collect photo IDs for editing. As a simple solution, I have a very small dialog box that contains a text field, an "Add Another" button, a "Done" button, and a "Cancel" button. When the user clicks "Add Another," I want to save the entered photo ID and re-display the dialog. If they click "Done," the currently entered ID should be saved, and the dialog should vanish. Finally, if they click cancel, the current ID does not get saved, and the dialog disappears.

I have set up the DialogResult of the "Done" buttton to return with DialogResult.OK, and "Add Another" to return with DialogResult.Yes. Here's the code in the UI controller responsible for coordinating this: public IEnumerable GetPhotoIds() { AddPhotoDialog dialog = new AddPhotoDialog();

DialogResult result = dialog.ShowDialog(this.parentWindow);

while (result == DialogResult.Yes || result == DialogResult.OK) { yield return dialog.PhotoId;

if (result == DialogResult.OK) { yield break; }

result = dialog.ShowDialog(this.parentWindow); } }

Then, in the top-level UI controller, I can simply do: AddPhotoUIController controller = new AddPhotoUIController(this.mainForm);

foreach (string photoId in controller.GetPhotoIds()) { PhotoInfo info = this.FlickrApi.PhotosGetInfo(photoId); this.CurrentBatch.Containers.Add(new SinglePhoto(info)); }

Pretty slick. I wish they had made this feature more general, though, and implemented full-on continuations.

I recently had to do some mangling of a dump of my personal Subversion repository. Basically, I had to modify some paths and revision copy numbers before re-importing to a clean repository. However, the dump was in one huge 300 MiB file, making it really difficult for to open for editing.

Normally, the solution would simply be to re-dump the repository using the --revision option to the svnadmin dump command. Unfortunately, in a flash of stupidity, I deted my old repository before I had the new one working. So I wrote a little Perl script to split the dumpfile into seperate files.

#!/usr/bin/perl

open(REV, "> repo") || die("Unable to open $!");

while (<>) { $line = $_;

if (/^Revision-number\:\s+(\d+)$/) { open(REV, "> rev.$1") || die("Unable to open $!");

if (!$header) { open(REPO, "< repo") || die("Unable to open $!");

while () { $header = $header . $_; } }

print REV $header; }

print REV $line; }

Ah...short and sweet, just the way a Perl script should be. Normally, I would immediately delete any Perl that I might happen to write, as I think the language is too flexible to be properly maintained over any length of time. However, since I can't find anything else like this out on Teh Intarweb, I figure I'll leave it here for posterity.

I've updated my post on Making Mantis with Basic Authentication Not Suck to fix an oversight I discovered in the original version.

Basically, direct links into Mantis would not work, since most of the Mantis pages redirect to the login page when a user has not yet authenticated. The modification was simply to modify the login page to detect basic authentication and redirect to the previously modified login script.

At work, I've been doing a lot of system setup and administration. While it's definitely beyond my job description as a Software Architect, there's nobody else than can do it right on my team - and thus it falls to me.

When administrating a large number of servers, it becomes quickly apparant that a centralized directory of credentials is absolutely necessary. The obvious solution is OpenLDAP. After creating a directory, though, it becomes necessary to integrate the various applications into the directory so that the users may use the same credentials across all applications. (I'm not even talking about single sign-on. That's a whole other kettle of fish.)

In some cases, it's built right in. For example, our Linux servers use pam_ldap and nss_ldap to tie in the operating system. For Apache, the modules mod_ldap and mod_auth_ldap provide the necessary integration; and some web applications, such as Subversion fall in line quite naturally.

Many web applications, though, support their own authentication via a web page, and those applications often do not fit in quite the way you would hope. Two such applications we use are MediaWiki and MantisBT. In the case of MediaWiki, I found a series of modifications that pretty much worked. I had to tweak a few things, and while I'm not going to write about them here, feel free to contact me if you'd like to know what I did.

Mantis, though, ostensibly supports basic authentication. The authentication section in the manual has a specific BASIC_AUTH option. The problem is that it doesn't work quite right.

I am protecting my Mantis directory with an Apache authentication directive. In the .htaccess in the Mantis directory, I require basic authentication, and the authenticated user must be a member of the cn=Mantis group in the directory.

SSLRequireSSL

AuthType Basic AuthName "Mantis"

AuthLDAPEnabled on AuthLDAPUrl ldaps://ldap_server/ou=people,dc=example,dc=com?uid?sub?objectClass=account

AuthLDAPGroupAttributeIsDN on require group cn=Mantis,ou=groups,dc=example,dc=com

order deny,allow deny from all

The Mantis configuration directive to use basic authentication is set in the config_inc.php file.

# --- authentication settings --- $g_login_method = BASIC_AUTH;

So in this scenario, you would never be able to get to the login page without authenticating with the web server. Mantis, rather than treating the basic auth as authoratitive, it takes the username and password received and then tries to log in using them (at least in version 0.19.3). I'm not sure the when that would be the right thing to do, but it certainly isn't in this case because it requires me to keep the password in the directory synchronized with the password in the Mantis database.

So our goal is to make Mantis ignore its own password and simply think it has successfully authenticated when g_login_method is set to BASIC_AUTH.

As a nice feature, the core/authentication_api.php file has support for auto-creation of a user if it doesn't exist when logging in via basic authentication. We want to keep that functionality. However, it inserts the basic authentication password into the database. While it really shouldn't hurt anything, I'd like to keep it seperate for clarity. So on line 77, I've modified the call to user_create() in the auth_attempt_login() to generate a random password and insert that into the database.

# Modified to generate a random password. # Since the basic authentication should be authoratative, then # this password is just a dummy password, and should never be used. # -- BCV $t_cookie_string = user_create( $p_username, auth_generate_random_password() );

The next step is to make auth_attempt_login() succeed when basic authentication has been used. Modifying the if block around the auth_does_password_match(), we use some short-circuiting to skip the check (and therefore always succeed) if the login method equals BASIC_AUTH.

# Since basic auth is authoratative, and assuming that this page # cannot be viewed unless it has already succeeded, then don't # bother checking a password, and just do a valid login. # -- BCV if (BASIC_AUTH != $t_login_method && !auth_does_password_match( $t_user_id, $p_password ) ) { user_increment_failed_login_count( $t_user_id ); return false; }

The final step is to redirect to the login.php script at certain appropriate points. The first is in the default page, named index.php. Normally, it redirects to the login page if the user is not authenticated. We're going to modify that behavior when we're using basic authentication so that it skips right to the login script instead. That way, it will automatically attempt to log in the user.

if ( auth_is_user_authenticated() ) { print_header_redirect( 'main_page.php' ); } else if (BASIC_AUTH == config_get( 'login_method')) { print_header_redirect( 'login.php' ); } else { print_header_redirect( 'login_page.php' ); }

Finally, we don't really want them to be able to log out, since that would require a login again. The modification I've chosen does a logout, and then immediately attempts to log in again.

auth_logout();

if ( HTTP_AUTH == config_get( 'login_method' ) ) { auth_http_set_logout_pending( true ); }

if ( BASIC_AUTH == config_get( 'login_method' ) ) { print_header_redirect( 'index.php' ); } else { print_header_redirect( config_get( 'logout_redirect_page' ) ); }

And that's it! Remember, the entire premise of these modifications is that the web server authentication is fully trusted, and that access to the underlying PHP scripts cannot occur unless the user has already successfully authenticated. If that's no the case in your scenario, then these modifications won't be secure.

Feel free to take these modifications for yourself. I'm going to see about submitting them as a patch to the official Mantis codebase. Oh, and if you find anything that I've missed, or anything that's insecure, please let me know.

So one of the big concerns with moving from an old blog installation to new software is the possibility of breaking links from around the Internet to content that I have created. At first, one might think that this kind of thing might generate a massive SEP field, but there's more at stake than just a broken link.

What is that, you ask? PageRank. The content on my site has garnered a non-trivial score in Google's view of the world, and such a score is valued, as it determines not only how high your own sites appear in any given Google query, but also how high people to whom you link appear. It is valued so much, in fact, that people often resort to comment spamming even mildly popular sites in order to increase their own search results.

So making sure the old link /article.php?story=20040727104257410 correctly maps to the new link /failed_sandbox_peer_launches_or_something.html is an important migration step.

So what's the tool of choice? Well mod_rewrite, of course. The mod_rewrite package is an Apache module that performs some black magic of rewriting URLs from one form into another. In my particular case, the requirement was to map a Geeklog story ID (SID) into a Drupal node ID (NID). Surprisingly, that's relatively easy to do.

The .htaccess file in this application's directory contains the following declarations. The first line examines the incoming query string for a variable called story, and it captures it using a regular expression. The second line looks for a request beginning with article.php, and then maps it to a value pulled from a mapping called geekmap. Notice the %1, which is a backreference to the SID captured in the previous line. The question mark at the end strips off any other query strings. Finally, the L stops all further mod_rewrite processing, and the R causes the new URL to be returned as an HTTP 301 response code, meaning a permanent redirect.

The two-part rewriting is required because mod_rewrite doesn't support examination of query strings inside of a RewriteRule.

# Do Geeklog to alias mapping. RewriteCond %{QUERY_STRING} story=(\d+) RewriteRule ^article.php /${geekmap:%1|}? [L,R=301]

So that begs the question: Where is that map called geekmap defined? It can be found In the httpd.conf.

# Add a rewrite map for Geeklog to Drupal RewriteMap geekmap txt:/etc/apache2/sid-to-alias-rewrite-map.txt

That's pretty self-explanatory, except for that txt part. It just specifies the simple text-file format for a rewrite map. There are other formats, that can be found in the documentation.

The map file is also pretty simple. It's just a series of tab seperated lines. The left column is the key, and the right column is the value.

20040727104257410 failed_sandbox_peer_launches_or_something.html 20040808110934404 three_tequila_night.html 20040809092923819 i_dont_even_know_her.html ...

And that's it! I do similar mappings for the old static page links, as well as the old RSS feed, but those are all much simpler cases. Finally, there's some other Drupal magic on the backend to convert those long underscored URLs into node ID URLs (such as node/136), as well as further mod_rewrite magic to make the Drupal URLs pretty - but that's beyond the scope of all this.