I upgraded the Openfire IM Gateway plugin today. The biggest new feature in the upgrade is support for using Google Talk via the Gateway. We've supported connections to Google Talk via XMPP federation for a while, but now if you want to chat with your Google Talk buddies using your Google account, but within Spark, you can now! You'll need to download Spark 2.5.6 to support this - it's not on the Software Center yet, but you can download the Case-skinned version here. There are also some other new gateway features (support for SIP/SIMPLE, other XMPP servers, hooking into an IRC server for MUC channels), but we don't have any plans to use them yet.

When I first tried out Twitter, I was eager to try out the IM integration it offered with our IM server. But then I found out that it didn't work. You could add twitter@twitter.com to your roster, but it would stay in a sort of "pending" state, so you couldn't send it the claim code. My guess was that their XMPP server seemed to be requiring SRV records of us, which is only really required if your XMPP domain name doesn't match the server's DNS. Since im.case.edu is both our XMPP domain and our DNS name, it should have worked. I sent a bug report to Twitter about it, quoting the pertinent bits of RFC 3920. Today, I was notified that this was fixed in their updates last week, so you can all update your Twitter status via your Case IM account, if you're into that sort of thing.

Being able to embed your IM presence in a web page has been available for a while, and it works just fine as it is. However, embedding the text with the <OBJECT> tag is kind of clumsy and never really looked nice in the sidebar of my blog's index page. So I improved upon this with some JavaScript magic (and a little back-end Perl): <script language="JavaScript" type="text/javascript" src="http://im.case.edu/api/presence.cgi?jid=NetworkID"></script> Replace NetworkID with your Network ID. You'll still have to add yourself to your Case IM buddy list for this to work.

Ever try to add an avatar to your Case IM profile, but got an error that you didn't have permission to do so? That's fixed now. I installed Hannes Wüthrich's LDAP VCard Avatar plug-in for Openfire. What this does is allow the avatar to be stored in the Openfire database, but all the rest of the user's information is still pulled from the LDAP server. So, you can have avatars now. Go to it!

Some Spark users (mostly on Windows Vista, but some on Windows XP as well) have reported seeing incorrect timestamps on their IMs. This appears to be an issue in the Java Runtime Environment that Spark ships with. We've come up with a workaround that seems to help:

• download Spark.vmoptions.
• copy/move it to the directory containing Spark.exe (on my installation of Windows XP, it's in C:\Program Files\spark)
• Restart Spark

The Spark.vmoptions file hard-codes the time zone used by the JRE to EST. If you're in another time zone, you'll need to edit the Spark.vmoptions file and set it to the appropriate time zone. The Mac OS X version of Spark is unaffected by this issue, since it uses the OS's JRE to run. It's unclear if the Linux version is affected.

As I mentioned before, a web-based IM client is part of the newer versions of Wildfire/Openfire Enterprise. It's now deployed and available at im.case.edu.

The upgrades I mentioned in my previous post are installed. There may be some post-installation tasks left to do. If you notice anything amiss or any problems, let me know. It took less than the full three hours I alloted myself, too.

OK, the long awaited upgrade to the Case IM server is coming! It should happen some time next week - probably Thursday morning.

• New Features:
• A web-based IM client (may not be available immediately, but should be shortly after the upgrade)
• Voice Chat (using Jingle)
• Many bugfixes and improvements

Also, the latest version of Spark is going to be in the Software Center soon. There may be some more "surprise" features as well, but they'll require some post-upgrade testing and configuration.

...since it made today's Case Daily. We also topped out at 116 simultaneous users today. We'll probably start looking into upgrading to the latest version of Wildfire soon - that way we'll have a web client, and be in position for the upcoming Jingle-enabled version of Spark coming soon. I'm also starting to play with the Fastpath feature. On my index page, there's now a click-to-chat button, which uses Fastpath. We won't be able to do this for individuals (too much manual setup), but groups interested in having this kind of functionality can contact im-admin@case.edu for more information.

The IM service is now very nearly production ready. Unfortunately, we were unable to set the JIDs to "userID@case.edu". The problem is SSL related (it seems that Entrust wouldn't let us issue a cert for "case.edu".) Instead, all of the JIDs are of the form "userID@im.case.edu". We also now have Case-branded versions of Jive's Spark client - you can get them right now at the rather sparse im.case.edu, and they'll be available in the Software Center soon. As a result of this, the "messenger.case.edu" moniker is now deprecated in favor of "im.case.edu". You can still connect to the server using the hostname messenger.case.edu, but all the JIDs are @im.case.edu, and there's no way to do JID aliasing that I know of, though that is a good idea...) This shouldn't affect most people unless they've been doing server-to-server connections already.

1. Oracle Messenger has been officially retired. This actually happened a few weeks ago, but I didn't get around to announcing it until now due to vacation, etc. 2. Because of (1), we've now got the Wildfire server listening for SSL connections on the correct SSL XMPP port (i.e. 5223). If you're using a client that needs to use the old SSL port, adjust accordingly. We're also working on getting a real SSL certificate installed, which should stop iChat from giving errors upon making SSL connections. 3. The "change-the-JIDs" move I keep talking about is going to happen. This may also be combined with some other changes that will improve reliability. Stay tuned for details. 4. We have now purchased the Enterprise version of Wildfire. Because of this, we will be soon distributing a customized version of Spark. 5. I'm also starting to look at the FastPath functionality in the Enterprise Server. This will allow specially-defined workgroups to do "click-to-chat". I've already got one group interested in this, which partially came out of a conversation I had during my vacation(!).

We're making good progress with the IM beta so far. Here are some notes on the current status of the service: 1. SRV records - Hopefully, we'll soon have SRV records in DNS. This will improve server-to-server traffic, as there are some Jabber servers(supposedly this includes GTalk(?)) that are known to not try s2s to servers that don't publish XMPP SRV records. 2. Domain name - Right now, the JIDs issued to users are of the form "NetworkID@messenger.case.edu" - I'm wondering if changing it to "NetworkID@case.edu" would be better? This would require a little bit of work - as then the SRV record would move from being a nice addition to being required, and any instances of "messenger.case.edu" in the database would have to be changed. 3. Web Services - At some point, we'll hopefully be making a web-based IM "presence" service available. This would allow users to embed an image or text on a web page (such as a blog indicating their current IM status. 4. Web-based IM - I'm still thinking about deploying some kind of web client for the service. The most likely choice is currently Claros Chat. I'd love to know if there are any good alternatives out there. One other thing that I haven't mentioned before - the IM server is available to Alumni and anyone else with a valid Network ID, so check it out.

ok, so the mail server crashed twice today. Want to know why? We knew that the initial cause was the IMAP server dumping core. We may have even had an inkling as to what may have caused it. Here's a little bit of the output of a pstack on the core file: ----------------- lwp# 12 / thread# 16 -------------------- ff129394 txn_abort (0, f30a14d8, f30a14c0, ff12e164, 0, 0) + 4 00085fbc mboxlist_trans_finish (0, 16, f30a14d8, f30a14c0, 0, d) + 24 00091f94 mboxlist_maybeRecoverSubsFromFile (1e25994, 2f, 0, 0, 0, 2f) + 68c 0008ebdc mboxlist_findsub (1799550, 1, 0, 1e25994, 1c7b538, 625b0) + 6c 00053bb0 cmd_list (1e25920, 1e2f0b8, 1, 11995d8, 1799550, 0) + 288 00047d90 cmdloop (1e25920, 708, 2000, 2710, 0, 0) + 3058 000434d8 sock_readable (1e25920, 0, 7c2b48, 0, 0, 0) + 580 ff1860bc GDispCx_Dispatch (221204, faa47c, 7bed4c, fffffff8, 0, 6d47c1) + ec ff1864b8 GDispCx_InternalWork (7bed4c, ff186188, fee6c000, c, 7c2b48, 0) + 330 ff0b2ad0 _pt_root (7c2b48, ff0cc24c, 0, fee78d04, 0, 2) + a4 fee5b03c _thread_start (7c2b48, 0, 0, 0, 0, 0) + 40 So something weird was going on and causing the IMAP server to crash, which then left something locked in the store daemon (the process that actually touches the mail, whether via POP, IMAP, webmail, or SMTP), which then caused further problems. After the second crash under similar unusual circumstances, I went digging into the IMAP log files, hoping that something would turn up. Something did: [17/Feb/2006:16:08:16 -0500] northuist imapd[33856]: Account Information: login [x] uid proxyauth [17/Feb/2006:16:08:16 -0500] northuist imapd[33856]: Store Critical: Mailbox database error: missing or empty key value specified Well, that's interesting. Let's see if that error turned up in the earlier crash: 17/Feb/2006:14:05:59 -0500] northuist imapd[31550]: Account Information: login [x] uid proxyauth [17/Feb/2006:14:05:59 -0500] northuist imapd[31550]: Protocol Information: uid created mbox UM-Messages [17/Feb/2006:14:05:59 -0500] northuist imapd[31550]: Store Critical: Mailbox database error: missing or empty key value specified Note that I've cleverly disguised the user's network ID as "uid" here. But interestingly, both crashes involved this same person's mailbox. It looks like they were being provisioned for Unified Messaging on the earlier crash, and attempting to access voicemail on the second. Time to look at their mailbox! northuist%ls =+U+M-+Messages/ store.idx store.sub store.usr Nothing terribly unusual from the initial peek. There's no actual mail in the INBOX (but it turns out they forward their mail, so that makes sense) Let's look at the UM-Messages folder: northuist%ls -l total 16 -rw------- 1 mailsrv nobody 7856 Feb 17 16:41 store.idx Very normal looking too. Let's go back and take a closer look: northuist%ls -l total 526 drwx------ 2 mailsrv nobody 96 Feb 17 16:41 =+U+M-+Messages/ -rw------- 1 mailsrv nobody 7856 Feb 17 16:41 store.idx -rw------- 1 mailsrv nobody 259656 Sep 3 2003 store.sub -rw------- 1 mailsrv nobody 37 Feb 17 15:48 store.usr hmmmmm... That store.sub file looks a little big. It's just supposed to be a list of all the folders you're subscribed to (i.e. your own folders) This user only has one folder, so why is it so big? Let's take a peek: northuist%more store.sub /hBOcdctpYVDO46Y W16kL91GMdfKlokIlr+9qDX31vApE+z7ENLl+8pyz4TuZe3YCN2Bbnml9DKtxzImBtBjCiM8SSiF jbJ2c4rGfW+KTC+VtSq+x6vK/qXINxqtfl+58EqPHrEj9SQ9bN7TWh4TiZ7f3pm6C4Vj9lIcG2+q 9z9ErtDxB2F56fBzHb+fsOXhtnNQ00M4ht8Z+PbW7tpiBq3ygXCm0Yg2/CHY/xAvWL9ro247Yu6l jUKZhFeCFl8mZLln2gOdx9bnfirC6zCkbUPnzW82to03ij1Q31EZWjj+P0bQjX5Xy2fo2i1LWAXK CwnxMmqkbMWSG5D3vbrgezE3I7dR7nhB0bo9p6X0h/LodW42MwPsfX3bt1Es93xc9xp1b3jnHqVT That doesn't look good. For comparison, here's an excerpt from my own store.sub file: user/sdh7/Sent user/sdh7/Sent Items user/sdh7/Trash user/sdh7/UM-Messages user/sdh7/JunkMail A-ha! A little magic later (i.e. rm'ing the file and re-massaging the mailbox database) and I think everything should be happier now. We'll see what happens. Update: We've finished running a database reconstruct and all still seems well. I did just happen to notice that that bogus file dated from 2003, so these crashes appear to be a holdover from an old crash, rather than a recent corruption. Given that more people are going to be set up for UM soon, we should probably run a check for things like this.

For a couple of reasons(speeding up migration time, testing out some enhanced spam control facilities for mailing lists), we migrated another alias from the master alias file to Sympa. The process was similar to what was done the other day, though this was a "simple" alias, rather than an Admin Alias. The procedure is now down to about six minutes (a couple of which were spent tracking down the appropriate crontab entries to run by hand). For some extra background, here are some of the different kinds of things we have lurking in our master alias file:

• simple aliases - this is just as it sounds, a run-of-the-mill everyday sendmail alias. alias: address
• Administrative aliases - these are the kinds of aliases you create here. At present, when you fill out that form, it actually sends an e-mail to our Network ID office, who then create the alias largely by hand - they enter the names into a text file on our list-server system, and run a script which puts the text file into place, and creates an entry in the master alias file pointing to the file looking a bit like this:

  foobar-annc-list: :include:/fs1/mail/lib/foobar-annc-list

  There may also be some additional aliases which go with these, which have a similar function to majordomo's owner aliases. Admin aliases are good for situations where you want a departmental alias, but don't want to tie it to a specific user via a Personal Alias They don't have the moderation capabilities of a Majordomo list, and they can only be modified by the Network ID office or Postmaster.
• Majordomo lists - These are the lists that you create here. Similar to admin aliases, submitting that form sends an e-mail to the network ID people, who create the subscriber files and run the script to create the pertinent entries in the master alias file. This creates the list itself (which uses program delivery to call the majordomo process), request, and owner aliases. In general Majordomo lists are pretty flexible, but difficult to manage (since changes to list configuration have to either be done via e-mail or by pestering the Majordomo administrator to do it for you) They've also proven somewhat unwieldy due to changes in the primary e-mail address we use here - mailing lists using a subscriber-post-only policy get really confused by Webmail users whose primary e-mail address has changed. This was made worse by the workaround we attempted to deal with @po.cwru.edu/@cwru.edu addresses a while back.
• archive aliases - some internal aliases wanted archiving capabilities - we accomplished this by setting up program delivery to deliver the message to a file.

Note that Personal Aliases aren't part of this. When they were initially deployed, they were stored in the Master Alias file, but those have now been migrated to LDAP. We eventually plan to migrate most of the aliases listed above to Sympa, though that's probably going to be a somewhat long and drawn out process. I suspect that admin aliases may actually be the hardest, since ownership of the aliases isn't as strongly defined as majordomo lists (most admin aliases are actually "owned" by our Network ID people).