The other side of the moon

Back in 2003, I'd published a report on the state of computer mediated collaboration at the time. The report did not contain any original research, but was a list of references to other research on the topic. I was also working on ayttm and doing some work on automated project management at the time, which led to my talks on Fallback Messaging and Project Management with Bugzilla, CVS and mailing lists.

The state of technology has changed a lot over the years, and we're getting closer to the Unified Message InBox. The idea has been mulling in my mind for a while, and I've prototyped various implementations as a subset of what I call a stream of collaboration.

Communication

As technologically aware humans, we communicate in a variety of ways. Face-to-face, through the grapevine, handwritten letters and post-it notes, instant messaging, SMS, telephone calls, email, discussion boards, twitter, blogs, smoke signals, morse code using signalling lights, semaphore flags and more. Some of us prefer one form over another, and some of us will completely boycott a particular form of communication purely on principle, some of us won't use a form of communication because simpler methods exist and some of us will use a particular form of communication purely because it isn't the simplest one available. That's what makes each of us uniquely human. It also pushes us into groups and cliques that may or may not intersect. Who among you has a separate group of friends on twitter and facebook even though the two media are not vastly different? Do you also belong to a HAM radio club and a book reading group?

Now some forms of communication are well suited to automated archiving while others might end up being a game of Chinese whispers. Each form of communication also results in a different signal to noise ratio. Depending on the context of the conversation, accuracy and efficiency may or may not be a concern.

Degrees of Communication

Collaboration and Cooperation

Collaboration between a group of people involves communication, often requires archiving of all communications, and a high signal to noise ratio. Discussion lists, IRC logs, wikis, whiteboards, video/audio conferences, project trackers, bug trackers, source control commits, and sometimes email all provide good archiving capabilities. With proper time-stamping of each interaction, the archiving service can interleave events showing the exact chronological order of decisions being made. A Bayesian Filter, similar to the ones used for classifying spam can be used on a per topic basis to hide (but not remove) off-topic sections in an attempt to increase the signal to noise ratio. Once archived, even synchronous communication turns asynchronous [1]. Readers can later comment on sections of a communications log.

While some of the tools mentioned above are aimed at technical collaboration, many of them may also be used effectively for collaboration and support in a non-technical context [2,3] where immediate dissemination of information that can be archived and later used for reference is important.

Social Interaction

A form of communication that is more tolerant of a low signal to noise ratio, and in many cases does not require archival is social interaction. For example, at a party to watch a ball game, conversation may range from the actual events of the game to something completely irrelevant, and they're all socially acceptable in that context. Similarly, the corridor conversations at a conference may only be partially relevant to the subject matter of the conference. How does this translate to a scenario where people are geographically separated and need to communicate electronically rather than face to face?

Social television experiences [4], Co-browsing, LAN parties and digital backchannels [5] are examples where individuals communicate online while simultaneously engaging in a common task at geographically disparate locations. Computer mediated collaboration allows these individuals the ability to come close to the full party experience.

Casual Conversations

With more people moving their lives online [6,10], we're at a point where the volume of casual electronic conversations far exceeds that of technical collaboration. Fallback messaging is a good starting point to tie different forms of communication into a single thread, but it ignores current reality.

For starters, the fallback messaging idea assumes that users would use the same interface to communicate synchronously and asynchronously. In reality people use different methods for each. Asynchronous communication offers the ability, and often creates the necessity of longer messages and a larger amount of time devoted to communicating the message [7]. Should I say "Dear ..." or do I lead in with "Hi"? Should my background be pink or yellow? Do I want hearts, unicorns, birthday cakes or plain white in the background? Do I sign off with "Sincerely", "Regards", "Cheers" or "ttyl"? A different interface for each mode of communication makes it possible to customise the interface for the most common uses of that mode.

Fallback messaging was initially only applied to conversation between two persons, however a lot of casual communication happens between groups with the conversation sometimes forking into side-band conversations between a few (possibly two) members of the group and then merging back in to the parent conversation [7]. Some times the child conversation is made public to the group and some times it isn't. A messaging system must take this into consideration.

Enabling the Stream of Collaboration

Borrowed Ideas

The main idea behind fallback messaging was that service providers, communication protocols and user accounts were an implementation detail and the user should not be concerned with them. Users should only need to think of who they're communicating with, identifying them in any way they find comfortable (for example, using their names, nicknames, an avatar or favourite colour). The service needs to be able to determine based on context and availability, which messaging protocol to use.

Another idea that comes out of the original SMTP specification, is the now obsolete SOML [8] command. The SOML (Send Or MaiL) command would check to see if the recipient of the message was online when the message was sent, and if so, would echo the message to the user's terminal. If the user wasn't online at the time, it would append the message to their mailbox instead. Services like Yahoo! Messenger offer offline messaging capabilities that will hold a message on the server until the user comes online at which point it is delivered to them.

The problem with both these approaches is that they expect the user to create a message appropriate for the underlying service rather than the other way around. An entire email message in the case of SOML or a short text message in the case of Yahoo! Messenger. What we really need is a service that can decide based on what the user does, what kind of message needs to be sent, and how that message should be presented.

Facebook, GMail and Yahoo! Mail all offer a service where instant messaging and mail style messages can be sent from the same page, but with a different interface for each. Additionally, GMail provides the ability to see archived email messages and chat messages in the same context.

Proposed Interface

A messaging system is most useful to the user if it acts as a single point for them to reference and act on all past conversations, and provides an easy gateway to initiating new ones. The read interface must list all past conversations chronologically, with the ability to filter them based on topic and other participants in the conversation. It should be able to show where a conversation forked into separate threads, some of which may have been private, but all of which involved the user, and where these threads merged back into the primary stream.

The interface should include all kinds of communication including email, instant messages, co-browsing sessions, whiteboards, IRC, and anything else. Integrating with a service such as Google Voice, Skype or other VoIP solutions also allows it to tie in telephone conversations. Twitter and facebook notifications would tie in to this timeline as well.

The system should not rely only on message headers and meta-information, but also on message content to determine the topic and participants in a conversation [9]. Some participants, for example, may be involved indirectly, but not be privy to the details of the conversation, however it is useful to the reader to be able to filter messages with these details. Content analysis can also be used to identify messages as common Internet memes and tag them accordingly, possibly providing external links to more detailed information on the topic. Lastly, as has been proposed with fallback messaging, the system needs to aggregate all accounts of a contact across services into a single user defined identifier. It should still be possible for the user to identify which service is being used, but this should be done using colours, icons or similar markers.

Where are we today?

All major electronic communication services already provide some level of API acess to their messaging systems [11,12,13,14,15,16,17]. Services like Tweetdeck provide a single interface to multiple short message services (like twitter, identi.ca and the facebook wall), and Threadsy is supposed to unify your online social experience. Facebook seeks to unify email, IM, texting and Facebook messages through their own messaging service [18] which, like fallback messaging, is supposed to abstract out user accounts so that all you see is your contacts. I haven't seen the new messaging service yet, so I don't know if it also integrates with things like GMail, twitter, Yahoo! and other services that compete with Facebook [19]. If it does, that would be pretty cool. If it doesn't, there's an opportunity to build it. There are still other services that need to be tied in to enable full collaboration, but it doesn't seem too far away.

References

1. Terry Jones. 2010. Dancing out of time: Thoughts on asynchronous communication. In O'Reilly Radar, October 26 2010. http://radar.oreilly.com/2010/10/dancing-out-of-time-thoughts-o.html
2. Leysia Palen and Sarah Vieweg. 2008. The emergence of online widescale interaction in unexpected events: assistance, alliance & retreat. In Proceedings of the 2008 ACM conference on Computer supported cooperative work (CSCW '08). ACM, New York, NY, USA, 117-126.
3. Sutton, J., Palen, L., & Shklovski, I. 2008. Back-Channels on the Front Lines: Emerging Use of Social Media in the 2007 Southern California Wildfires. In Proceedings of the Conference on Information Systems for Crisis Response and Management (ISCRAM).
4. Crysta Metcalf, Gunnar Harboe, Joe Tullio, Noel Massey, Guy Romano, Elaine M. Huang, and Frank Bentley. 2008. Examining presence and lightweight messaging in a social television experience. ACM Trans. Multimedia Comput. Commun. Appl. 4, 4, Article 27 (November 2008), 16 pages.
5. Joseph F. McCarthy, danah boyd, Elizabeth F. Churchill, William G. Griswold, Elizabeth Lawley, and Melora Zaner. 2004. Digital backchannels in shared physical spaces: attention, intention and contention. In Proceedings of the 2004 ACM conference on Computer supported cooperative work (CSCW '04). ACM, New York, NY, USA, 550-553.
6. Donna L. Hoffman, Thomas P. Novak, and Alladi Venkatesh. 2004. Has the Internet become indispensable?. Commun. ACM 47, 7 (July 2004), 37-42. http://cacm.acm.org/magazines/2004/7/6471-has-the-internet-become-indispensable
7. Rebecca E. Grinter and Leysia Palen. 2002. Instant messaging in teen life. In Proceedings of the 2002 ACM conference on Computer supported cooperative work (CSCW '02). ACM, New York, NY, USA, 21-30.
8. Jonathan Postel. 1982. RFC 821: Simple Mail Transfer Protocol. IETF Network Working Group RFC Draft. http://www.ietf.org/rfc/rfc0821.txt, (August 1982).
9. Dong Zhang; Gatica-Perez, D.; Roy, D.; Bengio, S.; "Modeling Interactions from Email Communication," Multimedia and Expo, 2006 IEEE International Conference on , vol., no., pp.2037-2040, 9-12 July 2006
10. Rana Tassabehji and Maria Vakola. 2005. Business email: the killer impact. Commun. ACM 48, 11 (November 2005), 64-70. http://cacm.acm.org/magazines/2005/11/6081-business-email
11. Yahoo! Mail API. http://developer.yahoo.com/mail/
12. Yahoo! Messenger SDK. http://developer.yahoo.com/messenger/
13. Twitter API. http://dev.twitter.com/doc
14. Facebook Developer API. http://developers.facebook.com/docs/
15. GMail API: http://code.google.com/apis/gmail/
16. Google Voice APIs: http://thatsmith.com/2009/03/google-voice-add-on-for-firefox, http://code.google.com/p/pygooglevoice/
17. Jabber protocol (Google Chat & Facebook Chat): http://xmpp.org/xmpp-protocols/
18. MG Siegler. 2010. Facebook's Modern Messaging System: Seamless, History and a Social Inbox. Techcrunch, Nov 15 2010. http://techcrunch.com/2010/11/15/facebook-messaging/
19. Alexia Tsotsis. 2010. Between Gmail, Twitter and now Facebook There is no Universal Inbox Yet. Techcrunch, Nov 22 2010. http://techcrunch.com/2010/11/21/facebook-messages-is-people/

One of the things that drew me to Everybuddy, its USP really, was fallback messaging. I haven't seen any other client (other than eb's offspring -- ayttm and eb-lite) implement this feature, which is why I've never switched to another client.

So, what is fallback messaging?

Consider two friends who communicate via various network oriented means (IM, Email, SMS, etc.), we'll call them bluesmoon and mannu (because they are two people who communicated this way for several years before meeting IRL). Now, said friends are extremely tech savvy, and have accounts on virtually every server that offers free accounts, and then some.

So, you've got them on MSN, Yahoo!, AOL, ICQ... um, ok, not ICQ because ICQ started sucking, Jabber, and that's just IM. They prolly have 3 or 4, maybe 5 accounts on each of these services, ok, maybe just one on AOL. Then they have email accounts. The standard POP3 accounts, 3 gmail accounts, a Yahoo! account for every yahoo ID, and likely no hotmail accounts (even though they have MSN passports) because we all know that hotmail is passé.

These guys also have lj accounts and one or two cellular phones on different service providers.

Ok, now that we have our protagonists well defined, let's set up the scene.

Act 1, scene 1

Mannu and bluesmoon are chatting over, umm, we'll pick MSN to start with. So mannu and bluesmoon are chatting over MSN, when all of a sudden <insert musical score for suspense here> a message pops up:

The MSN server is going down for maintenance, all conversations will now end

Sweet. So MSN decides that we're not allowed to talk anymore.

What are our options? Oh well, Yahoo!'s still online, so switch to Yahoo!. It's much nicer too because you can chat while invisible too. MSN (officially) doesn't let you do that.

So, they switch to Yahoo!, but... what the heck were they chatting about when the server went down? Context lost. They need to start a new conversation, most likely centred around cursing MSN. What's worse is that the earlier conversation was being archived because they may have needed it as a reference later. The new conversation can also be archived, but it's a pain to merge all these different archives later.

Anyway, they plough ahead. The conversation veers back on topic, ... but now the main net connection goes down. The only things that work are websites and email. What do you do? What do you do? Ok, Dennis Hopper I am not, so let's forget I said that.

The easiest option would be for bluesmoon to send a mail to mannu saying, "Hey dude, my net connection went down, gotta end the convo here.", or he could send the same in an SMS. But to do that he's gotta start yet another program and type out stuff out of context again, or worse, type out an SMS that he has to pay for!

So, here's where fallback messaging comes in.

Act 1, Scene 1, Take 2

<jump back to the MSN chat>

Where were we? Oh yeah, the MSN server goes down. Now, what if the chat client we were using was smart enough to figure this out, and do something about it. What's that something you say? Switch to the next available service. So, in this case, the chat program would automatically and seamlessly switch to using Yahoo!

There's several user centric pluses here. The people chatting do not need to know that a server went down, leave alone care about it and figure out what to do. Archives will be maintained across sessions. The context of the conversation will be preserved. Mannu and bluesmoon can go on chatting as if nothing happened.

If all the IM protocols go down, the chat client could switch to Email or SMS. Of course, mannu should have to tell it explicitly to use one of these, because the conversation will no longer be online. There's gonna be delays between sending the message and getting a response, so the chatters need to know about this.

So, how does your chat client know that you have buddies on multiple services, and about their email address and phone number?

Well, the chat client would have to group accounts on various services into a single contact. This kind of grouping also has other benefits.

Two people chatting with each other now don't have to think about user names and different services and what not. Mannu wants to chat with bluesmoon, he just selects bluesmoon from his buddy list. He doesn't have to care whether bluesmoon has an account on MSN, Yahoo!, AOL or whatever. Why should he care? So, I'd be chatting one to one with another person, without caring about what happens behind the scenes. Isn't that what makes for a good play?

Well, at some point mannu would have to care about services and user names, because he'd actually have to manually add and group all these accounts into one. Perhaps he could also set preferences of the order in which to fallback. That's all a one time set up. For the continuous ease of use to follow, I'd say it's worth it.

Final questions...

Is this really possible? Yeah, sure it is. You can thank Torrey Searle for that. Torrey implemented everybuddy to do just this, and threw in loads of sanity checking - thanks for that dude. It's what drew me to use and then work on the project for so long.

So, is this really possible? Probably not until IM companies decide that the network is just a transport, and it's the value a user derives from using that transport that makes them choose one service over another. It's why we choose the Mumbai-Pune expressway over NH4 that runs through the ghats, even though there's a toll.

Update I did a talk on fallback messaging at Linux Bangalore 2004