Unpublished paper meant for the popular press The Cyborgs are Coming or The Real Personal Computers by Thad Starner [email@example.com] (submitted to Wired) People look at me strangely when I walk down the street these days. However, I'm not particularly surprised; I have a box strapped to my waist with wires reaching out to my hand and up to my eye. I often hold silent conversations with myself, electronically taking notes on the world around me. Occasionally one of my observations triggers electronic memories and gives me new insights. No wonder people look at me strangely. You see, I'm one of the world's first cyborgs. We are on the edge of the next stage of human development: the combination of man and machine into an organism more powerful than either. Almost every user of a computer will be affected in some way: students, secretaries, lawyers, doctors, scientists, stock brokers, and CEO's, just to name a few. While the technology necessary for this merger may currently look strange, the hardware is obtainable from today's off-the-shelf components. For $3000 you can strap on prototype technology that makes present PDA's (personal digital assistants) pale in comparison. In mass production that price could fall to around $1000. Currently, the hardware consists of a small, light graphics display called the Private Eye(R) (the current version is 720x280) that fits over an eye in a pair of sunglasses, a one-handed chording keyboard (which functions like a full 101-key keyboard), and a small DOS-based computer which fits on the waist (in my case, the PC includes 85M of hard drive, 2M of RAM, and several ports including a PCMCIA). For a little more money, a cellular phone and modem can be added (for those net addicts). While almost plebeian in design, the combination of these inventions points to a very powerful paradigm in human-computer interactions. Furthermore, three multi-billion dollar product and service areas will be developed by such technology. The Vision Science fiction has foretold the merging of man and machine for many years. Cyborgs with minds partly consisting of silicon are almost commonplace in today's fiction. Usually, these characters are portrayed as the dark side of humanity, dependent on prosthetic neural circuitry to continue life. However, some writers have seen these devices as voluntary additions to the human host, augmenting but not supplanting the intelligence already there. This field of "Intelligence Amplification" (to borrow a term from Vernor Vinge) is the topic of this article. While such a term brings visions of direct brain interfaces, nothing so grandiose (and difficult) will be discussed here. Instead, a simpler interface will be described which has similarly powerful properties of persistence and consistency. In recent years, computers have gotten smaller, lighter, and more powerful while consuming less power. Fueling this trend is a great base of users who rationalize a need for computing power while traveling (or at least while roaming within an organization). Often these notebook machines are used for such mundane tasks as "To Do" lists, appointments, and business contacts. However, without such functions the user can be paralyzed. In fact, the pen-based computer community is trying to fill the need of the users who find notebook computers awkward for doing these tasks (unpacking, powering it on, booting, finding a place to type, etc.). However, even these machines are still inconvenient in the real world, for a variety of reasons that will be examined later. An ideal interface would be with the user all the time, listening to the user's real world interactions and updating appropriate files automatically. Going even farther, the computer should monitor the virtual world and notify the user when appropriate (important e-mail, the value of gold dropping $100.00, etc.). While this goal is extremely hard for many reasons, first level approximations to such man-machine relationships can be made using relatively simple hardware and software. The Private Eye Made several years ago, the Private Eye is one of the most unrecognized revolutions in display technology. This small, 1 oz. display uses a single row of 280 LED's and a scanning mirror to display a screen of 720x280 pixels to the user's eye. More modern versions have resolutions of 1024x768 pixels. The image is crisp, and the focus can be put anywhere from 10 inches to infinity. Since the display is worn close to the eye (for example, in a pair of sunglasses), the projected image is equivalent to a large screen display. Due to the "sharing" effect of the human visual system, the user can see both the real world and the virtual at the same time (some variations on this theme use two Private Eyes with half-silvered mirrors so that stereoscopic cues can be used and both eyes can see the virtual and the real at the same time). Furthermore, since there is no large glass or plastic surface to scratch or bend (the actual display surface is ~ 1" x 1"), the Private Eye is more robust than most other portable displays. The unit is designed to withstand a three foot drop, and, in my experience, can handle even rougher treatment. In fact, the Private Eye would not be too difficult to ruggedize for use in the military. Additionally, since the display is kept near the user's head where humans are much more careful with respect to impacts, the Private Eye is much less likely to be exposed to damage than the LCD screens in the stereotypical PDA. While this graphics display is not as powerful as the direct brain connect interfaces described in fiction, the visual system can process an enormous amount of information and is thus a great primary interface for receiving data. In addition, the overlay of graphics on to the real world allows virtual annotation of real world objects. A Revolutionary User Interface: The Keyboard? So far, keyboards on notebook and pocket computers are either too large for convenience or too small to use. This is a direct result of assuming the standard QWERTY keyboard is good for portable computing. Manufacturers are afraid that it would take users too long to learn a new way of typing. However, beside me is a one-handed chording keyboard which anyone can be taught to use in 5 minutes. It is certainly much easier to learn than the QWERTY interfaces (the letters actually go in order-"abcdef..."-but are arranged so that speed is not particularly limited). In an hour, a beginner can be touch typing. In a weekend a speed of 10 words/minute can be obtained. Shortly, 35+ words/minute can be achieved (my personal rate is around 50 words/minute with a macro package). This is the Twiddler keyboard from HandyKey (addresses are included at the end of this article). It even includes a tilt activated mouse. The Twiddler is but one of many one-handed designs out there. Some designs allow instant access to both hands if necessary (the Twiddler straps on to one hand). This feature may be very desirable in medical fields. In any case, these devices allow the use of full-featured keyboards anywhere (including walking down the street in the rain). When finished, they can be stuck in a pocket or left on the belt for easy, instant access. Not only are these keyboards convenient, they do not require much CPU power (unlike handwriting), always correctly recognize a user's input, and can take an amazing amount of abuse (I have kicked mine into a door, stepped on it, and gotten it wet, etc.). Putting It Together When the Private Eye and a one-handed keyboard are combined in a computer interface, the result overcomes the limitations in screen-size, access, and user input imposed by many of today's PDA's. The user can continually see both the real world and the virtual in his everyday work. The virtual world can be accessed even while the user is walking down the street, attending a cocktail party at a conference, attending a patient, giving a PhD defense, or taking a quick lunch before going back to Wall Street for more trading. The interface is persistent and reliable (due to the simplicity and packaging of its parts). Just this beginning system has many possibilities, and I would like to dwell on these applications as well as marketability of the current system before moving on. Simple Applications By adding basic communications through radio or cellular technology to the base system, many applications present themselves. As a computer professional, I find the ability to log into my computer system anywhere, anytime a serious boon. Even if communications are not possible for some reason, the ability to edit text and read mail locally is a major asset. Computer system administrators could find such technology invaluable for detecting and fixing employer problems without having to be physically present. While such interactions are possible on a notebook computer, the added portability and the persistence of this interface allow for better access. Students can take notes in class without having to glance down at their screen. Lawyers can be in communication with their office databases and support staff while cross-examining a witness. Repairmen can make inquiries and orders to the home office without interfering with their work. Health care providers can query databases for precedents or consult a remote physician while examining their patients. Brokers could transfer commodities, offer bids, or consult without shouting to be heard (and, maybe someday, even trade while away from the floor). Racing enthusiasts could bet and monitor their winnings without being at the track. These are but a few of the many applications that are possible. In fact, every day I wear my interface people find more uses for it. The Billion Dollar Hardware Business While this beginning system may seen clumsy, it is quite usable. At first, only a certain breed of technophile or time-critical information consumer would be interested in looking odd to gain the power and convenience such an interface would allow (with the current system, I am often mistaken for a telephone repairman). However, there are a growing number of individuals out there that qualify. In fact, marketing organizations have identified a new niche: the computer professional. These individuals use a computer every day and find it essential to their work. Furthermore, according to some estimates, 50% of these individuals earn over $100,000 a year. Even if the system was just sold as an expensive toy, a market may exist. More seriously, due to the unsuitability of handwriting interfaces for many tasks and the familiar DOS feel to my initial system, it may usurp the large market the pen manufacturers predict for their systems. Customization may leverage the concept into the service industry (inventory control, quality management, railway conductors, telemarketers, etc.) When improvements such as speech recognition, smaller designs, etc. come along the market will expand to a broader band of users, much like the notebook market has (notebook systems are presently outselling desktop models). Here is a chance for entirely new computer product lines, with upgrade paths every two years. Along with these lines come the necessary support hardware, such as digital modems and radio gear. The Billion Dollar Communications Business The Internet, television cable, and cellular telephone all started as very small systems. Today these communication mediums are almost institutions. Providing cheap and reliable wireless digital communication technology will become an incredible source of revenue. Even with just the notebook computer paradigm, many foresee a tremendous growth. With the addition of wearable computing (not to mention intelligence amplification), these figures can only improve exponentially. In fact, technology for both long range and very short range communication will be in high demand, since, after commuting to work, the user's wearable computer should automatically hook into the office work environment at higher bandwidths to help the user with normal chores. While this may not supplant the need wired interfaces to a powerful desktop system, the wearable can still help it's user operate these more powerful machines on a personal level, if only to separate more casual work (e-mail, weather updates, phone calls on the wearable) from concentration intensive work (CAD, accounting, visualization on the desktop). Another communications issue is the interfacing of the different parts of a wearable together. While the present interface is wired, it is easy to examine a low power communication system to wirelessly combine the keyboard, display, and computer. In fact, there has been some discussion of using the body itself as the communications carrier. According to some initial experiments by La Monte Yarroll, speeds as high as 1 Mbaud may be possible by driving a 5V signal across the skin. More conservative methods may include infared or low power radio frequency. The Billion Dollar Software Business While the initial systems may be DOS or Mac based, the new interface paradigm of persistency allows radical changes in software design. The new software should make the user interface simple and consistent in most situations. An improved level of user competency may arise from the increased use of the persistent interface (this is happening anyways as our children are growing up in a computer literate world), so these interfaces may become more complex than ever. A particular change of software design will be in determining when to interrupt the user for especially urgent incoming information or clarification of user input. The goal is to improve the user's productivity, not overwhelm his sensors. The research/software product field of Intelligent Agents may go far in addressing this issue. In fact, an artificial agent will be presented later as a tool for the author's wearable computer. This field will exist whether or not this particular hardware platform is created. With an increasing amount of information being generated, intelligent tools will be necessary in the coming world. Furthermore, until communications transponders become ubiquitous, software will be needed to make the transition from connected to disconnected use transparent. Much theoretical and practical work has gone into such systems already, but the commercial implementations lag behind. The Cyborgs Are Coming: The New Computing One of the simplest, yet most poignant, applications of this "wearable" technology is augmented memory. Today, many computer users already utilize the excellent memories of their computers for storing phone numbers, addresses, and "to do" lists. However, many of these users are then helpless away from their terminals (or have to lug out their notebooks or, at best, their palmtops, each time they want to check something). With durable wearable technology, these users can check and update their schedules wherever they may be and without interrupting whoever they may be talking to at the time (especially useful for storing e-mail addresses at conferences). In fact, reminders, meeting agendas, grocery lists, and lecture notes could be automatically or semi-automatically overlaid on to the real world as appropriate. These applications just scratch the surface of what is possible. Wearable computing allows a symbiotic relationship between computer and human which combines some of the strongest advantages of both: the creativity and intuition of a human with the precise storage and searching capacity of the computer. Suppose, that a reader of this article has the interface as previously described. As the reader scans the text (supposing, for the moment, that this article is in paper form and not on-line), he types in notes, unanswered questions, and comments in one window of his word processor. The reader's Remembrance Agent (RA) (an intelligent, adaptable piece of software that specializes to a user's needs) listens to the input and immediately conducts a search through the user's directories (local and/or remote) for files with similar contents. In another window of the word processor, the RA reports appropriate lines from files found in its search. These lines are ranked according to some measure of "usefulness" that is either directly programmed into the RA or learned over time. In this manner, the reader can quickly be reminded of similar pieces of information obtained in the past. Through these small memory assists, the reader can compare two people's views, confirm statistics, or generate entirely new ideas synthesized from the foundations laid by others. Furthermore, the Remembrance Agent can suggest files for the storage of this article and the reader's notes on it, possibly improving the reader's organizational skills. An initial implementation of this software has been completed, and a more sophisticated and powerful version is underway. The implications of such a system are tremendous. Imagine college students having immediate access to their education for the past 20 years, reporters and police detectives who can interactively and possibly automatically search for clues and leads, stock brokers whose systems automatically listen to news feeds for information that might affect prices, scientists with automatic access to a common storehouse of information which may spur new contacts and discoveries, CEO's with up-to-the-minute reports on their own and competitor's companies, lawyers whose Remembrance Agents discover a precedent based on a new twist in a court room trial, and doctors whose description of a patient's symptoms finds a match with a rare case reported on the other side of the world. The list goes on and on. Makers of PDA's have been suggesting similar possibilities for several years now. Many have recently toned down their claims. They have been duped by the concept of handwriting recognition and toy scenarios. Some have underestimated the problems of text retrieval, user interface, or intelligent agent design. What makes this scenario different? As the next section will show, the intimate, fluid relationship of man and machine and the large size of the information databases may change the situation. Having a wearable computer makes note paper obsolete. A searchable, organized environment where nothing is lost is very attractive to the users of note paper. With constant access to a computer screen and keyboard, the user can store all of his notes for the day (especially useful for students); take along a textbook, newspaper, or novel to read on the subway; play a video game; catch up on e-mail or netnews; debug programming; or compose his next piece of poetry wherever and whenever he wants. This is a very strong force for keeping everything on-line. Note that this particular interface reinforces this behavior much more than handwriting based PDA's where the awkwardness of unpacking, using two hands, and recognition errors limit the utility of the machine. Thus, the wearable computer can expect much more input from the user than more traditional machines. With this greater input directly from the user, especially over the period of years, a Remembrance Agent has a much greater likelihood of being useful. The Remembrance Agent could easily remind the user of something he typed several years ago (and subsequently forgot) which has pertinence to a present problem (even with low recall rates in unpersonalized text retrieval studies, automated recall is better than human recall when a database gets large or when the information is obtained over time). Furthermore, through this intimate, interactive relationship with the user, the Remembrance Agent can more easily learn the user's preferences. Another advantage is that, if the interface deals exclusively with plain text, both the hardware and the software can be upgraded many times without disturbing the knowledge gained in the past. However, neither may ever need to be upgraded for the functionality described. This would allows a revolutionary concept in the computer world: a life-long relationship between a user and a particular machine interface. As the machine and user adapt to each other over the years, a new, integrated being might emerge combining the best features of both. Imagine a policeman who never forgets a face (adding a digitizing camera and simple face recognition software), an architect who never forgets a structure, or a history teacher who remembers everything he has ever read or been taught. Augmented Reality Overlaying text on the real world in the augmented memory applications above can be thought of as a particular subfield in the realm of Augmented Reality. Augmented Reality refers to taking the virtual computer environment and combining it with the real. Wearable computing offers a simple, cost effective way to begin experimentation in this field. Using Private Eyes to overlay a mono or binocular image on the real world opens many possibilities. With the addition of a tracking system, the user could have a virtual desk overlaid in three dimensions on his real desk. Graphical user interfaces could add physical position to the descriptors of certain files. For example, a user could leave files at different locations in the office. These could act as reminders for certain actions the user has to perform. In addition, such a wearable with tracking might enable remote conference participants to be overlaid on the real world. Repairmen might get visual instructions overlaid on the devices they are supposed to fix. Architects and interior designers could have blueprints overlaid on a physical structure as they walk through it (a longer distance tracker like the Global Positioning System could be used). Construction engineers could visualize changes to a structure in the field. Doctors could visualize the inside of their patients before (or while) they operate. Note that these complicated graphics might not need to be rendered on the wearable. Instead, a base computer might be used to calculate the graphics necessary for the application and then transfer the information to the wearable for display. Several research efforts are already underway on these topics. However, the registration and tracking tasks necessary in some of these applications are difficult and may not be overcome in the near future. Knowledge Transfer One of the serious issues facing engineering companies today is the fast turn around of their employees. Often, by the time the employee is trained, he is looking for another job. However, if the employee used a Remembrance Agent to help keep notes on his training and work, his replacement can learn a great deal by simply copying the RA's files. In this way, the replacement can have access to a mini-expert for his new job even when the original employee has left. Intelligence Amplification Through Collectives Through the coupling of users with wearable interfaces, large intelligent collectives might form. The first implementation might be similar to an Internet irc channel, where several like-minded users congregate to talk. Such a channel might be used for real-time two-way communication from a conference attendee to remote participants who could not make it in person (possibly with images). A "help" channel might also be useful where users listen and answer questions during spare minutes for the common good (I repeatedly use such an interface for just this purpose at MIT, tapping into hundreds of other users). In this way, the power of a large group can be harnessed without much organization and without interruption of regular work. Another way to harness the power of a group is to allow access to members' Remembrance Agents. Thus, if I know that Chris is an expert on digital signal processing, I can just ask his Remembrance Agent about convolution without having to trouble Chris directly. So far, the collectives described have been loosely coupled and not personal. However, a tight collaboration can be formed between two people by dedicating a portion of each person's screen to the other's work. For example, let us imagine such a system between George and Chris, two computer scientists. Each time George looks at a file, the name of the file and the few lines around George's cursor appear automatically on Chris's screen. While Chris may not pay attention to these small disruptions (which are similar to what his Remembrance Agent may do), he has a constant idea of George's context. Next time George and Chris actively talk, Chris can be easily brought up-to-date on George's work. Furthermore, if something George types catches Chris's eye, then Chris can actively give advice (for example, Chris knows the location of a particular file or command which George seems to be searching for). Note that this system can also be asynchronous and filtered by an agent to avoid sending too many updates (keystroke by keystroke would be too disruptive) and to avoid displaying information when the receiving party is asleep. Simple extensions of this example can be applied to many fields. The Here and Now Unfortunately, the traditional computer companies have been ignoring this potential market, and the pen-based companies still hang on to the myth that handwriting recognition is the correct interface for PDA's. However, there are several research companies, universities, and independent inventors who have discovered wearable interfaces and have started prototyping the necessary hardware to become a "cyborg." Below are some of the companies and individuals that I have found instrumental in creating my current system and probably can be tapped to make copies. A wearable web page is being developed to provide more information on vendors. Doug Platt (showed up at the Media Lab with a working prototype when mine was still in pieces - my present unit was custom made by him and then revamped by me- has several ideas on chording keyboards as well as the unified technology,) firstname.lastname@example.org Select Tech (215) 277 4264 1657 The Fairway, Suite 151, Jenkintown, PA 19046 HandyKey Corp. (the one-handed keyboard/mouse) (516) 474-4405 141 Mt. Sinai Avenue Mt. Sinai, NY 11766 email@example.com Private Eye (display) Reflection Technology Inc. 230 Second Ave. Waltham, MA 02154 617-890-5905 FAX 617-890-5918 However, the marketers of the Private Eye are now Phoenix Group Plainview, NY (516) 349-1919 Park Engineering (the main base unit...their general version has a limited speech recognition board built in) Spokane, Washington (unfortunately this address has changed) As for my personal system, I am slowly evolving a software and hardware environment I need for everyday use. I am also working on a study of the long term effects of using this particular design (physiological, psychological, and productivity). Hopefully, as more wearable users appear (there are about 4 presently), I will be able to do studies on collaborative work as well. Acknowledgements and Disclaimer When I first began gathering equipment to experiment with wearable computing, I thought I was the first. Of course, this was wrong. Many people have worked on these systems both before and after my personal revelation. While I try to keep track of everyone who has influenced my opinions or given me facts to work with, I know that I can't possibly name them all. In particular, however, I would like to thank the following people for equipment, support, ideas, and criticism during this trek into a new frontier: Russ Hoffman (who probably started me thinking in this direction with his "silly" science project back in '86); Devon McCullough; Doug Platt; Steve Feiner; Pattie Maes; Henry Lieberman; Olin Shivers; Steve Roberts; Henry Fuchs; Mike Hawley; the folks at HandyKey Corp., Reflection Technology, and Park Engineering; the participants in sci.virtual-worlds and comp.sys.pen; and my co-workers at BBN and MIT who had to suffer through my enthusiastic outbursts and strange experiments these last few years. My opinions are my own. Sidebar: Why handwriting-based PDA's won't do it Personal Digital Assistants are supposed to be just what their names imply, personal and assisting. The PDA manufacturers would have you believe that you can (or will be able to) take these machines with you wherever you go, keeping notes, updating schedules, etc. However, today's machines have fundamentally bad interfaces for the following reasons: (1) Small screens. While the rest of the computer world has been migrating to larger and larger displays so that the user has enough room to use GUI's, the screens on PDA's have been getting smaller and smaller. Unfortunately, today's PDA's emphasize portability, which forces the smaller sized screens. Also, the handwriting interface most of the PDA's proclaim requires enough room for the user to write. This provides a fundamental limit on the physical size of the screen. (2) Awkward. All the PDA's on the current market require unzipping, unvelcro-ing, or otherwise unpackaging the PDA when you want to use it and then repackaging it when you are finished (while the Newton and the GRID Palmtop are small enough to be attached to the body, you still have to unvelcro the Newton from your pants or take out the pen for the Palmtop). Furthermore, almost all the PDA's require both hands for use (one to steady the tablet, the other to write). This is very inconvenient whenever simple one-line notes are required. Also, the user has to be careful to not damage the large LCD screen (for instance, don't put it in your back pocket). (3) Handwriting is a bad interface. The pen-based manufacturers claim that pen computing provides an intuitive interface with no training to operate. However, handwriting is NOT intuitive. We spend several years in school learning how to form our letters properly (some of us never learned). The pen manufacturers claim that this is still a lowest common denominator that is taught in the schools, and we can assume users will know how to write. However, in today's elementary schools, children are also being taught how to type. In fact, some claim that by the time today's first graders graduate, they will have typed 40,000 lines of code! Handwriting is not the wave of the future, it is the wave of the past. Assume then that handwriting recognition is a temporary measure (which many manufacturers claim, since speech recognition is now foreseeable). However, today's handwriting recognition simply does not work well. To get any useful work out of a handwriting system requires both user and computer training. So much for the walk-up interface! Pen manufacturers claim that this will improve with time, and indeed it will. Many research efforts in the area are now beginning to bear fruit. However, good handwriting recognition (writing a cursive paragraph with only one or two recognition mistakes) still requires most of the processing power of today's top workstations. With this amount of power, adequate speech recognition can be run just as easily! Why write when you can just talk? Even if one ignores the previous two objections to handwriting recognition, there is still a more basic problem. Handwriting is just too slow. Even assuming perfect, immediate recognition of handwriting, typing is faster for transferring information from a user to a computer. Of course, speech recognition is still faster than either handwriting or typing (in general). However, even assuming cheap, fast speech recognition, there will be times when speech is not convenient (privacy or when already talking with others). Even in a speech recognition future, keyboarding will still be useful by allowing another, possibly parallel, mode of communication between human and computer.