iBet uBet web content aggregator. Adding the entire web to your favor.
Link to original content: https://www.billbuxton.com/inputTimeline.html
Some Milestones in Computer Input Devices:
An Informal Timeline
Bill Buxton
Microsoft Research
First Draft: February 18th, 2011
Current Version: August 25, 2023
This page is a work in progress. It is an attempt by my to capture some of the key events, dates, and players that have helped drive the evolution of the range of input devices, technologies and techniques that we use today. I have organized the page in chronological order. And, be clear that my purpose here is to provide a brief overview, not a text book. I am not trying to chronicle the full history; rather the key highlights. If you are interested in diving in deeper, here are a few other resources on the topic of human input to computer systems that I have available on-line:
Finally, what I present here is almost by definition neither the whole story, or even the true story. It is a personal view, based on my current understanding and opinions - both of which are subject to change. One of the benefits of my making this public is that it forces me to be explicit in terms of what those understandings and opinions are. Hence, there is also a built-in mechanism that aids others in helping me learn, and thereby help those understandings and opinions evolve. This is, of course, a round-about way of soliciting feedback on what I have written. I hope my creating this page helps you. In exchange, I hope that you help me.
| c.a. 1950 | Light Gun: Robert Everett, Lincoln Lab, Lexington, MA. In the early 1950s, Robert Everett developed a light gun to read the position of a dot on the screen of the Whirlwind computer for diagnostic purposes. With the SAGE air defense computer, developed in 1956, the light gun was identify aircraft of interest by selecting them on the CRT. Lincoln Lab, Lexington, MA.
|
 Photo Credit: Computer Desktop Encyclopedia, © 2000 The MITRE Corporation Archives. |
| 1952 |
Trackball: Tom Cranston, Fred Longstaff and Kenyon Taylor, Ferranti Canada.
References
|
 Photo Credit: Ingenium - Canada's Museums of Science and Innovation, Artifact File 1977.0315. |
| 1954 | Digital Ink "Scope Input" Douglas T. Ross, MIT Lincoln Lab, Lexington, MA..
This is the first example that I can find of using analogue stroke to enter symbols, such as text, on a display. The technique is described as follows: A 'scope input' routine has been written which gives a type of two-dimensional analog input to the Whirlwind computer. The principle of the program is similar to the 'flying spot scanners' used on analog computers. The equipment used is a 16-inch oscilloscope under the control of the computer with a photocell mounted so that its field of view is the entire scope face. Then by programming a flying spot and asking whether or not the photocell 'saw' the spot, the program can be made to follow an opaque pointer as it is moved in a random fashion over the face of the scope. Since the program displays the spot by digital coordinates, the tracking of the pointer constitutes analog input to the computer. The italicized text above comes from a talk by Ross a version of which can be found in the following references in print and video:
|
 |
| 1957 | Stylator: A Specialized Digitizing Tablet for Stylus Input of Constrained Characters: T.L. Dimond, Bell Laboratories, Murray Hill, N.J.. This is a "next step" in stylus-based pen-input to a computer system. The 1954 work of Ross did enable the entry of text by writing on a scope. However. the hardware was very specialized and was difficult to replicate. Dimond's work on the Stylator enabled text to be entered on a small digitizing tablet by writing in a constrained manner (i.e., how characters were shaped) with a stylus. Some (see Wikipedia, for example) have declared the Stylator the first pen-operated digitizing tablet. I would credit the 1963 Rand Tablet (discussed below), with that distinction, since the Stylator, was not a general purpose tablet which could capture any pen-strokes written on it's surface. However, I do see the Stylator as a much neglected, but important, step between the 1954 work of Ross, nd the Rand Tablet.
|
 |
| 1957 | Lightpen: Ben Gurley, MIT Lincoln Lab, Lexington, MA.. A stylus shaped descendent of the Light Gun. The first technology that let you use a pen-shaped device directly on the screen.
|
 |
| 1960 | Glove-Based Keyboard: Robert Seibel & Nathaniel Rochester, IBM, NY. While really a chord keyboard for text entry, rather than a continuous controller as is seen in later devices, and more like a boxing glove than the gloves we wear in daily lift, this is the first glove-based input device that I am aware of. The patent was filed in 1960 and granted in 1963.
|
 |
| 1963 | Graphics tablet with stylus: The Rand Corp. The so-called “Rand Tablet” was developed under contract to ARPA. It paved the way to the graphics tablets still used today, such as those made by Wacom. It was 10"x10", employed a stylus, and had 100 dpi resolution. This tablet was later manufactured (in 1964) by Bolt, Beranek and Newman, Inc. (BBN) as the Grafacon 1010A Digitizing Tablet.
|
 |
| 1964 | 3 DOF Trackball Controller: Lincoln Lab, Lexington, MA. By this time, Lincoln Lab was using a 3 degree of freedom (3 axis) trackball for control of the PDP-4 computer. (See the entry for the first trackball in 1952, above.). |
|
| 1964 | Mouse. Doug Engelbart and William English, Stanford Research Institute (SRI), Palo Alto, CA. While the patent was not applied for until 1967, work began in 1964, and some of the ideas as early as 1963. With the cooperation of Engelbart and English, the design was already finding its way into other labs (such as the National Research Council of Canada) and computer science departments as early as 1969. Worth noting is that the Engelbart mice sensed motion using potentiometers rather than shaft encoders. Although the potentiometers were capable of multiple rotations before reaching their limits, this could still lead to problems of the wheels locking.
|
 |
| 1965 | Touch Screen Technology Described: E.A. Johnson of the Royal Radar Establishment, Malvern, UK. Describes a workable mechanism for developing a touch screen. What is interesting is that the touch screen used capacitive sensing - as do most consumer devices today. Photo from Johnson (1967).
|
 |
| 1966 | Three-Dimensional Wand: Larry Roberts, Lincoln Lab, Lexinton, MA. This was a 3D Ultrasonic Stylus developed for the TX-2 Computer. It could sense resolution of .2 in. in a 4’ x 4’ x 4’ volume. This is the first input device that I am aware of which employed ultrasonic sensing. Such technology was later used for digitizing tablets (such as by Science Accessories Corporation / SAC), head-tracking assistive technologies (Personics), 3D controllers for video games (such as the Mattel Power Glove), among other things.
|
 |
| 1968 | Ball Mouse: "RKS Rollkugel Steuerung ," Rainer Mallebrein, Telefunken, Germany First Mouse to Ship Commercially First Commercial Computer to Ship with a Mouse: Telefunken TR86 The "Rollkugel Steurerung" developed by a team led by Rainer Mallebrein was not only the first ball mouse, it was the first mouse to ship commercialy as well as the first to use shaft encoders rather than rotation-limited potentiometers. It was designed for the Telefunken TR86, which was a computer to handle input/output, and which sat in front of the larger Telefunken TR4, and later TR440, computers. As such, the Telefunken TR86 also became, the first commercial system to ship with a mouse. Despite these four firsts, very few have ever heard of the Rollkugel Steurerung. Instead, most sources credit Ron Rider's 1973 ball mouse as being the first ball mouse, and despite following Mallebrein's mouse by at least 5 years (and still using potentiometers rather than shaft encoders), it was awarded a patent on the idea (while Mallebrein's mouse was denied a patent due to lack of novelty. Too often, such is the nature of innovation. Until recently, few knew about Mallebrein's work, and it was the later, inependent "invention" that most impacted the industry. The ball mouse was a great improvement over the original design of Engelbart and English. Their original mouse sensed the rotation of two wheels protruding from the bottom at right angles - a bit like an Etch-a-Sketch (except the sides of the wheels were vertical rather than horizontal, relative to the mouse bottom). In contrast, the Rollkugel was an upside-down trackball that was rolled by moving the housing, such that the ball moved due to contact friction with the table. The name, "Rollkugel" is the German word for trackball, and the idea for this mouse - invented without knowledge of Engelbart & English's work - was inspired by using a trackball, and having the insight that one could make a different version by using a much smaller ball, and using it the other way around. You can experience this transformation between trackball-to-mouse yourself: just flip your trackball upside down to get a ball mouse, or the reverse to get a trackball. It is worth noting that this did not happen out of the blue. The work was preceeded not only by experience with a trackball, but also by testing a number of graphical input technologies for computer-based radar systems, including the light-pen, touch screen, ultrasonic surface waves (a kind of touch screen) and the trackball. They were not content just to find "a" solution to the pointing problem. They wanted to find the best solution. Hence. this work also represents a very early comparative study in Human Computer Interaction (HCI). Within all of this we have yet another lesson in the nature of innovation. Not knowing about a precedent does not mean that the precedent didn't exist. This is why I so strongly emphasize that innovation is not just about inventing the new, but also classical research and scholarship into the past.
|
  Photos Copyright: Computermuseum, der Fakultät Informatik, University of Stuttgart |
| 1972 | Touch Screen, F.A. Ebling, R.L.Johnson & R.S.Goldhor, University of Illinois, Urbana Champlain. This was the first widely deployed touch screen technology. It sensed position by the finger interrupting light beams that ran parallel to the surface of the computer screen. It was developed for the PLATO Computer Assisted Instruction system, which was deployed in schools throughout the state. The technology was commercialized by Carroll Touch. which was founded in 1973.
|
 Photo courtesy of the Archives of the University of Illinois, Urban Champaign.Found in RS: 39/2/20, Box COL 13, Folder COL 13-13 Computer Ed. Research Lab / PLATO 1952-74 |
| 1973 | Ball Mouse Redux: Ron Rider, Xerox PARC. While generally credited with having developed the first mouse that sensed movement using a bottom-mounted ball, this is not accurate. The first was the 1968 Mallebrein's Telefunken Rollkugel, discussed above. The vagaries of the computer industry being what they are, timing isn't everything. Also at play are the first 3 rules of real-estate: location, location, location. In this case, the location was Silicon Valley, and more specifically, Xerox PARC. Hence, while not the first, the ball mouse of English and Hawley was the most known and influential. Unfortunately, in terms of gaining recognition for their innovation, the Telefunken mouse has ended up as a footnote.
|
 A drawing from Ron Rider's ball mouse patent. Besides being a ball mouse, the form factor also resembles that of Mallebrein's 1968 ball mouse (which is a matter of ideas independently finding the same form, rather than copyig.) |
| 1973 | Active Haptic Input/Output Device: Frank Beck and Bent Stumpe, CERN. To the best of my research, the first use of active adaptive manual haptics in human-computer interaction was a rotary knob implemented in 1973 at the CERN Accelerator in Geneva. If correct, it is the first computer input device which was also, simultaneously, an active computer-controlled (haptic) output device. This single physical rotary knob could change its characteristics depending on what parameter it was controlling at the time. It could function as a rotational shaft encoder with dynamic computer control of the instantaneous coefficient of resistance, ranging from free-wheeling to locked. It could also emulate an isotonic rotary controller, that is, a controller which output positive or negative incremental values proportional to the amount of rotation, with computer controlled torque, which on release would “spring return” to its neutral "home" position. Finally, it could function/feel like a step-wise rotary controller with 16 detents, equally spaced around the 360˚ of rotation.
|
 Drawing from Beck & Stumpe (1973) illustrating the rotary controller with computer-controlled haptics developed at CERN.span> |
| 1981 | Optical Mouse. Independently developed by Steven Kirsch, Mouse Systems and Richard A. Lyon. The optical mouse avoided the problem of dust and dirt accumulating in the mechanisms of mechanical mice. Kirsch's mouse was commercialized by his company, Mouse Systems in 1982 (upper image to the right). A mouse based on Lyon's technology was incorporated into Xerox's 6085 "Viewpoint" workstation,which replaced the 8010 Star workstation in 1985 (lower image to the right). Unlike today’s optical mice, both of these early designs required a special mouse pad with a pattern on it that the optics used to track movement. The Mouse System's pad, seen in the first image to the right, was made of glass, while Lyon's, seen in the second image to the right, was made of paper (and hence one could make a replacement using a laser printer). You can print out your own copy of Lyon's paper pad from here. .
|
  |
| 1982 | Glove and Body Joint Sensor: Thomas Zimmerman, VPL Research, Palo Alto. In 1982 Zmmerman filed a patent (granted in 1985) for an optical flex sensor. In it he described how it could be sewn onto the fingers of a glove or a body suit in order to sense the amount of bend or flex of the joint covered. The technology described was employed in the VPL Datagloave which - when coupled with a 6DOF sensor, could determine the position and orientation of the hand, and pose of the fingers.
|
 |
| 1982 | Multi-Touch Input: Nimish Mehta, University of Toronto. This was the first documented multi-touch device for human-computer interaction. It was a touch-tablet rather than a touch-screen. The touch surface was a translucent plastic filter mounted over a sheet of glass, side-lit by a flourescent lamp. A video camera was mounted below the touch surface, and optically captured the shadows that appeared on the translucent filter. (A mirror in the housing was used to extend the optical path.) The output of the camera was digitized and fed into a signal processor for analysis.
|
 |
| 1983 | Rich Gesture Recognition, Myron Krueger. Video Place (and later Video Desk) was a vision based system that tracked the hands and enabled multiple fingers, hands, and people to interact using a rich set of gestures. Implemented in a number of configurations, including table and wall. Didn’t sense touch, per se, so largely relied on dwell time to trigger events intended by the pose. On the other hand, in the horizontal desktop configuration, it inherently was touch based, from the user's perspective. Essentially “wrote the book” in terms of unencumbered (i.e., no gloves, mice, styli, etc.) rich gestural interaction. This work was more than a decade ahead of its time and was hugely influential, yet not as acknowledged as it should be.
|
 |
| 1983 | PC with Touch Screen, Hewlitt Packard HP 150. Released in Septermber 1983, this was one of the first, if not the first (I have not found one earlier), PC commercially released with a touch screen. The computer ran MS-DOS, and the touch screen was standard on the first versions (although in later models, it was an option). The touch technology used was the same approach as that of the PLATO IV, discussed above, that is, the finger interupted light beams that ran parallel, and right above, the screen surface. The lack of success of the touch option is a good lesson in timing. At the time, graphical user interfaces were not widely used, in the system or applications, and even if they were, touch screen interaction is not generally well suited to desktop applications where one has to reach over the keyboard to the screen to select or point. As today's touch devices show, the HP 150 does not tell us that touch does not work, or is a bad idea. It just reinforces that everything is best for something and worst for something else. A touch screen is not the only duck that has to be lined up in order to have a successful product or application - something that is true for almost any single technology, touch or otherwise. |
 |
| 1984 | Multi-Touch Touch Screen: Bob Boie, Bell Labs, Murray Hill, N.J. The first transparent multi-touch screen overlay. |
|
| 1984 | Cordless Keyboard: PCjr "Freeboard" Keyboard, IBM. The PCjr computer, announced in 1983, and shipped in January 1984, employed a battery-powered cordless keyboard that employed IR technology. It was presented as working within a 6-foot range, however, like virtually all IR-based mice and keyboards to follow, it had limited success because it required unobstructed line-of-sight to operate (something frequently problematic on a desk-top). |
 Photo Credit: DigiBarn Computer Museum |
| 1984 | Tablet with Cordless Sylus: Wacom, Japan. The WT Series WT-460M, launched in January 1984, was the first tablet with an untethered stylus. |
 |
| 1984 | Cordless Mouse: Metaphor Computer. The first cordless mouse that I am aware of shipped with the Metaphor computer in September 1984. The computer also had a cordless keybard and function keypad, but the IBM PCjr shipped a cordless keyboard 9 months earlier (see above). The mouse was built for Metaphor by Logitech. It used infra-red (IR) signals to transmit mouse data to the computer. The existence of this mouse was apparently not well known, since two companies, Torringon in 1986, and Numonics in 1989 both advertised their products as having "The World's First Cordless Mouse". While coming from different companies, and having different form factors, both were named, Manager Mouse, the Numonics product presumably licencing the Torrington technology. Both of these mice also used IR technology, as did another 1989 cordless entry from a company called Practical Solutions. The problem with such devices that used IR technology was that, in order to work, they needed a clear line of sight between the mouse and the computer's receiver – a potential problem on a cluttered desk. Hence, cordless mice did not get traction until this problem was solved. This was accomplished by replacing IR with radio frequency (RF) communications. As discussed below, the first mouse to do this was the MouseMan Cordless, released in 1991 by Logitech.
|
 |
| 1985 | Scrolling Mouse: "Mighty Mouse". NTT, Japan & ETH, Switzerland. This is the first scrolling mouse that I am aware of - that is, a mouse with an integrated control from scrolling (albeit, a bi-directional skiding switch rather than a wheel, which became th emost common mechanism). It was not available commercially, and is not to be confused with later similarly named mice developed by Apple. The Mighty Mouse had five keys (one for each finger and the thumb) as well as a thumb wheel. The thumb and index finger keys were spring-loaded analogue keys that could either be binary, if activated by a fast click, or continuous analogue controls if pushed slowly. In addition, the thumb key could be slid forward and backward to change mode. The other 3 keys were binary switches. In contrast to modern mice, scrolling was not done with the wheel, but with the thumb key. Clicking the thumb key moved the document forward or backward a page at a time, while pushing gently on the thumb key smoothly scrolled the docuement or list forward or backward at a speed proportional to the pressure. The direction of paging and scrolling depended on if the thumb key was in the forward or back position. There were at least two versions of this mouse built and used. For scroll-wheel mice, see the 1989 Apple prototype and the 1995 Kye Systems mice below.
|
 |
| 1986 | Apple Desktop Bus (ADB): Apple Computer. Rather than an input device, the ADB, created by Steve Wozniak was an important new means for input devices to connect to the computer. It was a bus, rather than a serial port, and what it enabled you to do was connect multiple devices to it without needing a separate port on the computer for each device. This was because, as a bus, one could daisy-chain devices. It also brought with it the ability to plug-and-play. One could just plug in a trackball without rebooting the computer or doing anything else, and it could replace the function of the mouse - regardless of whether the mouse was still plugged in or not. But what was most interesting - but not that often exploited, was the ability of the ADB to distinguish which signals were coming from which device. Hence, if the application was appropriately programmed, one could map the signal from the mouse for one function, such as the normal pointing and selecting, and the signal from a trackball to scrolling. In many ways, the USB connector that we use is the ADB on steroids. USB replaced the ADB, but the ADB paved the way for it to do so. |
 |
| 1989 | Scroll-Wheel Mouse: D. Venolia, Apple Computer. This is the first scroll-wheel mouse - i.e., a mouse with a wheel-like rotary controller to enable scrolling - that I am aware of. It was developed by D. Venolia of Apple, and shown at the ACM SIGCHI Conference in 1989. It was a standard Apple ADB mouse augmented by a third shaft encoder. The roller attached to this encoder was manipulated by the thumb, rather than the forefinger, and was mounted on the thumb side of the mouse, rather than on the centre of the mouse top, as in most modern scrolling mice. The device worked on any Macintosh applicaton that also had a vertical scroll-bar. The first commercial mouse with an integrated scroll wheel was released in 1995 under two names, the Genius EasyScroll, and the Mouse Systems ProAgio. However, the first widely adopted scroll wheel mouse was the 1996 Intellimouse, the first scrolling mouse shipped by Microsoft. The genisis of this mouse began in 1993, with work done by Eric Michelman. Due to this work, Michelman is frequently credited with having invented the scroll wheel. However, depite his improving the technology, Venolia's work clearly preceeded his.
|
 |
| 1991 | Cordless Mouse with RF Communication: MouseMan Cordless, Logitech While the first cordless mouse was released in 1984 by Metaphor Computer (see above), it used infra-red wireless technology, which meant that the mouse and the computer sensor needed to maintain line of sight. Hence, any object blocking that would mean that the mouse did not work. The MouseMan by Logitech was the first commercially available cordless mouse to use radio frequency (RF) technology, which eliminates the line-of-sight problem. RF technology has been used since by virtually all cordless mice. |
 |
| 1995 | Commercial Scrolling Mouse: Genius EasyScroll / Mouse Systems ProAgio, Kye Systems Corp. While the first scrolling mouse that I know of was the 1985 Mighty Mouse, and the first scroll-wheel mouse the 1989 Apple prototype (both discussed above), I believe this 1995 mouse to be the first commercially available scrolling mouse. It was produced by Kye Systems Corp. of Taiwan, and released under two different brands and names: the EasyScrollMouse under their Genius brand, and as the, ProAgioMouse, under their Mouse Systems brand (Kye acquired Mouse Systems in 1990). The version that I have in my collection is the Mouse Systems ProAgio. |
 |
| 1997 | 2D Scrolling Mouse: TrackPoint Mouse, IBM Corp. This is the first commercially available mouse that directly supported both vertical and horizontal scrolling. Both the direction and speed of scrolling was controlled by the TrackPoint 2D isometric joystick mounted between the two mouse buttons. The direction of the force applied to the joystick controlled the direction of scrolling, and the amount of force controlled the scrolling rate. Since one apply the force in any direction in the 2D space controlled by the TrackPoint, one was not contrained to only horizontal or vertical scrolling. That is, one could also scroll diagonally; however, studies showed that, with the TrackPoint at least, that people rarely did diagonal scrolling, and that providing that capability sometimes resulted in unintentional vertical movement when only horizontal scrolling was intended, or vice-versa. Hence, in later models, the Scrollpoint was given a "saddle" type cap whose ergonomics helped reduce the problem, while at the same time improved the ergonomics of horizontal scrolling. In addition, a "dead zone" was put in the driver software so that near vertical forces resulted in vertical motion only, and likewise for near horizontal forces.
|
 |
| 1999 | Optical Mouse Without Mouse Pad: Intellimouse, Microsoft Corp. In 1999, Agilent Technologies (www.agilent.com), a spin-off of Hewlett-Packard, developed a new optical sensor that enabled optical mice to be made that did not require a special mouse pad. This paved the way for a new generation of optical mice, of which the Microsoft Intellimouse was the first. |
 |
| 2011 | Bi-Directional Pixels in Interactive Display: Microsoft PixelSense, Microsoft Corp. & Samsung Electronics. This device was originally released under the name of " Microsoft Surface V2." SInce then, it has been rebranded as "Microsoft PixelSense", and should not be confused with the Windows 8 "Surface" tablets. The system is a unique 4" thick, 40" diagonal 1,920x1020 display where the pixels are used to capture light as well as emit it. The hardware was developed by Microsoft in partnership with Samsung. The hardware is also branded the Samsung SUR40. One can think of the whole display surface as simultaneously functioning as both an LCD display and a CCD immager (like the chip that captures images in a video camera). Since it can "see" finger touches, it is a multi-touch display. But it can also "see" the shape of the object, or what is written on a page (within the resolution of the spacing of the pixels). The display emits visible light much like any colour LCD display, by means of triads of RGB pixels. However, each triad is augmented by a 4th pixel that emits invisible infra-red (IR) light. When something like a finger or object is placed on, or near, the surfce of the screen, the IR light is reflected back into the display, where its intensity is captured by the IR pixel. This enables the display surface to function like a real-time monochrome scanner, which has the same spatial resolution and the display. Adding the 4th IR pixel to the RGB triad enables the sensing to work even when the screen is blank (all RGB pixels not emitting light). However, sensing only reflected IR light is the reason that the device is monochrome. Finally, one place where the analogy to a camera's CCD breaks down is that cameras typically have lenses; PixelSense does not. Hence, only things directly against the display surface are captured in focus. As the distance from the object and the surface increases, the contrast and focus rapidly degrade. On the other hand, this also means that there is a certain capability to sense "hover", i.e., the presence and location of something near, but not touching, the display surface. This is not typical of multi-touch displays, and is on additional significant, but too often missed, attribute of the technology.
|
 |