Skip to main content

Pranav Mistry: The thrilling potential of SixthSense technology


'SixthSense' is a wearable gestural interface that augments the physical world around us with digital information and lets us use natural hand gestures to interact with that information.

We've evolved over millions of years to sense the world around us. When we encounter something, someone or some place, we use our five natural senses to perceive information about it; that information helps us make decisions and chose the right actions to take. But arguably the most useful information that can help us make the right decision is not naturally perceivable with our five senses, namely the data, information and knowledge that mankind has accumulated about everything and which is increasingly all available online. Although the miniaturization of computing devices allows us to carry computers in our pockets, keeping us continually connected to the digital world, there is no link between our digital devices and our interactions with the physical world. Information is confined traditionally on paper or digitally on a screen. SixthSense bridges this gap, bringing intangible, digital information out into the tangible world, and allowing us to interact with this information via natural hand gestures. ‘SixthSense’ frees information from its confines by seamlessly integrating it with reality, and thus making the entire world your computer.

The SixthSense prototype is comprised of a pocket projector, a mirror and a camera. The hardware components are coupled in a pendant like mobile wearable device. Both the projector and the camera are connected to the mobile computing device in the user’s pocket. The projector projects visual information enabling surfaces, walls and physical objects around us to be used as interfaces; while the camera recognizes and tracks user's hand gestures and physical objects using computer-vision based techniques. The software program processes the video stream data captured by the camera and tracks the locations of the colored markers (visual tracking fiducials) at the tip of the user’s fingers using simple computer-vision techniques. The movements and arrangements of these fiducials are interpreted into gestures that act as interaction instructions for the projected application interfaces. The maximum number of tracked fingers is only constrained by the number of unique fiducials, thus SixthSense also supports multi-touch and multi-user interaction.

The SixthSense prototype implements several applications that demonstrate the usefulness, viability and flexibility of the system. The map application lets the user navigate a map displayed on a nearby surface using hand gestures, similar to gestures supported by Multi-Touch based systems, letting the user zoom in, zoom out or pan using intuitive hand movements. The drawing application lets the user draw on any surface by tracking the fingertip movements of the user’s index finger. SixthSense also recognizes user’s freehand gestures (postures). For example, the SixthSense system implements a gestural camera that takes photos of the scene the user is looking at by detecting the ‘framing’ gesture. The user can stop by any surface or wall and flick through the photos he/she has taken. SixthSense also lets the user draw icons or symbols in the air using the movement of the index finger and recognizes those symbols as interaction instructions. For example, drawing a magnifying glass symbol takes the user to the map application or drawing an ‘@’ symbol lets the user check his mail. The SixthSense system also augments physical objects the user is interacting with by projecting more information about these objects projected on them. For example, a newspaper can show live video news or dynamic information can be provided on a regular piece of paper. The gesture of drawing a circle on the user’s wrist projects an analog watch.

The current prototype system costs approximate $350 to build.


Pranav Mistry is the inventor of SixthSense, a wearable device that enables new interactions between the real world and the world of data.

Pranav Mistry is a PhD student in the Fluid Interfaces Group at MIT's Media Lab. Before his studies at MIT, he worked with Microsoft as a UX researcher; he's a graduate of IIT. Mistry is passionate about integrating the digital informational experience with our real-world interactions.

Some previous projects from Mistry's work at MIT includes intelligent sticky notes, Quickies, that can be searched and can send reminders; a pen that draws in 3D; and TaPuMa, a tangible public map that can act as Google of physical world. His research interests also include Gestural and Tangible Interaction, Ubiquitous Computing, AI, Machine Vision, Collective Intelligence and Robotics.

Comments

Popular posts from this blog

Why Flash Player is the best plugin ever

Cross-platform Availability and Portability Flash Player is available for all the major operating systems: Windows Mac OS Linux Solaris Also, it comes preinstalled with Mac OSX and Windows XP . So the users of these platforms don’t even need to download it! There is also Flash Lite , which enables the viewing of Flash content on mobile devices like cellphones and PDAs. Just take a look at all the mobile phone manufacturers that are incorporating Flash Lite into their devices (currently more than 140 cellphones are Flash-enabled): Fujitsu Hitachi Kyocera Mitsubishi Motorola NEC Nokia Panasonic Sanyo Sendo Sharp Siemens Sony Ericsson Toshiba Still not convinced? Well, there is even more - the PDAs : Microsoft Pocket PC and Sony CLIE. Can there be more? Sure. Let’s not forget the consumer electronics : the iRiver U10 media player and the Kodak Easyshare-One Zoom Digital Camera are Flash-enabled, like some set-top boxes too. Oh, wait, I didn’t tell you yet which bro...

Net Telephony - Free local calls, STD at 15p/min

Thanks to the ubiquitous Internet, your telephone bill can be drastically trimmed. The Internet Service Providers Association of India (ISPAI) claims that STD calls can be offered at 15 paise/minute and local calls can be free on the same network, if the government permits unrestricted domestic internet telephony. The Internet Service Providers (ISPs) claim that they can offer free local calls in case of own network (that is, within the network of a particular internet service provider) and 10 paise per minute for calls on other IP network. “ISD call rates can further come down to 50 paise per minute on same IP network, and in other IP network 75 paise. And if you want to call on a mobile or fixed line from an IP phone, STD call rates can be offered at 50 paise per minute,” the ISPAI claims. The association has members such as Sify, Net4, HFCL, HCL Infinet, RailTel, Spectranet and Tulip IT Services. Currently, STD call rates on mobile networks are about Re 1 per minute while local rate...

Web Office: 2007 Year in Review

Over the next few weeks we'll be reviewing a number of Web product categories on Read/WriteWeb, summarizing what's happened in 2007 - and what to look forward to in 2008. We're starting with the Web Office, a market that underwent a lot of changes this year. Our definition of Web Office is: A Web Office suite is a combination of productivity, publishing and collaboration features. A Web Office both embraces the functionality of desktop office suites (e.g. Microsoft Office) and extends it by using Web Native features. Probably the biggest change was that Google Apps ramped up this year, starting with the release of Google Apps Premier in February. And 2007 continued the trend of acquisitions in this market, which started in 2006 with the likes of Writely and JotSpot. In 2007 Yahoo acquired Zimbra and Google acquired a number of small startups - including GrandCentral (online telephony service) and Zenter (presentations software). The biggest disappointment of the year in ...