A little over a month ago we announced our work developing an Augmented Reality Platform for the U.S. Government. While we can’t talk about contract negotiations as they are lengthy, sensitive, and tricky, we CAN share that our Bloom server is currently in an Alpha state and is already doing some amazing things. We are modeling, modeling, modeling from video and still photography and the output is amazing. The video above is a model of the front of my home. The video above represents only a single (but very important) step in the Bloom “Gathering Process.”
We are also in the process of optimizing the code to further improve efficiencies. As our effort continues we will be entering the second phase of our program that will include the development of the user interface for the iARM platform. That user interface will be a multichannel input / output leveraging gesture (hands/hand symbols), speech, and context. Our first evaluation will be the gesture based system. There are a number of open-source and closed platforms (this, and all of these…) that will likely play a part in this system but we are always looking for experts.
Please take the time to share if you have ideas you think would benefit this vision.
Remember that excited feeling you had at the beginning of a project where the sky was the limit and you were chomping at the bit to get started conceiving your next, greatest masterpiece? It’s now three-months later and you’re biting your nails raw in anticipation of the 3pm presentation of your concepts to your customer. It’s been a great exploration, you’ve done your best work, researched the heck out of the topic, and yet you are still praying they will love your ideas. What’s wrong with this picture? You may not know it but this situation is exactly the situation you don’t want to be in. The problem I’m illustrating is the conceptual disconnect that is causing you to lose sleep hoping they won’t dissect your concepts or worse combine them (not to mention that you are showing more than one concept). It’s the fact that the people sitting in the boardroom don’t know what they are about to see. Put bluntly, if your client is going to be surprised by your concept presentation deep into the project, you are risking failure. Sadly traditional methodologies encourage this pattern and the potential disconnects that often arise. To us, it’s just gambling.
How do we do it differently? We put our clients to work. A good friend of mine often refers to “Consultant’s Hubris” or the feeling that as a consultant we know more than the client. While we may be an expert at what we do, we hardly know the issues and opportunities the client faces on a daily (yearly) basis. We are not experts in their business, period. Face this and become more powerful.
Our contracts require client participation, sometimes significant hours weekly. When we kick-off we ask our client to designate a “steward” who will participate in the project heavily. The steward must have deep experience within the client organization and products we are working with. They are a Subject Matter Expert (SME).
We start with a working session to clearly define goals (or at least the first draft of the goals) and plan out how we are going to attack them. Following this are short 2-3 week work cycles (sprints) that involve twice-weekly working sessions where the client-steward and team are locked in a room discussing, researching, sketching and prototyping. Because our people are very experienced, they guide the steward’s ideas to merge with relevant trends and technology and result in beautiful solutions. At the end of each sprint we evaluate our collective success, present status to the larger client organization (gather additional feedback) and plan the next sprint. At the end of the project we have developed one concept with the steward that is not only extremely relevant but also co-owned. That’s right, the steward owns the solution because they co-developed it and they actually help ’sell’ our collective solution to their organization as we develop it. Of course this means they are circulating pencil sketches and prototypes well before the final presentation, but they are also translating the thinking into meaningful dialog with their colleagues. It’s very powerful to watch. The best part is we together are all heroes at the end and if you can make your client a hero, you’ve done your job correctly.
The first thing people ask me when I talk about this approach is how we handle the “bad” ideas stewards bring with them. The answer is communication. We’ve had stewards bring us wireframes of the application with full expectation that we would build exactly what they had specified. We embrace their concepts and talk directly about them with the steward. We explore the concepts that inspired them and the needs they are trying to solve. These concepts are immensely valuable windows into the mind our partner/client. During this conversation we share experience, industry trends, competitor approaches, and user-centric best practices. We do the drawing – together we are each the best at what we are. I’m proud to say that in every project we used this approach, the results have been stellar. The solution is something both we and the client are immensely proud of.
Because this method is designed to break down the walls between the client and our team it works very well with companies who have stakeholders with competing needs. Our process works very well in this “federated” situation because we incorporate stewards from each department in our working sessions. The debates happen in real-time and if they can’t be solved on the spot, they are resolved within the week with follow-up discussions.
There isn’t enough room on our blog to talk through every detail of this approach but if you’d like to learn more about this process we’re happy to help. Drop us a line or post-back here and we’ll be glad to share. If you like we can present our approach, train your team, and even collaborate on your next big thing!
If you invested 26 minutes into the video we posted here, then you already know where we are going with this post. Just as augmented reality (AR) is only one facet of an embedded interface, touch is only a portion of the entire vision for Natural User Interface (NUI). We are seeing a lot of touch and AR discussions today mostly because there have been advances in technology that are commercially (to we the consumer) available. Not to be a nag, but isn’t about time the problem drives the solution and not technology? Just as we learned during the age of artificial intelligence, technology (specifically improvements in processor speed in this case) will never solve the problem. Complexity must be replaced with simple and elegant solutions. Human’s with dreams must drive these solutions.
What is Natural User Interface (NUI)
So what is Natural User Interface? Natural user interface is the concept that you can touch (or yell at…) and manipulate digital objects much like you would objects in the real world. It’s a hard problem because we have been using mice and keyboards for so long we’ve forgotten what natural really feels like. On a recent cavort through the interwebs, I discovered the following concept and am thrilled to see discussions standardizing the way we talk about NUI.
Unfortunately, gestures like the two-finger-tap and the two finger hold are not natural interactions. We call these types of interactions “NUI Transitional.” They are interactions that are trained/learned that allow fingers to modulate information, but they are not “Present in or produced by nature.” An exaggerated example (talking to you Apple) would be if you have to use four fingers in a swiping gesture (versus 2 or 3) to interact in a specific way, you’re really not experiencing a natural interaction. More likely the software engineers ran out of natural metaphors, right?
A more natural approach
A hold-and-act-upon gesture is a good example of a truer NUI interaction for a touch surface. Specifically imagine you have a piece of paper on your desk and you want to slide it somewhere else. You touch the paper and, while holding your finger down, drag it to another location. Simple right? If you want to draw on that paper you wouldn’t use the same gesture. You also wouldn’t look around for a state-indicating button or icon to make sure you were in draw mode and not move mode, right? Instead you would put one hand (or finger) on the paper to hold it still and the second hand (perhaps with a writing implement) would move the drawing element over the surface of the paper.
This illustration uses Ron’s iconography to illustrate a stateless natural interaction. Specifically, we have one touch point from the left hand thumb holding the page still (a state in itself), while the right hand pinches (not the gesture) three fingers together much like holding a pen and drags across the surface. The result is a hard-to-explain but intuitive feeling writing gesture. I’ll disclaim that last statement in that we have no data to prove how “intuitive” that gesture is, but when we do it ourselves (give it a try) it feels pretty good. Mileage may vary… What do you think?
Channel 9 and some very cool folks at Microsoft (Marc Schweigert and James Chittenden) talked to Tim and I about our Touch.Codeplex.com open source code. It’s a long video (we take that as a compliment) but reveals quite a bit about our Natural User Interface (NUI) fanaticism.
Favorite Quote: “We want to allow people to touch data.”
Tanagram is proud to announce the launch of Touch.Codeplex.com. Our exploration into the state of the Natural User Interface (NUI) connected us with folks at Microsoft and together we identified an adoption issue. While the technology to build Touch applications exists, it is currently cumbersome to implement. Marc Schweigert and James Chittenden had an idea to use Expression Behaviors (literally drag-and-drop onto any object) to act as a bridge. Together we launch this humble beginning with a bold vision. We hope you join us as we expand this library in the months to come.
Writeboard Panorama of calculation debugging.
Project Overview
The APIs in WPF4 plus the Surface Toolkit for Windows Touch make building common touch scenarios easy. However, implementing many of the same touch scenarios using WPF3.5SP1 or Silverlight 3/4 involves writing a fair bit more code. Furthermore, the touch APIs across WPF4, WPF3.5SP1, and Silverlight are different.
Project Goals
The goal of this project is to simplify building common touch scenarios when using WPF 3.5 SP1 or Silverlight 3/4 by using Expression Blend Behaviors to provide a consistent way to implement these scenarios across WPF & Silverlight. Expression Blend Behaviors can be used within Visual Studio without a dependency on Expression Blend by downloading the Expression Blend 3 SDK. You can also find more Expression Blend Behaviors at http://expressionblend.codeplex.com/ and http://tinyurl.com/ExpressionGalleryBehaviors.
Project Roadmap
Beta Release of core Scroll and TranslateRotateScale behaviors to developer community. <– You are here
Revised Scroll and TranslateRotateScale behaviors
Gesture Behavior (repurpose awesome code from here)
FROM GOLD COAST BLOG: A common need for Windows 7 touch user interfaces is to react to a touch gesture. In this screencast, I demonstrate a very simple and easy way to interpret single touch gestures using MouseGestureTrigger from the Expression Blend Samples (http://expressionblend.codeplex.com/).
To learn more about Behaviors, Triggers, and Actions mentioned in the screencast, visit http://tinyurl.com/TriggersActionsBehaviors. If you are interested in touch/multitouch Behaviors, then have a look at http://touch.codeplex.com/. You can download even more Triggers, Actions, Behaviors, etc. at Expression Gallery.
If you do nothing else today spend 5 minutes and go to this site. You’ll need a printer, a pdf viewing application, a computer with a webcam (new Macs come with these by default), and a web browser with Flash 10 installed (you will get a prompt to install if you don’t have it).
Click on the “Launch Turbine” or “Launch Solar” link and permit Flash to access your camera (you’ll get a dialog box)
Hold up the printed PDF so it shows up in your camera view and prepare to be amazed.
This demo begins to show the promise of Augmented Reality on embedded display technology. What I mean is as displays get cheaper it will be more common for them to be placed in public or even private areas for your benefit. Imagine dressing room mirrors that can show you what you will look like without changing cloths. Imagine wearing digital accessories visible only to those who view you through a screen or through a Head Mounted Display (HMD). The possibilities are endless as we start to embed virtual objects in our reality.
Note: Tanagram has been awarded (we’re in the ‘negotiation’ phase) a grant to design a user interface concept for Augmented Reality. More on that soon!
I recently attended the Information Architecture Institute Idea 2008 Conference, October 7-8 in Chicago. The theme of the conference was ” . . . on designing complex information spaces of all kinds.”
What was intriguing about this particular conference was the diversity of people, both in professional and geographic terms. There were graphic designers, interaction designers, technical leads, managers, and oh yes . . . “information architects.” What was interesting about the attending information architects was that they came from so many backgrounds to become an information architect. There were actual classically trained architects that became IA’s, there were designers that were IA’s and so on.
When Richard Saul Wurman coined the term Information Architect in the late 1980’s, he was an architect that was designing travel books (Access Press) and came up with a term for designers that created information intensive artifacts. Since we cannot really agree on what the exact meaning of “information” and “architect”, I have come to the conclusion that merging the words into a concept would be difficult having hired several IA’s professionally. There is little agreement of what an IA is, and even what their outputs are. This was evident at the Idea conference, that there was no attempt in defining the term.
The first speaker was blogger David Armano who spoke on “Micro-Interactions in a 2.0 World.” A well-known and dynamic speaker, David took participants down a very rapid terrain of design, marketing and business through technological innovations. His central premise is that we are moving from passive consumers to active participants through existing social architecture technologies – not custom applications. Since all of our devices are internet enabled, the notion of a traditional browser experience is giving way to smaller more intimate digital apps that do one or two things. We as users cross-link these apps together. He used the term “life streams” to name this process of “engage, enable, and empower” our actions through a model of “usefulness, utility and ubiquity.” David also articulated new digital ecosystems such as the Nike Touch which uses “engagement” of “deposits” and “withdrawls” with several micro-functionalities bundled together. Social networks by their very nature amplify communications and he asked the audience what their “passion point” was.
Elliott Malkin, an artist from New York discussed “Information in Space.” His passionate and precise presentation went down a very indirect route that got me very excited. His initial metaphor was the hassidic concept of an eruv, or a physical demarkation between a secular world and a religious world using the same space. He referred to this psychographic space as having strong conceptual power for the intended group and for what for most people would not even notice. Unfortunately, I thought he was going to bring the metaphor back to digital technology and social architecture, but instead he discussed using digital technology to create a virtual eruv that could be monitored without rabbi’s going out to check if the eruv physical demarcations were intact. The implications of this metaphor in discussing how a shared space could have unique “functionalities” for different groups at the same time holds great promise.
Jesse James Garrett of Adaptive Path presented “Envisoning the Future of the Web.” They worked with the Mozilla Foundation in exploring the future of web browsers. Their concept is called Project Aurora. Now, when dealing with the future of anything, especially technology, it is difficult since we tend to use established conventions and behaviors and link it to a future that people can understand. Star Trek did it best by taking human behaviors and linking them to technologies that did not necessarily need to describe their inner workings. As viewers, especially hooked viewers, we understood the galactic federation model and the value system of the show to put the expressions and technologies in context.
With the future of a browser, Adaptive Path focused on augmented reality, or the overlay of digital information on the real world where there will be data abundance and the question will be how to we visualize, focus and manage all of it cognitively and socially. Processing power, storage capacity, bandwith and graphic capabilities of computers will impact how we interact with each other through the digital cloud. “Context awareness,” “natural interaction,” and “continuity” would allow for more natural collaborations. Each of us would have a semantic profile and with geolocation, would allow for very rich interactions between people where ever they are. Two main questions arose from the audience. What was the time horizon of Project Aurora? Jesse stated they had a 10 year window into the future (this would be the equivalent of 40 years in technological terms). He said they had to balance “compelling” with “plausible” in their vision. My view was that their vision was too contemporary and linked to current “plausible” scenarios. The second question was that their concept could be viewed that the browser was an operating system. Jesse made it clear that they did not want to address the operating system vs. browser question, but in my mind the two converge in their scenario.
Chris Crawford a former game designer for Atari presented an interesting perspective on “Linguistic User Interfaces.” His perspective on intelligent systems is that smart computers that could interact with humans using extensive language patterns is not realistic. This is due to the Sapir/Wharf hypothesis that inside the human mind language and reality exist together. Chris’ interesting take is that with games, a model for a computer to interact with humans is much more manageable since the worlds are much smaller. He further elaborated that software, verbs define the program and is core to the human/computer interface. With most current software, as the verb count increases, accessibility and expectability reduces. 100 verbs is the limit for most users. Chris is currently developing a linguistic user interface (LUI) for programs that can create stories. I found his perspective very compelling.
Alberto Canas, of the Institute for Human and Machine Cognition (IHMC) presented a surprising presentation called “From Meaningful Learning to a Network of Knowledge Builders.” IHMC has created the popular language mapping program CMAP, which I have on my computer. What was informative about his presentation on CMAP were examples of its application (pun intended). Knowledge essentially are concepts that are linked together with prepositions to make a relationship (this was a wonderfully simple definition). Humans have created written language to describe concepts that cannot easily be illustrated. CMAP are concept maps linked by phrases to form propositions. I already knew this in principle. Alberto then showed the power of CMAP through an ongoing project with the Panamanian government giving school children CMAP to describe their lives. The power of CMAP is that users can link images and web page addresses to their maps and can also link concept maps to other concept maps. They are essentially mini-websites which are non-linear. I will not look at CMAP the same way and believe that its potential is not fully understood by a large cross section of users.
Jason Fried, founder of 37signals presented a lucid lecture called “Getting Real.” I had not heard Jason before, but am a heavy user of Basecamp and a real fan of its simplicity and how reliable it is. Jason is a real visionary and their development process flies in the face of every convention that most digital consultancies use. They do not “plan” anything, do no “specification” documents, and do not use “actor or personas.” They focus on building things and figure out how to do it over time. Keep things small, use sharpie markers as the finest resolution when sketching ideas, and only have a core set of functions (a simple core). Part of me was aghast, but I quickly saw the logic to their process. If you are designing for yourselves and then find users, the 37signals model is perfect. Unfortunately, if you collaborate with clients to define the problem and then facilitate understanding, then the 37signals model will not work. However (you knew there was going to be a “but”) I totally agree with his concept of “scratching your itch” and doing things with passion.
Aradhana Goel of Ideo gave one of the most thought provoking presentations called “Emerging Trends, Design Thinking, Service Innovation.” We have all heard of IDEO and it is one of the most influential innovative firms (along with Pentagram) merging design and engineering. Aradhana was trained as an architect and has only recently become involved with service design. Her perspective on human factors was clear and in alignment with my understanding. What was powerful was her ideas around linking human factors with trend factors. Human factors focus on digging into context, while trend factors find the context. She went on to compare and contrast these two areas and how service design is a logical next step in productizing intangible experiences.
Bill DeRouchey of Ziba Design gave a very direct and engaging discussion on “The Language of Interaction.” His deconstruction of everyday visual clues that we take for granted and their constant reinterpretation and reapplication to other situations was informative, common sense, but insightful.
Overall, there were several key themes that all speakers seemed to focus upon:
1) Browsers are giving way to other internet enabled experiences
2) Windows, Icon, Mouse, Pointer system is under stress
3) Transaction is more than money
4) Link several apps, not one killer app
5) Focus on experiences, not just interactions
Upon reflection, I found this conference very fulfilling and reinforced certain convictions, challenged others and provided a very positive mental workout for me (which is what good conferences should do). I would like to compliment IAI for the organization of the conference and the lunches with different groups of people was enjoyable. There were twelve speakers in two days, interspersed with group lunches around the Chicago loop.
The IAI did not go down an exististential vortex of what an information architect is or is not (though it came close at times), which would have been a divisive and somewhat unimaginative exercise. Instead they linked together several strains of interesting ideas and left the participants to decide what it meant to them.
NPR reports German social networking site StudiVZ is being sued by FaceBook for User Interface (UI) infringement. This suit marks a shift in thinking towards the value of a user interface. In previous posts we’ve talked about UI as not being defensible. If this lawsuit is successful, we’ll be eating crow. We’re thrilled! Bring on that savory crow fricassee!
Significant real-time data immersion is not available for the consuming public yet. More data is available than ever before, but the traditional web browser experience has to grow up quite a bit before it begins to tax the cognitive capabilities of the average human with regards to information flow and memory. Sure, bad design is overloading us already, and yes you can watch real-time posts to DIGG, but imagine 7 to 12 real-time feeds simultaneously informing the user of the state of a given system. According to Micah Endsley [1] we can handle it, now we need to design it. One of the core principles of real-time information design is the establishing and maintaining of situational awareness.
Wickens defines situational awareness as “the continuous extraction of information about a dynamic system or environment, the integration of this information with previously acquired knowledge to form a coherent mental picture, and the use of that picture in directing further perception of, anticipation of, and attention to current events” [2][3]. The Air Force Research Laboratory similarly, and perhaps more simply, defines situational awareness as “how accurately a person perceives his current environment relative to the reality of that environment” [4].
Applying situational awareness to the goal of improving user interface, Davenport identifies three key areas of awareness encountered by the human participant: systems awareness, task awareness, and spatial awareness [5].
Systems Awareness
This is the human participant’s ability understand the state of his or her equipment. In the cockpit, for example, systems awareness is often abstract and usually requires aggregation of various gauge indications. Knowing that the engine is running hot means nothing by itself, but combined with other systems indicators, may indicate a potential problem.
Task Awareness
This is the human participant’s ability to accurately obtain information relating to tasks relevant to his or her goals. Understanding the current state of all tasks that are underway is critical as poor task awareness increases cognitive load, diminishing overall situational awareness. Good task awareness also enables the human participant to make informed decisions when making changes to the planned task.
Spatial Awareness
Spatial awareness can be broken into two sub-categories; Global and Local. Global spatial awareness is an understanding of the position of the human participant and his or her equipment in the world at that moment. It is the ability to accurately determine relative relationship and trajectory of objects within a global 360-degree sphere of influence and often pertains to the human participant’s relation to a target destination, anticipation of upcoming objects, and other spatial directional judgments. Local spatial awareness pertains to the attitude (vector and velocity) of the human participant’s equipment. This is particularly important when dealing with moving platforms such as aircraft. During observations of pilots using simulator software, it was repeatedly noted that during increased times of cognitive load the first errors made were related to spatial awareness. As the local spatial orientation of aircraft changes rapidly and frequently, the related local spatial SA tended to be the first awareness lost.
Interestingly, because situational awareness is the process of aggregating understanding it becomes evident that a failure at any time during aggregation can cause a series of failures much like a highway pile-up. The aviation community calls this cumulative effect of related incorrectly executed action loops a “Chain of Errors.” The National Transportation Safety Board has documented that a catastrophic failure, like a plane crash, is seldom caused by a single incorrectly executed action but instead by the cumulative effects of multiple incorrectly executed actions.
Situational awareness becomes even more critical in dimensional immersive experiences but we’ll save that discussion for another day.
References:
Micah R Endsley et all, International Center for Air Transportation, Department of Aeronautics and Astronautics, MIT, September 1998 Situation Awareness Information Requirements for Commercial Airline Pilots.
Amy L Alexander and Christopher D Wickens, University of Illinois, Aviation Human Factors Division, Savoy, Illinois 2003 The Effects of Spatial Awareness Biases on Maneuver Choice in a Cockpit Display of Traffic Information
Amy L Alexander and Christopher D Wickens, University of Illinois, Aviation Human Factors Division, Savoy, Illinois 2004 Measuring Traffic Awareness in an Integrated Hazard Display
Michael T. Brewer, Major, USAF, April 2000 An Investigation of the Non-Distributed Flight Reference (NDFR) Ownship Status Symbology
Clark E. Davenport, Air Force Institute of Technology, Wright-Patterson AFB OH, May 30, 1997 Displays for Spatial Situation Awareness: The Use of Spatial Enhancements to Improve Global and Local Awareness
Recent Comments