Multiple User InterfacesCross-Platform Applications and Context-Aware Interfaces phần 9 pot

310 SIMON LOCK AND HARRY BRIGNULL Finally, the SMS gloss used in the creation of new messages (which cannot be illus- trated visually) is basically an SMS parser which checks incoming SMS, looks for new message requests, creates new message bubbles and initiates distribution as required. 14.6. IMPLEMENTATION OF INFRASTRUCTURE AND DEVELOPMENT FRAMEWORK The infrastructure described in this chapter has been designed to support the construction of distributed, heterogeneous multi-device, multi-user, interactive applications. A proto- type version of the infrastructure is currently available and has to this date been used to construct and test a number of different multi-user interactive applications. The infrastructure is built on top of Jini and RMI technologies from Sun Microsystems. Jini provides all of the basic facilities for the description, registration and discovery of device services. This is based around one or more centralised lookup services which store details of all currently available services. A lookup service is first identified by multicast communication; once identified, new services can be added to the registry or it can be queried to identify a suitable device for a particular purpose. The infrastructure also provides a programmer’s API and set of common device services to aid in application development. The infrastructure takes on many of the important and common tasks required to build such applications. In addition to this, the infrastructure gives programmers access to a selection of varied and diverse input and output devices through a high-level service- consumer architecture. Services execute on hardware to which devices are directly or indirectly (i.e. through a proxy) attached. Code level adapters called ‘Consumers’ give programmers simple and easy access to the features of remote services. In this way heterogeneous, remote devices are presented to the programmer as a consistent set of local consumer classes. RMI is used by the bubbles infrastructure to achieve communication between the distributed components of an application written with bubbles. Communication within bubbles involves control and data synchronisation, the distribution of user interface events and the passing of messages for various internal housekeeping activities. Such commu- nication is achieved with the aid of RMI daemons running on the various devices which handle the actual low level transmission of messages across the available interconnec- tion mechanism. The Java programming language is used to bind together and build additional fea- tures and facilities on top of these low level mechanisms in order to create a complete implementation of the bubbles run-time infrastructure. A developer-level API has also been provided for the Java programming language to allow applications to be built which utilise the bubbles infrastructure. This developer-level framework will be discussed in the following section. A main aim of the infrastructure is to encompass and support the integration of a wide variety of heterogeneous devices. Depending on the resources and capabilities of a particular device, it may be integrated into the infrastructure in one of two ways. Devices with Jini and Remote Method Invocation (RMI) support can be connected directly to the RUN-TIME INFRASTRUCTURE FOR DISTRIBUTED, MULTI-USER/DEVICE INTERACTIVE APPLICATIONS 311 Dynamo infrastructure. Devices with no Jini/RMI support must be connected indirectly, via a special device proxy. Where relevant in this chapter, descriptions of infrastructure operation will appear for both of these types of device. 14.7. OPERATION OF THE INFRASTRUCTURE The aim of this section is to provide a more concrete description of the infrastructure by describing the various features of dynamic operation. This should shed light on how many of the features and functions of the infrastructure are achieved in practice. 14.7.1. DYNAMIC DEVICE SERVICE REGISTRATION The infrastructure utilises a Jini lookup service as a centralised registry for all enabled and currently available device services. In many multi-device interaction scenarios, there is a constant ebb and flow of devices as people move around and equipment is switched on and off. The use of the Jini lookup service assists the infrastructure in supporting the real-time registration, reconfiguration and removal of device services. All Jini/RMI enabled devices integrated into the infrastructure are represented as one or more Jini services. When a device is switched on, plugged into the network or comes within range, it notifies the registry to indicate that it is able and willing to handle requests from consumers. Non Jini/RMI enabled devices must use the Java Surrogate Architecture (SA) [Sun Microsystems 2001] in order to participate in interactive applications built using the Dynamo infrastructure. When employing the surrogate architecture, a Jini/RMI enabled surrogate host server is used as a proxy for a non Jini/RMI device. The host server bridges the gap between the main Jini/RMI protocols used by the infrastructure and the proprietary communication protocols of non Jini/RMI device services. As with Jini/RMI enabled devices, the Jini registry is still used to maintain a list of all the currently enabled and available devices, however the entries in the registry refer to the proxies rather than the services themselves. When a device is switched on, plugged into the network or comes within range, it uploads its proxy onto the surrogate host. This proxy then registers with the Jini registry to indicate that it is able to handle the requests of consumers. 14.7.2. DYNAMIC DEVICE SERVICE SELECTION When an application needs to engage in a particular form of user interaction, a device service which can provide suitable interaction is first identified by querying the Jini lookup. The infrastructure allows devices to be dynamically identified and utilised by applications on the fly. This dynamic linking makes it possible for applications to maximise interaction quality by having access to all currently available devices. To simplify the identification and selection of devices, no application interfaces directly with the Jini registry. Instead, all applications interact with a utility class, known as the ‘Arbitrator’. An application developer specifies what type of interaction they require at a particular point in an application, using query templates to specify the desired phys- ical properties of a suitable device. At run-time, the Arbitrator will then provide a set 312 SIMON LOCK AND HARRY BRIGNULL of currently available services which are most appropriate to the needs of the applica- tion. On the basis of this information, a linkage between application and device can be made. The infrastructure then supports communication between the main application and chosen devices. The Arbitrator abstracts over the Jini registry, but also provides additional, more advanced functionality for device selection such as: • Selecting given combinations of device properties • Selecting given combinations of interfaces and users • Enforcing access privileges on user interfaces • Handling private viewing and public announcement concerns • Load balancing between interaction devices • Minimum resource arbitration combined with upgrading and degrading of service To aid in the selection of device services, each one is associated with a number of description parameters which describe their specific abilities and constraints. When the Jini registry is queried, these parameters can be used to identify services which are suitable for use by the application. The set of parameters which may be used in describing device services is extendable and can be added to by the developer as and when new parameters are required. A few of the basic device description parameters for a number of common devices are shown in Table 14.4. 14.7.3. APPLICATION SERVICE LINKAGE Once a device service has been selected, a device consumer is then created and linked to it in order to allow an application access to the features of the service. With Jini/RMI enabled devices, communication between the service and consumer is achieved using the RMI features of Java. To achieve application-to-device communication, a consumer must first interface with the RMI code on the application side. RMI then uses the standard TCP/IP network protocols to communicate with RMI on the remote (device side) machine. The device side RMI then interfaces directly with the device service code to invoke its services. This architecture is illustrated in Figure 14.6. Devices without support for RMI, and indeed devices with no support for Java, may still be integrated into the infrastructure. Communication with such a device’s services must be performed using proprietary mechanisms since RMI-based communication is Table 14.4. Device description parameters. All devices Graphic display Pointing device Audio output mobile [bool] size [int*int] button count [int] bit rate [int] owner [string] resolution [int] accurate [bool] encoding [string] authorised user [string] employment level [int] fast [bool] stereo [bool] privacy type [int] number of users [int] usage range [int] RUN-TIME INFRASTRUCTURE FOR DISTRIBUTED, MULTI-USER/DEVICE INTERACTIVE APPLICATIONS 313 Application side Service consumer RMI TCP/IP Device side Device service RMI TCP/IP Figure 14.6. Device service with RMI support. Application side Service consumer RMI TCP/IP Surrogate host Device surrogate RMI TCP/IP TCP/IP Device side Device service TCP/IP Monitor Figure 14.7. Device service without RMI support. not possible. The architecture employed in the connection of non-RMI devices into the infrastructure is shown in Figure 14.7. The figure shows that in order to make use of a particular service, the application side cannot contact the device side directly since no RMI support is available. Instead, a consumer interfaces with RMI on the application side as normal. However, this RMI code must connect to a surrogate ‘proxy’ rather than directly to the device side. This is still done using TCP/IP network protocols between the two RMI elements. All RMI requests received by the proxy must then be converted into an appropriate representation that can then be forwarded on via a raw TCP/IP connection to the service side to be handled by the relevant device services. Service monitoring by the surrogate must take place to ensure that appropriate action takes place when the device is switched off, breaks down, is disconnected, or is otherwise unavailable. 14.7.4. BUBBLE SYNCHRONISATION A particular bubble may be distributed to more than one device at the same time, resulting in the simultaneous presentation of multiple instances of the bubble in various locations. Different instances of the same bubble presented on different devices are automatically synchronised by the infrastructure. Thus, any changes which take place in the data element 314 SIMON LOCK AND HARRY BRIGNULL Bubble Synchronised dataItem DataChange consumer Data registry Bubble Synchronised dataItem DataChange consumer User interaction DataChangeService Figure 14.8. Data Synchronisation in Dynamo. of one instance of a bubble are propagated to all other instances to maintain consistency. This data synchronisation mechanism makes use of a number of special “Synchronised- DataItem” classes (e.g. SynchronisedString, SynchronisedVector etc). These classes are adaptations of existing standard data types, but with additional data communication and integration facilities. The mechanism used to propagate data change is illustrated in Figure 14.8. The diagram shows that when one copy of a data item is altered (via user interaction with a bubble for example), a DataChangeConsumer is first notified. This consumer passes on the notification to a centralised DataChangeService which maintains a registry of all copies of all synchronised data items. When the DataChangeService receives a notification, it fires out an update notification to all copies of the originally altered data item. This update is received by the DataChangeConsumer which is local to each copy of the data item. Once the data items have updated their internal values to reflect the propagated change, any bubbles which present the values held in the data item are also updated. Distributed data synchronisation achieved in the manner described above has a funda- mental empathy with the architecture of the infrastructure which is parallel and distributed in nature. Distributed synchronisation also provides some resilience to temporary loss and regaining of connection, common in the type of applications targeted by the infrastruc- ture. In matters relating to the synchronisation of data values, the described infrastructure implements an open policy based mechanism. This allows developers to produce new synchronisation policies and ‘plug’ them into the infrastructure, thus allowing any syn- chronisation algorithm to be used. 14.8. INFRASTRUCTURE UTILISATION We have thus far described the infrastructure in some detail, we now however extend this discussion by demonstrating the utilisation of the infrastructure for the example RUN-TIME INFRASTRUCTURE FOR DISTRIBUTED, MULTI-USER/DEVICE INTERACTIVE APPLICATIONS 315 User Handheld Kiosk PC Cell phone SMS gateway Graphical gloss SMS gloss Creator Interacts with Realised on Realised on Realised on Submits to Interacts with Interacts with Has gloss Has gloss Figure 14.9. Message creation sub-architecture. MUI scenario. The scenario has two main use cases, the creation of a new message and the viewing of an existing message. Figures 14.9 and 14.10 show the architecture of the system, split between these two use cases. This partitions the system logically and makes the architecture simpler to understand compared with a single, more complex representation. Due to the dynamic nature of device registration and selection, the archi- tecture diagrams encompass a number of different variations for achieving the desired functionality. Entities from the diagram are as follows: • User – the application end user who interacts with the system. • Handheld – a handheld interaction device (e.g. a palm pilot, an IPAQ, etc). This device has display, pointer and character entry services. • Kiosk PC – a desktop PC provided for use by the general public. This device has display, pointer and character entry services. • Cell phone – personal mobile phone with standard audio and SMS capabilities. • SMS gateway – an existing subsystem that permits the transmission of SMS messages. • Creator – a bubble which encapsulates the interaction required to create a new message. • Graphical gloss – a presentation of the creator bubble suitable for realisation with graphical display, pointer and character entry services. • SMS gloss – a presentation of the creator bubble suitable for realisation via SMS on the SMS gateway service. 316 SIMON LOCK AND HARRY BRIGNULL Message board Handheld Kiosk PC Cell phone Voice synthesiser User Folded gloss Graphical gloss Audible gloss Message Realised on Realised on Has gloss Has gloss Has gloss Realised on Realised on Connects to Interacts with Interacts with Interacts with Interacts with Figure 14.10. Message viewing sub-architecture. Entities from the diagram are as follows: • User – the application end user who interacts with the system. • Message board – a large scale, public screen (plasma, projected etc), providing both display and pointer services. • Handheld – a handheld interaction device (e.g. a palm pilot, an IPAQ, etc). This device has display, pointer and character entry services. • Kiosk PC – a desktop PC provided for use by the general public. This device has display, pointer and character entry services. • Cell phone – personal mobile phone with standard audio and SMS capabilities. • Voice synthesiser – a gateway that transforms text into speech for rendering on a mobile phone. • Message – a bubble which encapsulates a single message. • Graphical gloss – a presentation of the message bubble suitable for realisation with graphical display, pointer and character entry services. • Audible gloss – a presentation of the message bubble suitable for realisation on a mobile phone via a voice synthesiser gateway service. • Folded gloss – a presentation of the message bubble suitable for realisation with a large scale, public display service. 14.9. APPLICATION USAGE SCENARIOS We now demonstrate the operation of the message board system and infrastructure using a number of possible usage scenarios. For the sake of simplicity, no exceptions to the normal RUN-TIME INFRASTRUCTURE FOR DISTRIBUTED, MULTI-USER/DEVICE INTERACTIVE APPLICATIONS 317 Start Register Detected End Received Notified Realised Entered Register handheld Present gloss Registration detection Send creator bubble Send message bubble Data change notification Enter data Figure 14.11. User adds message to board using a handheld. thread of interaction with the system are presented here. This includes such occurrences as the inability to find a suitable available service, the execution of the system when a user attempts a particular action without inappropriate access privileges, the operation of the system if a user cancels the creation of a message part way through, and so on. Stages from the process model shown in Figure 14.11 are as follows: • Register handheld – the handheld device is switched on (or device-side components booted) causing the device’s services to initialise and be registered with the Jini lookup. • Registration detection – the application detects that a new device has been added and links it in using a new consumer. • Send creator bubble – the application sends the message creation bubble to be realised to the handheld so that the user may create a new message. • Enter data – the user enters data into the message creator bubble using the graphical gloss which is presented on the handheld’s display. • Data change notification – when the message has been created (and accepted), a data change notification is received by the application. • Send message bubble – the application then sends a completed message bubble to the large scale public display (we assume the public display has previously been identified and linked into the application). • Present gloss – the public display presents the bubble graphically, using the ‘folded’ gloss to maintain privacy. Stages from the process model shown in Figure 14.12 are as follows: • Enter SMS text – the user enters the text of an SMS message on their phone. • Send SMS – once happy with the message, the user sends the SMS to the appropriate service centre. • Parse message – on receipt of the message, its contents are parsed to identify dis- tinct elements. • Insert data – the parsed data items are inserted into a new message. • Data change notification – when the message has been filled, a data change notification is received by the application. • Send message bubble – the application then sends a completed message bubble to the large scale public display. 318 SIMON LOCK AND HARRY BRIGNULL Start Created Received End Received Notified Parsed Inserted Enter SMS text Present gloss Send SMS Parse message Send message bubble Data change notification Insert data Figure 14.12. User adds message to board using a cell phone. Start Logged in Selected End Received Verified Validated Found Login Present gloss Select message Validate action Send message bubble Verify access Find realiser Figure 14.13. User views public message from board. • Present gloss – the public display presents the bubble graphically, using the ‘folded’ gloss to maintain privacy. Stages from the process model shown in Figure 14.13 are as follows: • Login – the user logs into the public display with their username. • Select message – the user then selects a folded message from the display. • Validate action – the system checks to ensure that the action is valid (i.e. the user has access to open the folded message). • Find realiser – since the message selected is a public one, a suitable public display is identified for the presentation of the message. • Verify access – the system checks to ensure that the user has access to the cho- sen display. • Send message bubble – the application then sends the message bubble to the pub- lic display. • Present gloss – the public display presents the bubble using the unfolded gloss which is most suitable. Stages from the process model shown in Figure 14.14 are as follows: • Register handheld – the handheld device is switched on (or device-side components booted) causing the device’s services to initialise and be registered with the Jini lookup. RUN-TIME INFRASTRUCTURE FOR DISTRIBUTED, MULTI-USER/DEVICE INTERACTIVE APPLICATIONS 319 Start Registered Detected End Received Verified Mapped Selected Found Validated Register handheld Present gloss Registration detection Map to board pointer Select message Send message bubble Verify access Find realiser Validate action Figure 14.14. User views private message from board using handheld. • Registration detection – the application detects that a new device has been added and links it in using a new consumer. • Map to board pointer – the pointing service provided by the handheld is mapped to a pointer on the public display, allowing the user to interact with the public display using their own stylus. • Select message – the user then selects a folded message from the display using the stylus-based pointing mechanism. • Validate action – the system checks to ensure that the action is valid (i.e. the user has access to open the folded message). • Find realiser – since the message selected is a private one, a suitable private display is identified for the presentation of the message (this may well be the display service on the handheld, if it is suitable). • Verify access – the system checks to ensure that the user has access to the cho- sen display. • Send message bubble – the application then sends the message bubble to the pri- vate display. • Present gloss – the private display presents the bubble using the unfolded gloss which is most suitable. Stages from the process model shown in Figure 14.15 are as follows: • Register phone – the user dials a pre-defined number to register the phone and invoke the message checking service. • Registration detection – the application detects that a new device has been added and links it in using a new consumer. • Check for messages – the current set of valid messages is queried to find if there are any messages for the user. • Find gateway – if any messages for the user are present, a voice synthesis gateway is identified. • Send message bubble – the application then sends the message bubble to the gateway for realisation. [...]... December 2000 Wang, W., Dorohonceanu, B., and Marsic, I ( 199 9) Design of the DISCIPLE synchronous collaboration framework Proceedings of the 3rd International Conference on Internet, Multimedia Systems and Applications, October 199 9, 316–24 Part VI Evaluation and Social Impacts 15 Assessing Usability across Multiple User Interfaces ¨ Gustav Oquist1 , Mikael Goldstein2 and Didier Chincholle2 1 2 Uppsala University,... ensure that suitable interfaces are provided for presentation on different devices, in different usage contexts and for different users REFERENCES Abrams, M., Phanouriou, C., Batongbacal, A.L., and Williams, S.M ( 199 9) UIML: an Applianceindependent XML User Interface Language Computer Networks, May 199 9, 31(11–16), 1 695 –1708 Brignull, H (2000) An Evaluation of the Effect of Screen Size on Users’ Execution... November 199 6, 30–38 Streitz, N.A., Geißler, J., Holmer, T., et al ( 199 9) i-LAND: An interactive landscape for creativity and innovation Proceedings of CHI 99 Conference, May 199 9, 120–27 Sun Microsystems (2001) Jini Technology Surrogate Architecture Overview http://developer.jini org/exchange/projects/surrogate/overview.pdf, May 2001 Tandler, P (2000) Architecture of BEACH: The Software Infrastructure... April 2000, 323–31 Rekimoto, J ( 199 8) A Multiple Device Approach for Supporting Whiteboard-based Interactions Proceedings of CHI 98 Conference, April 199 8, 344–51 Schuckmann, C., Kircher, L., Schummer, J., and Haake, J.M ( 199 6) Designing object-oriented synchronous groupware with COAST Proceedings of CSCW 96 : ACM Conference on ComputerSupported Cooperative Work , November 199 6, 30–38 Streitz, N.A., Geißler,... groupware applications Proceedings of the International Conference on Software Engineering, May 2001, 475–84 Masui, T and Siio, I (2000) Real-World Graphical User Interfaces Proceedings of the International Symposium on Handheld and Ubiquitous Computing (HUC2000), September 2000, 72–84 Myers, B.A., Stiel, H., and Gargiulo, R ( 199 8) Collaboration Using Multiple PDAs Connected to a PC Proceedings of CSCW 98 :... Buxton, W., and Smith, K.C ( 199 7) Reactive Environments: Throwing away your keyboard and mouse Communications of the ACM, 40(7), 65–73 Dix, A., Finlay, J., Abowd, G., and Beale, R ( 199 8) Human Computer Interaction, 2nd edition 398 9 Prentice Hall Fox, A., Johanson, B., Hanrahan, P., and Winograd, T (2000) Integrating Information Appliances into an Interactive Workspace IEEE Computer Graphics & Applications, ... mobile interface solutions in a MUI We will end with a brief discussion about our findings and some concluding remarks Multiple User Interfaces Edited by A Seffah and H Javahery  2004 John Wiley & Sons, Ltd ISBN: 0-470-85444-8 ¨ GUSTAV OQUIST, MIKAEL GOLDSTEIN AND DIDIER CHINCHOLLE 328 15.2 MULTIPLE USER INTERFACES: MULTIPLE CONTEXTS OF USE The MUI concept enables unified access to information across a... and output on mobile devices Traditional Input Output Dynamic Ubiquitous Senseboard SmartView Tegic T9 Adaptive RSVP FJG Keypad TactGuide ASSESSING USABILITY ACROSS MULTIPLE USER INTERFACES 337 For input, we exemplify the traditional paradigm with the Senseboard text entry interface [Goldstein et al 199 9; Alsi¨ and Goldstein 2000], which is based on sensors picking o up finger depressions when the user. .. keyboard Senseboard consists of two pads positioned in the user s palms Each pad captures the motion of the fingers and the hand, thus enabling keyboard typing without a keyboard The design is based on the fact that users who have acquired the skill of touch-typing can accomplish this without the keyboard being present [Goldstein et al 199 8, 199 9; Alsi¨ and Goldstein 2000] The first o proof of concept prototype... modalities as well Mobile phones already combine sight, hearing and touch in the way they communicate, and their users are therefore quite 345 ASSESSING USABILITY ACROSS MULTIPLE USER INTERFACES accustomed to devices that blink, beep and vibrate Since mobile devices are handheld, they may also make use of the hand’s sensitivity to touch, and thus use the tactile sense as a channel of communication Currently, . N.A., Geißler, J., Holmer, T., et al. ( 199 9) i-LAND: An interactive landscape for creativity and innovation. Proceedings of CHI 99 Conference, May 199 9, 120–27. Sun Microsystems (2001) Jini Technology. different users. REFERENCES Abrams, M., Phanouriou, C., Batongbacal, A.L., and Williams, S.M. ( 199 9) UIML: an Appliance- independent XML User Interface Language. Computer Networks, May 199 9, 31(11–16), 1 695 –1708. Brignull,. Applications, October 199 9, 316–24. Part VI Evaluation and Social Impacts 15 Assessing Usability across Multiple User Interfaces Gustav ¨ Oquist 1 , Mikael Goldstein 2 and Didier Chincholle 2 1 Uppsala