Standards and standardization are important factors in developing user interface management systems (UIMS). An outline of an ISO graphical standard is given in the context of controlling and improving computer graphics programming.
Software for managing user interfaces comes in a variety of guises. The purpose of such software is to assist application programmers to provide good quality human interfaces for their programs. A number of techniques are needed to fulfil that purpose. Interaction Technique Libraries provide a software toolkit of techniques that an application programmer may employ, which may include:
A user interface management system (UIMS) [1, 2, 3, 4] provides some or most of these techniques, but integrates them into a single system. In addition, a UIMS provides some method of controlling the sequence of user operations. It is common for much of the dialogue structure to be specified outside the application program, since it is likely that the dialogue designer may wish to experiment with the dialogue structure without needing to recompile the whole application program. A project in which the author was involved [5] had some of the characteristics of a UIMS in that it separated dialogue control from the application modules. By present day standards, the facilities were not rich: it provided menu-building, error/prompt tools and a fixed screen layout.
In this paper, we will want to refer to software that could be in either category, UIMS or Interaction Technique Library, and will simply refer to such software as user interface software.
It is important that such software can support good quality user interfaces. For example, there is much interest in interfaces which does not compel the user to follow a sequence of steps predetermined by the dialogue designer. Instead the user is presented with a series of software-supported controls that can be directly manipulated to achieve the desired result [6], [7]. Until recently [2], there have not been UIMS to support direct manipulation but toolkits are being produced [8].
Although standards for user interface software and user interfaces as such are not yet being developed, there are standards for one of the major components which is the graphics software. One of the reasons for using a standard computer graphics programming interface is to encourage the interchange of application modules and complete programs using computer graphics.
However, using a standard in general can carry the risk of excluding modern techniques. We would like to gain the benefits of standardization and still be able to produce good user interfaces.
The Graphical Kernel System (GKS) became an ISO standard for computer graphics programming in 1985 [9] and language bindings are being standardized language by language. [10]. Since there are introductions to GKS elsewhere [11] and to the bindings elsewhere [12], this section will merely highlight some main points that are relevant to designing user interfaces.
GKS controls workstations which are abstractions of devices. One useful example of this abstraction is on a single user system with a window manager. With a good window manager, each window can be controlled by an application program, which need not be affected by where its windows are placed or by the obscuring of its windows by others. It is convenient to map a GKS workstation to a window, so that an application program may control several graphical windows by opening several GKS workstations. This illustrates how a concept in the standard has been extended to cover a use which was not envisaged when the first drafts were being prepared.
Apart from simply adding to a picture by further graphical output, picture changes can also be achieved by manipulating GKS segments. The application programmer anticipates changes by grouping graphical output in segments, enclosing them by calls to create segment and close segment. The program then has a variety of techniques with which to make the changes. For example a segment can be made invisible and later visible again or its geometry may have a linear transformation applied to it. It is possible for a segment to be deleted, after which its name can be re-used. To give the user control over these operations graphical input can be used to identify a segment (picking it).
On some devices, these changes are straightforward. Such devices generally include a display list mechanism. Other devices cannot manage the changes at all and the entire picture must be regenerated to achieve the correct up to date image. For such devices, GKS allows the application program to delay the regeneration i.e. implicit regeneration is suppressed. Many devices fall between the two extremes, and bit map devices not supported by display lists are a good example. In principle, it is possible for GKS implementations to provide fast support for segment changes on such devices using built-in bitblt operations [13] but at present it is rare for this to be provided.
An important goal of the graphical input functions in GKS is to encourage portability of application programs. This is achieved through the concept of the logical input device. An example of a type of logical input device is locator: the user's input returns a point in world coordinates and a number identifying the viewpoint in which the input was located. There are six different types of logical input devices altogether (including pick and choice which are used later in the article) and each type can return its own type of value. Normally if input is available at all, it may be assumed that a program has access to at least one instance of every type of logical input device. There are two exceptions to this. The standard allows the implementation to spread those logical input devices across more than one workstation, although this is not common. Also a workstation that only accepts graphical input is allowed.
A GKS input device may be in any of three operating modes which determine the nature of the interaction with the user.
When a device is in request mode and a request is underway, the interaction is under the complete control of the logical input device in GKS. It is not possible for the program to intervene until the user indicates that the interaction is complete or is to be interrupted before completion.
When the device is in sample mode, the application program can read the current input value, which is delivered without waiting for the user to trigger the input in any way.
When the device is in event mode, event reports triggered by the user are queued, along with reports from any other device in event mode. The effect of having several devices in event mode is to give the user the choice of which devices to use at any time.
With sample and event modes, the application program need not relinquish control. However sample and event modes often imply several threads of control, which listen to the physical device and generate the input echoes if that has to be done in software. For this reason, sample and event appear in a higher level of GKS (level c) than request (level b).
By contrast to the input functions covered so far, the process of initialising an input device is device dependent. The control information thus supplied is used in subsequent interactions. Among the parameters are a prompt and echo type, which is an integer, and a data record which has a structure that depends on the device. The former selects a prompt and echo type from among those implemented and sets it. For example, GKS defines the following prompt and echo type value for locator:
<0 implementation dependent 1 must exist: prompt and echo is device dependent 2 echo using cross hair 3 echo using tracking cross 4 echo using rubber band line 5 echo using rubber rectangle 6 echo by displaying digits ≤7 values are subject to registration
Lists in GKS such as the one above may be extended by ISO through a Graphical Registration Authority. Registration takes place after the standard is agreed and extends its meaning. For example, an implementation of a particular locator device may provide prompt and echo types 1, 2, 3, 24 and so initialise locator for that device may select any of those.
The data record may be used to convey further control information to the device. For example, an implementation may allow a bit map to be sent thus defining a locator cursor shape. Where a choice device is represented by a menu (prompt and echo type 3 for choice input), an implementation accepts a list of strings to define that menu.
To allow the application programmer the advantages of a standard graphics system and to allow the user the benefits of a good user interface, how should GKS and the user interface software relate to each other?
The model that has GKS calling the user interface software, as part of the GKS implementation (see Figure 1), has several disadvantages.
The requirements for user interface software are still the subject of experimentation, whereas GKS provides a minimal set of building blocks in a standard interface. It is not reasonable to expect that the standard interface has encapsulated all present and future requirements. This model also precludes the user interface software from being the main program which calls application modules (termed an external control UIMS in Thomas [4]).
A model in which both the application program and the user interface software call GKS is considered in Figure 2. This allows both the application code and the user interface software to be portable. As can be seen from the diagram, the user interface software can be either master or slave. For the remainder of this paper this model will be assumed.
There are a number of implications which need to be explored further.
If only request input is available, then typically any client of GKS has very little control. This is illustrated by showing how an item may be selected from a menu, using a language devised solely for the purpose of the examples, but bearing some resemblance to PASCAL and ADA.
When using request, all we can do is select which logical input device to use from among those available (we select choice device d on workstation w). The example sets up the menu (prompt and echo type 3 in initialise), waits for the input to complete and collects the input value (in item_ selected):
INITIALISE_CHOICE (w, d, 1, 3, echo_area, ('shift', 'rotate', 'scale', 'quit'));
REQUEST_CHOICE (w, d, status, item selected);
In that example there is very little for the program to do, because GKS and not the program has control over the details of the interaction. The following example uses sample mode. It repeatedly uses pick to select an item box of a menu. The box is highlighted by thickening its border. The example assumes that the device has a bitmap model and removes the thickening by simply rewriting in the background colour. By looking at a choice device on each cycle of the loop, the GKS client can find out whether the interaction should end or be repeated.
Two procedures which assist with the output are introduced but not elaborated:
# d_menu_item sets up the display for one menu item
# and saves it in a GKS segment using
# CREATE SEGMENT and CLOSE SEGMENT
procedure d _menu_item
(int_ colour, # colour of interior
write colour, # colour of border and text
box, # record containing text and
# coordinates of box
seg) # segment number to be used
# thicken box provides the output necessary to
# thicken a particular box
procedure thicken box
(write colour, # colour to be used
box) # box record
Now the procedures can be used:
# Display the menu items (the number of them is n_items)
SET_TEXT_ALIGNMENT (centre, half);
for item: = 1 to n items do
# bk and fg are the background and foreground colours
d_ menu_item (bk, fg, box [item], item);
pickstatus: = none;
already_reversed:= 0;
# Make the selection. Remember to undo the thickening;
# already_reversed holds the number that was last thickened
loop
if pickstatus = ok then
thicken box (fg, box (item));
already_reversed:= item;
end;
SAMPLE PICK (pickstatus, item, pickid);
if already_reversed ≠ 0 then
thicken_box (bk, box (already_reversed));
already_reversed:= 0;
end;
SAMPLE_CHOICE (choicestat, choice);
until choicestat = ok and choice= done;
item selected:= item;
Improvements to the example would detect whether the box number was unchanged and that there was no need to unthicken and thicken the same box. However even as it stands, the example illustrates that a high degree of control over the details of the interaction is possible when sample input is used (and although not illustrated, this also applies to event input).
In the example above, one facet of the interaction that could not be controlled is the composition of the pick and choice devices used. The fixed composition of a logical input device in GKS includes the physical device used and the nature of any output given to the user in the course of the interaction. This is predefined.
The advantage of the logical input device method is that a GKS client can use a logical input device on different installations with different sets of hardware. A locator might be implemented by a tablet or by typing coordinates on a keyboard. A criticism of the logical input device concept is that poor interfaces result from using unsuitable physical input devices. That criticism as it stands actually misses the mark, as it is difficult for the software to repair the user interface if the physical input devices are wrong. Using a logical input device at least allows the application program to be tested in an alternative environment. There are other criticisms of the concept however that repay some consideration.
First, if the physical input device is perfectly suitable, can the client of the logical input device make use of it? The implementation might prevent a perfectly reasonable use of the device. For example a set of three buttons might be combined in a single choice device which returns a number indicating the button pressed. This does not give the client full control over the button status. One solution is to implement three choice devices each with two values. It is necessary for implementations to include among the logical input devices a selection that maps as closely as possible to the physical ones.
Second, from those available, can the client of GKS find the logical input devices and prompt and echo types which provide the most suitable user interface? Remember that several devices of the same class might exist. In general, the composition of a logical input device cannot be changed and is not available for inquiring. One solution is to take advantage of the registration of additional prompt and echo types. Using the ISO registration process, it will be possible to give new standardized meanings to additional prompt and echo types and their associated data records. Once this is done, it will be possible for GKS clients to find out if these additional prompt and echo types are present and to adapt their behaviour.
It is worth emphasizing that the data records in the initialise functions allow control information to be sent control information could be is a subject for useful investigation. This data is usually associated with parameters such as a cursor bit map or a list of strings for a menu. However the data could be a string representing a program, to be interpreted when the device executes. This program could be used to configure the behaviour of the logical input device. The definition of the News Window interface from Sun Microsystems allows Postscript programs (constructed by the application) to be obeyed in the window system and this is a possible vehicle for configuring logical input devices.
A user interface which supports direct manipulation implies the use of many objects (or spaces) on the display, ready to receive user input and react to it in some way. This multiplicity needs the parallelism of GKS's sample and event input. That parallelism might extend to the application program, as a possible method of implementing direct manipulation is to attach separate processes to each object being manipulated. Care should be taken since GKS operations would have to be atomic in order to preserve the current GKS state. In fact, whole sections of code in the application would have to be atomic to prevent mutual interference. For example, the following sequence deletes a segment and recreates it with different output:
DELETE_SEGMENT (OBJ1) CREATE_SEGMENT (OBJ1) ... output functions CLOSE SEGMENT
This sequence should not be interrupted, because if it were, it could not be predicted whether the interrupting process was in an open GKS segment or not. It would be necessary for application processes to declare locks around sensitive sequences of graphics code, just as they would need to do around other (non-graphical) sensitive resources.
Using GKS with user interface software has been discussed. To allow a user interface designer complete control, a GKS implementation needs to provide closer control over the physical input devices than is guaranteed in the standard. Implementations need to provide logical input devices mapping closely to physical devices, as well as those needed for portability.
Level c input in GKS (i.e. sample and event mode) gives the client fine control over the interaction, unlike level b input (which allows only request mode).
Passing an interpretive program to a logical input device using the initialise data record may allow the interaction to be controlled by an application procedure in the logical device. This may be workable even with request input.
To date much applications software on bit-mapped graphics workstations has tended to bypass GKS, but the above discussion has shown that the use of GKS is compatible with high quality user interfaces, and brings with it properties of portability and generality of potential benefit to interface designers.
[1] Foley, J D User interface tools Siggraph '85 Tutorial Notes for 'Advanced topics - human factors in computer graphic systems' (1985)
[2] Myers, B A and Buxton, W Creating highly interactive and graphical user interfaces by demonstration Computer Graphics, Vo! 20 No 4 (1986)
[3] Pfaff, G R User Interface Management Systems Springer-Verlag, FRG (1985)
[4] Thomas, J J and Hamlin, G Graphical input interaction techniques: workshop summary Computer Graphics Vol 17 No 1 (1983)
[5] Gallop, J R and Shaw, W D PIGS: A command system for interactive graphics Proc. Eurocomp (1975)
[6] Shneiderman, B Direct manipulation: a step beyond programming languages IEEE Computer Vol 16 No 8 (August 1983)
[7] Spence, R Human-computer interaction user interface design Eurographics '86 tutorial notes (1986) Springer-Verlag, FRG
[8] Williams, A S An architecture for user interface R & D IEEE Computer Graphics and Applications (July 1986)
[9] Computer graphics - Graphical Kernel System Functional Specification ISO 7942
[10] Computer graphics - Graphical Kernel System language bindings Part 1 FORTRAN, Part 2 PASCAL, Part 3 ADA, Part 4 C (still drafts) ISO 8651
[11] Hopgood, F R A, Duce, D A, Gallop, J R and Sutcliffe, D C Introduction to the Graphical Kernel System (GKS) 2nd ed. Academic Press, London, UK (1986)
[12] Sparks, M R and Gallop, J R Language bindings for computer graphics standards IEEE Computer Graphics and Application Vol 6 No 8 (1986)
[13] Slater, M Segments on Bit-mapped Graphics Displays Software Practice and Experience Vol 16 No 11 (1986)