Currently accepted models of interactive graphics software may be exemplified by those underlying recent standards proposals. That of the ACM-SIGGRAPH Core [GSPC 1977] consists essentially of two bi-directional channels (Fig. 1). One links the applications program to its output devices via a view transformation and a segmented, transformed display file. The other links the program to its input devices. There is also a connection between these two channels at the device end to provide low-level echoing, but the form of this echoing is determined by the system implementor. This link between input and output is only under applications control in so far as it may be turned on or off.
The model underlying GKS [DIN 1978] is more complex (Fig. 2). It also consists of two bi-directional channels, but the output channel contains not merely a segmented, transformed display file, but also another segmented file, the GKS file. Segments can be read back from this file, their primitives transformed and added to another segment. Because GKS is only a 2D system, it places in the input channel the inverse of the view transformation, and thereby provides the application with a more consistent interface, since both input and output are referred to the same coordinate system. The Core's attempt to maintain complete compatibility between 2D and 3D prevents this, since the inverse of the projection part of the 3D view transform is one-many (Problems are also caused by overlapping viewports.). GKS' treatment of the link between input and output is even more cavalier than the Core's:-
"Low level prompting and echoing are device and implementation dependent. High level prompting and echoing should be modelled by the applications program."
The scrutiny that these models have undergone as part of the standardization process has revealed deficiencies in two main areas. The models fail to provide an adequate basis for the formalizations which are desirable, both to improve the specification of the standards, and to permit certification of implementations. They also appear either inadequate or inelegant when applied to interactive as opposed to passive graphics applications.
The major impediment to a complete formal description of a device-independent graphics system is the difficulty of defining what is meant by the same picture drawn on different devices. However, there are two other difficulties peculiar to the models outlined above, namely:-
The problems of these models for interaction may be summarized as :-
The new model (Fig. 3) consists of two uni-directional channels. As the model is intended to apply to 2D graphics, the output channel includes the view transformation, and the input channel includes its inverse. There are thus only two kinds of function available to the applications program; output functions effecting a change in the state of the graphics system, and inquiry functions describing the state of the graphics system (cf. Eckert's O and V functions [Eckert 1979]).
There are two stores of state information in the model. The first stores the current view transform and is accessed by the view transform and its inverse, from the output and input channels respectively. It is possible for output functions to change the transform definition, and to cause replies describing the current transform to be transmitted on the input channel. The second store contains the definitions of a set of macros, and is accessed by a macro processor in each channel. This store corresponds, as far as the output channel is concerned, to the segmented, transformed display files of conventional models. However, in this model it is involved in both input and output operations. It not merely describes the link between them which is implicit in other models, but also describes a process whereby the link may be fully under applications program control. It is possible for output operations to alter the contents of the macro file, and to cause replies based upon its current contents to be transmitted on the input channel.
Rather than expound the model in the abstract, its features will be illustrated by using it to describe various concepts embodied in current graphics systems.
The use of macros to achieve machine independence has long been a recognized technique in programming [Brown 1974]. It has also been used to great effect in a field more closely related to graphics. Macro facilities in text formatters permit the matter of a document to be output in a variety of styles, using techniques appropriate to different devices [Kernighan et al 1978].
In the model, device independence is achieved by defining the standard primitives of the output channel as macros in terms of the specific primitives of the output device. Changing devices involves re-defining these macros, which can be done by appropriate output operations. The reverse process can be used to achieve input device independence, by defining the specific input primitives as macros in terms of the standard primitives of the input channel. As an example, the difference between the handling of storage and refresh output can be described by defining a macro which draws the whole of the current picture, and arranging to invoke it in the storage case at every end_batch command, and in the refresh case at every clock event.
Note that in this model the device drivers no longer have the main responsibility for device independence; their function is restricted to translating to and from a form acceptable to the macro processor. This form can vary from device to device, obviating the need for the handshaking used by GINO drivers to adapt to varying device capabilities. The importance of permitting applications program control over this adaptation has been stressed by Vinberg [Vinberg 1978].
Segments in graphics serve two functions. The first is that of grouping primitives into a compound entity which can be given manipulatable attributes, such as visibility and detectability. The second is that of structuring the picture as a set of repeatable sub-units.
The first function can be described if each segment is defined as a macro consisting of calls to primitives, preceded by calls to macros with names of the form:-
<segment_name><attribute__name>
which serve to establish the values of the appropriate attribute. Attribute manipulation output functions then simply re-define these attribute macros.
The second function is described by permitting segment macros to include calls to other macros.
It has already been shown that input device independence can be achieved by defining the input primitives of the device as macros. These input macros can, in addition, include calls which re-define macros forming part of the picture, since the file of macro definitions is shared by both the input and output macro processors. This enables the model to describe the link between input and output operations. Further, since the input macros are subject to re-definition as a result of output operations, the link between input and output may be completely controlled, even dynamically controlled, by the applications program.
One aspect of current models which has not so far been described is their operation in the time dimension. The Core model provides for asynchronous input; that is it permits the applications program to enable one or more input devices, and then to continue processing until a specific await_event function is called. It should also include an inquiry as to the number of events awaiting processing. The GKS model uses synchronous input; that is it provides for the applications program to request a number of inputs from a device, and then waits until that number has been received, or the user has indicated end_of_input.
The Core goes further than this. It divides input devices into event devices (pick, button, keyboard) and sampled devices (locator, valuator). A number of sampled devices may be associated with an event device, in such a way as to include in the report of the event the values of the sampled devices at the time of the event.
The model is equally capable of describing both types of input. Asynchronous input with device association is modelled by including in the macro invoked by the event calls to macros returning the values of the appropriate sampled devices. Synchronous input is modelled by using individual input operations to append to the macro invoked by end_of_input the values they return.
The rapid development of standards for data communication networks makes the specification of a protocol or, more properly, a layered structure of protocols for the transmission of graphical information a matter of urgency. Many output protocols have been defined [Enderle et al 1978, Groot 1975, Conley et al 1978] and there are striking similarities between them. There was also an attempt to set up an interactive protocol for the ARPANET but this was abandoned for lack of funds [Sproull & Thomas 1974, Sproull 1978].
The role of a protocol definition in the model is clear. It serves to define the form of the calls to, and the names of, the output macros, and to define the results of the input macros. This model is especially suited for discussion of network protocols, since it requires only two uni-directional channels and therefore fits network access techniques easily.
Those current models of input which follow Wallace [Wallace 1976] in regarding input as fundamentally separate from output, and echoing as merely incidental, seem to view the user as interacting with the model, or at least with the applications program. Although the new model is capable of reflecting this view, it is also capable of describing systems based upon a view of the user as interacting with the picture. Because the file of macros is shared between the input and output macro processors, the picture which it describes can be viewed as a resource to be shared by the user and the applications program (Fig. 4). Two different styles of interaction are possible in this approach, implicit input, in which the applications program retains control of the interaction process, and picture inquiry, in which the picture is shared on an equal basis between the user and the application.
Implicit input is a style of interaction in which input operations occur as the result of output operations with missing parameters. These are filled in by the results of the implicit input operations, whereupon the output takes place, and the completed output primitives are returned to the application. Examples of such output operations are:-
Draw a line from the CP to the locator. or Make the next segment picked invisible.
In this way the effects of user actions upon the picture are specified by the applications program before the event, but executed by the graphics system after the event. The advantages of this approach in a distributed or multi-process environment are obvious.
Implicit input can be even more powerful if it is combined with the ability to re-define the macros invoked by the user's input. A common interaction [e.g. Bergeron et al 1978] involves a hierarchy of menus. When the user picks an item in the higher-level menu, it is made invisible and the appropriate one of a number of lower-level menus is made visible. This can be modelled by defining each of the pick-ids of the higher-level menu as a macro which, when invoked by the pick, re-defines the visibility attribute macros of the appropriate segments. In this way, a whole interaction sequence can take place under applications program control, but without applications program intervention. The link between input and output is under applications control, but operates autonomously.
Implicit input can take place either synchronously, if the application permits only a single pending macro, or asynchronously. Typically, a number of implicit input macros would be defined, and those corresponding to visible, detectable pick-ids would be enabled. The one macro actually invoked by a pick could disable any inappropriate macros by changing visibility or detectability attributes.
One important feature of the new model is that it permits the applications program to inquire about the contents of the picture. This models, for example, the <seg_readback> commands of the ARPA protocol, which cause the terminal to reply with a sequence of output primitives sufficient to represent the segments inquired of. The autonomous operation of implicit input can be extended using this facility. If an appropriate set of picture manipulation macros are defined for a particular application, they can operate without reporting the changes to the picture directly. Instead, the application discovers the results of the user's operations by inquiring about the picture, and the user discovers the results of the application's operations by observing (i.e. inquiring about) the picture.
It is noteworthy that in both these cases the contents of both input and output channels between the graphics functions and the macro processors are output primitives. They both give rise to symmetric graphics protocols. This symmetry is appropriate because, unlike conventional models, the subject of the communication along both channels is the same, namely changes to the picture.
The fact that a macro file viewed as a resource shared by both the input and output channels, and by the application and the terminal, leads to a symmetric protocol has strong analogies with recent work on network virtual terminal protocols. One such symmetric protocol [Schulze & Borger 1978] is based upon a description of the contents of a notional screen (= display file). This is accessible to both parties, and the protocol provides facilities for transferring control over parts of the description (= segments) from one party to the other.
An alternative to current models of interactive graphics software has been presented which is capable of describing a wider range of interactive techniques. The model, based upon well-known techniques of text processing, unifies many of the concepts underlying current software by indentifying them as aspects of graphics macro processing.
I should like to thank Malcolm Sabin and my other colleagues on BSI DPS13/WG5 (Graphics) for their stimulating and constructive criticism of an earlier paper addressing some of these issues.
1. Bergeron R.D, Bono P.R. & Foley J.L "Graphics Programming Using the Core System" ACM Computing Surveys 1_Q 4 (Dec. 1978).
2. Brown P.J. "Macro Processors and Techniques 1 - Portable Software" Wiley (1974).
3. Conley B, Puk D. & Reed T. "Basic Graphics Package Intermediate File Format" Los Alamos Scientific Laboratory (Mar. 1978).
4. DIN "Proposal of Standard - Information Processing Graphical Kernel System (GKS) - Functional Description" DIN 00 66 252 (Dec. 1978).
5. Eckert R. "Specification of Graphics Systems" Workshop on Man-Machine Interaction, Seillac (May 1979).
6. Enderle G. et al "The AGF Plotfile - Towards a Standard for Storage and Transportation of Graphics Information" Computer Graphics (ACM-SIGGRAPH) 12 4 (Dec. 1978).
7. Groot D. "GPGS 16 bit Device Independent Picture Code" Delft University (Oct. 1975).
8. GSPC "Status Report of the Graphics Standards Planning Committee" Computer Graphics (ACM-SIGGRAPH) 11 3 (Fall 1977).
9. Kernighan B.W, Lesk M.E. & Ossanna J.F. "Document Preparation" Bell System Technical Journal 57 6pt2 (July 1978).
10. Schulze G. & Borger J. "A Virtual Terminal Protocol Based Upon the 'Communication Variable' Concept" Computer Networks 2 (1978).
11. Sproull R.F. & Thomas E.L. "A Network Graphics Protocol" Computer Graphics (ACM-SIGGRAPH) 8 3 (Fall 1974).
12. Sproull R.F. Private communication (May 1978).
13. van den Bos J. "Definition and Use of Higher-Level Graphics Input Tools" Computer Graphics 12 3 (Aug. 1978).
14. Vinberg A. "Position Paper on Graphics Standards" Computer Graphics (ACM-SIGGRAPH) 12 4 (Dec. 1978).
15. Wallace V.L. "The Semantics of Graphics Input Devices" Computer Graphics (ACM-SIGGRAPH) 10 1 (Spring 1976}.
