This paper outlines a taxonomy for graphical input and interaction. The taxonomy is based on one central concept termed positioning. First a number of basic concepts and basic functional capabilities are described, which are assumed to be available. Next, a graphical input system is built up using these basic notions and basic functions. Also the possibilities for interactive work with such a system are investigated.
This paper describes a taxonomy for graphical input. The taxonomy is based on one central concept termed positioning. The concept of positioning though not new at all is used in a way similar to the concept of viewing. Viewing serves to distinguish so-called modelling functions (i.e. complicated, high level, application dependent functions to generate a graphical model of an object) from graphical kernel functions with respect to output. The output kernel should only serve to visualize the graphical model.
It will be investigated whether positioning can be used to distinguish kernel functions and modelling with respect to input.
Intuitively speaking, positioning is a function which puts a graphical object in the proper place in the world of the model. A modelling system accepts graphical objects annotated with positioning information. Positioning specifies in a unique way the size, orientation and place of the object with respect to the model.
For the time being it will not be required that a distinction between kernel and modelling is given. Graphical input and interaction will be constructed around the concept of positioning. Identification of kernel functions, if any, should come out as a result.
When designing a graphics system it is necessary to build upon some fundamental notions. It is equally necessary to postulate basic resources and basic functional capabilities. Before elaborating further on positioning, an outline will be given of the fundamental notions and resources that are taken for granted.
A graphical object is a structured entity containing, apart from information about the structure, only values which have a geometrical interpretation (e.g., coordinates). Graphical objects can be manipulated as part of the input process. One may assume that the structure can be anything from single entities to hierarchical and even cyclical constructs. In all these cases however, the types of the geometrical values are from the same, preferably, small set.
Certain fixed combinations of structure and geometrical values may be dedicated to represent an often used basic graphical object(e.g., a line segment), but this is not essential here.
A positioning function accesses the geometrical values in a graphical object in a way controlled by the structure(and the positioning function itself, of course) and next transforms each value encountered into a new one of the same type. This is the basic notion for the semantics of positioning. However a positioning function although specified, does not have to be effectuated immediately.
A model in the universe of some application is a mixture of graphical and non-graphical information, structured in some way. The following basic notion defines how the non-graphical and the graphical component are to be related within a model.
Since we are dealing with a graphical model, we will assume that the graphical component is the backbone of the model and that the non-graphical information is so-to-speak centered around this component. The important consequence of this is that the graphical component also provides the structure skeleton of the model. In other words, in addition to the geometrical values of the previous section, the structure also should have a graphical interpretation. To meet this requirement positioning will be proposed as means of structuring.
Typical for graphical objects is that they may have a multilevel structure. On each level one can distinguish graphical objects from a class associated with that level. Moreover, this structure may have been created in order to be able to associate with each level application dependent non-graphical information which is organized in a similar multilevel way. whatever the nature of the graphical objects on each level may be, the class of positioning functions will be the came for each level. This goal of having one class of positioning functions directly mirrors the intuitive notion of positioning an object, as given above, which is completely clear without knowing what the object is.
Complex input objects cannot be created by a user in one action. They require a whole sequence of actions interrupted by correctness checks on the sequence as a whole or the most recently produced subsequence.
The order in which the subobjects are created is not necessarily the same as the order in which the subobjects in a complete complex will be accessed. In particular the error checking and recovery mechanism might require the ability to jump over the most recently produced objects and to correct a more ancient one.
Problems encountered here are very well known in connection with language processors (e.g., parsers and compilers). It will be assumed that a parser is available which checks correctness of input against a predefined syntax. This parser will be able to carry out user initiated back-up sequences. No restrictions are imposed on the input syntax (e.g., strictly LL0, etc.). Simplifications of the syntax should only be carried out for the benefit of the user.
To summarize, the following functional capabilities are assumed to be present in the system:
The input system at first will not attempt to cover every possible form of input.
Text will be passed on from one level to the next, unless it is consumed explicitly at that level by non-graphical operations. In a later stage the possibility to transform a piece of text into a graphical object will be considered. It is expected that text is not essential for graphical input.
Non-graphical information associated with graphical objects will not be considered. This information typically deals with the aspects of the model for which no graphical representation exists. To the graphics input stream such information will appear as escaped sequences to be skipped over.
A user at a graphics terminal is supposed to have a geometrical working space for creation of graphical objects. The geometrical space where the object will have to be positioned is called target space.
An object can be positioned basically in three ways:
The following analysis of these three methods attempts to show that the final result in terms of internal data structure can be the same. Moreover, it tries to show that the three methods can be combined freely. This is important as it is essential for using positioning as a basic concept.
Given an object o in working space, the object corresponds to o' = pos(o) , in target space. pos() denotes the positioning function. Internally the resulting position will be represented as a geometrical transformation: pos(o) -> t * o.
The mapping between working and target space can be defined per object by a position. In the case of multilevel structured objects, target space is a relative concept, leading one step up in hierarchy.
At target space level, objects can be added one after the other to the same scene. A scene will be represented by a node in a tree structure.
A completed scene can be added as object level scene.
If four objects in a scene would have been positioned the tree would look like:
If the objects are not positioned, but simply added, this is reflected in the absence of structure. The objects are combined into a sequence with the same position for each element. In the tree structure these sequences add only one branch. This effect can be accomplished in two ways: either objects are positioned first and the position is next dissolved by actually transforming (e.g. pushing the transformation downwards), or, objects are created on the spot with no (unit) transformations stored.
It follows that the user may take advantage of positioning facilities without this having to result in more complex objects.
An interesting way of positioning (especially for coupling) is by using a handle. A handle is a sub-object which (on a lower level in the structure) can be isolated as object. This implies that upon creation of the complex object the handle has been added by positioning. Positioning of a complete object can be accomplished by (re)positioning the handle.
Suppose a handle h, is added at position t, to the object o.
Now h can be repositioned in two ways:
In the first case the position p of the object follows from the equation
p.t = t' or p = t' . t-1 .
In the second case for a two level structure the adjustment for h equals the position of the object.
The result of creating complex objects by positioning can be represented as expression type input, where position is the operator and object the operand. Apart from this we only need the concept of a sequence of objects with the same position. It is now also clear that we have chosen expressions rather than relations (or sets). Creating more complicated structures than oriented graphs (e.g. nets) is definitely considered modelling.
For viewing one generally considers a two level structure (segments) sufficient. In these cases the input cannot be more complex, since the viewing system would not preserve anything more complex. The use of handles might be the answer here.
It is felt that for a user to grasp a more complicated data structure from a visual representation is impossible without leaning on the associated application dependent meaning. This is the reason for using expressions instead of relations.
Concluding this section the following positions are taken:
As soon as manipulations of objects become part of interaction, feedback during manipulation is needed. Feedback is more than echoing. Echoing suffices to show what values are going to be supplied to the program. Feedback is a means to find and therefore to define values.
Good interactive facilities should be based on the concept of feedback. The position operation of the previous section is an outstanding candidate for application of feedback. Feedback can be looked upon as a program structure that will be called a feedback procedure in the sequel. A feedback procedure can be associated with manipulation as follows. The header of a feedback procedure declaration might look like
"feedback proc" (object): "result"
The parameter, object, is a value parameter allowing the feedback process to show the state of the process without actually changing the object. The keyword result denotes the value that has to be delivered by the process (a position in our example). In the body of the procedure input expressions define the actions expected from the user. There also will be an echoing mechanism defined, in direct relation with the input expression. Last but not least, the conditions for return and the value delivered are part of the declaration.
The mechanism proposed here is in rudimentary form already present in many interactive systems. It is considered to be a basic tool and should therefore be part of the kernel of any interactive system.
Current technology allows for sufficient local intelligence to implement it.
