Applying the notion of the Consumer Report Paradigm (cf. ISO 9126) to the overall process of evaluation, it is obvious that the starting point is to find out which are the relevant facts, features, tests and judgements in the context of translators' aids. There are two key elements in this process, i.e. the object of evaluation and the user (cf. section Towards formalisation and automation). Featurization in general, therefore, involves the integration of the user and the object aspects while at the same time considering the quality characteristics and attributes involved (cf. ISO 9126).
Carrying out user surveys according to the method recommended in section Translators' aids: user profiles leads to a set of facts denoting the basic needs of users with respect to the environment in which the tools are applied. Carrying out product surveys (cf. section Product typology) leads to a set of facts describing the basic functionality of the tools under question. Matching both sets of facts one arrives at a catalogue of facts relevant to users interested in these products. Such a catalogue of facts may range from e.g. the operating system (DOS/Windows/Windows NT/ UNIX/Sinix/Macintosh...) to the help facilities offered (context sensitive help/hypertext help/systematic help file...).
The next step in the overall process of preparing an evaluation involves assigning specific features to the facts which generally characterize such systems. The detailed nature of such feature lists mirrors the intention behind them, i.e. to elucidate those peculiarities of a software tool which might turn out to distinguish between tools of the same type. Feature lists are compiled in the awareness of the possibilities individual systems offer and with the aim of demonstrating the differences between similar tools.
In section Specification/inspection catalogues a catalogue will be described, the goal of which is to assess to what extent a software tool fulfills the user's expectation/specification, i.e. the user's needs in general terms. In section Feature checklists of translators' tools we will describe what has to be considered when compiling feature checklists for translation memories and terminology management databases.
The specification/inspection catalogues were developed in the course of the ESPRIT projects 2315 and 6005 TWB (Translator's Workbench I and II). They are based on a previous user requirement study which was performed in the framework of TWB I. In addition to user requirements, the catalogues which have been developed for term banks, translation memories, machine translation, machine-assisted terminology work, and checkers take into account the latest technical and functional achievements in the respective areas.
The catalogues serve as the basis for i) the specification of tools according to the needs of users and ii) the evaluation of to what extent the tool conforms with the user's needs or expectations. In general terms each catalogue comprises facts relevant to the software and related to a certain quality characteristic. Among the quality characteristics tackled in the ESPRIT catalogues are task adequacy, error tolerance, execution efficiency, ease of use, ease of learning etc. Users can tick items which are relevant to them, give them an individual priority and moreover specify this priority by rating its relative importance (0-100) compared to other items of the same type (target).
The specification/inspection catalogues also help to establish user profiles, since they demonstrate that the relative importance of particular items differs between different user groups, e.g. in the context of translation it is pretty obvious that a free-lance translator who works with a stand-alone PC will not give networkability a very high priority, while a translation department certainly will.
Having specified the relevant items and rated their importance, the answers can now be compared with the actual functionality of the software. The comparison between the previous specification and the actual functionality of the software works on the boolean basis of availability: yes/no. When an item specified beforehand is actually available, the full score will be given to the system, whereas no score will be given if the tool does not cover the item wanted. A numerical value, which can be calculated by comparing the target specification and actual functionality, demonstrates the percentage to which the software in general complies with the user's needs.
As `facts' in the Consumer Report Paradigm are only the first step of the overall framework of decision taking, such catalogues have to be complemented by feature checklists (cf. the following section), tests and judgements (cf. section Evaluation procedure).
Any feature checklist in the context of evaluation needs to be standardized to make sure that it is applicable to any such tool and that the results are independent of situational variables. Moreover, the relation between the availability of a certain feature and the quality aspect it denotes has to be defined carefully.
Depending on the nature of the tool, only particular quality characteristics have to be considered for translation memories and terminology management databases respectively. Feature checklists should be based on the assumption that each feature mentioned in the list can be assessed and evaluated on the basis of
The principal aim of any feature list is to deliver evaluation results without involving a large number of real users and extensive tests. The feature checklists are filled in by evaluators (probably with the assistance of real users) for a number of tools of the same kind. Thus the feature checklists for the different tools will be of help to real users who need some information on the systems under question. On the basis of the comparison of the different checklists, the user can finally choose the most suitable system for his personal environment.
A number of comprehensive feature checklists have been elaborated in the framework of EAGLES, i.e. checklists that focus on basic features relevant for a broad concept of term bank and translation memory. All feature checklists elaborated so far in the context of EAGLES need to be further integrated and harmonized according to the common framework, which will be an ambitious task for future work.
Having collated valuable work in this first phase of EAGLES, the primary concern of featurization in the next phase is to
The main focus of the EAGLES evaluation/translation subgroup lies in providing worked-out checklists, mainly for translation memories and terminological databases. These checklists will be handed over to people active in the area of Linguistic Engineering, translators and system engineers. They will be asked to comment on the items and their relation to specific quality characteristics and to judge the usefulness of such checklists when it comes to buying one out of the plethora of tools available. Their comments will be considered and a final version of the checklists will be elaborated.