Personal Glossaries on the WWW

5. Results

[skip to list of contents]

Navigational Links

1. User Feedback

   During the debriefing sessions, users were encouraged to express their opinions about their experiences with the glossary tools they were given. Most users expressed satisfaction with the tools. The majority of users also said that they would like to have such a tool available when working with unfamiliar topics.

   The details of the protocol (including the debriefing) are in the detailed methodology section.

   Many users commented that they would have made more extensive use of the glossary if they were more concerned about understanding the material in the articles. A number of these users cited studying for quizzes or exams that would count towards their course marks as examples of such situations.

   Users that were given the updateable tool admitted that although functionality was present that allowed them to modify the glossary, they felt no need to do so. The most commonly stated reason for this was that they did not feel it was worth the effort because they would not be able to access the glossary ever again. Users said that in situations where this was not the case, they would consider the use of these features more helpful.

2. Analytical Results

   Speed and performance on the tests given to users were used as measures of the usefulness of the glossary tools. These measures were compared for each glossary tool in order to determine which of the two had been more useful to the users.

   1. Performance (effectiveness)

      The purpose of our glossary tools is to increase users' understanding of the text that they are reading. We measured this understanding is measured by the scores that users receive on sets of questions about the topics of the articles that they read.

      Table 1: Test Scores (%)

      		Range	Mean	SD	Mode
      No Glossary	Before Reading	11 – 78%	55%	17%	44%
      	After Reading	0 – 100%	60%	17%	60%
      Simple Glossary	Before Reading	0 – 90%	45%	22%	56%
      	After Reading	40 – 100%	78%	16%	80%
      Updateable Glossary	Before Reading	22 – 67%	44%	13%	44%
      	After Reading	70 – 100%	83%	12%	70%

      There was a significant difference between the performance of users with a glossary tool and without a glossary (t=5.505, df=39, p<0.05).

      The presence of a glossary tool significantly improved users' performance during the study. We conclude therefore that glossary tools of the type we developed will help readers to better understand texts.

   2. Speed (efficiency)

      1. Total Time

         User speed is used as a measure of users' efficiency while they read the given articles and answer the associated sets of questions.

         The users' mean speed of 603.01 seconds while using the glossary tool was slower than their own mean speed of 548.99 seconds without a glossary tool (t=2.550, df=39, p<0.05).

         Table 2: Time To Read Text & Answer Questions (in seconds)

         		Range	Mean	SD
         No Glossary	302.6 – 778.8		549.0	114.1
         Either Glossary	306.2 – 880.2		603.0	131.4

      2. Time to read

         Another way of examining the change in speed is to consider users' speed reading the actual article separately from the speed of answering the questions after the articles. No difference was found between the mean lengths of duration participants needed to read articles with or without glossaries (t=-.648, df=39, p<0.05). The mean time of 255.12 seconds spent reading articles with glossaries is not significantly differently from the mean time of 267.35 seconds spent reading articles without any glossary tool.

         Table 3: Time To Read Text (in seconds)

         		Range	Mean	SD
         No Glossary	131.9 – 501.4		267.4	95.4
         Either Glossary	70.1 – 476.5		255.1	96.4

      3. Time to answer

         The mean speed of 187.69 seconds to answer questions after reading an article with a glossary tool is slower than the mean speed of 139.69 seconds to answer questions after reading an article without a glossary tool (t=4.522, df=39, p<0.05).

   3. Difference between glossary types

      One glossary tool allowed users to update or add entries. Descriptions of the tools are in the detailed method section.

      Two types of glossary tool were developed for this study, an automatic tool and a user updateable tool that allowed users to create, edit or delete terms and entries in the glossary.

      The participants of the study did not make use of the updateable glossary's functionality. Thus, there was not enough data available to determine if there is a real difference between the two types of glossary.

End of Section
Where to Next?

References

References for all works cited are available in a separate chunk.

& .

[Up to navigation links]