QUASS - a tool for measuring the subjective quality of real-time multimedia audio and video
Anna Bouch, Anna Watson and M. Angela Sasse
Department of Computer Science,
University College London
Gower Street, London
WC1E 6BT, England
{A.Bouch, A.Watson, A.Sasse}@cs.ucl.ac.uk
ABSTRACT
There is currently no adequate method of measuring the subjective quality of audio and video experienced by users in desktop videoconferencing, meaning that objective audio and video quality requirements cannot be effectively derived. This paper introduces a new measurement tool, QUASS (QUality ASsessment Slider), which is designed to address this problem.
KEYWORDS
Desktop videoconferencing, subjective quality, assessment, measurement.
INTRODUCTION
Desktop videoconferencing has the potential to enable large and distributed audiences all over the world to participate in conferences and meetings without having to leave their respective physical locations. However, despite the huge potential of this communication technology, uptake has been slower than expected. One reason for its slow uptake must be the potential users' concern over whether the quality of the audio and video will be good enough, at a price they can afford. There has been no systematic investigation of the audio and video quality required for different videoconferencing tasks to be accomplished successfully. The complexity of the issues involved in evaluating these media in videoconferences is discussed in Watson & Sasse (1996), but what is required immediately is a method by which subjective quality ratings can be gathered in a dynamic, continuous fashion as a videoconference proceeds. Different videoconferencing tasks and sub-tasks will have different quality requirements, and these must be identified, so that quality thresholds and guidelines can be established.
MEASURING PERCEIVED QUALITY
Assessment of speech and video quality has traditionally been in the domain of bodies such as the ITU (International Telecommunications Union) and the EBU (European Broadcasting Union). Assessment methodologies and rating scales have been developed and standardised across the world by these bodies. However, desktop videoconferencing is a new area of communication and traditional quality rating methods are not suitable for use in videoconferencing assessment. The main reasons for this are: the range of vocabulary on ITU recommended rating scales is not applicable to low-cost videoconferencing quality; conditions over some networks can fluctuate, meaning that quality ratings gathered at the end of a conferencing session can be subject to primacy or recency effects; measuring only speech or video quality alone does not produce very meaningful results for generalising to multimedia conferences where the different media can and do interact. These issues are discussed in greater depth in Watson & Sasse (1998).
What is required is a new type of subjective measurement tool that takes into account the dynamic and interactive nature of videoconferencing speech and video quality. This tool must be able to gather real-time continuous assessment results as conferences, and their various sub-tasks, proceed.
ESTABLISHING REQUIRED QUALITY USING QUASS
Through questionnaires and focus groups held after various desktop videoconferencing trials, we at UCL have identified a number of different dimensions that play a role in forming opinions of overall quality, such as packet loss and 'unpredictability' for speech, and speed and 'blockiness' for video. We have developed a software measurement tool, QUASS (QUality ASsessment Slider), which allows users to move a slider up and down an unlabelled continuous scale as a speech file varies in quality along one of the formative quality dimensions. Measurements of the position of the slider are pumped to a file every second, allowing us to compare the objective speech quality with the subjective rating at that instant. The tool can also be used to dynamically controlthe quality of the speech that the user is receiving, along a particular dimension. In order to prevent the user setting the quality to its maximum level at all times, this condition also provides the user with a 'budget', which decreases according to the quality that is demanded. The expenditure is recorded by the software in order to provide insights into the relationship between payment behaviour and the requested Quality of Service (QoS).
Although at present QUASS is being used only in a laboratory setting to investigate speech quality dimensions, we believe the technique will also be suitable for the investigation of subjective video quality, and in interactive, multimedia conference situations so that different task requirements can be identified. We believe that QUASS will prove to be a sensible means of assessing subjective quality, and will allow HCI researchers to make qualified recommendations to users, developers of new systems and network managers as they begin to implement network resource reservations.
ACKNOWLEDGEMENTS
Anna Watson and Anna Bouch are supported by EPSRC CASE awards with BT.
REFERENCES
Watson, A. & Sasse, M.A. (1996) Evaluating audio and video quality in low-cost
multimedia conferencing systems. Interacting with Computers, 8(3), 255-275. Watson, A. & Sasse, M.A. (1998) Measuring perceived quality of speech and video in
multimedia conferencing applications. To be presented at ACM Multimedia '98.
