Experimental Evaluation of
Usability and Accessibility
of Heading Elements

Takayuki Watanabe

W4A (Banff, May 7 2007)

This Slides: http://www.comm.twcu.ac.jp/~nabe/2007/W4A2007/

[Slide-view On/Off (access key:D)]

Why did I carry out this experiment?

Resarch-Based Web Accessibility is important.

I want to evaluate the effect of heading elements experimentally.

1. INTRODUCTION

Web content accessibility guidelines require the markup of content structure.

1. INTRODUCTION (cont.)

Heading elements

1. INTRODUCTION (cont.)

In order to make use of heading elements:

But they mostly do not do these. Why?

1. INTRODUCTION (cont.)

The objective: to demonstrate how content that is well marked up improves usability and accessibility.

2. METHODS

2.1. Components of Web accessibility

W3C/WAI: How the Components (of Web Accessibility) Relate

to examine the effect of structure markup, all other components must be accessible:

2.2 Subjects

2.3 User agents

2.4 Experimental Design

Pilot study showed strong learning effect and large individual differences in task completion time.

Mixed design:

Two structured (marked up with heading elements) sites and another two unstructured sites were used for the experiment.

2.5 Web sites

Experiment: instructing the subject to accomplish the given tasks by using the Web sites.

Hierarchical structure of the Web site A

2.5 Web sites (cont.)

Table 1. Category order for four sites
  Site A Site B Site C Site D
1st h2 ingredients calories cooking time purpose of the dishes
2nd h2 purpose of the dishes cooking time calories ingredients
3rd h2 cooking time purpose of the dishes ingredients calories
4th h2 calories ingredients purpose of the dishes cooking time

2.6 Tasks

The same 4 tasks for each site. (Some tasks details changed)

2.8 Procedure

Half of the subjects saw the structured site first
and the other half saw the unstructured site first.

  1. collect demographic information
  2. training session (to accustomed to experimental setup)
  3. practice session (to accustomed to site image and tasks)
  4. Site A, B, C, D
  5. questionnaire and interview after every site

3. RESULTS

This paper used last two sites (Site C and D) for analysis
because the subjects were more accustomed to the site and tasks in these sites.

3.1 Analysis of Task Completion Time

3.2 Results of Sighted Subjects

3.2 Results of Sighted Subjects (cont.)

Fig 3a. Average task completion time and SD for sighted subjects in Ordering 1 (users first accessed structured and then unstructured site)

Fig 3b. Average task completion time and SD for sighted subjects in Ordering 2 (users first accessed un-structured and then structured site)

3.2 Results of Sighted Subjects (cont.)

Table 2. Average "structure speed-up" ratios for sighted subjects
  speed-up ratio ± SD
Task 1
0.49 ± 0.22
Task 2
0.90 ± 0.28
Task 3
0.80 ± 0.21
Task 4
0.81 ± 0.35

3.3 Results of Blind Subjects

3.3 Results of Blind Subjects (cont.)

Fig 4a. Average task completion time for blind subjects in Ordering 1 (users first accessed structured and then unstructured site)

Fig 4b. Average task completion time for blind subjects in Ordering 2 (users first accessed unstructured and then structured site)

3.3 Results of Blind Subjects (cont.)

Table 3. Average "structure speed-up" ratios for blind subjects
  speed-up ratio ± SD
Task 1
0.56 ± 0.23
Task 2
0.83 ± 0.54
Task 3
0.71 ± 0.20
Task 4
0.48 ± 0.37

3.4 Differences between Sighted and Blind

Table 4. Summary of two-way ANOVA of task completion time for 4 tasks
  structured or not subject (sighted or blind) interaction
Task 1 F(1,17)=53.20, p=0.000 F(1,17)=153.3, p=0.000 F(1,17)=21.72, p=0.000
Task 2 F(1,17)=8.86, p=0.008 F(1,17)=30.62, p=0.000 F(1,17)=1.80, p=0.050
Task 3 F(1,18)=24.70, p=0.000 F(1,18)=17.25, p=0.001 F(1,18)=7.46, p=0.014
Task 4 F(1,18)=21.23, p=0.000 F(1,18)=45.48, p=0.000 F(1,18)=13.02, p=0.002

 

3.4 Differences between Sighted and Blind Subjects (cont.)

Fig 5. Ratio of average task completion time of blind subjects over that of sighted subjects

4. CONCLUSION

Experimental results are interpreted with respect to the questions:

4.1 Reduction of Task Completion Time

4.2 Differences between Sighted and Blind Subjects

4.3 Effect of Structure Markup for Blind Subjects

4.4 Effect of Heading Elements in Terms of Usability and Accessibility

From this experiment and the discussions above, it can be said that

  1. Heading elements improve usability because task completion time was reduced considerably.
    (This finding was supported by questionnaire data that showed higher user satisfaction with the structured site.)
  2. Heading elements also improve accessibility because the benefits of structuring are sometimes stronger for blind users.

last slide


This Slides: http://www.comm.twcu.ac.jp/~nabe/2007/W4A2007/