Why did I carry out this experiment?

Resarch-Based Web Accessibility is important.

• Guidelines are developed mainly by discussion.
• But we need enough evidence (academic approach):
  • experimental approach
  • non-experimental approach (social approach, ecological validity)
  • technological approach

I want to evaluate the effect of heading elements experimentally.

1. INTRODUCTION

Web content accessibility guidelines require the markup of content structure.

• WCAG 2.0: "Ensure that information and structure can be separated from presentation."
  • techniques: "H42: Using h1-h6 to identify headings"
• JIS X 8341-3: "5.2 a) Web content shall define document structure using heading, paragraph, list, and other elements."

1. INTRODUCTION (cont.)

Heading elements

• show the hierarchical structure of the content.
• show the topic of the section.
• Sighted users use them as a cue for distinguishing between content blocks.
• Blind users also can use them as a cue if properly marked up.

1. INTRODUCTION (cont.)

In order to make use of heading elements:

• content developers should mark up content with heading elements.
• user agents (browsers, AT) should have functions to make use of heading elements.
• users should know how to use these functions.

But they mostly do not do these. Why?

1. INTRODUCTION (cont.)

The objective: to demonstrate how content that is well marked up improves usability and accessibility.

• Q1. How much is the task completion time reduced by marking up heading elements?
• Q2. Do the benefits to task completion time of marked up heading elements differ between sighted and blind users?
• Q3. Is the difference in task completion time between sighted and blind users decreased when structure markup is both available and utilized by user agents?

2. METHODS

2.1. Components of Web accessibility

W3C/WAI: How the Components (of Web Accessibility) Relate

to examine the effect of structure markup, all other components must be accessible:

• accessible Web content (aside from marking up heading elements)
• User Agents with enough capability to utilize marked up Web structure
• skilled users who know how to use the UA's functions

2.2 Subjects

• 16 sighted subjects: 20-22 years old, female, use Web almost everyday
• 4 blind subjects: 30-34 years old, male, born blind, use JAWS for several years, use Web almost everyday

2.3 User agents

• Sighted subjects: Firefox with "Document Map" addon (Demo)
  • display the content's heading structure in the sidebar.
  • a user can move to any heading element by clicking that heading in the sidebar, allowing rapid navigation.
• Blind subjects: JAWS 6.2 or 7.1 (Japanese version)
  • read aloud the number of heading elements when loading the page
  • heading navigation functions, can skip to next/prev. h1-h6.

2.4 Experimental Design

Pilot study showed strong learning effect and large individual differences in task completion time.

Mixed design:

• Within-subject factor: site type (structured or not), task number (1-4)
• Between-subjects factor: subject type (sighted or blind)

Two structured (marked up with heading elements) sites and another two unstructured sites were used for the experiment.

2.5 Web sites

Experiment: instructing the subject to accomplish the given tasks by using the Web sites.

• "Summer Dish Recipes" sites
• Both structured versions (Fig. 1a) and unstructured versions (Fig. 1b) were prepared for the same site.
• CSS was used to make the visual appearance of both versions look the same.

Hierarchical structure of the Web site A

• h2: Select from ingredients
  • h3: Vegetable
    • 4 h4 level subcategories
  • h3: Fish and shellfish
    • 3 h4 level subcategories
  • h3: Meat
    • 3 h4 level subcategories
  • h3: etc
    • 2 h4 level subcategories
• h2: Select by purpose
  • h3: Main dish
  • h3: Sub dish
    • 3 h4 level subcategories
  • h3: a la carte
• h2: Select by cooking time
  • h3: 5 min.
  • h3: 10 min.
  • h3: 11 to 20 min.
  • h3: Over 21 min.
• h2: Select by calories
  • h3: less than 100 kcal
  • h3: 100 kcal range
  • h3: 200 kcal range
  • h3: 300 kcal range
  • h3: 400 kcal range
  • h3: 500 kcal range

2.5 Web sites (cont.)

• The site has 27 links to other recipe pages.
• These links were classified into the appropriate subcategories of the h3 or h4 level.
• Recipe pages showed details of the recipe (ingredients, cooking time, and calories).

2.6 Tasks

The same 4 tasks for each site. (Some tasks details changed)

• Task 1: Yes or No question. Answer if a particular category existed in the h3 level.
• Task 2: Study 8 or 9 recipes of h3 or h4 levels and then choose one which best matched the conditions.
• Task 3: Choose 3 recipes which matched the conditions.
• Task 4: Count number of recipes which matched each of two conditions and then indicate which conditions was matched by more recipes.

2.8 Procedure

Half of the subjects saw the structured site first
and the other half saw the unstructured site first.

1. collect demographic information
2. training session (to accustomed to experimental setup)
3. practice session (to accustomed to site image and tasks)
4. Site A, B, C, D
5. questionnaire and interview after every site

3. RESULTS

This paper used last two sites (Site C and D) for analysis
because the subjects were more accustomed to the site and tasks in these sites.

3.1 Analysis of Task Completion Time

• After the experiment, subject actions such as task start and task completion (and in many cases, scroll, jump, click, key press, and page name) were recorded with a timestamp.
• Task completion time was calculated as the time difference between the start of the task and the end of it.
• Some tasks were evaluated as "not completed" because the answer was wrong.

3.2 Results of Sighted Subjects

• Two-way ANOVA: if task completion time differed between structured and unstructured sites.
• Two within-subject factors: structured vs. unstructured, task ordering (1st, ... 4th).
• Result: task completion time was affected by the structure factor, F(1,13)=10.29,p=0.007.
• Result: task was also significant factor, F(1.57, 20.4)=54.74, p=0.000.
• Result: no interaction between two factors, F<1.

3.2 Results of Sighted Subjects (cont.)

Fig 3a. Average task completion time and SD for sighted subjects in Ordering 1 (users first accessed structured and then unstructured site)

Fig 3b. Average task completion time and SD for sighted subjects in Ordering 2 (users first accessed un-structured and then structured site)

3.2 Results of Sighted Subjects (cont.)

3.3 Results of Blind Subjects

• Two-way ANOVA: if task completion time differed between structured and unstructured sites.
• Two within-subject factors: structured vs. unstructured, task ordering (1st, ... 4th).
• Result: task completion time was affected by the structure factor, F(1,3)=33.37,p=0.010.
• Result: task was also significant factor, F(3, 9)=10.37, p=0.003.
• Result: no interaction between two factors, F<1.

3.3 Results of Blind Subjects (cont.)

Fig 4a. Average task completion time for blind subjects in Ordering 1 (users first accessed structured and then unstructured site)

Fig 4b. Average task completion time for blind subjects in Ordering 2 (users first accessed unstructured and then structured site)

3.3 Results of Blind Subjects (cont.)

3.4 Differences between Sighted and Blind

• Two-way ANOVA: if task completion time differed between sighted and blind subjects.
• One within-subject factor (structured or not) and one between-subjects factor (sighted or blind)
• Result: both factors were found to affect the task completion time.
• Result: interaction between these two factors were significant for every four tasks.

3.4 Differences between Sighted and Blind Subjects (cont.)

Fig 5. Ratio of average task completion time of blind subjects over that of sighted subjects

• Task 1 (Yes or No question) took more time for blind subjects than sighted subjects.
• Except for task 1, blind subjects needed roughly twice the time required by sighted subjects when the site was structured.
• Task4 : difference between the structured and unstructured for task completion time tended to be greater.

4. CONCLUSION

Experimental results are interpreted with respect to the questions:

• Q1. How much is the task completion time reduced by marking up heading elements?
• Q2. Do the benefits to task completion time of marked up heading elements differ between sighted and blind users?
• Q3. Is the difference in task completion time between sighted and blind users decreased when structure markup is both available and utilized by user agents?

4.1 Reduction of Task Completion Time

• Sighted subjects: the speed-up ratios (structured/unstructured) is 50 to 90%.
• Blind subjects: the speed-up ratios (structured/unstructured) is 50 to 80%.
• the speed-up ratios vary among tasks.
• Thus, task completion time is reduced about 10 to 50% when heading elements are properly marked up.
• ANOVA results show task completion time is strongly affected by tasks.

4.2 Differences between Sighted and Blind Subjects

• ANOVA results showed that sighted/blind factor affects the task completion time.
• Differences between sighted and blind subjects differ from task to task.
  • Task 2 and 3: blind subjects tend to take about twice as long to complete a task for both structured and unstructured sites.
  • Task 4: the impact of lack of structuring on blind subjects seems to be more severe.

4.3 Effect of Structure Markup for Blind Subjects

• The average ratio of task completion time with blind subjects over that of sighted subjects was about 2. (except for task 1)
• ANOVA results show a significant interaction between type of subject (sighted or blind) and structure (structured or not).
• Ratio of task completion time (blind/sighted) seems to be smaller for structured sites.
• Thus, it can be said that the sites marked up with appropriate heading elements decrease the gap between sighted and blind users.

4.4 Effect of Heading Elements in Terms of Usability and Accessibility

From this experiment and the discussions above, it can be said that

1. Heading elements improve usability because task completion time was reduced considerably.
   (This finding was supported by questionnaire data that showed higher user satisfaction with the structured site.)
2. Heading elements also improve accessibility because the benefits of structuring are sometimes stronger for blind users.

last slide

• Web browsers should implement heading navigation functions.
• More content should be marked up with heading elements.
• WCAG 2.0 and JIS X 8341-3 is correct. :-)

This Slides: http://www.comm.twcu.ac.jp/~nabe/2007/W4A2007/