Issues in Questionnaire Design

One of the problems that researchers encounter when they try to validate a questionnaire is the scoring ( ). The majority of questionnaires found in the literature use ordinal ranking data for each item (e.g. 1 to 5). However, sums of ordinal data, at face value, have no meaning, and individuals with the same score can represent very different characteristics. Then, various statistical models ( ) can be used for validation purposes, such as the Rasch model ( ). The Rasch model helps to transform the raw data into a meaningful interval scale through logarithmic transformations and probabilistic equations ( ). Rasch analysis ( Fig. 4.1 ) does not assume that all items have the same level of difficulty and that all persons have the same ability but weights them accordingly ( ). Rasch analysis is a probabilistic model that helps to define the difficulty of an item independently of the population, and the ability of a person independently of the items resolved by this person ( ).

A schematic representation of Rasch ‘person-item’ analysis of the results of a vision-related quality of life questionnaire. The questions/items are ranked by their visual difficulty, and the respondents/persons are ranked according to their functional ability. iStock.com/zuperia; iStock.com/IconicBestiary; iStock.com/Lyudinka; iStock.com/BRO Vector; iStock.com/jossnatu; iStock.com/intararit; iStock.com/SurfUpVector; iStock.com/Sudowoodo; iStock.com/Dmitrii Musku.

Clinically, a measure of QoL is going to be useful if it is: reliable, valid and appropriate, responsive, interpretable, simple, quick to administer and complete, and easy to score ( ).

Reliability refers to the consistency of a set of measurements or measuring instrument used to describe a given property or behaviour and has a number of aspects ( ). Test-retest reliability is the reproducibility of a given measure and refers to the degree to which scores stay the same over time, in a situation where no change is expected ( ). The participants should score similarly on the items between two assessments in a relatively short period of time ( ). Intraclass correlation coefficients (ICCs) should be higher than 0.60 over a 2-week period or less ( ). Reliability does not imply validity; a measure can be reliable and not valid. Scores can be internally consistent and stable over time, and yet the instrument does not measure what it is supposed to determine.

Validity is the extent to which an instrument measures what it is supposed to be measuring ( ). Reliability is precision, whereas validity is accuracy ( ). An instrument is said to have content validity if it covers all the aspects of a domain (e.g. social support: tangible, affectionate, instrumental, informational and positive social interaction) ( ) and face validity if it seems to be measuring what it is supposed to measure: whether it ‘looks like’ is going to assess what it says, rather than it ‘has been shown to work’. It refers to the degree to which a measure is sensitive to the participants’ interests ( ).

According to , a questionnaire has good content validity when the authors design the items by examining the literature, as well as consulting patients and/or experts. They consider the consultation of patients a prerequisite, because the questions are about them or their feelings and how they perceive their problems. An initial (large) number of suggested questions should be reduced during the development process by removing those that are not applicable to a large section of the population; are too easy for the participants; ask about a different concept; or are redundant (with content covered by the rest of the questions).

Convergent (or concurrent) validity measures the degree to which an instrument correlates to other measures of the same or similar construct (the results of the measure are compared to the results of the gold standard obtained at approximately the same point in time) ( ). Discriminant validity is the capacity to differentiate between cases and controls or disease severity groups ( ). proposed that an r value of 0.10 was a low correlation, 0.30 was moderate, and 0.50 was a high correlation. Construct validity estimates how well a measure correlates with other indicators of similar related constructs (i.e. reflects the ability of an instrument to measure an abstract concept, or construct. There is not always a gold standard) ( ). Construct validity can be measured by the infit and outfit statistics (indices of measurement accuracy). Infit statistics detect unexpected patterns on responses to items (by individuals) that are targeted on them. Outfit statistics detect unexpected patterns on responses to items that are considered relatively easy or very hard for the individual.

Internal consistency is the extent to which all items measure the same construct and is tested using Cronbach’s formula for coefficient alpha ( ). Cronbach alpha should be higher than 0.70 for group comparisons and test-retest reliability ( ).

When checking for subscales in the questionnaire, items that do not load (i.e. do not correlate with any other items measuring a similar construct) on any factor or that load in multiple factors should be eliminated. Cronbach alpha is a statistic commonly used to express this correlation ( ) and should be between 0.70 and 0.90.

also described the term Responsive-ness as the extent to which the instrument can identify variations in individuals known to have a change in visual functioning. This will be particularly important if the instrument is to be used to assess the effect of a particular intervention. Interpretability is the ability to understand the scores on a measure ( ), and the degree to which it is possible to assign qualitative meaning to quantitative data ( ). For interpretability, mean scores and standard deviations should be given for the population on which the instrument was tested. Respondent burden ( ) is the demand imposed on the participants (i.e. usually mostly time, but also energy and emotional stress). Some participants may not complete the full questionnaire, and the proportion of missing values can be an indication of respondent burden.

A related issue which is particularly relevant to a visually impaired population is how the questionnaire will be delivered. The usual method is self-administration, often in written format. However, this can create difficulties for those with a VI or carry the risk that another person completes it on their behalf. In-person administration by the researcher or clinician can be time-consuming and lead to bias. Online completion and telephone interview with an independent person are possible alternatives that may be useful in some populations.

Questionnaires Designed for People With Visual Impairment

The Impact of Visual Impairment (IVI) questionnaire ( ) was developed to assess the vision rehabilitation needs in the context of participation restriction and was supposed to be useful when measuring intervention outcomes.

According to , the IVI questionnaire is an instrument targeted at people with VI and it covers several dimensions of QoL, such as functional, social and psychological. The authors described a high content validity and reproducibility for this scale, whilst interpretability and respondent burden were considered to be average. Nevertheless, the IVI appears to address questions which relate to visual functioning of the patient and their limitation in their daily activities (e.g. reading labels and instructions in medicines) and patients’ feelings (e.g. feeling embarrassed) and not just participation restriction aspects (e.g. ‘I have taken part in social activities as and when I have wanted’). With such a wide-ranging content, it is difficult for a questionnaire to be responsive to simple rehabilitation strategies such as providing a magnifier.

The Low Vision Quality of Life (LVQOL) questionnaire aims to identify and quantify the impact of visual impairment on aspects like independence, mobility, time for oneself, and daily tasks ( ). This questionnaire was developed using patients with a wide variety of conditions causing untreatable vision loss. The authors reviewed the literature in order to gather all the subjective questionnaires previously designed to measure QoL. The resulting items were combined, so no repetition occurred. The questions were then revised by a group of experts (optometrists, ophthalmologists, orthoptists, occupational therapists, welfare officers, audiologists and those with low vision), with items not relevant to the majority of persons with low vision excluded. The 74-item questionnaire was conducted on a random sample ( n = 150 from the database of the Vision Australia Foundation) of subjects with low vision.

The LVQOL questionnaire is supposed to differentiate persons with low vision from those with ‘normal’ vision. For this purpose, the study included an age-matched and gender-matched control group of 70 subjects attending the optometric clinic (with minor eye conditions such as early cataracts, blepharitis, or dry eye). Internal consistency was calculated using the item-total correlation and Cronbach’s α and reliability using the ICC. The authors determined that a homogeneous scale (internal consistency) was that which focused on the different aspects of the same attribute. Consequently, items should be moderately correlated to each other and to the total scale score. Furthermore, when the items are highly correlated, questions add little information (redundancy). However, when the correlations are very low, the questions refer to different attributes. Four questions were removed using this principle (Cronbach’s α). The questionnaire was completed four times in a 4-week interval (short period for a potential change in patients’ situation and long enough, so the participants do not remember their prior answers) to assess test-retest reliability. Those items that remained unchanged were selected. Using these methods, the scale was reduced to 25 items. The LVQOL questionnaire was confirmed to have high internal consistency (Cronbach’s α = 0.88) and reliability (0.72). The questionnaire was designed to produce a summed score of the patients’ QoL (between 0 and 125, with a higher score indicating better QoL).

Varimax rotation identified four principal factors (i.e. general vision, mobility and lighting issues; psychological adjustment; reading and fine work; and ADLs) in the LVQOL, each with internal consistency (Cronbach’s α >0.80). The questionnaire also appeared sensitive at differentiating people with low vision and those with normal sight. Less than 20% of the subjects with low vision scored 95 or higher in the questionnaire, compared with 65% in the control group. This scale is a suitable clinical tool to quantify the QoL of patients with low vision.

It takes 5 to 10 minutes to complete and it is easy to understand by the subjects. Furthermore, on the systematic review of QoL questionnaires conducted by , the LVQOL questionnaire ( ) was considered to have high quality concerning psychometric aspects. Based on their research, the scale has good content validity (i.e. selection of items, item reduction, checking for subscales and internal consistency) and fair reproducibility and interpretability. The LVQOL was considered to be the gold standard at that time ( ) and therefore has been used in many studies ( ), when measuring QoL in people with low vision.

Another questionnaire specifically measuring visual functioning is the 48-item Veterans Affairs Low-Vision Visual Functioning Questionnaire (VA LV VFQ-48) ( ). This instrument measures visual efficacy in terms of overall ability, mobility, visual information processing and visual motor skills ( ). Theoretically, the questionnaire consists of 48 items (activities) that the subject has to categorise from not difficult to impossible to accomplish. But there are two further questions (is it difficult because of your vision?/how do you usually perform this activity?) that must be applied to every item of the questionnaire. Hence, the 48 items prove to be 144 questions, and as observed takes around 35 minutes on the telephone. The short form of this questionnaire is only 20 items, again theoretically. In practice, these 20 items actually become 60 questions. The authors did not state how long the interview would last for this short form, but it would probably take about 15 minutes. The VA LV VFQ-48 scale is a very comprehensive tool to obtain accurate measures of the visual ability when time is not an issue: it has been designed to be sensitive to rehabilitation. The items were developed and chosen by a multidisciplinary team of clinicians and researchers as well as persons with vision loss. The items were calibrated with Rasch analysis to reflect their difficulty. The analysis includes standard errors of each estimate and reliability coefficients in order to calculate measurement precision. Test-retest reliability was assessed by readministering the questionnaire to 30 patients after 3 or 4 weeks (ICC = 0.98) ( ). established the sensitivity and validity of the VA LV VFQ-48 as a measure of low-vision rehabilitation outcomes. In this study, the questionnaire was administered to 71 veterans (individuals with low vision and legally blind), before and 4 months after receiving rehabilitation. The results showed improved person measures postrehabilitation in legally blind patients. On the other hand, patients with less severe VI (near normal visual acuity) seemed to remain constant, before and after rehabilitation. The instrument appeared to be a sensitive measure for those patients with severe VI. As the VFQ has been Rasch analysed, it is possible to select only some of the items, and then the questionnaire responses can be scored using previously published algorithms ( ). Based on this, a 15-item ‘near vision’ questionnaire was devised (NV-VFQ-15) by selecting appropriate items from the VFQ-48 questionnaire for a study which assessed the effectiveness of portable electronic vision enhancement systems (p-EVES) in addition to optical aids for patients seen in a hospital clinic ( ).

designed a visual function questionnaire (the Activity Inventory [AI]) which has the feature that each patient responds to an individually tailored set of questions. The questionnaire is divided into three objectives (daily living, social interactions and recreation), under which lie 50 goals and each goal (e.g. cooking) has a collection of tasks (e.g. reading a recipe). The patients have to rate the importance and difficulty of each goal as well as the difficulty of the tasks that serve these goals. The result of the patient’s answers provides a functional history and the data needed to estimate the patient’s visual ability. The reasoning for designing this questionnaire is interesting because it is true that when using a fixed-number-of-items questionnaire, some of the questions are not relevant to some patients (e.g. respondents are allowed to skip items based on responses to screening questions, or by permitting them to respond that the item is not applicable or that they do not do the activity described for reasons other than vision). The skipped answers lead to difficulty handling these missing data in the final analysis, particularly when using conventional Likert-type scoring algorithms. Although Rasch analysis can overcome this problem, the authors argue that it would be even more reasonable to use a tailored questionnaire for each patient in addition to Rasch analysis.

The AI can be helpful in clinical practice to address goal-oriented components of the rehabilitation plan, as a tool to plan rehabilitation, evaluate patient progress and measure outcomes. The key is that it is possible to use this questionnaire to obtain specific information for a particular patient and plan and adapt the rehabilitation according to the patient’s priorities and goals.

Other Vision-Related Questionnaires

There are other instruments described in the literature, such as the Visual Function Self-report (VF-14) ( ), the Activities of Daily Vision Scale ( ), the Visual Activities Questionnaire ( ), the National Eye Institute Visual Function Questionnaire (NEI-VFQ) ( ), the Visual Disability Inventory ( ), the Visual Disability Assessment ( ), the Michigan Commission for the Blind Functional Assessment Scale ( ), and the Functional Assessment Self-Report Inventory ( ).

Some questionnaires use a mixture of self-report, and of the clinician grading the performance of the individual whilst actually carrying out some of the tasks (e.g. Functional Low-Vision Observer Rated Assessment (FLORA; ) and the Melbourne Low Vision ADL Index ( )).

However, some of the available instruments have been developed to evaluate medical treatments in clinical trials (e.g. the NEI-VFQ) or are disease-specific (e.g. MacDQoL ( ), the VF-14 ( ), or the Daily Living Tasks Dependent on Vision (DLTV) ( )). A number of these questionnaires are time consuming ( ) or have been designed for specific and likely treatable conditions ( ) and not to address the overall impact of visual loss in a general low-vision population ( ). The FLORA ( ), for example, was developed to assess treatment or vision restoration in people with ultra-low vision, blinded with retinitis pigmentosa.

As noted earlier, the original NEI-VFQ was validated for an adult population with common eye diseases ( ) but not specifically for a low-vision population. This questionnaire, however, has been used widely ( ) and for different age groups and degrees of visual loss ( ). An abbreviated version was also developed and validated (the 7-item NEI-VFQ) for use in evaluating the Welsh Low Vision Service ( ).

Questionnaires may also be validated for a specific population (e.g. hospital sample) and then used on a different one (e.g. community settings or primary care practice) ( ). The LVQOL questionnaire was validated for self-administration, as well as in-person or telephone interviews and was specially designed for a low-vision population. Generalisation and robustness are more likely when the questionnaires have been validated for several populations and different forms of administration.