External Validity
Methodologies and Research Design 18:
Research Design IV: External Validity and
Sampling
Objectives • to unpack different types of external validity
• to describe crucial issues in sampling - precision vs. representativeness vs. cost - probability vs. non-probability / 'judgement '
• to describe stages in the process of sampling, and the possible intrusion of ‘bias’
• to describe methods of probability sampling and methods of non-probability sampling
• to be able to estimate your desired sample size, - using rules of thumb - or charts (e.g. in deVaus)
• to describe the problem of non-response, and how to minimise it
• to consider methods used for ‘sampling’ and generalisability in qualitative / ‘flexible’ designs External Validity, or Generalisibility
Population Validity - generalisibility from selected sample of cases
to population of interest
Ecological Validity - generalisibility / applicable from particular • time • place / setting, incl. conditions of the research • event(s)
to other settings,
to other time-periods
to other “events”, OR
to other forms of the “treatment” (independent variable),
to other measure of the outcome (dependent variable)
(1) Random Sampling for Representativeness
Basic trade-off: Validity vs. Cost
esp. External
esp. Population
Representativeness: avoiding (sampling) bias
Precision: source – sampling variation measure – standard error aims: to estimate to minimise
Cost (per interview) • contact costs
• fieldwork costs (incl. travel, postage)
• coding and analysis costs
Stages in the Social Process of Sampling
1. Define the General or Target Population
* 2. Specify the Working Population: “list”
3. Gain Access: to list, to members of population
* 4. Select the Sample: probability or non-prob. ?
* 5. Contact members of the Sample
* 6. Gain their
Research Design IV: External Validity and
Sampling
Objectives • to unpack different types of external validity
• to describe crucial issues in sampling - precision vs. representativeness vs. cost - probability vs. non-probability / 'judgement '
• to describe stages in the process of sampling, and the possible intrusion of ‘bias’
• to describe methods of probability sampling and methods of non-probability sampling
• to be able to estimate your desired sample size, - using rules of thumb - or charts (e.g. in deVaus)
• to describe the problem of non-response, and how to minimise it
• to consider methods used for ‘sampling’ and generalisability in qualitative / ‘flexible’ designs External Validity, or Generalisibility
Population Validity - generalisibility from selected sample of cases
to population of interest
Ecological Validity - generalisibility / applicable from particular • time • place / setting, incl. conditions of the research • event(s)
to other settings,
to other time-periods
to other “events”, OR
to other forms of the “treatment” (independent variable),
to other measure of the outcome (dependent variable)
(1) Random Sampling for Representativeness
Basic trade-off: Validity vs. Cost
esp. External
esp. Population
Representativeness: avoiding (sampling) bias
Precision: source – sampling variation measure – standard error aims: to estimate to minimise
Cost (per interview) • contact costs
• fieldwork costs (incl. travel, postage)
• coding and analysis costs
Stages in the Social Process of Sampling
1. Define the General or Target Population
* 2. Specify the Working Population: “list”
3. Gain Access: to list, to members of population
* 4. Select the Sample: probability or non-prob. ?
* 5. Contact members of the Sample
* 6. Gain their
References: Williams, M. (2000), ‘Interpretivism and Generalisability’, Sociology, 34, 2, 209-224. Williams (2000) considers: Proposition 1: Interpretivists do generalise and this is inevitable – though they may deny the possibility of generalisation, or ignore the issue. Geertz (1979), ‘The Balinese Cockfight’ – generalises from a particular ritual in Balinese life, to Balinese culture in general Fisher (1993) generalises from a typology of young ‘fruit machine’ gamblers to ‘what it is to gamble’ Williams (2000): examples of : • Moderatum generalisations: aspects of research situation are seen to be instances of a broader recognisable set of features... Roper I. and Cunningham I. (2003). ‘Working beyond Contract: Employer Views on Family Friendly Employment Relations’. MUBS / HRM. ‘… where approval is given, … this may be fragile: … may be given in principle, yet in terms of ‘practicality’, these same people may object.’ Scase and Goffee (1989) [ Bryman & Bell, 2003, p52]