principles of large-scale social science
standards for governmental goals and reporting
science is about real life.
can ask how far goals and reports correspond to real life.
important the project, the clearer the words need to be.
standards are for specialists and others.
intention is to provide a list of minimum standards for goals and reports.
help people outside government or social science to think about large-scale
knowledge is required to ask questions about social science.
real-life experience sometimes have a more realistic perspective than those without.
standards are for economic goals and reports.
- Clarify the reasoning.
Distinguish between “ deduce”, ”
infer”, ”hypothesise”, “judge”, “ speculate”.
Look especially for points in the argument where these are implied or
- Clarify terms. Only use a
word if you a) understand precisely what it means to you and b) have a
good idea what it will mean for your audience. Watch out for
excuses not to think this through. If your reaction is “of
course I know what it means”, this may indicate either that you have never
stopped to think about it, and/or you may learn by considering what the
word means to other people. Errors of description can be worse
than giving the wrong numbers.
- Ask how the goal or measure might
be meaningful in real life. Take time to think about real-life
people and situations.
- Understand where the burden of
proof lies. A lazy thinker says the burden of proof is on the
sensible questions about reliability:
- Thinking about real-life
situations, estimate margins of error for data, giving a) reasons and b)
possible real-life sources of random error and/or systematic skewing.
- Thinking about real-life
situations, estimate margins of error for conclusions, giving a) reasons
and b) possible real-life sources of random error and/or systematic
- Survey answers and inferences
as to what happened;
- Sample data and inferences on
- Population trends and aggregate
trends for people;
- Spending and income;
- Spending and consumption;
- Level and adequacy;
- Incidence and prevalence;
- Prevalence and degree;
- Trading activity and income
(e.g. GDP per capita and average income);
- Income and profit;
- Prices and judgements as to
which prices were relevant to people in the target group;
- Data on prices and judgements
on the cost of living (relevant prices x needs)
- Consumption and personal
- Personal resources and
- Available resources and
judgements on well-being.
- Take time to imagine self
and/or others in real-life situation of subjects of research.
- Limit assumptions, methods and
claims to those you would apply to yourself and/or people close to you if
you were research subjects.
This principle applies to descriptions of
b) inferences and
If you would not tolerate a similar assumption or conclusion about you on
similar evidence, throw out the goal, method or both.
This kind of test is perhaps best begun by considering
i) various kinds of extreme case (“what if X happened?”)
ii) what circumstances might produce
extreme cases, and
iii) whether these factors are worth bearing in mind for the present
Again, the burden of proof is not on the person who points
out that factors might be important, but on the
scientist who wishes to make a factual claim without excluding the possibility
of alternative explanations of the numbers.
comparisons of people in different categories:
- Use only descriptions which
apply to distinct and meaningful groups.
- Statistical significance and
real-life importance. To argue from the first to the second
requires more information.
33 Howard St, Oxford OX4 3AY England
[Note: This document published on mattberkley.com 14
February 2006 is identical to the version of 21 January except for corrections
to numbering scheme and fonts].