Fundamental principles of large-scale social science

 

Proposed standards for governmental goals and reporting

 

New draft list

21 January 2006

 

Matt Berkley


 

Notes: 

 

Social science is about real life.  

Therefore, anyone can ask how far goals and reports correspond to real life.

The more important the project, the clearer the words need to be.

These standards are for specialists and others.

The intention is to provide a list of minimum standards for goals and reports.

They may help people outside government or social science to think about large-scale claims.  

No knowledge is required to ask questions about social science.  

People with real-life experience sometimes have a more realistic perspective than those without. 

Some standards are for economic goals and reports.

 

 

Overarching principles:

 

  1. Clarify the reasoning.  Distinguish between “ deduce”, ” infer”, ”hypothesise”, “judge”, “ speculate”.
    Look especially for points in the argument where these are implied or assumed.
  2. Clarify terms.  Only use a word if you a) understand precisely what it means to you and b) have a good idea what it will mean for your audience.   Watch out for excuses not to think this through.   If your reaction is “of course I know what it means”, this may indicate either that you have never stopped to think about it, and/or you may learn by considering what the word means to other people.   Errors of description can be worse than giving the wrong numbers. 

 

  1. Ask how the goal or measure might be meaningful in real life.   Take time to think about real-life people and situations.  
  2. Understand where the burden of proof lies.   A lazy thinker says the burden of proof is on the sceptic.  

 

 

Ask sensible questions about reliability:

 

  1. Thinking about real-life situations, estimate margins of error for data, giving a) reasons and b) possible real-life sources of random error and/or systematic skewing.
  2. Thinking about real-life situations, estimate margins of error for conclusions, giving a) reasons and b) possible real-life sources of random error and/or systematic skewing.  


Distinguish: 

 

  1. Survey answers and inferences as to what happened;
  2. Sample data and inferences on whole populations; 
  3. Population trends and aggregate trends for people; 
  4. Spending and income; 
  5. Spending and consumption; 
  6. Level and adequacy;
  7. Incidence and prevalence; 
  8. Prevalence and degree; 
  9. Trading activity and income (e.g. GDP per capita and average income);
  10. Income and profit;
  11. Prices and judgements as to which prices were relevant to people in the target group;
  12. Data on prices and judgements on the cost of living (relevant prices x needs)
  13. Consumption and personal resources; 
  14. Personal resources and available resources;
  15. Available resources and judgements on well-being. 

 

Credibility test:

 

  1. Take time to imagine self and/or others in real-life situation of subjects of research.  

 

 

Ethics test:

 

  1. Limit assumptions, methods and claims to those you would apply to yourself and/or people close to you if you were research subjects. 

    This principle applies to descriptions of
    a) data,
    b) inferences and
    c) conclusions. 

    If you would not tolerate a similar assumption or conclusion about you on similar evidence, throw out the goal, method or both.  

    This kind of test is perhaps best begun by considering
    i)   various kinds of extreme case (“what if X happened?”)
    ii)  what circumstances might produce extreme cases, and
    iii) whether these factors are worth bearing in mind for the present purpose.

 

Again, the burden of proof is not on the person who points out that factors might be important, but on the scientist who wishes to make a factual claim without excluding the possibility of alternative explanations of the numbers.  

 

For comparisons of people in different categories: 

 

  1. Use only descriptions which apply to distinct and meaningful groups.  

 

 

Distinguish:

 

  1. Statistical significance and real-life importance.   To argue from the first to the second requires more information.

 

 

 

 

 

33 Howard St, Oxford   OX4 3AY    England     

+44 (0)1865 726768

www.mattberkley.com       

matt@mattberkley.com  

 

 

[Note:  This document published on mattberkley.com 14 February 2006 is identical to the version of 21 January except for corrections to numbering scheme and fonts].