Evidence Corner

Welcome to the Evidence blog. I'll use this space to bring you a selection of material on evidence and policy I come across. I very much hope you will find it useful, and maybe even bring your own contributions.

Media fail to report crime survey data correctly (again).

Friday, January 20th, 2012

It’s that time of year again – British Crime Survey data are released, and the media misinterpret them. This time the problem is that old chestnut ’statistical significance’. Journalists have looked at the figure for all crime, seen that the survey reports a rise of 4%, and then written their copy. A typical line, this one from the BBC is ‘ Separate figures from the British Crime Survey (BCS) – measuring people’s experiences of crime – suggest crime went up by 4%. Not wanting to single out the BBC, the Telegraph did no better, reporting that ‘the British Crime Survey showed an increase’ [in crime]. The Telegraph went one better though. Having failed to interpret the BCS data correctly, their article goes on to cast doubt on the survey data (collected via face-to-face interviews with over 40,000 people), preferring instead to rely on what we can only characterise as ‘common sense’: ‘People are not stupid. They know that it stretches credulity for official figures to suggest that 2011 was a good year for crime-fighting.’. Who needs data when we can just rely on people’s impressions?

Top marks to the Guardian on this occasion though, who correctly reported that the survey figures show ‘no statistically significant change’ in overall crime.

Is the concept of statistical significance really so difficult to grasp that it eludes the cream of British journalism?

‘Independent’ Battle of Ideas Blog

Monday, December 5th, 2011

Throughout October and November, the Independent Online has been partnering with the Institute of Ideas Battle of Ideas festival to present a series of guest blogs from festival contributors. One of the contributors, Ben Pile, posted an intriguing article entitled Against evidence-based policy making. In it, he engages in one of those slightly annoying reductionist arguments where he rails against a position that no one apart from himself would support. However, he finally comes around to concluding something many would support, that an ‘….over-emphasis [my italics] on evidence in policy making is corrosive to a public debate about the values which inform the interpretation of evidence’. An interesting article and even more interesting set of responses can be found here

Using evidence to set healthcare priorities

Friday, September 23rd, 2011

The Nuffield Trust and the Health Services Management Centre (HSMC) at the University of Birmingham have carried out a national research study into how primary care trusts (PCTs) have gone about setting priorities for health care spending. Setting priorities in health maps the activities taking place across the NHS and explores the effectiveness of these practices within specific local contexts.

http://www.nuffieldtrust.org.uk/sites/files/nuffield/setting-priorities-in-health-research-report-sep11_0.pdf

Evaluating summer camps for children in need

Friday, September 23rd, 2011

Although not widely reported in the press, we noted with interest more policy on the hoof from Nick Clegg at the 2011 LibDem September conference.

The idea of providing catch-up summer classes for children “most in need” prior to starting secondary school may have some intuitive appeal, but where similar schemes have been properly evaluated, questions have been raised as to their effectiveness. Is it a good idea to invest £50m of tax-payers money in a scheme with no proven efficacy?

One of the most well-known evaluation studies (amongst students of social policy at any rate) is the Cambridge Summerville Youth Study:

http://www.childtrends.org/lifecourse/programs/cambridge.htm

It was a community based program for children and adolescents in eastern Massachusetts in the 1950’s. Children in the program received a counsellor who visited with them and their family twice a month. These counselors are on call for problems that the family may have been having and referred children to a variety of different programs including tutoring, medical treatment, psychiatric treatment, summer camps, Boy Scouts, YMCA, or other community programs.

Unusually, the programme was robustly evaluated using a randomized controlled trail methodology. The results from a 30 year follow-up showed conclusively that the programme had no impacts on juvenile arrest rates measured by official or unofficial records. The programme also had no impacts on adult arrest rates. There were no differences between the two groups in the number of serious crimes committed, age at when a first crime was committed, age when first committing a serious crime, or age after no serious crime was committed. A larger proportion of criminals from the treatment group went on to commit additional crimes than their counterparts in the control group.

Other studies have come up with similar findings. Generally they conclude that programmes like this, unless they are of very high quality (well-staffed by trained professionals), tend to act as ‘bad behaviour academies’, where like-minded children get together to exchange ideas on how to be anti-social.

Without wanting to dismiss the idea of summer schools purely on the basis of a flawed programme implemented a generation ago, it would be nice to see a commitment to ring-fence part of the proposed £50m budget to some robust (RCT) evaluation.

Simple guide to crime statistics for England and Wales

Tuesday, March 8th, 2011

Home Secretary Theresa May has asked National Statistician, Jil Matheson, to review how crime statistics are collected and published. The review aims to improve transparency and public trust in the numbers.
In response, we have published a short guide to the statistics as an example of what might be done to help the public gain a better understanding of crime and the criminal justice system.

Using the government’s own British Crime Survey and police recorded crime data, our booklet reports that crime in England and Wales fell by 50% between a peak in 1995 and 2009/10—nearly 10 million fewer crimes were committed in 2009/10 compared to 1995. The downward trend is continuing despite the recession, which many experts predicted would see a rise in the figures.

The Matrix summary also shows just how the police fare when it comes to detecting crimes. In 2009/10, 1.3 million offences were brought to justice, a rise of 19% on 1999/00—and this at a time when the numbers of offences being committed was falling. But still fewer than one in three crimes reported to the police end up with a suspect being identified. For all crimes, not just those reported to the police, detection rates fall to less than 13%.

Also striking is the fact that young people under the age of 20 account for a huge proportion of crime in England and Wales. Nearly 6% of all 17-year-olds have been found guilty or cautioned for committing offences. At a time when local authorities are looking to make savings by cutting services to young people, these figures suggest they might want to consider the impact cutting youth services may have on local crime rates.

Launch of rigorous evidence newsletter

Friday, February 4th, 2011

With the strap line ‘Distinguishing Effective Evidence-Based Programs from Everything Else’, the US Coalition for Evidence-Based Policy has launched its monthly newsletter. Circulated as an e-newsletter, it summarizes key findings from high quality (the focus is on well-conducted RCTs) evaluations across all areas of social policy. The Coalition follows guidelines issued by the US National Academies on evaluation designs.

The newsletter provides results from the Coalition systematically monitoring the literature to identify all new RCTs – and other rigorous nonrandomized evaluations – published or posted on-line across all areas of social policy. It includes (i) summaries of the few findings identified by an expert panel as meeting the highest (Congressional “Top Tier” or near Top Tier) evidence standards; as well as (ii) summaries of promising findings that are not yet ready for Top Tier consideration (e.g., due to only short-term follow-up). The Coalition views these latter findings – promising but not conclusive – as valuable for identifying programmes that merit testing in more definitive RCTs.

Looks like another valuable resource for the evidence-based policy community?

Beyond evidence-based policy: Outcome-based government

Friday, January 28th, 2011

The Centre for Social Justice this week published a report entitled Outcome-based Government: How to improve spending decisions across government.

In what should be essential reading for anyone in the business of producing evidence and analysis for policy, the report sets out how both local and national government needs to up their game when it comes to applying the principles of evidence-based policy. As an example of the failure of government to use evidence and analysis adequately, it quotes the Public Accounts Committee as observing: ‘The Government spends £1.2 billion a year on measures aimed at tackling problem drug use, yet does not know what overall effect this spending is having.’

The report provides an excellent analysis of why evidence has not been applied consistently to the development and evaluation of policy. One observation has particular resonance for anyone who has worked in government research:

‘The late involvement of analysts in policy development led one civil servant to describe the process as “policy-led evidence making”, with the methodology often used to support ministerial initiatives rather than appraise policy options.’

As with any good analysis, the report offers a menu of practical solutions to the problem. Top of the list comes the need for government to articulate a specific set of outcomes that define its priorities. It’s all very reminiscent of the line argued in this blog of the value for local and national government of having a clear theory of change for policy interventions. Having robust empirical evidence for links between outputs and outcomes is an essential pre-requisite to effective monitoring and evaluation.

Whilst the report recognises the value of methods described in the Treasury’s Green Book, it notes that in too many cases, the guidance is not followed consistently. As anyone who has discussed the ROAMEF model with policy makers will know, in cases where its principles are applied, it is often too late after key policy decisions have been made.

Among the twenty, very practicable, recommendations the report makes, perhaps the most standout (for a right of centre think tank) is the establishment of a new, independent body to scrutinise the extent to which policy decisions in local and national government are adequately informed by robust analysis. The proposed new body, The Office of Spending Effectiveness (OSE), would be responsible for setting standards of decision making, ensuring policy was adequately informed by evidence-based business cases, and evaluated through the collection and analysis of robust cost and outcome data. Such a scrutiny function could potentially make great strides towards delivering evidence-based policy.

Whilst you may not agree with all of the proposals made in the report, it certainly reads like a refreshing new approach to the perennial problem of how to get evidence consistently integrated into the process of government, both locally and nationally. A valuable contribution to the debate?

Evidence and employment policy

Friday, January 21st, 2011

Picking up the theme of evidence in the area of employment policy from the previous post (see below), the impressive ‘twenty-first floor’ site explores the evidence underpinning a Coalition announcement of changes to employment protection legislation. The proposals aim to extend the period after starting work within which employees cannot make claims of unfair dismissal from one to two years, and to require workers to post a fee when lodging an employment tribunal claim.

In essence, the theory of change underpinning this move is that giving employers greater flexibility to dismiss employees will stimulate the jobs market.

So what is the evidence? Two sources are relevant: First, the CBI reported findings from an employer survey showing that 46% felt employment tribunals were ineffective, and 50% claimed to have seen a rise in the number of vexatious claims. Second, a Trade Union Congress sponsored report, ‘Flexible with the Truth’, looks in detail at the relationship between labour market flexibility (LMF) and economic performance. It concludes the evidence that relaxing employment protection laws stimulates economic performance is equivocal at best. To quote: ‘…it is unlikely that a substantial reduction in EP …… would deliver drastically enhanced labour market performance’.

Author of the 21st Floor article Keir Liddle declines to take a definitive position on the grounds of insufficient evidence. How would you call it?

evidence about evidence-based policy

Monday, January 17th, 2011

The Centre for Science and Policy, University of Cambridge is launching a consultation to identify the key research questions on the relationship between science and policy.

The work is predicated on the assumption that we know very little about the relationship between the production of evidence and the making of policy. To date, research on integrating evidence into the policy making process has focussed either on the production of evidence, or on the mechanisms underlying the development of policy.

The Centre is taking up the challenge of building a research community whose remit will be to look closely at how scientific advice for policy audiences is produced. It wants to build alliances across academic disciplines, and also, crucially, with policy practitioners who work in this area, and also with those experts who have practical knowledge of what it is like to serve as expert advisers.

In the meantime, I came across this brief parliamentary interchange, care of the donpaskini blog:

Nic Dakin: To ask the Secretary of State for Work and Pensions what evidence he used to determine that planned changes to housing benefit for those out of work for over 12 months will increase employment levels.

Steve Webb: We did not make any specific assumptions about the impact on employment levels of this measure. Research shows that the reasons for long term unemployment are complex. However we believe reducing housing benefit after 12 months will provide an additional financial incentive for jobseekers to take up work.

Evidently some work to be done over at DWP on developing evidence-based policy!

Big society and healthcare: evidence for the impact of public involvement

Friday, January 7th, 2011

Big Society has become a signature policy of the current administration. While simple definitions are elusive, David Cameron has variously described it in terms of liberalism, empowerment, freedom and responsibility. In the context of delivering health and social care, an Institute for Government seminar concluded that health professionals will have to accept the public’s growing role in determining their own service needs.

But what is the evidence for greater public involvement improving services? Two recent papers provide some interesting insights.

Martin Marshall and Vin McLoughlin of the Health Foundation claim the available evidence does not unequivocally support the assumption that patient choice is a significant driver of quality improvements. Because patients typically do not behave like rational consumers when it comes to their healthcare, current mechanisms for providing them with information on care quality is not proving to be a strong lever for change. Marshall and McLoghlin suggest that changing the way information is presented to the public could change relationships between patients and providers for the better, but is unlikely to lead to improvements in quality of care in the short term.

A systematic review from the RAND Corporation found that while it may have some intrinsic value, there is little evidence to suggest public involvement in healthcare activities has had any positive impact on decision-making or policy. The report suggests that greater public involvement in health decisions may even threaten equitable access to services. It concludes that we in fact know too little about how people become involved as active service users, how the healthcare system influences patient expectations, and the impact of social and political values.

While the rhetoric may be popular, it would appear there is still work to be done in building evidence of effective practice when it comes to involving the public in policy and practice in the healthcare field.


Warning: include(sidebar2.php) [function.include]: failed to open stream: No such file or directory in /home/10770/matrix2/www.matrixknowledge.co.uk/public_html/evidence/wp-content/themes/evidence/category-23.php on line 66

Warning: include(sidebar2.php) [function.include]: failed to open stream: No such file or directory in /home/10770/matrix2/www.matrixknowledge.co.uk/public_html/evidence/wp-content/themes/evidence/category-23.php on line 66

Warning: include() [function.include]: Failed opening 'sidebar2.php' for inclusion (include_path='.:/usr/share/pear') in /home/10770/matrix2/www.matrixknowledge.co.uk/public_html/evidence/wp-content/themes/evidence/category-23.php on line 66