Ricky Taylor and Lan-Ho Man: Using administrative data to evaluate the Troubled Families Programme
20 January 2020

The Troubled Families Programme is one of the biggest social policy programmes in government. It costs £920m, is run across England in 149 upper tier local authority areas and provides services to 400k families with multiple, high-cost problems including worklessness, domestic abuse, mental and physical ill health, school truancy and anti-social behaviour and offending.

How did we evaluate the programme?

The Troubled Families Analysts were given a project brief to include all upper tier local authorities and to carry out a robust impact evaluation (with a counterfactual) and cost benefit analysis – all within a limited budget.

Impact evaluations like ours aren’t that common. A National Audit Office study (2013) reviewed 6,000 analytical reports produced by government and found that only 305 of these were impact evaluations (5%). Of these studies only 126 (2%) were assessed as good quality in that they included a counterfactual or comparison group, and only 70 (1%) included a cost benefit analysis.

The challenging brief for an evaluation of this scale was a catalyst for innovation as traditional data collection methods, such as surveys, would have been prohibitively expensive.

We decided to use nationally held administrative data to conduct an impact analysis of the programme. The project included the collection of personal identifiers, including names, dates of birth and addresses of individuals and families either on the programme or those awaiting a place (these formed a comparison group for the evaluation) and matching this information to various datasets including the Police National Computer, National Pupil Database and Work and Pensions Longitudinal Study. 

We then used propensity score matching to compare outcomes of those on the programme with those in the comparison group. Other policy programmes had been evaluated in a similar way, but never to the same scale, or in combination with so many government databases or other sources of information.

What challenges needed to be overcome to evaluate the programme in this way?

It took two years to navigate the data protection legislation, identify and agree the legal gateways to share and use the data and get data sharing agreements in place.

Initially we had to work closely with the Information Commissioner’s Office to ensure we had considered the rights of data subjects. We considered the ethical, legal and data security issues in full and set these out in data sharing agreements and a full Data Privacy Impact Assessment. These documents were agreed with our key partners before we could press ahead.

We learned along the way that innovation brings risk which can make people (quite rightly) wary. This meant keeping key stakeholders (such as local authorities) updated, outlining the benefits and providing reassurance on the risks.

What other challenges did we face?

We spent quite some time quality assuring our data before carrying out our impact evaluation.  

Once we started receiving data, we realised quickly that there was a lot of work to do on data quality and our ONS colleagues (acting as a trusted third party) worked with local authorities to improve their data. Our Technical Advisory Group were also keen to ensure that the programme and comparison group had been selected in a similar way, as this could cause problems when attributing impacts to the programme itself (establishing a causal link).

Was the project worth the effort?

Eventually our efforts paid off. The project saw one of the biggest data sharing exercises in Whitehall and we have data on 864k individuals in 250k families on the programme. We have processed several terabytes of data with rigorous security and data processing controls to ensure that the data is protected.

The national impact evaluation has been well received. It is considered robust and has successfully demonstrated that the programme has had a positive impact, showing that, after two years:

• the proportion of children going into care reduced by a third; juvenile convictions were down by 15 per cent; juvenile custody was down by 38% and adult custody was down by a quarter; and
• the number of working age Jobseeker’s Allowance claimants reduced by 11%.

The cost benefit analysis found for every £1 invested by central government in the programme there was an economic and social (or public value) benefit of £2.28, and a fiscal benefit of £1.51.

These are impressive rates of return on a social policy programme and something to celebrate. In fact, at the start of this year, the Communities Secretary, Rt Hon Robert Jenrick, announced a one year extension to the programme and said the Department wanted to build on the success of the programme. We would not have been able to demonstrate this success without a robust impact evaluation and the continued support of our colleagues in central and local government.

Click here if you want to find out more about the evaluation and the key findings.

Written by
Ricky Taylor and Lan-Ho Man
Department for Communities and Local Government