The Perfectly Balanced Sample

In a well-known sketch in an episode of the 1980s political satire Yes Prime Minister the scheming senior civil servant Sir Humphrey Appleby deftly explained to his junior colleague Bernard Wooley how the outcome of opinion polls can be spun to support any position you care to take depending upon the choice of questions asked and highlighted later. Essentially, his advice boiled down to sardonic instruction “don’t mention the first five questions and publish the last one…”

Sir Bernard’s cynical manipulation of consultation and polling data to support “policy-based evidence-making” laid bare to television audiences a generation ago that the devil is very much in the detail of all opinion polls and that “a perfect balanced sample” is rarely what it seems.

The City of Edinburgh Council published this week the results of its latest consultation on the future of Edinburgh’s Winter Festivals. The Council’s official press release breathlessly proclaimed “overwhelming support for continuation of the city’s Christmas and Hogmanay events”, from the 8600 individuals and 35 organisations that participated.

We ourselves were one of those organisations, offering our own continuing caveated support for the Winter Festival’s overall contribution to the city’s vitality and vibrancy during the depths of the winter months.

The Leader and Deputy Leader of the Council welcomed the overall headline claim that there was an 87% and 86% approval rating for the Christmas and Hogmanay events, with Cllr Day commenting that the report showed a “clear desire to keep the winter celebrations.” But what happens when you dig a little deeper into the data?

Readers are told in an early paragraph that “The survey sought to establish support for having the two events”. The survey should, of course, have sought to establish “the levels of support” for the two events. A Freudian Slip that Sir Bernard would certainly not have approved of!

Of the 8614 individuals surveyed, just 2% of respondents (a mere 168 individuals) were residents of the city centre of Edinburgh, the area most directly impacted by the installation of these events. Widened out to the next three postcodes, that number only increases by a further 14% (1180 people). Meaning that 84% of the individuals who participated in this consultation are not directly or personally impacted by the imposition of the subject that they are commenting upon.

Suddenly that 86/87% approval rating starts to come into a sharper focus. Worryingly, the polling company itself acknowledges that:

“Edinburgh respondents were less positive than respondents from the rest of Scotland and the UK about the winter celebrations being welcoming, enjoyable, unique, affordable, beneficial to business and environmentally sustainable.”

They continue:

“When asked about previous celebrations, views were mixed. While many of the comments were positive, a larger number were negative. The main concerns focused on the central location and impact on residents.”

Even more damningly the consultants also noted:

“The majority saw the celebrations as being welcoming for tourists, but they were less likely to be welcoming to residents. Most agreed that they are good for businesses, but few perceived them as affordable or environmentally sustainable. The main reasons for not attending Christmas and Hogmanay celebrations in the past were overcrowding, ‘designed for tourists’ and too expensive.”

The consultants very helpfully broke this data down into more granular detail. Unfortunately, the most imposed-upon citizens, city centre residents, were not given their own column but barely one in three (35%) of residents surveyed from the city’s most central four postcodes believed these festive events are “welcoming to Edinburgh residents.” A whopping 88% of them felt that they were principally aimed at “tourists and other visitors.”

Other disappointing and negative factors were similarly highlighted deep in the appendices, including the respondents’ perceptions of how incredibly expensive this publicly subsidised Winter Festival event has become (the Council currently makes “£813,456 available annually for delivery of the two events”), potentially excluding families and individuals on low incomes:

Concerns were also raised about the event’s unsustainability and its negative environmental impacts:

Concerns about overcrowding, lack of local produce, the negative impacts on permanent businesses and residents, unnecessary barriers to those with disabilities and other issues were also mentioned in the report.

The consultants also acknowledge that direct engagement with Edinburgh’s individual businesses about the tangible impacts that the Winter Festival has on their activities “is very small”. Despite this, they were still confident enough to claim that the majority of businesses say the Winter “Festivals have increased their turnover.”

Following closely Sir Bernard’s mantra not to “mention the first five questions…” Paul Lawrence, the City’s Director of Place, has placed a report on the consultation before Councillors, for adoption next week by the relevant sub-committee. Among the desired outcomes for the Place Directorate the report states:

“There is a clear desire for a city centre activity for Christmas, which not only supports businesses by attracting people to the city centre, but also delivers the celebratory elements of Christmas, including a food and beverage offer and activities for families with an emphasis on affordability and environmental sustainability. A procurement exercise will be undertaken to appoint a producer to deliver this event, using the clear feedback from the consultation to ensure that it meets the needs of residents and businesses and is of appropriate scale.”

And:

“For wider winter activity, it is proposed to appoint an individual or organisation to a Winter Festivals Director role, reporting directly to the Council. They will have responsibility for developing an overall Winter Festival programme based on a theme of Light and incorporating the desired elements from the consultation: fireworks, lighting installations and live music. The inclusion of local artists, communities, performers and creatives is essential in the development of this programme.”

The future guarantee of more space for local artists and performers, the recognition of the significant financial barriers presented by the high costs to participate in a publicly funded event and the acknowledgement of the evident unsustainability of certain aspects of the offer are all very welcome.

But we also call for a properly independent investigation and analysis of the actual tangible benefits and competitive threats this heavily subsidised event provides to local permanent businesses. We are also calling for a more sincere acknowledgement of and actioning upon the concerns of those city residents who are being asked to bear the brunt of any negative impacts.

98% of respondents to this survey offered their opinion about an event that they travel to, an event that is imposed upon the other 2% of respondents. From such poor polling weighting, even Sir Humphrey would struggle to argue with a straight face that the heralded “87% support” strapline is a “perfectly balanced sample”!