Why are historic censuses not free and open to all?

POSTED ON July 16, 2021 BY Prof Richard Rodger

A guest blog from noted urban historian and Cockburn trustee Prof Richard Rodger questioning why Scotland’s historic censuses remain behind a costly paywall

Have you come across such terms as ‘Open Data’, ‘Open Access’ or even ‘Open Government’? ‘Accountability’ and ‘transparency’ are bandied about by local government and Scottish government bodies. But saying it does not mean doing it. Repeating it is not enough. Good intentions need execution.

So you might think that when in 2019 the Scottish Government published A Culture Strategy for Scotland it might have had something to say something about ‘Archives’ – the written cultural gold mine of our nation. But no! Not a single reference to ‘archives’, ‘historical records’, or the National Records of Scotland in A Culture Strategy for Scotland.

So is that it? Don’t we value our records any longer? No culture strategy for written assets? Or worse, are our top civil servants in the National Records of Scotland so weak they meekly waived through a document with no reference to the historical assets they are there to protect. And should not our history professors and other senior cultural figures have something to say about this? That’s right. Register House, a purpose built repository for the documentary record of Scotland of such importance that it was the first thing to be built in the New Town is not worth a single mention in a major policy document. Shameful.

But that’s not all. You might think that with a commitment to Open Data, Open Access, blah blah …from the Scottish Government and Local Councils we would be able to consult Census records from 150 years ago. You might think that having paid your taxes, local and national, that you would be able to consult the digital version of the Census online. Locked down as a nation we might have been able to consult historical records. Instead we were locked out! Or do I mean lucked out?!

Well local historians will tell you – you can consult records on line. True – up to a point. You can pay through Ancestry.com and Findmypast and you can consult individual records on a pay per view basis. This is OK if you (a) have the money (b) have some initial leads (c) accept the errors and imperfections in the transcribed records (d) want to consult English census records, because they include 1911 (unlike Scotland) and also provide digital versions of the originals (unlike Scotland). You also need to be vigilant. The quality of the farmed out transcription is really dreadful. For example, in Edinburgh, ‘Bucclench’ is used For Buccleuch Street look for Bucclench; for Darnaway Street look for ‘Warnaway.’ OK – not too difficult to spot if you are from Edinburgh. Here was an excellent chance to farm out checking personal names and Scottish place names to local historians and genealogists. Dereliction at the contractual level, and at the implementation stage.

But historical research is more than pursuing individuals. It is about understanding society. What was the average family size; most common occupations; what percentage of women were household heads; and many other interesting questions? These cannot, I repeat, CANNOT be addressed with current arrangements and so access to the Census actively impairs an understanding of Scotland and Scottish society.

You can download UK Census data from this digital census databank. But there is an approval system to see the entire dataset. Two ‘safeguarding’ tests have to be passed. One is whether access to names and addresses is prejudicial to ‘personal’ interests. This is rarely invoked for the historical census. The other is safeguarding the ‘commercial’ interests at stake!! In the case of the historical census, ‘commercial interests’ are cited in order to protect the digital investment made under a contractual agreement between The National Archives (TNA, Kew, London) and DC Thomson Media, to which the National Records of Scotland (NRS) at General Register House, Edinburgh is party.

Yes, you did not mis-read that: access to the UK census is dependent, admittedly at one remove, on the owner of those other national institutions, The Beano, The Dandy, and the Sunday Post newspaper and its national treasures, “The Broons” and “Oor Wullie”.

Nowadays, as in the past, we are routinely asked for our ‘name and address’ for almost all transactions and relationships. Linking common data fields is a key tool to enriching historical analysis and to forfeit ‘name’ and ‘address’ is to diminish the productivity of many non-Census sources which can be nominally linked. That happens everyday to us so where is the concern for historical records say between 1851 and 1911? And rather than safeguarding commercial interests should not the contract to digitise the Census be safeguarding our historical interests?

In a recent paper I have written about these issues, “Making the Census Count: Edinburgh 1760-1900″ for the Journal of Scottish Historical Studies, three examples are used to show how access to names and addresses enriches historical understanding. One example utilises the Census of Edinburgh to identify the ‘Top 10’ popular male and female names in 1861 (see below). This is impossible without access to the full digital census. You will never have seen anything like this analysis. Another example considers Alfred McNightingale’s addresses over forty years to understand his employment patterns. A third illustration considers a selection of Old Town Closes in 1861 and 1891 in order to reveal how household size and composition was affected, if at all, by the demolitions initiated under the ‘Improvement’ Act of 1867. None of these strands, and many others, can be undertaken without access to nominal records. I am fortunate to have been given access to the names and addresses. But why should others be excluded?

The top ten most popular forenames in Edinburgh in 1861

The interests of local historians, genealogists, students and academic historians are adversely affected by current arrangements. The inability to harvest quantities of data from the Census impairs social, economic, cultural, demographic and social science history generally. It limits local interests and civic engagement by denying a legitimate interest in local themes and general trends. School projects and undergraduate dissertations are limited and, at a time when digital resources are a godsend, the general public cannot engage with Census data to advance their own historical interests. Of greater significance perhaps are questions such as: How can the histories of a nation be written without access to the core data – the inhabitants and their locations?

Support the Cockburn

Donate
Join Us

Why are historic censuses not free and open to all?

POSTED ON

A guest blog from noted urban historian and Cockburn trustee Prof Richard Rodger questioning why Scotland’s historic censuses remain behind a costly paywall

Have you come across such terms as ‘Open Data’, ‘Open Access’ or even ‘Open Government’? ‘Accountability’ and ‘transparency’ are bandied about by local government and Scottish government bodies. But saying it does not mean doing it. Repeating it is not enough. Good intentions need execution.

So you might think that when in 2019 the Scottish Government published A Culture Strategy for Scotland it might have had something to say something about ‘Archives’ – the written cultural gold mine of our nation. But no! Not a single reference to ‘archives’, ‘historical records’, or the National Records of Scotland in A Culture Strategy for Scotland.

So is that it? Don’t we value our records any longer? No culture strategy for written assets? Or worse, are our top civil servants in the National Records of Scotland so weak they meekly waived through a document with no reference to the historical assets they are there to protect. And should not our history professors and other senior cultural figures have something to say about this? That’s right. Register House, a purpose built repository for the documentary record of Scotland of such importance that it was the first thing to be built in the New Town is not worth a single mention in a major policy document. Shameful.

But that’s not all. You might think that with a commitment to Open Data, Open Access, blah blah …from the Scottish Government and Local Councils we would be able to consult Census records from 150 years ago. You might think that having paid your taxes, local and national, that you would be able to consult the digital version of the Census online. Locked down as a nation we might have been able to consult historical records. Instead we were locked out! Or do I mean lucked out?!

Well local historians will tell you – you can consult records on line. True – up to a point. You can pay through Ancestry.com and Findmypast and you can consult individual records on a pay per view basis. This is OK if you (a) have the money (b) have some initial leads (c) accept the errors and imperfections in the transcribed records (d) want to consult English census records, because they include 1911 (unlike Scotland) and also provide digital versions of the originals (unlike Scotland). You also need to be vigilant. The quality of the farmed out transcription is really dreadful. For example, in Edinburgh, ‘Bucclench’ is used For Buccleuch Street look for Bucclench; for Darnaway Street look for ‘Warnaway.’ OK – not too difficult to spot if you are from Edinburgh. Here was an excellent chance to farm out checking personal names and Scottish place names to local historians and genealogists. Dereliction at the contractual level, and at the implementation stage.

But historical research is more than pursuing individuals. It is about understanding society. What was the average family size; most common occupations; what percentage of women were household heads; and many other interesting questions? These cannot, I repeat, CANNOT be addressed with current arrangements and so access to the Census actively impairs an understanding of Scotland and Scottish society.

You can download UK Census data from this digital census databank. But there is an approval system to see the entire dataset. Two ‘safeguarding’ tests have to be passed. One is whether access to names and addresses is prejudicial to ‘personal’ interests. This is rarely invoked for the historical census. The other is safeguarding the ‘commercial’ interests at stake!! In the case of the historical census, ‘commercial interests’ are cited in order to protect the digital investment made under a contractual agreement between The National Archives (TNA, Kew, London) and DC Thomson Media, to which the National Records of Scotland (NRS) at General Register House, Edinburgh is party.

Yes, you did not mis-read that: access to the UK census is dependent, admittedly at one remove, on the owner of those other national institutions, The Beano, The Dandy, and the Sunday Post newspaper and its national treasures, “The Broons” and “Oor Wullie”.

Nowadays, as in the past, we are routinely asked for our ‘name and address’ for almost all transactions and relationships. Linking common data fields is a key tool to enriching historical analysis and to forfeit ‘name’ and ‘address’ is to diminish the productivity of many non-Census sources which can be nominally linked. That happens everyday to us so where is the concern for historical records say between 1851 and 1911? And rather than safeguarding commercial interests should not the contract to digitise the Census be safeguarding our historical interests?

In a recent paper I have written about these issues, “Making the Census Count: Edinburgh 1760-1900″ for the Journal of Scottish Historical Studies, three examples are used to show how access to names and addresses enriches historical understanding. One example utilises the Census of Edinburgh to identify the ‘Top 10’ popular male and female names in 1861 (see below). This is impossible without access to the full digital census. You will never have seen anything like this analysis. Another example considers Alfred McNightingale’s addresses over forty years to understand his employment patterns. A third illustration considers a selection of Old Town Closes in 1861 and 1891 in order to reveal how household size and composition was affected, if at all, by the demolitions initiated under the ‘Improvement’ Act of 1867. None of these strands, and many others, can be undertaken without access to nominal records. I am fortunate to have been given access to the names and addresses. But why should others be excluded?

The top ten most popular forenames in Edinburgh in 1861

The interests of local historians, genealogists, students and academic historians are adversely affected by current arrangements. The inability to harvest quantities of data from the Census impairs social, economic, cultural, demographic and social science history generally. It limits local interests and civic engagement by denying a legitimate interest in local themes and general trends. School projects and undergraduate dissertations are limited and, at a time when digital resources are a godsend, the general public cannot engage with Census data to advance their own historical interests. Of greater significance perhaps are questions such as: How can the histories of a nation be written without access to the core data – the inhabitants and their locations?

Support the Cockburn

Donate
Join Us