“Sensitive” ScotGov documents cast new light on Ministers’ & SPADs’ FOI role

Newly released documents, obtained as a result of two appealed FOI requests, show that Ministers and Special Advisers have been deeply involved in the withholding of information from the public, sometimes despite the protests of civil servants – including those in the government’s own FOI Unit.

As this is rather a long post I’ve broken it into sections.


Requests for Information

The New Documents

Inside the Briefing Documents

Some Final Thoughts

Download the Documents


Back in 2017 journalists from across Scotland signed an unprecedented open letter condemning the Scottish Government’s handling of FOI requests. The letter was published by CommonSpace and The Ferret on 1 June, 2017 – it lead to a debate in parliament (following which a motion criticising the Scottish Government was passed unanimously) and, ultimately, the commencement of a high-level investigation from the Scottish Information Commissioner (SIC).

That investigation found that the Scottish Government had been operating a two-tier FOI system in which requests from journalists (as well as political opponents) were expressly subject to additional ‘clearance’, a practice with no basis in law. The role of Special Advisers was also scrutinised. The government was required to make changes to various aspect of its policies (the first attempt at which was rejected by the SIC).

You can read the full intervention report here.

Requests for Information

On the 3rd of August 2018 I submitted a request to the Scottish Government seeking the release of the following information:

All internal communication involving Scottish Ministers, Special Advisers and communication staff in relation to:

  1. The letter, published 1/6/17, raising concerns of journalists regarding the Scottish Government’s FOI policies
  2. The SIC intervention into the Scottish Government’s handling of FOI requests, announced in November 2017 and for which the final report was published on 13/6/18

The time frame for this request is 01/06/2017 to 20/06/2018.

(NB: I had originally sought to include senior Civil Servants in this request but was advised that the government would refuse to respond to this request as it would apparently breach the cost threshold for FOI requests)

The government responded to this request on the 31st August 2018, dealing with each part individually.

In relation to part one – regarding the open letter from Scottish journalists – they said this:

While our aim is to provide information whenever possible in this instance the Scottish Government does not have the information you have requested because there was a nil return on thorough searches conducted.

And for part two – regarding the Scottish Information Commissioner’s investigation into the government – they said:

While our aim is to provide information whenever possible in this instance the Scottish Government does not have the information you have requested because there was a nil return on thorough searches conducted.

This is a formal notice under section 17(1) of FOISA that the Scottish Government does not have the information you have requested.

Some of the information you have requested is available from the Scottish Government website https://beta.gov.scot/publications/foi-18-01655 . Under section 25(1) of FOISA we do not have to give you information which is already reasonably accessible to you. If, however, you do not have internet access to obtain this information from the website listed then please contact me again and I will send you a paper copy.

So, aside from the contents of some correspondence with the Scottish Infomation Commissioner (which is the topic of the FOI response linked to above) the Scottish Government claimed to hold absolutely no internal communications in relation to either of the topics featured in my requests. Nothing at all.

Unsurprisingly I found this difficult to believe. So, on the 6th September 2018, appealed both decisions, pointing out that: “I don’t think that any reasonable person is going to believe that ministers, their special advisers and communications staff did not internally discuss either the letter in question or the SIC intervention over the course of a full calendar year.” I also asked the government to be clear about whether or not such information had simply been deleted.

I received a response to these appeals on the 2nd October 2018 in which the government once against insisted that it simply did not hold any of the information I had requested, even when “deleted items” had been searched. Once again, I simply didn’t believe that this could possibly be accurate.

Nil response after review
One of the two review responses confirming the government’s position – that it held nothing in relation to my requests. The second review response used a near-identical form of words.

But this wasn’t the end of the story because, when I submitted the above requests to the Scottish Government on the 3rd of August, I also submitted two parallel requests. I asked for exactly the same information but added 5 words which, it turns out, were absolutely crucial:

All internal communication held by the FOI unit involving Scottish Ministers, Special Advisers and communication staff in relation to:

  1. The letter, published 1/6/17, raising concerns of journalists regarding the Scottish Government’s FOI policies
  2. The SIC intervention into the Scottish Government’s handling of FOI requests, announced in November 2017 and for which the final report was published on 13/6/18

The time frame for this request is 01/06/2017 to 20/06/2018.

To be absolutely clear, this specification should not have been necessary. The FOI Unit is part of the government so the original requests should have caught any information held by this particular department. But, for some reason, they didn’t.

The government responded to this request on the 3rd September 2018. Having previously told me that no material was held, TWO-HUNDRED AND THREE PAGES of material were suddenly made available to me.

Of course the government had redacted a lot of the information so I submitted an appeal, arguing that the exemptions had been applied over-zealously and that the public interest would clearly favour greater disclosure given the nature and seriousness of this issue.

The government (finally) responded to this request for a review on the 22nd October 2018. Here is the full response:

20181022 FOI_181_2110 and FOI_18_02111 review response - final - GWC-page-001

20181022 FOI_181_2110 and FOI_18_02111 review response - final - GWC-page-002

20181022 FOI_181_2110 and FOI_18_02111 review response - final - GWC-page-003

20181022 FOI_181_2110 and FOI_18_02111 review response - final - GWC-page-004

20181022 FOI_181_2110 and FOI_18_02111 review response - final - GWC-page-005

20181022 FOI_181_2110 and FOI_18_02111 review response - final - GWC-page-006

20181022 FOI_181_2110 and FOI_18_02111 review response - final - GWC-page-007

I think a few things are worth noting here:

  1. I make extensive use of FOI and I can just about follow all of the information provided. This “final” response has been made incredibly complicated by the government’s earlier actions, and I wonder whether someone coming to FOI for the first time would have had the slightest chance of getting through all this (I certainly couldn’t have a couple of years back).
  2. Just look at the scale of the corrections required here, every one of which is a sign that the original response to me did not comply with the law.
  3. A significant quantity of new material – originally deemed too sensitive for publication – has now been released…

The New Documents

Of the information now available to me about the government’s handling of this (ongoing) controversy, by far the most interesting is a collection of documents labelled “OFFICIAL – SENSITIVE”. They are briefing documents, issued to Ministers and Special Advisers in advance of their interviews with the Scottish Information Commissioner.

These documents provide “information about individual cases” which investigators had indicated they may wish to ask about and provide clear timelines for a number of contentious FOI cases.

You can view each of the documents as they were provided to me by clicking on the appropriate link at the bottom of this post at the bottom of this post; however, there are some specific parts of these documents that are particularly note-worthy, with screenshots of these sections – and commentaries – available below.

But before we get to that there are a few of things to bear in mind:

  1. These documents were created for the specific purpose of preparing individuals for interviews with the Scottish Information Commissioner. The documents make clear that the contents are the FOI Unit’s assessment of several FOI cases identified as being of interest to the Scottish Information Commissioner.
  2. These briefings do not seem to have been provided to all interviewees, a full list of which is also available: List of SIC interviewees. It seems that they were provided to individuals who would be asked to answer questions on cases identified by the SIC.
  3. Even redacted versions of these documents do not seem to have been included in the original FOI response, meaning that the government has now – having been challenged – accepted that it did not apply the law correctly in completely withholding this “sensitive” information.
  4. The documents contain explicit examples of SPADs blocking the release of information and of the Scottish Government knowingly and deliberately breaching FOI law.
  5. These documents don’t seem to have been released when communication between the Scottish Government and the SIC was requested at an earlier stage, which suggests that they may not have been available to investigators. Although they would have had access to the source material used to generate these documents, it is remarkable to see just how candid the government’s own officials were about failures to uphold FOI law. Consequently, it is also worth asking whether knowledge or sight of these documents might have altered either the line of questioning during the interviews or even the final contents of the commissioner’s overall intervention report into the Scottish Government’s handling of FOI.

Inside the Briefing Documents

John Swinney

The briefing provided to John Swinney concerns three FOI requests – in all three cases I was the person seeking the information. The document is incomplete as the government has withheld some information, specifically the “FOI Unit assessment of potential matters for discussion” – this will soon be the subject of a further appeal to the Scottish Information Commissioner.

The first of these requests (FOI/16/02132) concerned the Scottish Government’s standardised testing system and highlights the significant role that Special Advisers appear to have played in an (ultimately unsuccessful) attempt to keep material out of the public domain:

Swinney 1

Swinney 1-added

It is clear from this information that, had it not been for the influence of Special Advisers, the information from the ‘risk register’ in question would have been released to me in response to my initial request. It is worth noting that this material was, eventually, released to me after I took the case to the Scottish Information Commissioner, who was critical of the government’s handling of this case, confirming that the government’s application of the law had been incorrect (and that the original case handler, not the Special Adviser, had been correct).

The second case (FOI/17/01247) in Swinney’s briefing concerns a request in which I sought records of meetings and communication between the Scottish Government and Teach First, an issue which generated headlines in national media. In this instance, John Swinney personally intervened in an attempt to prevent the release of correspondence between the government and Prince Charles (as well as his private secretary, Mark Leishman). The email from Swinney’s private office stated that he would “prefer” is this information were withheld:

Swinney 2

You can read more about this matter here. Once again, it is important to note that this information was eventually released to me, leading to further scrutiny of both the Scottish Government and the Prince of Wales.

When asked about this matter during his interview with the Scottish Information Commissioner, Swinney blamed his Private Secretary, whose “unfortunate wording” had “caused concern”. The DFM claimed that he was simply concerned that officials had failed to “consider” the Royal Household exemption (when dealing with material from the Prince of Wales’ Private Secretary).

Swinney extract 1

By way of follow-up Swinney was asked whether he believed that the exemption did apply, but an answer to this question does not appear to have been forthcoming.

Swinney extract 2

The third and final (FOI/17/01769) case from Swinney’s briefing document concerns what is now sometimes referred to as a ‘meta-FOI’. In it, I asked the government to provide me with information about the role of senior civil servants, communications staff, special advisers and ministers in handing a number of my own FOI requests. It was, ultimately, the starting point for much of the current scrutiny of the role of Special Advisers and ministers in the FOI process.

Long story short: the government tried to withhold the key information and lost because the Scottish Information Commissioner was “unconvinced” by their attempts to apply an exemption.

Swinney 3

Download the Documents

Keith Brown

This briefing, like John Swinney’s, concerns three specific FOI responses. Interestingly, however, the “Potential matters for discussion sections” have not been redacted (suggesting that there is something particularly significant being withheld from the documents concerning the DFM).

The first case (FOI/17/00406) outlined in this briefing relates to the Enterprise and Skills Review, with the requester (a Lib Dem researcher) seeking copies of correspondence and a list of meetings between the Scottish Government and the Scottish Funding Council.

Brown 1

There are two clear issue here. The first is that information was withheld from the requester “in accordance with Stewart Maxwell’s views”. The other is that Keith Brown’s office appears to have sat on a response for nearly two months, despite the statutory response deadline have already been breached when they received the relevant documents on the 20th of March 2017. Though I can’t be sure, it seems to be the case that the requester did not submit an appeal against the withheld information – based on this document they certainly should have, but of course they couldn’t have known that at the time.

The second case (FOI/17/00844) dealt with in Brown’s briefing requested “a copy of the agreement which saw Transport Scotland/the Scottish Government buy Glasgow Prestwick Airport from Infratil for £1 in 2013, including any contract or letter of agreement which indicates the terms of the deal.”

The main issue with this case seems to have been the failure to respond within the statutory timescale (it was more than a month late) and this is certainly what the FOI Unit identified as the likely matter to be raised during the investigation, but the use of the National Security clause is also, at the very least, an interesting additional detail.

The third case (FOI/17/01567) referenced in the briefing to Keith Brown is a big one. It came from a Guardian journalist and focuses on information linking Prestwick Airport with Donald Trump, his businesses and the US Government.

The FOI Unit suggested that questions may be asked about delays in dealing with the case, and that may be true, but there’s another, arguably more pressing, issue on display in this particular instance.

Brown 2
NB: These are extracts from the same document but, as you can see from the dates above and below, part of the process in question is not shown here.

Brown 3

Obviously, the usual questions about Special Adviser influence (note the underlined part in the bottom box) and general FOI handling are as relevant as ever, but it is particularly concerning that, on two occasions, this FOI response was influenced without proper records being kept.

SPAD comments that are “not on file” are bad enough, but questions really must be asked about Brown’s role in answering both the initial request and the subsequent review (this may also apply to Stewart Maxwell, but the documents available do not make clear whether a different Special Adviser was involved at each stage of handling this request). This is particularly important when part of this interaction took place via telephone calls where the details have, unsurprisingly, not been recorded.

Download the Documents

Shona Robison

The briefing documents for Shona Robinson deal with just two FOI requests, although the second actually concerns decisions made by Aileen Campbell (the topic of this request was deemed to be “within Ms Robison’s portfolio responsibilities”).

The first of these (FOI/17/00947) concerns a request for the release of “reports received by Paul Gray’s office from the NHS Chief Executives’ Group from 2013 to September 2016.” The Scottish Government has, so far, refused to release this document in full, redacting key comments regarding the decision-making process. Nonetheless, the material available shows clear evidence of political influence over FOI releases.

Robison 1

At this stage, with two relevant documents have been identified, the FOI Unit has, it seems, been absolutely crystal clear that both should be released. It’s especially important to note that they reject the application of ‘section 29(1)(a)’ exemptions. Why? Well, take a look at what happened next…

Robison 2

Although we don’t (yet) know exactly what Robison said, because the government hasn’t (yet) released this information, we do know that, in the end, the two documents in question were withheld despite the advice of the FOI Unit. It also appears to be the case that section 29(1)(a) exemptions were applied, perhaps at the insistence of Robison, and – bizarrely – that officials were asked to offer “contributions to explain” this.

The requester refused to accept this and demanded a review, and then things get even more interesting.

Robison 3

Once again, front line officials and the FOI Unit believed that the requested documents should be released; once again, key comments have been withheld by the Scottish Government; once again, we see that a political individual – in this case a Special Adviser – intervened and tried to have both kept secret using Section 29(1)(a) exemptions.

In the end, after a battle which seems to have been entirely unneccesary, and which appears to have been brought to a conclusion only when the threat of a judgement by the Scottish Information Commissioner was brought into play, one of the papers was released in full while the other was redacted (I cannot find the documents for this request on the government’s website so do not know how extension this redaction was).

The second case (FOI/16/01545) from Robison’s briefing was actually answered by Aileen Campbell, former Minister for Public Health and Sport (this area was within Robison’s overall portfolio responsibilities).

Campbell’s role – and understanding of the law – is worth highlighting (look at the end of the timeline below) but, to be honest, this case is mostly about the actions of Special Adviser Davie Hutchison.

Robison 4

Robison 5

It is very clear from this information that the response in question was significantly delayed because of Hutchison’s insistence that some material be kept secret, despite the fact that both the case handler and the FOI Unit believed that it should be released. In the end, although some information was released, he seems to have been successful in having other information withheld.

Download the Documents

Davie Hutchison

Since we’re already talking about Hutchison’s involvement in FOI it makes sense to look at this briefing document next.

The first of these (FOI/16/00813) is, to be blunt, absolutely damning – and not just for Hutchison himself. The request centred around correspondence between the government and NHS Lothian regarding the ‘treatment time guarantee’. It was initially submitted in May 2016 but the response was delayed by a request for clarification of the terms of the request.

Hutchison 1Hutchison 2

It’s worth noting – as the FOI Unit did in preparing this briefing – that the application of exemptions became “more extensive throughout January and February” as Hutchison kept on trying to withhold more and more information.

What happened on 9 February, however, is simply outrageous.

As we can see in the screenshot above, the FOI Unit explicitly informs the Special Advisers that the changes they have requested would mean that the response would “not properly comply with FOISA” (ie. it would break the law). They also laid out their reasons for this, although they have not been made public (yet).

Incredibly, however, the FOI Unit seems to have decided to simply allow Hutchison to have his way “rather than further delay the response”, noting that if they received an appeal they would be forced to release the information as requested. So the end result is that the Scottish Government issued an FOI response which it knew did not comply with the law, on the basis that if the requester complained they would change their approach. According to the files available, the requester did not seek a review.

Obviously, the officials in question should not have made these changes but it is clear that they had come under significant pressure. If the Minister in question (presumably Shona Robison) was aware of this, yet still cleared the response, then they have very serious questions to answer. Either way, the role Davie Hutchison played in this case demands some serious scrutiny.

The second case in Hutchison’s briefing has already been dealt with as it is also the second case from Robison’s briefing (scroll up to see it).

The third case (FOI/16/02137) from this briefing concerned the Vale Community Maternity Unit at Vale of Leven Hospital. Once again, Hutchison was heavily – and repeatedly – involved in an attempt to block the release of information, despite the views of the case handler and the FOI Unit.

Hutchison 3

At the first stage of this process the case handler not only plans to release all relevant information, they even note that “there was nothing particularly sensitive or contentious” contained within the materials. That didn’t stop Hutchison from ‘suggesting’ extensive redactions.

Hutchison 4Hutchison 5

It is once again quite clear that the “advice” being offered by Hutchison is, at least in the view of the officials dealing with the case, not optional but rather an instruction. And once again, the government issued a response to an FOI request knowing that, if its approach were challenged, they would have to release the information in question.

This case was eventually considered by the Scottish Information Commissioner who accepted that two sentences in one of the documents could be withheld – the government was forced to release the rest, as the FOI Unit had expected.

The fourth case (FOI/17/00171) from Hutchison’s briefing documents concerns government communications with an unidentified individual regarding ‘NHS Scotland Productivity’ and the ‘2020 Vision Advisory Board’. Once again, Hutchison is seen to be exercising extensive control and seeking to block the release of information.

Hutchison 6Hutchison 7

As you can see, the case handler’s response to Hutchison’s position has been redacted, but it is quite clear that they did not agree and attempted to fight their corner. The very next day they had a meeting with Hutchison and the SPAD Private Secretary (PS), following which the PS “revised the draft response”, then telling the case handler that they could submit the draft for Ministerial approval “subject to the changes proposed being taken on board.” It would require a very generous interpretation for this to be seen as anything other than an instruction.

Hutchison new1Hutchison new2

After the ‘proposed changes’ had been made, and following the clarification of two details, the Minister (Shona Robison) cleared the response for release; however, the document still had to be changed again before being released after the FOI Unit intervened, raising particular concerns about the covering letter to be issued.

But – after all that – when the case reached the Scottish Information Commissioner the government decided to release “all information requested”. For those who have never interacted with the SIC, I should explain that this does not necessarily mean that the government simply came to its senses – they may have simply been advised by the SIC that they faced an unfavourable ruling and decided to cut their losses.

Ironically, during his interview with the Scottish Information Commissioner, Hutchison talked about having a “clear obligation to get it right rather than put something out we know to be wrong.” That obligation does not seem to have applied to this particular case.

Hutchison was also asked how he keeps his role as a civil servant separate from that of a political adviser while handling FOI requests. He replied: “They’re not separate. They’re the same thing.”

Oh, and by the way, this is from Davie Hutchison’s LinkedIn page:Hutchison Linkedin

Download the Documents

Liz Lloyd

Lloyd is the Chief of Staff to the First Minister, Nicola Sturgeon, and also runs the Special Advisers office of the Scottish Government.

The first case (FOI/17/00179) concerns a letter sent by Nicola Sturgeon to Hilary Clinton after she lost the 2016 Presidential election to Donald Trump.

Lloyd 1

As the FOI Unit has noted, there is a significant issue with a delay in this case, with SPADs clearing the response on 31 January (by which time the response was already late) but the response not being issued until 1 March. What happened? Well, we simply don’t know – because records of the process have not been kept.

As the documents for this FOI do not appear to be online we don’t know what information was released or what exemptions may have been applied; the government’s failure to keep track of the case itself means we don’t know the extent of Special Adviser influence on the process.

The second case (FOI/16/01335) in Lloyd’s briefing pack focuses on the work done by the Scottish Government on a written constitution for Scotland prior to the 2014 independence referendum. It reveals what looks like an extremely serious breach of FOI law.

Lloyd 2

The first issue here is that the deadline for responding was not met despite the case handler having completed a draft response more than a week in advance. Perhaps most striking, however, is the confirmation that “the case had been discussed at a meeting with special advisers”. Despite such high-level attention, however, and regardless of strict legal deadlines, it is noted that a response to the official handling the case is “unlikely before 7 November.”

Lloyd 3

Lloyd’s comment on 16 November is extraordinary and extremely serious (and, thanks to the official who produced this document, a direct quote rather than a paraphrase). Having been told by the official that they plan to issue the response on 18 November, the First Minister’s Chief of Staff replied:

“I’m not committing to it going out on Friday at this point. It’ll go out when we’ve had a chance to ensure it’s got a proper handling plan.”

This is a clear example of a Special Adviser blocking the release of an FOI response until lines have been prepared. In effect, she is saying that the information cannot be sent until they’ve decided how they might spin it.

Lloyd was asked about this case directly during her interview as part of the FOI investigation and tried – without any success – to explain away this glaring breach of the law.

Lloyd extract 1

I’m not sure why Lloyd thinks this explanation makes things any better. Her clarification in fact confirms that, unless press lines were ready for Friday, then the FOI response would not be issued on that day. In this case a clear link is being drawn between the issuing of an FOI response and the development of a ‘handling plan’ for the information, with the former clearly dependent upon the latter.

FOI Unit officials themselves flagged the “impact of arranging a comms handling strategy on issuing the response” as a likely matter of interest for investigators.

This is made particularly striking by Lloyd’s response when asked about a potential “lack of understanding about the role of SPADs”:

Lloyd extract 2

“SPADs can advise but can’t instruct.” Right.

The third case (FOI/16/01014) connected to Liz Lloyd also involved another SPAD (who wasn’t interviewed) – Katy Bowman. The requester sought the release of briefing material prepared for Nicola Sturgeon in advance of a meeting with EU officials and MEPs.

Lloyd 5

What is interesting here is that Bowman initially asks the case handler to release more information, but when Lloyd becomes involved the conversation turns to “more extensive redactions”. Her response to the FOI Unit’s advice has also, ironically, been redacted, as has the reply sent by officials:

Lloyd 6

The government’s decision to withhold material from this section of the documents makes it difficult to be sure of what happened, what was withheld or how it was justified.

What we can see, however, is that the issuing of the response was delayed for more than 2 weeks (5 August to 22 August) because the comms team had decided not to proceed until a “related request” had received “clearance from FM”.

The fourth and final case (FOI/16/01414) in Lloyd’s briefing came from Sputnik News and concerned physical security breaches at government premises. It provides explicit evidence of SPAD exerting enormous control over FOI responses and also raises questions about the role of the Communications team.

Lloyd 7

As we can see above, the ‘Corporate Comms’ team got involved early on in this process, contacting the official responsible for the request and, strangely, inserting themselves in the process by asking that the draft response be sent to them first, at which point they would “forward to special advisers for clearance”.

On the face of it this is probably just about defensible – if they were just asking for sight of the material so that they had plenty of time to develop their response that would probably, just about, be alright.


Lloyd 8

The intervention of the Comms team on 24 October, where they state that “they disagreed with the proposed approach”, is concerning. Communications staff should have no role in making decisions around FOI requests – they should simply prepare lines in response to the information.

The events on 6 December are also striking because, as with cases already explored in this blog post, the case handler is concerned that the alterations being requested by SPADS “meant the response was not in line with the initial request”.

And then things get even more interesting…

Lloyd 9

It is 100% clear here that the Corporate Comms team has not only blocked the progress of an FOI request, it has also demanded SPAD “clearance”. The (unnamed) Deputy Director’s email is a clear attempt to fight against undue influence includes pointed criticism not just of the apparent clearance process but also of the changes being demanded by SPADs (which seem to be making the request unnecessarily complicated and, as a result, far harder to answer.

The response was finally issued to the requester on 4 January 2017, with no request for review appearing in the records.

This is one of the cases where a more extensive release of information – including the email exchanges between different departments – would clearly be in the public interest.

Download the Documents

Colin McAllister

As Head of Policy, and Senior Special Adviser to John Swinney, McAllister is a powerful figure in the Scottish Government. His briefing pack focused on four FOI requests, although the second of these (FOI/16/01335) has already been examined above, having been included in Liz Lloyd’s briefing.

The first case (FOI/15/02007 plus FOI/15/02008, FOI/15/02009) related to McAllister dates back to 2015 and concerns Alex Salmond, Nicola Sturgeon and John Swinney’s meetings with Angus Grossart (although his name has been redacted from the documents below the original requester has confirmed that meetings with Grossart were indeed the subject of this FOI).

The process starts badly, with the government failing to meet the legal deadline for a response and failing to explain the delay.

McAllister 1

And things continue in that vein…for nearly a year!

McAllister 2

More than 2 months after the legal deadline to respond the FOI Unit is having to push for a date on which information can be released, warning that the failure to do so will simply add further problems.

But it gets them nowhere.

McAllister 3

It being “difficult for Colin to give a date”, and “impossible to commit to a date for responding”, clearly highlights the control being exerted by McAllister in this case. The case handler can be seen repeatedly attempting to get a response to the requester and, it seems, getting nowhere.

Note also that the Private Secretary to the Special Advisers office explicitly talks about the need for McAllister to have “cleared the amended response”.

The process continues for another couple of weeks…

McAllister 4

Unsurprisingly, the requester took this matter to the Scottish Information Commissioner. This happened on 14 April, with the appeal highlighting the delay, the lack of explanation for that delay, and the claim that information about these meetings was not held.

The SIC found that the government was right to state that information about these meetings was not held (hardly a surprise, given the Scottish Government’s track record in that area) but was absolutely scathing about the handing of the case:

McAllister 5

The Scottish Information Commissioner’s full judgement on this case can be accessed here.

As already noted, the second case in the McAllister briefing (FOI/16/01335) has already been examined, having also appeared in Liz Lloyd’s briefing pack.

The third case (FOI/17/01547) deals specifically with the issue of Special Advisers ‘clearing’ FOI responses.

McAllister 6

Despite the case handler warning against an overly-narrow interpretation of this request, we can see above that Colin McAllister has been instrumental in an attempt to withhold the identified materials because, it is claimed, “special advisers do not clear FOI requests.”

But of course in the earlier case from McAllister’s briefing, we can see the Private Secretary to the SPADs office explicitly that a date for a release of information cannot be given until McAllister has “seen and cleared” the contents.

Anyway, the rest of this goes pretty much as expected. The requester asks for a review of the case and the new official, while explicitly asking for a check that other relevant material has not been omitted, proposes to “reinstate the case handler’s original approach as sent to special advisers on 18 July”.

McAllister 7

The highlighted information above is absolutely extraordinary when we remember that, as part of this response, Special Advisers had previously insisted that they do not “clear” FOI responses – yet on 14 August those same Special Advisers are instructing officials “not to send the submission to the Minister”, and on 17 August they adopt the position that the material can only go to the FM and DFM “subject” to their changes being accepted.

The fourth and final case (FOI/17/02107) in McAllister’s briefing relates to a request that I submitted in September 2017 in connection with my investigation into Teach First’s attempts to expand their operation in Scotland.

McAllister 8

Once again the influence of Colin McAllister on an FOI response is clear to see. On this occasion, he sought to have the entire document in question withheld from me using an exemption that was “not applicable”. The FOI Unit then “gave advice on attempting” to apply a different exemption, but even at this stage advised that this wouldn’t stand up if challenged.

Unsurprisingly, I did challenge the use of the exemption; equally unsurprisingly, I won. The document was subsequently released in full, despite the interference of McAllister.

McAllister 9

Interestingly, it appears that the response to my appeal was not submitted to SPADs such as McAllister for clearance (or, if it was, this has not been recorded) – instead, the official seems to have gone straight to the DFM, John Swinney, who cleared the release.

You can read the story that came from this disclosure here.

During his interview with the SIC, McAllister actually raised the events of this case. It is striking, not to mention concerning, to see that such a senior figure in government, with significant influence of FOI disclosures, still doesn’t understand why he was wrong – a fact made clear by his attempted, and wildly flawed, explanation.

McAllister 1

As it happens, the assertion that this one story stopped Teach First bidding for a contract, though flattering, is rather wide of the mark. Scotland’s universities had unanimously agreed not to work with the organisation, and serious questions were being asked about their lobbying operations, in particular the use of Prince Charles to help them gain access to ministers, as well as the organisation’s attempts to circumvent the tendering process entirely. As far as Teach First themselves go, they said at the time that they were pulling out because the Scottish Government had rushed the development of the new programme. They also raised concerns about the financial aspects of the contract.

Download the Documents

Stewart Maxwell

This briefing document contained information on two cases. The second has already been examined (it was also the first case in Keith Brown’s briefing), while the first (FOI/16/02147) is closely linked to the same topic and minister.

Maxwell 1 new

Maxwell 2Maxwell 3Maxwell 4

Maxwell “suggested” that the case handler withhold more information, but it ended up being disclosed after an appeal. As the FOI Unit has highlighted, the “reasons for withholding information” are therefore of interest, given that they were ultimately too flimsy to survive once the matter had been referred to the Scottish Information Commissioner.

Keith Brown’s apparent wish for the material to be released on a specific day is also a matter of concern – despite the information being cleared on a Tuesday, it was not issued to the requester until the Friday (as Brown had apparently wanted).

And, of course, the ridiculous delay in dealing with the case, significantly breaching the legal deadline for doing so, is entirely unacceptable.

Download the Documents

SPAD Office

As noted at the beginning of this post, the name (or names) on the final briefing document are redacted but, based on information contained in several documents, we can be relatively sure that this material was prepared for the member of staff in the SPAD Office (presumably the Private Secretary?) in advance of their interview with the Scottish Information Commissioner.

The first case (FOI/16/00432) relates to the “potential creation of a Scottish Monetary Authority”.

SPAD 1It gets off to a bad start when the request is, it seems, simply ignored. After that…well…things just get worse.


Where to even begin with this? Between the 17th and 29th of March the new case handler (reviwer) and the FOI Unit worked together to produce a draft response to go to the requester – that response remained unsent for two and a half months following the introduction of SPADs and Ministers to the process. There is no explanation for “when, how or why the response was deemed to require the attention of the First Minister after her the DFM had already ‘cleared’ it. Most interestingly of all, the response was still “being considered” by one or more SPADs after the First Minister’s clearance had been given.

And amidst all of this the requester (quite rightly) went to the Scottish Information Commissioner, who pretty much immediately issued a judgement stating that the Scottish Government had failed to comply with the law.

But that wasn’t the end of the matter, because the requester then went back to the SIC seeking a review of the actual information disclosed to them.


Then, on 1 March 2017, more than a year after the initial request had been submitted, the SIC ruled in favour of the requester and forced the government, finally, to hand over the information that had been wrongly withheld.


And what did the FOI Unit make of all this when they were preparing the briefing? Well it seems they had the same questions as I do:


The second case (FOI/16/01736) from this briefing focuses on the planned deal between the Scottish Government, Sinofortune China Railway and China Railway No 3 Engineering Group. That deal, readers may remember, was aborted following human rights concerns, causing considerable embarrassment for the Scottish Government.


As with one of the Liz Lloyd cases, the obvious problem here is a near-total lack of information, making it impossible to know what on earth caused a delay, and consequent breach of the law, of more than 8 months. All we know about this case – aside from the delay – is that the process ended with SPADs and Comms staff arranging a handling plan, with the response itself issued four days later.

But if you want to see that response, here it is.

The final case (FOI/17/00171) in the SPAD briefing has already been covered as it was covered in the documents provided to Davie Hutchison. 

Download the Documents

Some Final Thoughts

There are two likely defences for all of this. The first is that the government is changing its policies in response to the SIC investigation – but in some crucial ways this is untrue. On the matter of SPAD and Ministerial involvement, for example, the government ultimately proposes to maintain the same flawed system, but within a more formalised structure.

The government would also likely attempt to remind us that Ministers and SPADs can play a role within the FOI system without automatically breaching the law, but this rather misses the point. The question is not whether they can be involved, but whether they should be.

The answer to that question depends on whether or not one believes that explicitly political actors can act is an entirely impartial manner when dealing with requests for politically sensitive information. I’d suggest that the answer to that question is pretty obvious.

During the SIC interviews the matter of ministerial clearance of FOI responses came up repeatedly. Various individuals defended it on the basis that Ministers need to know what information is being released, particularly if it is likely to be raised in parliament or by the press. This is, of course, true – but it does not follow that Ministers must also be involved in deciding what is released.

In defence of their FOI role, SPADs tended to argue that no one else in government possesses the same breadth of knowledge, and that their insights are, therefore, invaluable. Then again, they were hardly going to talk down their own importance, were they?

The Scottish Government has repeatedly asserted that SPADs don’t ‘clear’ FOI responses (at one point a government minister told parliament that they only check for accuracy) but, as these documents show, this is categorically not true. It may be the case that they do not technically possess any formal clearance powers, but the amount of influence they exert over the release or otherwise of sensitive information, and their function as ‘gate-keepers’ between officials and ministers (at points refusing to allow officials to submit material to ministers until specific changes have been made) means that they are in effect clearing FOI responses.

As one SPAD (Colin McAllister) said during his SIC interview, he and his colleagues are “precluded by the SPAD code from instruction” – yet there are numerous examples of SPADs doing exactly that in the cases above, which are themselves just a tiny snapshot of the total FOI requests handled by the Scottish Government.

There may well be times when one or more needs to be consulted (like any other government employee) as part of the process of answering a request for information, but that is a very long way from being a justification for the sort of routine, far-reaching interference on display in the documents above. Confirming to a case handler that some potentially relevant policy-development work is ongoing, for example, is not at all the same thing as telling them to apply policy development exemptions to a request before it can be issued.

The Scottish Government is responsible for ensuring the effective operation of Freedom of Information systems within their organisation but there seems no real requirement for this responsibility to be exercised in such a ‘hands on’ manner (either directly or in a delegated manner via SPADs). A huge range of non-governmental organisations are subject to FOI requests in Scotland and most seem to manage just fine.

A better approach, I believe, would be to regard Ministers as being responsible for ensuring that the right systems, people, training and support are in place to allow properly impartial civil servants to make decisions about the release of material – sensitive or otherwise – in line with the law. Such a change – preferably combined with far more transparent record-keeping – would be a significant step towards making information genuinely free.

Download the Documents

Click on the links below to download the briefing documents and interview notes featured in this post.

John Swinney: Briefing Document —- Interview Notes

Keith Brown: Briefing Document —- Interview Notes

Shona Robison: Briefing Document —- Interview Notes

Davie Hutchison: Briefing Document —- Interview Notes

Liz Lloyd: Briefing Document —- Interview Notes

Colin McAllister: Briefing Document —- McAllister interview notes

Stewart Maxwell: Briefing Document —- Maxwell interview notes

SPAD office: Briefing Document —- SPAD Office interview notes

Click here to download all referenced files at once

Click here to download all interview notes from the website of the Scottish Information Commissioner

Teach First, Ask Questions Later – full report & emails | Blog

The Herald is today [Sat 28 October] running two exclusive stories about Teach First, the fast-track teaching organisation expected to bid for a Scottish Government tender to deliver a ‘new route into teaching’.

The first of these articles – on the front page – concerns a highly critical report by Glasgow University which was produced “at the invitation of the General Teaching Council of Scotland and the Scottish Government”. You can read the story here.

The second article reveals that the Scottish universities currently involved in ITE (Initial Teacher Education or teacher training) have unanimously agreed not to work with Teach First. This comes despite repeated attempts by Teach First to establish a relationship with these institutions. You can read the story here.


The document in question was written by Dr Stephen Parker and Prof Trevor Gale, both of the University of Glasgow, in April 2016 (which is prior to Teach First’s meetings with both Nicola Sturgeon and John Swinney).

I am making a full, unredacted copy of the report available here: Glasgow University report – Teach First.

A summary of the key introductory points is provided here:

Image 1Image 2Image 3



Teach First has been attempted to establish a link with a Scottish university for some time. A previous investigation – which led to stories about a detailed Teach First proposal for Scotland and secret lobbying on behalf of the organisation by Prince Charles – revealed that the organisation asked Scottish Government officials to contact Morag Redford (chair of the Scottish Council of Deans of Education) on their behalf.

Email 1 – sent from James Westhead (Teach First) to Stuart Robb (Scottish Government) on 17 March, 2017:

Image 5 TF to SG

Email 2 – sent from James Westhead to Stuart Robb on 4 April, 2017:

Image 6 TF to SG

In recent weeks, Teach First stepped up its attempts to secure a partnership with a Scottish university. I submitted FOI requests to all eight universities involved in teacher education, allowing me to confirm the following:

  • Education departments at Aberdeen, Edinburgh, Stirling and UHI have had no contact with Teach First other than that regarding the recruitment of students into the programme
  • Dundee, Glasgow and UWS were contacted by Teach First but declined their requests for meetings (in the case of Glasgow, Teach First wrote directly to the author of the report covered above)
  • Strathclyde was contacted by CBI Scotland, who unsuccessfully attempted to arrange a meeting including Teach First

Email exchange – Teach First & University of Dundee

Email exchange – Teach First & University of Glasgow

Email exchange – Teach First & University of the West of Scotland

Email exchange – CBI (on behalf of Teach First) and University of Strathclyde


Tartan Teach First? A look inside the Scottish Government’s plans | Blog

Last week the Scottish Government finally released the tender documents for its ‘new route into teaching’. Ever since plans to tender for what looks like a fast track teaching programme were announced, questions have been asked about whether this will allow Teach First to realise its long-held ambitions of expanding north of the border. In recent week I have published a number of stories highlighting key aspects of Teach First’s Scottish proposals (see here) and lobbying operation (see here).

Here I take a look at the specifications of the tender for the proposed new programme with a particular focus on the similarities between the Scottish Government’s plans and the systems currently operated or proposed by Teach First.

The Money

The first rule of any investigation is simple: follow the money. In the case of the Scottish Government’s “new route into teaching” this will mean following up to £250k, which is the top end of the available budget for a project that will last just a couple of years.

TD-Costs 1

But it’s not just about the headline figures. As shown below, the government expects employers (ie. councils) to fund the salaries of anyone working through this new scheme, and has also stated that the existing Scottish payscales must be respected. Interestingly, however, they also state that they would be willing to consider “any innovative incentivisation that can assist recruitment.”

TD-Costs 2

This could mean just about anything but the government will have to be extremely careful to avoid a scenario where an “incentive” is deployed to create a de-facto unqualified teacher’s salary, as is available in England and utilised by likely-bidders Teach First (who suggested something very similar to the Scottish Government a few months ago). This is definitely an area to keep an eye on, especially with John Swinney having now announced £20k bursaries for people switching career to become teachers of STEM subjects.


The Timeline

The tender documents also contain a timeline for the development and delivery of this new route into teacher. Bids must be submitted by the end of November 2017, with the government awarding the contact at the start of the new year (ignore the typo in the tender documents). From that point, the programme will have to be designed, marketed, accredited and verified within, at the very most, 12 months (although it seems likely that most organisations would seek to start the programme in August or even – in the case of Teach First – July, giving just 6 months to get things up and running). As the screenshot below shows, the new course will be advertised to prospective students before it has even been accredited by the GTCS.



As has already been reported in The Herald, even Teach First felt that this time scale was too narrow, although – and as is highlighted later – this was very much in the context of seeking favourable terms for themselves.



As the tender documents make clear the winning bidder is not designing a programme that they will operate for an indefinite period; instead they are creating a system which will be given, in its entirety, to the Scottish Government.

TD-Handover 1

This means that as of December 2020 (at the latest) this new route into teaching will be brought in-house and, presumably, operated by an arm of the Scottish Government from that point forward (although the government could in theory re-tender for a new operator while retaining the model).

TD-Handover 2

Given that the SNP has previously described the tendering process (in the case of CalMac) as “the most expensive exercise in futility” it seems likely that questions will be asked of this approach. If this new system is to ultimately be operated in-house, and the government knows, at least in general, what it is looking for, then why not simply run it on a fully public basis from the beginning rather than, potentially, hand £250k to a private organisation? The total cost of the tendering process, and this figure relative to the overall cost of the programme, will clearly come in for scrutiny further down the line.


University link

Another notable aspect of the tender surrounds the need to link with a university as part of the programme. A few days after the tender was opened a question was submitted, with an answer that is likely to raise eyebrows:


This opens the possibility of an organisation with no experience of Initial Teacher Education in Scotland, supported by an organisation with no experience of Initial Teacher Education in Scotland, winning a £250k contract to establish a new form of Initial Teacher Education in Scotland. It’s not difficult to see why this might cause problems, and hard questions would have to be asked of any organisation which, in seeking to enter the Scottish education system, had been unable – or unwilling – to secure a partnership with a Scottish university. In the case of Teach First this is particularly relevant because documents obtained under FOI laws suggest that they have been finding it difficult to convince a Scottish university to work with them (more to come on this in the next few weeks).



As The Herald has already reported (see here) this new route into teaching will involve a very small number of people:


Whatever else this programme may be, then, it is clearly not an answer to the increasingly worrying recruitment crisis in Scottish education. It is also worth noting that Teach First recommended a cohort of around 40 students for the first year of a new Scottish programme as far back as 2012:



Another interesting aspect of this section of the documents is the reference to “innovative advertising and recruitment techniques”.


This phrase immediately reminded me of the following paragraph, which comes from an HMIE “information-gathering” visit to Teach First from 2011 (obtained via FOI):


This is just one of a number of areas of the tender which aligns very closely with what we – and the Scottish Government – already know about Teach First.



Another similarity between the tender documents and the Teach First approach is found in the focus on “leadership”.

These screenshots come from the Scottish Government’s tender documents.

TD-Leadership 1

TD-Leadership 2

The images below come from documents submitted to the Scottish Government as part of Teach First’s pitch for permission to bring their teacher training model to Scotland.




It’s also worth highlighting that the existing Teach First programme is in fact called the ‘Leadership Development Programme’. On the website it is described as follows:



Focus on deprivation

Which brings us to what is, without doubt, the most interesting aspect of this tender: the focus on getting new teachers into schools in areas of deprivation. Indeed, bidders are given explicit instructions that new student teachers must work in “schools with high levels of deprivation” and that the winning organisation must work with areas being “supported by the Attainment Scotland Fund”, part of “a targeted initiative focused on supporting pupils in the local authorities of Scotland with the highest concentrations of deprivation” (Glasgow, Dundee, Inverclyde, West Dunbartonshire, North Ayrshire, Clackmannanshire, North Lanarkshire, East Ayrshire and Renfrewshire). (http://www.gov.scot/Topics/Education/Schools/Raisingeducationalattainment)

These are taken from the tender documents:


This might not seem particularly surprising – after all, the government has spent the last couple of years insisting that tackling the fact that those from deprived background suffer a significant educational disadvantage (the so-called “attainment gap”). In Scotland, however, problems around teacher numbers are not concentrated in areas of deprivation – instead, we struggle to attract and retain teachers in rural areas and in specific subjects (although evidence suggests that the number of subjects facing such problems continues to increase).

The following images are taken from Scottish government briefing papers prepared in advance of meetings between both Nicola Sturgeon and John Swinney and representatives of Teach First:


It would seem, then, that the Scottish Government is about to spend up to a quarter of a million pounds, plus the costs of the tendering process itself, tackling a problem that they themselves clearly and categorically state does not exist – but which Teach First was explicitly set up to tackle:




With Teach First widely expected to bid for this new contract, the organisation’s history of lobbying the Scottish Government is acutely relevant.

Scottish Government documents make absolutely clear that they have been under no illusions about this lobbying. Ministerial briefing papers explicitly mention the organisation’s “long held ambitions of exporting their model of Initial Teacher Education to Scotland”.


Despite this, government ministers held several unminuted meetings during which Teach First appears to have received advice regarding a future bid to the Scottish Government. Full details of these meetings can be found here: https://jmcemedia.wordpress.com/2017/10/08/revealed-meetings-between-the-scottish-government-and-teach-first/

Furthermore, Severin Carrell (The Guardian’s Scotland editor) and I recently revealed that Prince Charles has also lobbied the government on behalf of Teach First, but that the Scottish Government refuses to reveal details of this lobbying. You can read that story here: https://www.theguardian.com/uk-news/2017/aug/14/scottish-ministers-refuse-to-make-public-prince-charless-lobbying-letters-teacher-training-rules-scotland

It is plainly unacceptable that an organisation which was seeking a change in government policy, and which is expected to bid for a subsequent government contract, was able to hold unminuted meetings with the former Education Secretary, the current Education Secretary (and DFM) and the current First Minister, and questions should be asked about the extent to which this lobbying influenced the final design of the tendering documents now available.

But it’s not just the secretive meetings that are concerning. The Herald has already reported on Teach First’s attempts to both delay the tendering process (in order to improve their chances of a successful bid) and convince the government to allow them to bypass it altogether (see here).

Below is a screenshot of one of Teach First’s emails to the Scottish Government:


Here we see them ask the government if it would be willing to pursue an “alternative arrangement” to the “public tender process”. This was actually part of a lengthy email in which Teach First pushed hard to convince the government to follow a path that would be better suited their organisation.


Although these requests appear – at least thus far – to have been rejected, they still raise questions about any potential bid from Teach First for this government tender.

Revealed: meetings between the Scottish Government and Teach First | Blog

In recent weeks both The Herald (see here and here) and The Guardian (see here and here) have published stories regarding communications between Teach First – an organisation specialising in ‘fast-track’ teacher training – and the Scottish Government.

Below is a list of meetings dating back to 2012 along with links to the original documents from which this information has been drawn (released following a Freedom of Information request).

Those familiar with the Scottish Government’s record when it comes to transparency will be unsurprised to discover that no minutes were taken during three separate ministerial meetings with Teach First, despite official briefings being prepared and civil servants also being in attendance.

16/5/2012 – Scottish Government officials met with Teach First representatives

This meeting took place the morning after Teach First had hosted a reception at the Scottish Parliament. Then education secretary Mike Russell attended the reception. No names of participants are confirmed but an email suggests Brett Wigdortz (Teach First) may have been present.

Mike Russell asked “to be kept closely informed of developments…”.

Brief record of meeting (redacted by Scottish Government)

30/1/2014 – Mike Russell met with Brett Wigdortz & James Westhead (Teach First)
Also present: Fiona Robertson, Ian Mitchell, Rachel Sunderland (Scottish Government)

A full briefing for this meeting was prepared but no minutes were taken and no further notes of what was discussed are available.

Official briefing docs (redacted by Scottish Government)

No minutes

22/4/2016 – Stuart Robb & John Gunstone (Scottish Government officials) met with James Westhead & Reuben Moore (Teach First)

This meeting seems to originate in a conversation between James Westhead (Teach First) and Mark Leishman (Private Secretary to the Prince of Wales). A Scottish Government official makes a specific request for notes of the meeting but receives no reply. The email chain ceases at this point.

Email chain

No minutes or other records

26/8/2016 – Stuart Robb & David Roy (Scottish Government) had a telephone conference with Jonathan Dando (Teach First)

This discussion seems to focus on the Scottish Government’s plans for “new forms” of teacher education in Scotland, and serves as preparation for the meeting with Nicola Sturgeon.

It includes the following:

“We agree to ensure [Teach First] had the opportunity to tender if we go down that route.”

Notes of conversation

21/9/2016 – Nicola Sturgeon met with Paul Drechsler (Teach First)

Also present: Clare Hicks (Scottish Government)

A full briefing for this meeting was prepared but no minutes were taken and no further notes of what was discussed are available.

The official briefing document classes this as a “private meeting”.

Official briefing docs (redacted by Scottish Government)

No minutes.

10/1/2017 – John Swinney met with James Westhead (Teach First) & Peter Duncan(Message Matters)

Also present: Stuart Robb (Scottish Government)

A full briefing for this meeting was prepared but no minutes were taken and no further notes of what was discussed are available. Prof Lesley Sawers (Equality and Human Rights Commissioner for Scotland) is advising Teach First but was unable to attend this meeting.

Official briefing docs (redacted by Scottish Government)

No minutes

30 or 31/3/17 – John Swinney met with Brett Wigdortz (Teach First)

This meeting took place during the International Summit on the Teaching Profession 2017 in Edinburgh. Reference to the meeting appears only in a heavily redacted email exchange

Email exchange (redacted by Scottish Government)

NB The Scottish Government has also been lobbied by Prince Charles on behalf of Teach First, but refuses to release details of this communication. You can read about this story on The Guardian website.

As highlighted in this article for The Guardian, documents released by the Scottish Government also suggest that former First Minister Alex Salmond discussed Teach First with the Prince of Wales during a meeting on the 12th of June 2013

ScotGov refuse to release Prince Charles lobbying letters | The Guardian

First published on August 14, 2017 by The Guardian: https://www.theguardian.com/uk-news/2017/aug/14/scottish-ministers-refuse-to-make-public-prince-charless-lobbying-letters-teacher-training-rules-scotland (shared byline with Severin Carrell)

Prince’s office supported proposals from charity that could financially benefit from changes to Scotland’s teacher training rules

The Scottish government is refusing to release correspondence showing the Prince of Wales lobbied ministers to loosen up strict rules on teacher training for a charity that stands to earn money from the changes.

Ministers have rejected three requests to publish letters and documents from Prince Charles’s office supporting proposals from Teach First, a charity he helped set up 15 years ago, to introduce fast-track teacher training in Scotland.

It comes at a time when the Scottish government is on the brink of announcing a £1m scheme to accelerate training in Scotland amid a shortage of suitable graduates, allowing new providers to enter the market.

Teach First said it has expressed its interest in bidding for the new contract to supply teachers in Scotland, and would earn a fee for every person it recruits if it wins. In England, where the charity has recruited more than 10,000 teachers for state schools and academies, it earns £2,600 for each trainee.

Prince Charles has served as Teach First’s patron since it was founded in 2002 with the aim of recruiting teachers for inner city schools that struggle to find teachers. It fast-tracks university graduates through a six-week training course, who then continue training in the classroom.

A handful of the prince’s “black spider” handwritten letters, written to UK ministers in 2004 and 2005, were released after the Guardian won a 10-year legal battle to have them disclosed under the Freedom of Information Act. But the law in England and Wales on the disclosure of royal correspondence has since been tightened substantially, meaning they no longer have to be made public.

Scottish ministers have said the Scotland’s FoI rules on royal correspondence remain more open than the rest of the UK, and were not tightened after the black spider memo case. But civil rights activists say that claim is undermined by the Scottish government’s decision in this case.

Maurice Frankel, the director of the Campaign for Freedom of Information, said there was a clear conflict of interest if the prince was lobbying on behalf of a charity he patronised and which would gain financially from that lobbying.

“There is a powerful case for access to his advocacy correspondence, full stop,” Frankel said. “We’re talking about a specific policy which it turns out will financially benefit the organisation about which he is expressing those views. That says ‘conflict of interest’ in very large letters.”

An investigation by the Guardian shows the Scottish government was first lobbied by the prince in April 2012. That same day Teach First also sent the then Scottish education secretary, Mike Russell, a briefing advocating significant changes to Scotland’s teacher training system.

Russell attended a Teach First reception at Holyrood four weeks later, according to Scottish government records that are publicly available. He instructed his officials to keep in contact with the charity while they talked to the General Teaching Council Scotland about its proposals. One document notes Russell “would like to be kept closely informed of developments”, although progress was slowed by the independence referendum in 2014.

The issue of changes to teaching in Scotland is a highly charged one. Although the English and Welsh systems have been deregulated, the Scottish profession is highly protective of its state education system and the strict rules governing teacher training.

Scottish secondary school teachers must have a university degree in the specialist subject they teach and a one-year graduate diploma taught at a university. Teach First has proposed instead a five-week summer course before its Scottish trainees are placed in the classroom, similar to its English and Welsh model.

Iain Gray, Scottish Labour’s education spokesman, said: “The pattern of communication with Teach First directly and with the Prince of Wales on their behalf can only raise questions of policy being made as a result of pressure, and these concerns must be dispelled or otherwise by disclosure of the correspondence.”

The official records show Teach First kept lobbying the Scottish government. Civil servants have redacted further correspondence and briefing papers from 2012, 2013 and 2014 on the grounds they relate to the Prince of Wales. One of those is a letter from Russell to Teach First; another is a Scottish government briefing pack on the charity.

Even though Scottish ministers insist they weigh up the public interest very carefully when it comes to correspondence from the royal household, in this case they have applied a unique exemption under section 41(a) in the Freedom of Information (Scotland) Act 2002, on all documents mentioning the Prince of Wales.

Two requests were made to the Scottish government to release the correspondence under the act, followed up by a media request from the Guardian to make them public. But the Scottish government said it had applied the royal exemption because the material related to “communications with HRH the Prince Charles, the Duke of Rothesay”.

Officials at Clarence House in London also refused to release the material voluntarily, and refused to comment. The prince’s officials are sticking to their policy not to release correspondence from him which they say is private, and refuse to deny that the letters came from his private office. Teach First referred inquiries about his role to Clarence House.

A Scottish government spokesman said: “The primary aim of the development of new routes into teaching is to broaden the range of people entering the teaching profession, not fast-tracking the qualification of new teachers.

“Any new route into teaching must be accredited by the General Teaching Council for Scotland and will require a partnership with a university.”