Tom Watson: he who wields the dagger WILL inherit the crown

Let me let you into Westminster’s worst kept secret. There is no chance – absolutely no chance – of Jeremy Corbyn leading Labour into the 2020 General Election. Corbyn is 66 now; at the time of the next election, he’ll be a few weeks short of seventy-one. The demands of modern political life make leadership no place for a septuagenarian.

But even if Corbyn were younger, there’d be no chance then either. This is someone who’s spent his entire political life on the backbenches, immersed in the politics of protest. He’ll never have even dreamt of making it to the Labour front bench, let alone the leadership; more to the point, he won’t have wanted to either. His performance (sic) so far as leader is an ample illustration why: Corbyn knows he’s been wildly over-promoted; that he isn’t up to the task of a job which requires authority and control over his party and its message. This is The Peter Principle writ large.

The Tories know it: that’s why they’ve already started insinuating that not Corbyn, but Labour en masse, are a “threat to security”. Dan Hodges knows it; Nick Cohen has called for it; Stephen Bush, as he hinted on Thursday, knows it too. Absurd though it might sound only a week into Corbyn’s leadership, the question for Labour is (1) How to replace him; (2) When; (3) With whom?

Two theories are doing the rounds. One involves Tom Watson (as she was the first to suggest it, even while the leadership contest was still ongoing, let’s call it the Mensch Thesis); the other, Alan Johnson. Let’s call that the Pedley Thesis, and take the second one first. On Thursday, Keiran Pedley set out his view that the genial Johnson, longtime Tony Blair loyalist, former General Secretary of the Union of Communication Workers, and who can now often be found making merry on Andrew Neil’s comfy sofa, could be the man Labour coalesce around midway through the Parliament, much as the Tories did with Michael Howard in 2003.

In my view, since Denis Healey was fatefully rejected in 1980, Labour have only produced three individuals who could have won a General Election in most circumstances: Blair of course, David Miliband… and that man Johnson. A major part of why Blair hung on so long after the Iraq debacle was his desperation that someone emerged who could stop Gordon Brown making a Horlicks of all his and New Labour’s good work. With his innate, preternatural political instincts as sharp as ever, Blair knew Brown wasn’t remotely successful Prime Ministerial material; but one man in his Cabinet was.

When, following an internal coup against him fronted by Watson in 2006, Blair announced preparations for his departure the following year, who was the friend by his side? Johnson. This wasn’t a coincidence. Over the nine-and-a-half months which remained, given Miliband was still too wet behind the ears, Blair must’ve hoped that Johnson could rise in prominence enough to credibly challenge Brown for the succession – but there was a problem. An insurmountable one. Johnson didn’t want the job. He never has.

British political history is full of counter-factual talking points. If Margaret Thatcher hadn’t been forced out by her own party, and instead allowed to lead it to defeat in 1992, the fratricidal nature of her exit wouldn’t have poisoned the Tories for so long afterwards. If Labour had won the 1992 election, but then faced Black Wednesday just five months later, goodness knows what existential damage this would’ve wrought. If John Smith had lived, Labour would never have moved so far rightwards and alienated (until last weekend, that is), so much of its core support. If Brown had called a snap election in 2007, David Cameron would almost certainly have been defeated, ousted by his party in favour of a right-winger, and Labour might well still be in power even now.

That Brown didn’t – he bottled it because of Cameron’s inheritance tax pledge – remains the most significant political watershed of this generation. His paralysis in the face of a glaring opportunity confirmed Blair’s worst fears and left Labour on the rack. They’ve never recovered since.

But in Summer 2009, as Brown reeled from catastrophic local and European results, there was another chance. Various ministers resigned; The Guardian called for the Prime Minister to do the honourable thing. With Labour hopelessly placed in the polls, perhaps recognising what a hospital pass the premiership seemed to represent, now – pace Michael Portillo 1995 – it was Miliband’s turn to dither. And again, Johnson clearly wasn’t interested.

Between then and the 2010 election, the horrendously unpopular Brown turned an insurmountable 20-point deficit into, with the aid of the Clegg bounce, a hung Parliament. Imagine how much better the outcome would’ve been for Labour had either Miliband or Johnson grasped the nettle: as a minimum, they’d have surely been the largest party. No Cameron and Osborne; no Liberal Democrat wipe-out this year; no Tory-led slash and burn, but Labour-led trim and singe instead.

Even then, after the wrong brother won the Labour leadership, there were further opportunities to oust the poorly performing Ed during the last Parliament. With David having exiled himself in New York, once more, the only man Labour could realistically have united around was Johnson; once more, loyalty to his party and his own very real humility trumped any ambitions which, to reiterate, he’s never held anyway.

At any point between 2007 and last weekend, Alan Johnson could’ve been the Howard-esque unifying figure which Labour will desperately require post-Corbyn. Had he succeeded Blair, Johnson – with broad appeal across the country and throughout his party, as well as a compelling backstory – could’ve been Labour’s John Major: a vastly more successful one too. But last weekend – and here’s where the Pedley Thesis collapses – the world tilted entirely on its axis.

By opening the leadership contest up, Ed Miliband’s unwitting legacy to the Labour Party was, in effect, its death; and the immediate, Phoenix-like emergence of a new, grassroots-based movement of the left. A long overdue fusion of traditionalism and radical populism. Labour was never going to be miraculously immune from the anti-austerity, anti-neoliberal forces which have propelled the rise of Syriza, Podemos, the SNP or Bernie Sanders; nor the sterilisis which has plagued its social democratic sister parties across Europe. As Britain’s increasingly absurd electoral system strangles new parties at birth, this fusion was the only answer for the left. A personal view is its long term outcome – 10 or 15 years from now, with or without Scotland – will be the implementation of proportional representation.

What does this mean? Put simply, the Blairite wing of the party is finished, for good. That’s why we’re hearing rumours of disgruntled Blairites crossing the floor: they know the game is up. Although he’d furiously protest to the contrary, I suspect this is part of what’s put Cohen’s nose so out of joint; it certainly has with John “who cares about the grassroots?” McTernan. But Johnson, however popular and likeable he remains, is himself a Blairite; and would certainly be seen as such by an infuriated, much expanded membership in the event of Corbyn’s departure.

On Wednesday, Hodges wondered whether “there’s a choice for Labour other than simply embracing Corbynism or returning to Blairism?” The always perceptive Telegraph columnist has already sensed which way the wind’s blowing. What’s happened isn’t just about Corbyn at all; the changes in Labour’s make-up are seismic and irreversible. Centrifugal forces cannot be resisted: there can never be a return to the top-down triangulation of the past.

Where I differ from Stephen Bush, though, is I’d argue that Labour’s soft left were in full-blown retreat after Blair’s accession in 1994; all but vanished altogether; then started making small, baby steps forward during Miliband’s ill-starred leadership. Now, not so much the hard left (who, other than John McDonnell, is of the hard left and among Corbyn’s more senior Shadow Cabinet members?) as the soft left are triumphant.

Not for nothing did Neal Lawson state Labour were as good as dead only to publicly declare for Corbyn, to say nothing of Anthony Barnett and Jeremy Gilbert chiming in with their support: along with, in the latter’s case, his vehement disdain for Blairism. The often mischaracterised Owen Jones, Zoe Williams, Paul Mason and Aditya Chakrabortty are all in the vanguard of the modern soft left; all are hugely supportive of Corbyn, and especially energised by the long term opportunity his victory represents.

Longer term, there’s nothing to stop, say, Stella Creasy, the epitome of the soft left, becoming Labour leader after 2020. She recognises how greatly the party needs to change, always emphasises the need for a grassroots movement, is – hallelujah – in favour of electoral reform, and as her vote in the Deputy Leadership contest proved, is popular among the membership: just not as much as one individual. The man at the centre of the Mensch Thesis, Tom Watson.

Few individuals have brought down one Labour leader, let alone two. Even fewer have gone on to inherit the crown themselves. But equally: few have held as much power within the Labour Party as Watson now does. Massively popular among the membership thanks to his brilliant campaigning on the Westminster paedophile ring, questioning of the Murdochs, and ousting of Blair, Labour’s fixer commands respect across the political spectrum; and to judge by his comments during the Deputy Leadership contest, has at least modified his machine-like ways of the past.

He is also a personal friend of Louise Mensch, Twitter Tory attack dog-in-chief. Mensch is frequently a hate figure for the left (even, on occasion, a joke figure); but this is to seriously underestimate her very real qualities of political cunning and nous. She’s an increasingly shrewd observer of the Westminster scene.

On August 9, Mensch explained how Watson could very quickly replace Corbyn as leader: even as early as Christmas. On Tuesday, she went much further: noting that trade union leaders were already subtly distancing themselves from Corbyn after his disastrous first few days in the job; and reminding us that Watson, not the new leader, is the man who actually has most of them in his pocket.

On Thursday, Bush characterised Watson as on the party’s ‘soft right’. I think that’s a mistake. Just as Creasy is sometimes strangely referred to as a Blairite (she’s not; she’s a non-dogmatic, labels-rejecting, all-inclusive Labourite), Watson is sometimes oddly seen as on the right of the party. In fact, he’s a reformed Brownite, parked right in the centre not only of the Parliamentary Labour Party, but the membership too. The unions’ man; the PLP’s man; the membership’s man. He could not be more strongly positioned.

And that he’s arguably the only figure capable of bridging the yawning chasm separating much of the bewildered, blindsided PLP from the now massively leftist membership is of critical importance. Whenever Corbyn goes (even if it’s genuinely of his own volition, or follows electoral meltdown next year or in 2017), many members will be furious and out for blood. Meanwhile, David Cameron’s reforms to trade union funding – a naked attempt to destroy the Labour Party, as Hodges has set out – mean Labour literally cannot afford to lose those members. It needs them, and many more of them. The only way it can survive, change and ultimately prosper is by broadening its reach across the British left, and eventually uniting it. Corbyn’s victory was merely the first step in that long, hugely overdue process.

How can Labour avoid civil war among the membership and make itself electorally credible again? Just as the Tories did with Howard: by crowning Watson a year or (more likely) two from now, without a contest. Watson already has a mandate; as Corbyn’s Deputy, it’s natural he should succeed. Then, just as Howard didn’t change much of Iain Duncan Smith’s message – he merely tweaked it at the edges, while communicating it infinitely more effectively – Watson wouldn’t need to change all that much of Corbyn’s message either. Somewhere to the right of Corbyn and the left of Miliband – a very distinctive message, voiced in a credible way by a competent, strong leader – is what would result.

Could a Watson-led Labour win in 2020? No, especially given the boundary changes and gerrymandering which the Tories will push through. But he can emulate Howard in overseeing the start of the recovery, going down to reasonable defeat, and handing over to a younger, longer term figure afterwards: either Creasy or Dan Jarvis, in all likelihood.

Of course, a great deal can change in two years in politics. Britain might well be mired in recession by 2017; new figures may have emerged in the party. But right now, it is incredibly difficult to see who other than Watson can replace Corbyn and keep Labour together. Everything points to him; and Blairites most certainly need not apply. Louise Mensch 1, Keiran Pedley 0.

Advertisements

The polls and (all but one of) the forecasts WERE wrong. Ed Miliband was nowhere near becoming Prime Minister

Thursday May 7, 2015. Britain goes to the polls at what is universally – entirely wrongly – believed to be the closest General Election in fully two generations. The opinion polls are deadlocked, and have been for months. The choice facing the country is by far the most stark since 1992: this wasn’t, contrary to what 99.99% of people assumed, a close election – but it was certainly a watershed.

On Tuesday morning, I set out why I believed that almost all forecasts and predictions were wrong: Ed Miliband’s strategy had been hopelessly flawed; his party would find itself squeezed from all sides; and above all, that the methodology employed by every single opinion polling company was wrong. Alarmingly wrong. Disastrously wrong. Inexcusably wrong.

At this point though, I want to highlight that, in no small part, I was wrong too. While I foresaw an enormously disappointing night for Labour, I did not anticipate such an apocalyptic one. Whereas I expected that the Scottish National Party (SNP) surge would hold, and count horribly against Labour not only in Scotland, but even more in England, where it had become the central plank of Conservative strategy to frighten voters into giving David Cameron a proper mandate, I did not foresee the almost complete wipe-out which resulted. And contrary to my belief that the Liberal Democrats’ positioning themselves as a sensible coalition partner would enable their support to hold up to some extent, it instead entirely melted down: Nick Clegg’s party squeezed even more horrendously than Labour amid the strange, Byzantine beast of First Past The Post (FPTP).

True, despite all that, I still did considerably better than any of the forecasting websites, any of the polling companies: I got the big picture right, you might say. But even I stopped short of stating the Tories would win a majority; I thought a further Tory-Lib Dem arrangement was likely, and never foresaw that half of this equation would all but vanish from the electoral landscape altogether.

A little later, I’ll be introducing readers to the man who did get it right: the only man prior to the formidable exit pollsters who got it extraordinarily, almost impossibly right. But if you’ll forgive me, on this awful morning for progressive politics across England and Wales (if certainly not Scotland), I want to indulge in a spot of something so often beloved of those on the left: self-flagellation.

You see, what happened at this election – this supposedly cataclysmic shock – was supposed to happen all along. The government, while embarking on a series of horribly divisive, almost entirely unnecessary economic policies, had experienced no disasters. The Prime Minister’s approval ratings remained good; at times, remarkably good. Those of the opposition leader were, thanks to his palpable lack of gravitas and the toxic reputation of his party (blamed both for Iraq, entirely justifiably; and the 2008 financial crash, rather less justifiably) poor: at times, extraordinarily so given dramatically rising food bank use, rising inequality, and a prolonged fall in living standards unparalleled since the Second World War. Most of this directly affected those who would, surely, represent Miliband’s core constituency – yet any sense of cut-through remained elusive.

Then consider that, for reasons I set out last month, the nature of Britain’s electoral system has made it impossible for any party standing on anything resembling a truly radical, progressive agenda to get anywhere near winning an election since 1979 (in fact, since 1974): with the early 1980s split on the left meaning that, in effect, the entire voting system was dragged ever further to the right, a self-perpetuating process which is still ongoing and shows no signs of slowing down. And above all, on by far the most important indicator of any party’s readiness for government – economic competence – the Tories had remained well ahead of Labour ever since the crash; considerably because of the latter’s extraordinary failure to challenge a narrative about ‘austerity’ which isn’t only misleading – but is fallacious and increasingly dangerous to Britain’s medium and long term future.

This narrative, parroted relentlessly by the increasingly hysterical Tory press, the BBC, and both the Tories and Lib Dems, meant that when Miliband said perfectly reasonably that no, Labour had not over-spent before the crash, most viewers were horrified. How could they trust someone so irresponsible, not even prepared to apologise; who’d been part of a government which, so everyone insisted, had ‘run out of money’?

Never mind that no country in charge of its own money supply can ever run out of money (it simply prints more); never mind that Britain wasn’t even remotely imperilled in the manner of southern European countries trapped in the euro zone and crucially, without control of their money supply or economic policy; never mind that the effect of coalition-imposed austerity was simply to remove huge amounts of liquidity from the system, grind the economy to a dead halt, and it only began to recover when those policies were significantly ameliorated; never mind that almost all macro-economists around the world (notably the Nobel Prize Winner, Paul Krugman; the Merton College, Oxford Professor, Simon Wren-Lewis; and even the International Monetary Fund (IMF) itself) had rejected austerity as a busted flush; never mind that not Labour, but the coalition, had doubled the national debt, and left it massively more exposed to an increasingly possible second crash; never mind that the economy had been growing rapidly when Gordon Brown was forced out of office; never mind that borrowing costs are historically low, and inflation is at zero; never mind that the welfare state itself had been built by the postwar Labour government at a time the country was technically bankrupt (so it simply borrowed instead, investing in infrastructure and setting a course for the Keynesian consensus); never mind that the now immortalised Liam Byrne note was a playful aside to his successor in the manner of long established Treasury traditions; never mind that, mindbogglingly, the Tories were proposing a more extreme version of the very policy which had failed so completely in the first place… none of this mattered.

 

Liam Byrne’s note was immortalised at this election – by a coalition playing cheap, low, dirty politics

If a lie is repeated often enough, it becomes the truth. Thus both coalition partners asserted that Labour’s much more balanced approach to deficit reduction would “pass our debts on to our children and grandchildren”, even when Tory policy will, by preventing growth or re-balancing, actually do that very thing; both continued to espouse the risible nonsense that Britain’s debt (which remember, they had doubled) was somehow comparable to a credit card debt, or that running a country is akin to running a household budget.

The press, run by barons who benefit enormously from the continuous upward funneling of wealth to the super-rich, and who would be personally impacted by a mansion tax, the return of the 50p tax rate, and especially the removal of the absurd protection of non-doms, hammered the message home again and again: Labour would endanger everything. A shockingly economically illiterate public (so illiterate that this itself poses an increasing threat to public policy, and certainly to the UK’s fiscal health) would inevitably acquiesce: despite policies which do most of them ongoing financial and social harm. And once the ‘danger’ posed by a party with the brass neck to have huge numbers of MPs democratically elected by Scottish voters was thrown into a wholly disingenuous, toxic mix, the die was cast: with public minds panicked into nonsensical comparisons with the 1970s, told that Nicola Sturgeon would ‘drag’ Miliband to the left… despite the SNP actually standing for slower, more drawn out austerity than Labour.

But throughout the last five years, Labour themselves have been horribly culpable: for failing to challenge a false narrative, or set out their own plans in any convincing way. When Miliband defended the Brown government’s record during the televised debates, he needed to assert why it hadn’t over-spent – but in keeping with serious communication issues which dogged him throughout his leadership, he couldn’t. Instead, like a rabbit in the headlights, hoist by the petard of his own foolish commitment to austerity, he froze – and his failure to ‘take responsibility’ will undoubtedly have hung particularly heavy in undecided voters’ hearts in the polling booths yesterday.

 

The rabbit in the headlights look comes naturally to Ed Miliband

That the public have continued to blame Labour for hardship caused by the coalition is a huge part of the reason why Miliband’s results at local, European and by-elections were so poor; and those results, as we shall see, represented an enormous, critical warning: not only to Miliband, but the pollsters. Both ignored them (in the latter case, incomprehensibly so); both will have plenty of time to reflect and repent on this now.

To recap: miles behind Cameron on approval ratings, public credibility, and especially economic competence; lacking in authority or leadership skills; leading a party with a toxic image (and with a Shadow Chancellor who embodied this in public minds more than anyone else); standing on a progressive platform the like of which hadn’t succeeded at any general election in 40 years; overseeing continually poor electoral fortunes despite mid-term ballots almost always providing a huge boost to any modern day opposition (and for that matter, failing to pull into anything like the kind of mid-term lead which any opposition needs in order to win the big one); and up against a government regarded by most as perfectly competent, how could anyone possibly have believed that Ed Miliband’s Labour Party stood the remotest chance of being returned yesterday?

Every single key electoral indicator was against it and in favour of the Tories. The apocalyptic, beyond seismic fall-out from the Scottish referendum – incorporating a Scottish left feeling completely betrayed by the last-minute announcement of The Vow and Cameron’s apparent shifting of the goalposts the moment a NO vote was confirmed; furiously mobilised against a Scottish Labour Party who had stood alongside the hated Tories and therefore confirmed itself once and for all as ‘Red Tories’, who neither represented their interests nor had delivered anything resembling social justice despite 13 years of government – which, true to form, wasn’t so much ignored as entirely missed by Labour’s laughably oblivious London headquarters until it was already far too late – was only the latest of many factors which, combined, could only lead to one conclusion. Miliband would lose; and he would lose big.

Thus, a week after the referendum, a time when most commentators were incomprehensibly expecting a Labour majority, and had not picked up on what was happening in Scotland in any way, I stated that he would inevitably lose in this post on The Guardian. When challenged, I even forecast the share of the vote: 37 or 38% for the Conservatives; 31 or 32% for Labour – and posited:

I think we’ll start the general election campaign, ie. a month before polling day, with everyone anticipating a hung Parliament. We’ll finish it, as a minimum, with the Tories as the largest party – and probably with a majority.

To me, this was the only logical outcome. As we moved towards the election, I treated any Tory lead as a sign of the inevitable; but was bewildered as Miliband appeared (according to the fatally, catastrophically flawed polls) to overtake Cameron during the campaign, then as things remained level pegging all the way to polling day. I simply grew more and more suspicious: what was wrong with the polls? Why were they all saying something which I could scarcely even conceive of being true?

Then, as May 2015 noted the difference between telephone and internet polls, things began to make a little more sense. The Guardian/ICM poll revealing a huge difference in Sheffield Hallam when voters were quizzed on their nationwide choice (Lord Ashcroft’s technique in constituency polls, which plainly formed the bedrock of all forecasting models), but were then asked to factor in the local candidates (as they would on election day) confirmed to me categorically that I was right: the polls WERE wrong, and were both understating the incumbency vote and overstating the progressive one.

Yet even then, I didn’t follow this through to its logical outcome. In my article on Tuesday, everything I highlighted should have left me forecasting a Tory majority: not merely that Miliband could not possibly become PM. I should also have better appreciated that, given a hitherto merged party encompassing both the Liberals and Social Democrats had effectively cut its latter half away by joining the Tories, but would be facing those very same Tories (who the polls were under-stating) in most of its key seats – who basked in public approbation for economic competence, while the Lib Dems played the Aunt Sally role of being blamed for any of the coalition’s more pernicious policies – Clegg’s party wouldn’t just lose half its support. It would inevitably haemorrhage a whole lot more: and be lambs to the slaughter when up against the Conservatives.

Instead, naturally enough I suppose, I asked myself, “in 2015, with all their sophisticated techniques and given all the lessons of the past, could ALL the pollsters – even including Nate Silver, for goodness’ sake – really be THAT wrong?” So I restricted myself to predicting another Tory-led coalition, rather than extrapolating in conclusion the very thing which all my arguments should have led me to. My forecast remained much more favourable to Cameron than almost any other anywhere; but still, mea culpa. A lesson learned.

 

Even Nate Silver’s number was up at this election

Then on Wednesday night, as the nation slept before crunch time, and the likes of May 2015 asserted preposterous levels of hubris about the likely outcome, everything suddenly became clear. I had chanced upon the one and only forecasting hero of this long campaign: whose services will surely be in the highest demand in the years ahead, who had conducted comprehensive, demonstrably proven psephological research, and whose findings drove an absolute coach and horses through every single professional forecaster and polling organisation… as well as sending an unforgettably cold chill down my Labour-supporting spine. For the Conservatives, his work spelt Nirvana; for the opinion polls, it spelt Nemesis.

In an extraordinary tour de force, the best, most counter-intuitive piece of electoral research I believe has ever been conducted in the modern day (better even than Silver’s in 2008 and 2012), the website Number Cruncher Politics (NCP) explained that, according to all possible indicators and variables, the polls were telling a wildly different story to that accepted by just about everyone.

As I had done, NCP strongly suspected a repeat of 1992-style Shy Tory Syndrome; but unlike me, as the man behind it is possessed of the scientific expertise necessary to trawl through exhaustive amounts of data going back some 50 years and model it in various ways, he had the ability to conclusively prove it. By examining electoral data covering the previous 35 years, toplines from the last half century, and opinion polls from this Parliament, he identified a very clear statistical pattern, which repeated itself through three separate models. His main findings were as follows:

1. As I had noted in my article, opinion polls at British General Elections are usually biased against the Tories and in favour of Labour. The exception, when both the Conservatives and Labour were a little understated in 2010, while the Lib Dems were wildly overstated, was in all likelihood, a one-off (and to me, easily explicable by how vulnerable the soft progressive vote invariably is under FPTP. The same demographic of voters inflating the Lib Dem position 5 years ago were, I intuited, having an identical effect on Labour this year).

2. The unusual fluidity of the electorate during this last Parliament (so many Lib Dem voters abandoning the party; Tory and Labour voters heading over to Ukip) had almost certainly undermined – perhaps even entirely negated – the adjustments made by pollsters after the debacle of 1992. Especially when it came to the usual method of reallocating ‘don’t knows’ – because far fewer of these would vote for their traditional party, but amid a new, multi-party landscape, how could pollsters possibly determine who would, and who would not?

 

The rise of Ukip was one of many factors giving pollsters all sorts of extra difficulties

3. Every single one of 16 opinion polls ahead of an election over the last 2 years had fallen prey to a pro-Labour bias, at the same time as late swing to the Tories was occurring: unnoticed until the election itself. At the 2014 European elections, Labour’s lead had been overstated by 3.3 points; while all by-elections since early 2014 had displayed a huge shy Tory factor, averaging 5.5 points.

4. Conflicting internal polling – reports of which I noted in my article, and clearly precipitated a dramatic change in strategy over the last week as a panicking Labour campaign desperately courted the notoriously unreliable young vote (again, the same voters who failed to turn out for the Lib Dems in 2010, despite so many of them having promised their support) – bore uncanny resemblances to 1992.

5. Since 1992, the overstatement of Labour and Lib Dems had been almost uniform: ranging between 2.3 and 2.9 points.

6. Serious problems in weighting data from online panels; other problems in reallocating undecided voters in phone polls, not to mention poor response rates to the latter.

7. Polling error averaging 5 points net over all General Elections between 1983 and 2010.

8. Relative to the Tories, Labour’s local election performances in non-General Election years under Miliband had been the second worst by any opposition over the last 35 years.

9. All three models – based on adjusted topline numbers; polling internals; and actual votes – were telling an extraordinarily similar, unbelievably alarming story about this year’s election; and combined, when tested against the polls at every General Election since 1983, outperformed them on all but one occasion (2010, when the difference was a mere 0.2 points).

10. What was that story? Namely, that the Tories were heading for a victory by between 6, and more likely around 8 points – despite the opinion polls all suggesting things were dead level.

The sheer, overwhelming exhaustiveness of the research left no room for doubt. I was awestruck by what I had just read, and given my habitual sense that the polls were very, very wrong, knew it could only mean one thing. The Tories were heading either right to the cusp of an overall majority; or their first majority since, you guessed it, 1992.

Three final opinion polls were now due to be published before 7am. Inexplicably, none were; all appeared several hours later instead, along with a fourth too. Had the polling companies seen the NCP research, appreciated the catastrophe it foretold for them, and reappraised their data? I have no idea, but can only suspect not. I tweeted it to May 2015, but got no response: instead, the New Statesman’s forecast site happily updated the news that the final polls still suggested a tied race, and Miliband premiership.

Section 66A of the Representation of the People Act, 1983 states the following:

(1)No person shall, in the case of an election to which this section applies, publish before the poll is closed—

(a)any statement relating to the way in which voters have voted at the election where that statement is (or might reasonably be taken to be) based on information given by voters after they have voted, or

(b)any forecast as to the result of the election which is (or might reasonably be taken to be) based on information so given.

Above, point 1(a) refers to exit polls: which of course, are never released until 10pm. But 1(b), while not explicitly doing so, appears at all elections in the past to have been taken to cover opinion polls released after voting has begun. This was the sixth General Election I’ve experienced – and never before have opinion polls been published after 7am on polling day. Why were they on this occasion?

In fairness to the pollsters in question, the Act was hardly designed for an internet age dominated by social media and, at this election, by forecasting websites: all of which continued to apply the latest data despite point 1(b); while on Twitter and Facebook, various candidates re-tweeted endorsements from voters and voters declared who they had voted for. As The Telegraph notes today, some form of mild campaigning still seems to have been going on beyond that.

A mess, then, which the Electoral Commission would do well to resolve ahead of future ballots – but there are two vastly more serious points here. First, for the duration of the campaign (and, for that matter, a good 2 years prior to that, according to NCP), all polling companies and forecasting sites have been publishing wildly, at times ludicrously inaccurate information which inevitably influenced the race, and the outcome. When parties are assumed to be tied, their strategy changes; media treatment of them substantially changes (never more so than in Labour’s case this year); public responses change too. In the latter case, how many more voters are likely to vote a certain way out of fear that those they’re opposed to are in touching distance of victory? And under FPTP, how many vote tactically who would otherwise not have done – and vice versa?

In practice, as the NCP model conclusively demonstrates, Labour were never ahead, and have probably been several points behind throughout the campaign. Yet that’s not what the polling companies were saying. Should organisations with such disastrously flawed methodology and a consistent record of inaccuracy which, as exposed by NCP, dates back at least 32 years, be allowed to dominate the agenda in such a way; and above all, to have such heavy influence on debate and public discourse?

Remember: huge amounts of the Tory campaign were dedicated to frightening English voters into stopping a minority Labour government propped up by the SNP – but in practice, this was never the prospect it appeared, because Labour were doing much worse than was believed. Enormous amounts of discussion were put over not to policy, not to manifestos, but the electoral and Parliamentary arithmetic – but this bore no resemblance to the reality.

Tory scaremongering about Nicola Sturgeon was based on a false prospectus

Is it any wonder the British public have such little understanding of macro-economics when whole election campaigns – leading to a decision hugely determining the futures of them, their families, their loved ones – are given over to constant reactions to never-ending opinion polls: even when these polls are completely wrong? Other countries (most notably, France) do not allow polling firms to play such a huge role during the final week of election campaigns; why on Earth does the UK?

To make matters worse, the thesis I set out on Tuesday focused heavily on the role of Lord Ashcroft: whose constituency polling has been a huge feature of this campaign, and clearly became central to most forecast sites: most notably, May 2015 and Election Forecast. My piece concluded that his data was almost certainly awry; and in that, we now know that he was in very good company with all other polling firms.

At The Telegraph, virtually ever since Ed Miliband was elected Labour leader, Labour-supporting Dan Hodges has consistently (and we now know, quite brilliantly) forecasted Miliband’s demise. He must be experiencing very conflicting emotions this morning; as a fellow Labour traveller, I know I am. In February, he asked a question which the British polling world should have already been asking: “What does Lord Ashcroft want?”

As Hodges set out, over the last 5 years, Ashcroft has morphed in public persona from hugely controversial non-dom to friend of the political process: opening up the business of polling to the public in a manner never seen before. But Ashcroft, contrary to what so many must assume, is not a pollster: he buys in polling from other companies, publishes the results, but won’t reveal who these companies are. He himself is not a member of the British Polling Council either.

Not only that, but he’s an extremely wealthy Tory peer, and former Deputy Chairman of the Party. During tbis campaign, he’s tweeted his admiration of Sturgeon – whose ‘danger to England’ just so happened, by purest coincidence, to constitute the central plank of the successful Conservative strategy. I have never known a Tory give such regular praise to a nationalist in the way Ashcroft has.

His final ‘snapshot’, released well after 7am yesterday, had Conservative and Labour tied, and only added to the bigger picture that the two parties were deadlocked. But ask yourself, purely hypothetically: if you were a Tory who naturally desired your party to win as convincingly as possible, would you want the final poll to have them well ahead… or locked in a race too close to call, which would encourage maximum possible turnout? More to the point: purely hypothetically, if you were a Tory who wanted maximum possible negative exposure of Labour throughout an election campaign, scaring the public into voting heavily against them, would you want them to be well behind… or seemingly on the verge of victory?

How far the obvious flaws in Ashcroft (sorry, I mean the companies which he buys his polling in from)’s data help explain the almost identical flaws in every other firm’s data, I couldn’t possibly begin to imagine. But how has such an obviously self-interested individual become the most influential figure in opinion polling? Why were at least two very well-publicised forecasting sites so dependent on deeply flawed data? Why haven’t more questions been asked about his motivations?

 

What does Lord Ashcroft want?

One man who has asked such questions – many, many such questions – has been the Labour peer, Lord Foulkes. He conducted his own online pursuit of his Tory contemporary throughout the campaign: leading to frequently entertaining exchanges, but never leaving observers in much doubt as to what he thought. Foulkes has hinted for many days that the polls were wrong; and in February, went a great deal further. Opinion polls, he fulminated, were increasingly:

“Being manipulated at the behest of people with money, whether they be the media or individuals, as part of the political process… What is clear now is the media in particular, but others as well, are demanding instant polling, determining when it should be done and how it should be done. The academic rigour that ought to be carried out isn’t being carried out.”

And to be sure, given what this article has set out, who could possibly argue true academic rigour had been carried out on a whole series of polls which haven’t just been a little wrong – but a lot wrong? Very wrong. Astoundingly wrong.

Similarly astounding, when you stop and think about it for a moment, is how, after half a century’s dominance of Scotland, the Labour Party have been swept away just like that via a speeding yellow flash; meltdown triggered, as noted above, by The Vow and its ongoing fall-out. But what’s so often forgotten about the referendum campaign – the democratic event that changed Scotland irrevocably forever – is that between August 2013 and polling day itself, just two opinion polls (and only one with a sample size of over 1000) put YES ahead. That latter poll, by YouGov and the Sunday Times, was publicised with unusual relish by Rupert Murdoch on Twitter, and sent shockwaves through the British establishment.

Its response to one solitary rogue poll? The Vow. As soon as it had been made, Labour were in no position to control what resulted; and when that involved Cameron cutting the rug away from Gordon Brown’s feet on the steps of Downing Street, suddenly, a party which for so long had dominated Scottish politics was faced by an oncoming train it (because it was in government in neither England nor Scotland) could do nothing about – which yesterday, flattened it completely, destroying its broader electoral hopes in the process.

Opinion polls, then, can be remarkable things with yet more remarkable consequences. Especially when they’re conducted for a newspaper de facto owned by someone with a surprisingly good relationship with the then First Minister of Scotland; even more when they help trigger the collapse of a party which had taken that someone on in a manner of no other British political party in my lifetime. What did that someone’s lead redtop do during the election campaign? Simultaneously support the SNP (against Labour) in Scotland; the Tories (against Labour, while banging home the message of dangerous, rebellious, left wing Scots coming to rule over the English) in England.

Rupert Murdoch, of course, was humiliated by the Leveson inquiry into phone hacking; deemed “not a fit person to exercise the stewardship of a major international company” by the British government, and has been investigated by both British and American authorities for bribery and corruption. This perhaps helps explain The Scottish Sun’s enthusiastic support for the SNP; The Sun’s vituperative opposition to Ed Miliband, whose ideas threatened to end any remaining influence of Murdoch over the UK media, and its voicing of a narrative which would inexorably pull England and Scotland apart, threatening the imminent break-up of a 308-year-old Union.

Through his actions, Murdoch has left no doubt regarding his open hostility to the UK and British establishment. Hell hath no fury like a global press baron scorned. At the same time as Murdoch has fallen from grace in Britain, Ashcroft’s star has dramatically risen. Which is, presumably, entirely unremarkable – but the examples of both provide a heavy reminder of what can happen when vested interests collide with the democratic process.

As Foulkes has noted, extraordinarily, the ever more influential polling business is entirely unregulated. In such a world, despite their consistent inaccuracy, all polls are effectively taken on trust: a remarkable state of affairs. Foulkes has therefore introduced a Parliamentary Bill intending to establish an independent regulator; which will also place restrictions on polls being published in the three weeks leading up to an election. In light of all that’s happened in this campaign, one can only wish his Bill speedy and safe passage.

To return, though, to yesterday. I now knew conclusively that the polls were hopelessly askew – but the dear old British public did not. Neither did the BBC presenters: who in tandem with the entire watching world (but not, at least, maybe two or three of us) greeted the exit poll with incredulity. The moment I saw that poll, I was entirely sure it was either (a) absolutely accurate; or (b) still underestimated the Tories’ position. The latter was how it proved. While frantic Labour supporters initially clutched at the straw of an exit poll that wasn’t – a YouGov re-contact survey confirming the previous day’s numbers, which had been arrived at via all the same mistakes as all the other polls – I simply told my friends that it was over. And it was.

We’ll never know whether those in charge of the exit poll had been alerted to NCP’s work. Either way, though, they at least did a magnificent job in hellishly demanding circumstances; they at least emerge from this complete fiasco with their heads held high.

As I type, David Cameron is now back in Downing Street, gleefully clutching a 12-seat majority. He and the Tories, though, will soon discover that in politics, as in life, we should be careful what we wish for; for with an EU referendum now certain, Cameron is likely to be even more boxed in by his lunatic fringe than John Major was after 1992. The chaos of that period destroyed the Tories’ public image, opening a chasm in the party which even the staunchly Euro-sceptic Cameron has been unable to close. With the economy now likely to be sent straight back into recession by even heavier austerity than before, I would be moderately surprised if the Prime Minister makes it as far as the referendum; and in the event of a vote to stay in, even more so if the Conservatives don’t do what they’ve been threatening to do for over a generation, and split completely and irrecovably.

For the other big winners of the night, the SNP, the path to independence is clear. How can a country with one Tory MP and 56 nationalist MPs possibly be governed from London by a majority Tory administration? Answer: short of immediate full federalism and an immediate change to proportional representation – preventing such a scenario ever occurring again – it cannot. The Union’s days are numbered.

And Labour? With Miliband having resigned with typically good grace, now the party must put right its awful, tragic error (for both men) in choosing the wrong brother in 2010; and resolve its ongoing internal debate between the kind of quasi-social democratic agenda offered by the now departed Ed, and something closer to the Blairism necessary to win under FPTP. Chuka Umunna would plainly represent the latter; but in truth, if Labour are ever to regain a foothold as a natural party of government (especially once what we know now as Britain becomes merely England and Wales), it desperately needs to reach out to and reconnect with the many millions of working class voters it has shed since 1997.

Tony Blair knew how to win; but did not know how to create something truly long lasting and self-sustaining. Only by shunning both Blairism and Milibandism can Labour hope to do that: it needs to forge a new non-statist, localist, communitarian approach, in which the ties of family, friendship and community are reborn; power is devolved downwards to localities and through federalism, and offers real remedies for Britain’s increasingly divided, fragmented, discordant, atomised society.

That huge bloc of traditional support (the latest chunk of which voted SNP and Ukip last night) must be won back; the sense of alienation they feel must be listened to and engaged with. Their aspirations – as well as those of small business or bigger business – must be met: a much larger, much more coherent coalition of Labour support than is ever generally appreciated is out there, but has been almost entirely neglected for a whole generation.

As the country discovered yet again last night, British politics only works for a few. As it regathers its energies from this latest devastating blow, it is Labour’s task and will always be Labour’s cause to ensure that, by offering new solutions to both new and old problems, it finally begins to deliver for the many.

 

No, not New Labour. Blue Labour.