I wonder how the Tories and other right wing parties even gained their fiscally responsible reputation to begin with. Because it doesn't hold up under any sort of scrutiny.
It were Blair/Brown who seem to have run the tightest ship economically, and prime ministers such as Atlee and I believe Wilson who are the ones who's administrative abilities were the most highly valued. And currently the Torries have thoroughly identified themselves with the extremely fiscally irresponsible idea of Brexit.
Same with the US where Clinton is the one with the reputation of economic miracles, where Obama and not Bush tackled the recession and where Democrat Rooseveld is the posterboy of how to solve a bad economy while Republican Hoover is the poster boy of how NOT to do it.
Even the fiscally well regarded Reagan and Thatcher don't really explain that mindset since its well established these days that the economic legacy they left behind was a house of cards that we now see collapsing.
The source of the modern reputation is the borrowing/debt incurred by Blair and Brown. Throughout their tenures, the British press hammered the government on the supposed irresponsibility of large-scale borrowing.
Then we had the global recession, and the press successfully tarred Brown as responsible for its impact on Britain. It was largely a false accusation, but it stuck.
Then, when the Cameron beat Brown in the election, Brown's Chief Secretary to the Treasury Liam Byrne left a (supposed-to-be-humourous) note saying "sorry there's no money left". Cameron hammered that as an example of Labour being cavalier about public finance. The public were also largely unaware that a Tory chancellor had made the same joke in the 60s.
Then we had the government of Cameron and Osborne, who heavily sold cuts to public services as a "responsible" necessity. The following Tory governments actually borrowed at a greater rate than Blair or Brown ever did, but-- again-- the public are generally unaware of that. Because the press only ever drone on about Labour doing it.
Tl;dr: it's the rabid right-wing press, and public ignorance.