Estimates of the excess deaths linked to climate change range from ~250,000 annually to the millions by next decade. Not to mention the wholesale flooding of countries in the global south, drastically lower life expectancy of urban people linked to pollution and poor air quality, increased likelihood of wildfires and other natural disasters, etc.
This isn't 'counterfactual'. It's already happening.
There's a mix of information there.
-Excess Deaths: Human deaths from natural disasters have plummeted over the course of the 20th/21st century, even as the human population has increased. It's something like a 98% reduction.
-Life Expectancy: Human life expectancy has skyrocketed over the same time period, and it's a trend that's true of every continent on Earth.
-Air Quality: 10 million people die every year due to indoor air pollution from not having clean fuel, so while I'm not disputing the problem of air pollution, I think in a lot of ways, this is literally case of "pick your poison."
This isn't an argument against climate change being an issue (which it is), or that we have to get off fossil fuels (which we have to), or that climate change is having a deletrious effect right now (which it is), or that LEDCs are more vulnerable than MEDCs (which they are), but if we're dealing with counterfactuals (say the Industrial Revolution never occurred), then the facts are that there'd be far fewer humans, living far poorer lives. By literally any measure of human wellbeing, humanity's never been doing better in the period we're living in now.
On top of that, it's unclear that's even true, as a warming climate could just as much increase arable land away from the equator and prevent some of the millions of annual cold-related deaths.
That's not a good argument to use.
Yes, technically, an increase in temperature could cause some areas to become more arable, but other areas are going to become less arable, and everything I've read on the subject indicates a net loss. There's also the fact that agriculture developed in a period of climatic stability that climate change could jeprodize. Furthermore, the people who lose access to arable land (i.e. what's already happening in Africa) are going to have to go somewhere. Even if one takes the position that doesn't matter (and I'll give you the credit of assuming that you agree that it does), then even from a purely selfish point of view, how do you feel like dealing with a refugee crisis of up to a billion climate refugees?
You're trying to isolate the use of fossil fuels from the civilization building consequences of their use, focusing on a single side effect of those fossil fuels, and still only considering the downsides.
There's countless downsides to fossil fuels - air pollution, global warming, finite resource, ecological destruction, etc.
Even if climate change wasn't an issue, there'd still be plenty of reasons to transition to alternate energy sources.
Downside is that the planet's arguably never been doing worse since humans climbed down from the trees either, so again, pick your poison.