I don't fancy your chances of being correct in any of your prognostications, but rather, chalk them up as more science fiction, and in a right wing world where anything (even the most morally reprehensible) is possible; including a return to virtual slavery and child labor.
Well the line on the profit graph must always rise, along with parasitic (drones) shareholder expectations!

I liked the Dunes trilogy, and the eventual outcome.

And planet moving tractor beams, which can even realign earth sized planets or moons, into new orbits.

The unexpected failure to change the human condition has its origins in corporate greed, and "minds" that serve their needs, rather than mankind!
Minds that oh so cleverly warp and twist the facts, or indeed, simply ignore science based evidence, simply because it doesn't fit the preferred belief system, or the corporate wish list! And if the cap fits?

Anyone can own their own opinion, but nobody can own the facts! Or ignore them at their peril!

The human condition is not aided either, when there are just too many of us, or one group believes they're born to rule; and therefore exploit the less well off shamelessly.

Simply put, our future will be little different than that of Venus, unless so called scientists, stop refusing the face the facts; as opposed to cherry picking from them, just to suit the conformation bias!

Something that ought to be forever banned from the world of real science!
Expose the literal facts, even the extremely unpalatable ones; then let the chips fall where they may!
Rhrosty.

'But the changes in our society since his boyhood are enormous, and they result from great productivity which in turn is the outcome of advances in technology and a much greater scientific understanding of the natural world. Those advances don't have to be linear.'

A fascinating train of thought. Of course advances in science, technology and in social evolution are not linear. But what is the trigger that starts a specific advance?
Human kind, that is homo sapiens, has been on the move out of Africa for over 100,000 years. Yet what we call civilisation can not be dated back much more than 5,000 of these years. Why the long delay? Why did we suddenly find a need to better understand our environment and our place in it?
After reading a story in Australian Geographics 123 "Messages from Mungo” by John Pickrell it suddenly hit me. It is the environment we create for ourselves that causes the change in us! Mungo Man died 40,000 years ago and to all intend and purposes he and his people lived much like the Aboriginal people lived when the white man invaded Australia 250 years ago. In 40,000 years they never built any cities, never had permanent homes. Because there was an overabundance of living space in this new continent. If crowding occurred someplace, they just moved elsewhere. Once living space becomes restricted and people are forced to live in cities, that brings out a desire to improve conditions and a need to understand the limits.

So, it is quite likely that in the distant future, when living space no longer challenges us, that social and technical evolution will settle at (or regress to) a new level.

Science fiction has been entertaining and enlightening for most of my life. Much of it has come to pass, especially that sub-section of it I refer to as social fiction. "1984" remains pretty much spot on, with the failure to predict computers being the only jarring note.
William Gibson is hard to fault.

My experience with Sci-fi has been similar, however I don't agree that
"the changes in our society are enormous...because...if there is a problem of any kind, we approach it to see what is its cause, and how we can best deal with it....I remain positive about humanity's capacity to...apply knowledge well. In most of the world, conditions for the ordinary person are very much better than they were twenty years ago, let alone a century ago."
There is no discernible difference between the way most human societies are organised and run today than in the past. It is still dog eat dog, profit at all costs, slash and burn and to hell with the consequences. There have always been individuals who can see the problems of human behaviour, but they have always been paid lip service and ignored. We know the causes of Middle-Eastern strife, but the solution doesnt suit our oligarchs.
There is a plethora of knowledge about how humans could live in plenitude and harmony, but that knowledge is ignored because that's not the way we evolved. If you think living in a vast megapolis with all the attendant problems is better than in a forest or on a fertile river plain with clean air, water, and an abundance of food for the taking, then I guess they're better off, but I know I would rather be dead that in the position of 99% of humans today.
For me the beauty of Sci-fi for me was that the best tales reflected the failings of the human psyche, but gave pointers to how it could be improved. Heinlein springs to mind.

Another interesting thread Don.

I used to find most science fiction very good on the yacht, when I was a long way from anywhere, with no TV reception, & difficult radio as well.

I found most short stories, & even many novels were very forgettable, particularly the plots. In just 3 to 6 month you had totally forgotten the thing, & could read it again as if a new novel.

Supply was always a problem, but then I got lucky. I picked up the 30 volume 15Th edition of the Encyclopaedia Britannia at a garage sale in Honiara, Solomon Islands. After a bit of hopping around it, I decided to read it cover to cover was the best idea. Bet not many have done that.

Funnily enough, it often felt like I was reading science fiction when reading it.

Encyclopaedia Britannica! God I hate this auto spell correction.