Mr Pascoe, you might like to expand your reading as there are obvious gaps. For example, you might start with Brezinski's The Grand Chessboard where he boasts about the US setting up al Qaeda in the 1970s to wage a campaign in Afghanistan to lure the USSR (as it then was) into occupying Afghanistan thus "giving them their own Vietnam". Al Qaeda means "the base" i.e. data base, and originally referred to a list of jihadis who could be recruited to be used as an arm of US foreign policy. That remains one of its central functions, as Peter Dale Scott and Nafeez Ahmed among others have pointed out in several scholarly treatises on the subject.

This is of course not a new phenomenon. Read for example Ganser's and Cottrell's books on Operation Gladio where, in pre-Muslim bogeymen days, left wing groups were used to carry out terror attacks in Europe, justifying, inter alia, repressive government measures to ensure the continued dominance of right wing (i.e. US friendly) governments. Plus ca change.

It is important also to note the local context in which radical groups arise. Algeria actually elected an Islamist government in a free election, only to have it overthrown by the army, and then endured decades of brutal repression. You will recognise the parallels with modern Egypt. Similarly al Shabaab, which you mention, arose as a consequence of the US backed invasion of Somalia by Ethiopian troops. There are many other examples, but you see my point.