There are lots of interrelated causes, but TL/DR we biffed the introduction of supersonics in the 1970s — building large SSTs before we had the tech to make them efficient — and then banned supersonic flight over land, busting the business case for supersonic development. With speedups off the table, big aerospace focused on incremental improvements to efficiency. Fortunately, technology and business case have converged, and a supersonic renaissance is finally just around the corner.
Elaborating:
Less than a decade after the dawn of the jet age, supersonic transport (SST) projects were launched in Europe, USSR, and the US. Unlike the majority of jet airliner projects, the SST projects were government-led with requirements set by Cold War oneupmanship rather than practical economics. So we got Concorde: a technological marvel with unworkable airline economics.
Concorde was a joint venture between the French and British governments. With afterburners and 1960s-era systems and aerodynamics, Concorde was a gas guzzler, requiring higher ticket prices — ultimately $20,000 (2016 dollars). Yet Concorde had 100 seats: it’s hard to find 100 people who can afford that price for routine travel. As a result, there wasn’t much of a market, and there were no economies of scale.
The American “Concorde-killer,” the Boeing 2707, would have been even worse. Taxpayers footed most of the development bill and the federal government set requirements for glory, not practicality: a 300 seat, Mach 3 airliner. The seats would have been impossible to fill at the required fares, and the SST had little hope of earning a profit for airlines. Boeing cancelled development in the early 1970s after Congress pulled the massive subsidy.
Then something rotten happened: supersonic flight over land was banned in the US. Ostensibly, this was about sonic boom noise concerns — but IMHO it was really about protecting US aerospace from Concorde competition: had noise been the real concern, there would have been a noise limit, not a speed limit.
It would be natural for supersonic travel to start with business jets (“SSBJs”): relatively small vehicles, easier to make quiet, aimed at a market willing to pay a premium for time savings. But the supersonic overland ban busts the business case for SSBJs: 75% of bizjet miles are flown over land and buyers are loath to pay top dollar for an aircraft not faster most of the time. Business jet makers like Dassault, Gulfstream, and Cessna have looked at the SSBJ market and concluded it doesn’t make sense until you can fly supersonic over land.
So, instead of private supersonic aircraft paving the way for mainstream supersonic travel, we’ve had to wait until the technology was ready to serve a large market, despite the overland supersonic ban.
Finally, that day has arrived: with carbon fiber composites, modern aerodynamics, and turbofan engines, one can build a small supersonic airliner profitable at business-class fares. Thanks to globalization-driven increases in travel, there is a market for over 1,000 such aircraft, even without supersonic flight over land. That’s a large enough market to justify the development costs.