The London Underground has been going since 1863. It was built at a time when there was no real alternativr of moving the working classes between the expanding fottpront of London and their place of work It was put underground as London was already a developed city from medieval times and putting a mass transit system above ground was not practical. I did a gig with London underground for 4 months back in the late 90s and they still used vacuum sealed resin coated relays for signal switching then They have since upgraded most of their signalling. It is a great system and once becomes accustomed to complaining when you have to wait more than there minutes for the next tube on some lines. It is. however, quite expensive if you live a normal commuting from the outer burbs.
Also, if I compare London to say Melbourne or Brisbane (the two cities I know best in Aus), then the footprint in London of what we would call the City is a lot larger. I work on the east side of London at Liverpool street station. This postcode is EC2 - East Central London 2.
This map is roughly what would be considered the CBD of London equivalent of Melbourne taking into account shopping and office/working precincts.
You could remove the stuff south of the river on the south eastern end of the map. It is about 6km wide and 2.5 km high. Maybe going out as far as Marylebone is a stretch, but in terms of central business, it is very densely build - much more on a relative basis than outside the CBD of Melbourne. Even if we argue about semantics, London's population swells to up to 10m on a working day and a lot are within that perimeter. So a mass transit system to move people about within the most densely populated business areas is needed, and overground buses won't cut it.
Also, fun fact, all of the early tube stations are colour coded. This is because in the early days, not a lot of the working class could could read, so they colour coded the signs to let them know when it was their stop.
As a person who lives in London 2 days a week now and up until COVID, 5 days a week and for 20 years before that in London, like every mass transit system, it has its issues. The reason why I went back to motorcycling is London started having more of its fair share of issues and it was costing me a lot of time. Less the underground, to be fair, but there were times where i was stuck on a station for over an hour, or got to Waterloo to see every train cancelled and the station concourse packed with people. It is still a great system, though.
However, Melbourne's inner transport system is excellent, too. The trains are a lot better; the most extensive tram system in the world provides excellent options, although they are slow if you need to go to, say Bundoora. I think there are still buses, but they seem a top up system. The greater Melbourne, as is most of Australia, is built for the car. There are pluses and minuses of both, but a city with a physical size as big as Melbourne, even though I think is the most populous city now of over 5m based on the classification of significant urban area makes it more difficult to have the network of public transport that a much higher densely populated area of London (according to Google London has 3.5 times the population density of Melbourne). Also the London road system which matured wwell before the car makes public transport a much less stressful than driving.