Why IPSec transport still exist if almost always it could be changed to tunneling and vice versa?
I don't see Transport mode IPSec used in the general population of networked device user today. I think it never built up enough momentum to be universally deployed. Software and network vendors had motivation to sell Tunnel mode implementations (plus extensive backends) to enterprise customers with remote access need, but didn't push Transport mode to anyone. The capabilities may have existed, but the ease of use still leaves much to be desired.
So it exists, but is it still relevant? Transport mode historically wasn't accessible to a large number of users. One exception was the free software folks.
History and implementation status of Opportunistic Encryption for IPsec
The above link describes the historical effort to put IPSec into use everywhere, and how those efforts were stymied. The reasons can be summarized as the insecurity of Internet infrastructure (i.e. DNS), and the relative complacency from those involved to change it.
why do all these approaches exist and duplicate each other?
All these approaches exist primarily due to to independent identification of and solution to variations of the need for secure remote access, at approximately the same period of time. A slightly improved version of your question might be "why are all these approaches still in use?"
You've answered your own question about why L2TP is still in use: accounting and configuration. (it may be more interesting to look at why other protocols, such as PPTP, are no longer in use.) In many cases, even if you don't care about accounting and configuration,
In other cases the answer isn't as clear. Take the gateway-to-gateway case. You could use pure Tunnel mode IPSec, or use GRE tunnels over IPSec (in fact, I believe they are over Transport mode IPSec). I don't know that there's any advantage one way or the other besides familiarity. Personally, I've never set up Tunnel mode IPSec on a Cisco router. I've always done encrypted GRE. Why? Because everything I know about plain GRE applies to encrypted GRE. So it's familiar to me.
Don't forget application-level VPNs/Tunnels, such as OpenVPN or Secure Shell. These generally have poorer performance than kernel- or appliance-level implementations. But they were (and are) generally easier to use and had the advantage of more easily getting through proxies and firewalls (at least until the advent of deep content inspection). Also, they often have fewer dependencies; it's much easier to compile OpenVPN on an old Linux server than to recompile the kernel to support IPSec.
Could you give me example of situation when one of these methods is "the only right one to use"?
In networking (as with so much in computing), you'll never see "the only right one to use". In most cases, you are stuck with what is feasible. For example, all Android devices work with L2TP-based VPNs. So that is feasible, even if you don't need configuration or accounting. The familiarity I have with GRE tunnels on Cisco devices makes them easier for me to implement than pure Tunnel mode IPSec. And I can make an OpenVPN or a SSH-based tunnel on an ancient Linux server that I can't upgrade (for some reason or other).
