The other answers are great but focus on mobile phones, so I'd like to talk about landlines.
Before even talking about phone lines or networks, let's talk about the phone itself. Most landline phones today are cordless phones you can get pretty much everywhere. They use DECT, a protocol with proprietary crypto that's broken and only requires a DECT NIC plus some CUDA GPU time to crack the crypto (if it's even enabled). No physical access is required. An attacker can eavesdrop on your calls, spoof your bank's number and do all sorts of other evil without even touching the phone network. This is as easy as cracking a WEP key, and the fact that these phones are so widespread doesn't make it any better.
Now let's talk about POTS (prehistoric analog phone line) and the new "fake" POTS offered by DSL and fiber providers.
With traditional POTS the audio is transmitted in the clear over phone wire, sometimes over several kilometers. Depending on your country and the state of the POTS infrastructure, the cables could be (relatively) secure underground, in the air on utility poles, or damaged, barely hanging off a electric pole somewhere. Eavesdropping on that is as easy as connecting another phone on the wires and recording, but this is still harder than breaking DECT and requires physical access. I'm not an electrical engineer but I suspect there may also be leaks in the form of crosstalk with other lines, possibly caused by old/damaged switching equipment, so using potentially expensive equipment someone may be able to listen in from a different line that shares the same switch or wiring.
With the new fake POTS, it's actually VoIP that gets turned into POTS with FXS hardware inside the modem/router (this is actually very stupid, as they force the user to buy often insecure wireless phones and lowers the quality instead of using an IP phone and secure crypto).
Compromising the modem/router is enough to listen in and impersonate your bank or any other number. No physical access is required, the attack can be done online as the modem/router is directly exposed to the Internet and often have disastrous firmware.
If compromising the modem isn't a possible, it may be possible to eavesdrop on the actual DSL line, cable or fiber, and from there attack the VoIP protocol (SIP or MGCP). Physical access is obviously required, and most equipment is relatively expensive so I wouldn't worry about banking data - someone who MITM's your DSL line definitely isn't after your banking data, as he probably already has much more in his own account than you do so if your bank data gets stolen it's unlikely that your DSL line is the culprit.
DSL eavesdropping requires expensive equipment (a DSLAM basically), but once that's done there's no encryption of the DSL traffic. Both PPPoE and IPoA are merely encapsulation.
Cable and fiber eavesdropping is easier. Both cable (DOCSIS) and consumer-grade fiber (PON) broadcast the downstream traffic to everyone connected to the same CMTS/OLT, it's supposed to be encrypted but I wouldn't count on it being enabled (France's largest DOCSIS provider didn't have it enabled last time I checked in late 2013). Listening on the downstream channel for cable only requires a basic DVB-C TV tuner card, and transmitting would require a rooted cable modem. PON eavesdropping requires a rooted ONT for the downstream channel, but I believe physical access to the OLT is necessary in case of fiber - the upstream channel may not be broadcast to everyone.
Once we assume the DSL/fiber/cable is broken and MITM'd, there's the actual VOIP protocol, SIP or MGCP. Both support encryption, but I doubt it's enabled. Most likely it isn't, so listening in would be very easy. If crypto is enabled you could still try and MITM the connection, chances are the awful SIP client running on the modem/router doesn't properly validate certificates and would accept your self-signed cert.
That's about the part that connects you to the phone provider (often your ISP). Once there, your call may go through lots of legacy and potentially insecure systems, and even the call center's infrastructure may be a disaster. Sadly, the DECT and DSL part may only be the tip of the iceberg.
About caller ID, it's spoofable by design. Phone providers can set the caller ID string to anything and it'll work just fine. Most providers often prevent end-users from setting their own caller ID, but I'm sure there are some that don't do these checks and attackers can use those providers to impersonate any number.