How does it even work
carp is a protocol, pretty much like VRRP, sharing IPs among several hosts.
If you were to run tcpdump on the interface on which your carp was configured, you would be able to see messages such as:
# tcpdump -vvni vlan1000 proto carp
tcpdump: listening on vlan1000, link-type EN10MB
19:34:59.688730 carp 1.2.3.4 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=10 advskew=5 demote=0 [tos 0x4] (ttl 255, id 25329, len 56)
tcpdump -vvni vlan42 proto carp
tcpdump: listening on vlan42, link-type EN10MB
19:35:56.610612 carp 10.42.42.5 > 224.0.0.18: CARPv2-advertise 36: vhid=42 advbase=10 advskew=5 demote=0 (DF) [tos 0x10] (ttl 255, id 13021, len 56)
Carp members would send messages to a multicast address, advertising their presence to each other. Only one member would be elected as a master on a given carp group.
In case of active-active scenarios, you could still have two or more hosts serving the same carp address, assuming pf states are shared (pfsync) among carp members.
Carp addresses would resolve to predictive hardware addresses, based on your carp VHID:
# arp -na|grep 'carp.*permanent'
10.42.40.1 00:00:5e:00:01:0b carp11 permanent l
10.42.41.1 00:00:5e:00:01:08 carp8 permanent l
10.42.42.1 00:00:5e:00:01:2a carp42 permanent l
10.42.43.1 00:00:5e:00:01:07 carp7 permanent l
10.42.44.1 00:00:5e:00:01:04 carp4 permanent l
10.42.45.1 00:00:5e:00:01:03 carp3 permanent l
10.42.46.1 00:00:5e:00:01:0a carp10 permanent l
10.42.242.1 00:00:5e:00:01:02 carp2 permanent l
10.42.252.1 00:00:5e:00:01:64 carp100 permanent l
10.42.253.1 00:00:5e:00:01:05 carp5 permanent l
10.42.254.1 00:00:5e:00:01:06 carp6 permanent l
1.2.3.4 00:00:5e:00:01:01 carp1 permanent l
192.168.10.254 00:00:5e:00:01:0c carp12 permanent l
For VHID 1, my carp address would have its mac set to 00:00:5e:00:01:01, for VHID 2 00:00:5e:00:01:02, ... and so on. Hence: regardless which is your carp master, clients in that network would not need to refresh their ARP cache, they would always contact the same hardware address.
The reason why having two hosts sharing the same IPs in general is "not pleasant", is because for one: they don't have the same mac. Based on who replied to your last arp request, you can switch from one to the other unexpectedly. Also, both your pfSense share a similar configuration, ensure the same functions in your network. While when mistakenly re-attributing an IP, you usually have two completely different machines.
Should CARP members also have physical IPs
It depends. Most of the time: no. I tend to set one on management or public interfaces, making sure that I can still reach a backup host (eg: monitoring, troubleshooting some connectivity issue, ...), or just setting up pfSync.
With FreeBSD, you may also be using hastd+ctld+ifstated sharing HA LUN devices. Then, I would also setup a physical IP on the main interface, as regardless of who's master, I'ld need hastd to sync my devices.
