Firstly, from their paper
Messages to WhatsApp groups build on the pairwise encrypted sessions
outlined above to achieve efficient server-side fan-out for most
messages sent to groups. This is accomplished using the “Sender Keys”
component of the Signal Messaging Protocol.
The first time a WhatsApp group member sends a message to a group:
- The sender generates a random 32-byte Chain Key.
- The sender generates a random Curve25519 Signature Key key pair.
- The sender combines the 32-byte Chain Key and the public key from the Signature Key into a Sender Key message.
- The sender individually encrypts the Sender Key to each member of the group, using the pairwise messaging protocol explained previously.
For all subsequent messages to the group:
- The sender derives a Message Key from the Chain Key, and updates the Chain Key.
- The sender encrypts the message using AES256 in CBC mode.
- The sender signs the ciphertext using the Signature Key.
- The sender transmits the single ciphertext message to the server, which does server-side fan-out to all group participants. The “hash
ratchet” of the message sender’s Chain Key provides forward secrecy.
Whenever a group member leaves, all group participants clear their
Sender Key and start over.
I would say the key point here is number 4: The sender individually encrypts the Sender Key to each member of the group, using the pairwise messaging protocol explained previously
It's a clever adaptation which builds on top of the one-to-one protocol: use it to distribute a shared key to each individual member of the group, so then the group can use the server in order to provide efficient "fan-out" (as well as blob storage) without the server being privy to the required private keys.
"So what security properties does the group chat protocol feature and how does it work (at a high level)?"
As to how it works, I think that paper is as much detail as you'll find available for now until someone else does a full (applied) analysis.
As to the features it provides...
Forward secrecy: the ratchet mechanism described provides this (or so they claim)
Passive: the "Sender Key" distribution method in point 4 makes passive attacks nigh-on impossible: bear in mind that a passive party would have to get through the noise pipes encryption layer (to the WhatsApp servers) for each of the parties in the group and would probably require multiple points of presence which facilitated access to the traffic for each party in the group.... so you can pretty much rule out anybody there unless they're a major backbone provider or NSA.
Active attack: I'd say a side channel attack on one of the devices in the group would be feasible, but if you have that level of invasive access to the device (and therefore most likely its owner!), a pair of pliers and 5 mins with the device owner is likely to be far more fruitful.
Transcript consistency: all the messages use HMAC to ensure integrity (step 7)
Plausible Deniability: if anything, all the extra security provided by their new implementation makes it very difficult to deny that a particular user sent a message....