Performance-wise that should not matter very much. Back in the LVM1 days a small PE size severely limited the maximum volume size, with LVM2 those restrictions do not apply.
Physical extents mostly matters if you choose to create lots and lots of small logical volumes. In that case you might waste some disk space, if a physical extent is larger than your typical logical volume. On the other hand, choosing a very small PE size can slow down LVM tools during LVM operations (such as resizing a VG) quite a bit.
Nowadays I usually don't spend too much time on thinking about the PE size, I let my distro to choose it. If you have time and willingness, you can of course try different PE sizes and benchmark them to find out if PE size matters in your case.