Okay, I've managed to get it figured out, so I'm going to answer my own question to the best of my knowledge.
The original error was caused by the fact that the quota format vfsv0 is unable to support quotas >= 4TiB. Quota has a (relatively) new format to support quotas >4TiB, called vfsv1. You need at least kernel 2.6.33 for kernel support for vfsv1.
You need(?) to use journaled quotas, which will work on ext4 and ext3(?).
In /etc/fstab you will need to add the following for quota support for your mount:
usrjquota=quota.user,grpjquota=quota.group,jqfmt=vfsv1
an example fstab line would be (here, the mountpoint is /):
/dev/md1 / ext4 grpjquota=quota.group,usrjquota=quota.user,jqfmt=vfsv1 0 2
Don't create the files quota.user or quota.group in your mountpoint. Afterwards do:
mount -o remount /
then do
quotacheck -avugm
which will create quota.user and quota.group, followed by:
quotaon -avug
At this point, you may get an error along the lines of quotaon: Quota format not supported in kernel.
This is because the kernel wasn't compiled with support for vfsv1, but not to worry, try
modprobe quota_v1
modprobe quota_v2
quotaon -avug
if that worked, then be sure to add quota_v1 and quota_v2 to /etc/modules
If everything worked, then you should be able to change the quota to something over 4TiB!