In terms of filesystem for the actual NexentaStor you're pretty much stuck between ext2/ext3 and ZFS and given NexentaStor is there for, and because of, ZFS then you're best sticking with that. As for what 'sharing protocols' you need (which is what I think you're really asking) then you've got a few choices;
- For Windows the default sharing protocol is SMB/CIFS for file-level sharing but it can also supports iSCSI for block-level when conbined with a clustered file-system, it can also use NFS if you add the extra code that comes with Windows (called SFU). It can also support Fibre-Channel with the right drivers and a few other protocols such as AFP if you add extra third-party code.
- For shared storage ESX supports Fibre-Channel, iSCSI and NFS - nothing else.
So it depends on how you want to connect your servers, I'm no expert but I believe NexentaStor supports FC connections as well as 1/10Gb Ethernet. If you want one sharing protocol for all then clearly it needs to be either iSCSI or NFS, personally I prefer NFS over iSCSI for ESX but then again Windows deals with iSCSI better than it deals with NFS (unless you buy extra code anyway).
If I were you I'd use ZFS for your actual file system and then share using SMB/CIFS to Windows servers and NFS to ESX servers, you should be able to do this all from a single file system share although I can't see why you'd want to.
Come back to us if you have any further questions.