It depends of which Windows Server you're using. For instance, if you're using Microsoft Windows Server Foundations, you can have up to 15 hosts accessing it without the need of CALs, or if you're using Essentials, you can have up to 25 hosts. The problem is that if you need more hosts to access, you're gonna have to buy another Windows Server's edition. There are other limitations, that are addressed in this article.
Basically, as far as I know, you're always gonna need both client and server's license, only CALs being optional, depending of your Windows Server's edition.
Cya!