Yes. Anti-malware will always slow a system down. Windows defender does a really good job (perhaps the best when it comes to performance) but still needs to tap into a lot of resources to get it's job done.
I personally don't run ANY of it within my VMs but I also don't put anything on them that is sensitive or questionable. Thats just me.. I am not advising you of anything.
The scanner doesn't own 100% of the system and Windows needs to balance thread priority, kernel calls and many other factors into who gets how much of the shared system's resources. There are a LOT more things to consider than just CPU.
In addition to thread scheduling and the like, consider that before X can run, Y needs to make sure that it is OK. Making sure that it is OK will require both signature and heuristic scanning of disk storage and then the memory while the exe loads and then finally executes.
Any time you have synchronous actions, your time increases even if the load on the CPU isn't maxed out. Even asynchronous actions require synchronization at points in the job they are doing so they can communicate their findings with other parts of the system.
A system NOT running AV will always outperform the same system running AV. If your system is fast enough.. you won't notice but it is still true. ;)
I hope this helps.
