If it's Windows based, which you said it is, I would. I would also try finding some form of host intrusion detection (a program that monitors/audits files that are changing on the server and alerts you to the changes).
Just because you aren't changing files on the server doesn't mean that there isn't a buffer overflow or vulnerability that will allow someone else to change files on the server remotely.
When there's a vulnerability the fact that there's an exploit is usually known within a window of time between discovery and fix distributed, then there's a window of time until you get the fix and apply it. In that time there's usually some form of automated exploit available and script kiddies are running it to expand their bot networks.
Note that this also affect AV's since: new malware created, malware distributed, sample goes to your AV company, AV company analyzes, AV company releases new signature, you update signature, you're supposedly "safe", repeat cycle. There's still a window where it's spreading automatically before you're "innoculated".
Ideally you could just run something that checks for file changes and alerts you, like TripWire or similar functionality, and keep logs on another machine that is kind of isolated from use so if the system is compromised the logs aren't altered. The trouble is that once the file is detected as new or altered you are already infected and once you're infected or an intruder is in it's too late to trust that the machine hasn't had other changes. If someone has cracked the system they could have altered other binaries.
Then it becomes a question of do you trust the checksums and host intrusion logs and your own skills that you cleaned up everything, including rootkits and Alternate Data Stream files that are possibly in there? Or do you do the "best practices" and wipe and restore from backup, since the intrusion logs should at least tell you when it happened?
Any system connected to the Internet running a service can be exploited potentially. If you have a system connected to the Internet but not actually running with any services I'd say you're most likely safe. Web servers do not fall under this category :-)