DNS is fun, isn't it?
I'll answer the last part of your question first, since it's the easiest to answer. When a DNS client looks to resolve your address, it will start the process of DNS recursion. The goal of this recursion is to end up with the NS record of the DNS nameserver that serves the domain. With your set up, multiple NS records will be returned - the one serving your IIS set up and the NS registered in cpanel.
If you'd like to see what the DNS recursion process returns for NS records, follow the recommendation of one of the commenters and perform the dig command with a specific parameter of "ns" (to specify ns query) and the domain name of your website.
From there, one of the NS records will be tried first (usually - it'll be the one that the DNS client receives first) and it'll be queried for the specific A record or whatever other record(s) it's looking for. If that NS can't come back with anything, it'll try the second NS. That's why if two records differ between DNS servers that both serve a specific zone, the first record that it can query for and get returned to it will be the one used by the system.
Now to the other part of your question - Yes, you should transfer your domain registration to another service (i.e. Google Domains) before you close your web hosting account. Most hosting providers, from my knowledge, won't even allow you to close your account with outstanding domain registrations. The function that cpanel provides in your system is to be a catalyst for web management of your website assets. You don't NEED cpanel, unless your hosting provider (which, I presume will be your own IIS set up) requires it.
I hope my answer serves to help you. Feel free to comment with any other concerns/corrections.