It depends on how exactly it was compromised.
Unlike your PC, the GPU has no dedicated area to keep the user-uploaded information persistent between reboots (but see below). Thus any possible infection shall not survive the power off, after which your GPU is a clean slate again. It is possible, of course, that the malware on your PC will reinfect the GPU again once it is booted up, but in this case the source of compromise is your OS, and you shall focus on disinfecting your OS.
However some GPUs have firmware which is flashable by user - comparing to, for example, factory-only flashable firmware. Thus it is possible that a malicious actor would patch a firmware for your GPU, and flash it. This would require significant effort, and at this moment you should be only concerned about it if you think of a reason a three-letter agency would be after you. If you're concerned about it, you can reflash your GPU using the original firmware (although in this case your ability to obtain a copy of clean firmware would also be limited).
It is also possible that the malicious code is injected into GPU on factory, and then shipped to you. In this case you'd have no means to detect this.
Saying so, GPU attack vector has its limitations. While it can read and modify the arbitrary memory data undetected (including ignoring memory protection flags), it has no access to MMU and no way to know which physical memory regions are mapped to virtual regions. Thus making the attack happen (for example granting a certain process admin privileges) would be very difficult to perform. And the lack of internal persistent storage and other means to exfiltrate the data makes it way more difficult than traditional malware.