Every blue moon my computer's time is slow, i.e. a second takes almost a minute to pass

Question

So, here's the thing, I am absolutely lost what could be the problem here. Let me walk you through the symptoms my computer is showing, and the steps I tried to fix and/or identify the root of the problem. I'll also add a bit of background info about my setup first.

Background information

• Self-built computer:
  • Mainboard: ASUS Maximus VII Ranger
  • Processor: Intel Core i7-4770S
  • Graphics Card: Palit GeForce GTX 1070 JetStream
  • Memory: 32GB
  • Hard drives: 4 SSD drives
  • OS: Windows 10
• Computer runs 24/7.
• Computer is about 3 years old, graphics card is a bit younger, but I don't have the exact buying dates at hand right now.

Symptoms

• Every blue moon when I try to logon in the morning, the Windows lock screen shows the wrong time. This morning at 9:00 it showed 5:14.
• When I press a key (mouse or keyboard), it takes several minutes for the password prompt to appear.
• I can type my password and login immediately! The visual fade-in effect is just what's slow.
• On the desktop anything that is instant (typing in a text editor, drawing a pixel in paint) happens instantly and without problem.
• Anything that uses a time-delay on purpose (fade-in effects, cursor flashing in a text editor, countdowns...) is super slow.
• Opening the clock shows that one second takes almost a minute to pass.
• The computer isn't overheated.
• The processor is at 0%.
• Memory is fine.
• The time is just slowed down.
• When I reboot, the effect doesn't immediately go away.
  • Boot delays take forever.
  • Time in BIOS is as slow as under Windows.
• The effect disappears without notice after rebooting several times. I then have to resynch the computer time, but otherwise the computer doesn't show any symptoms afterwards.
• The effect then reappears seemingly randomly after a few weeks.

What I tried

• Resetting the CMOS.
• Replacing the CMOS battery.
• Flashing the BIOS/Firmware.
• Running diagnostics on memory and CPU.
• Completely unplugging the computer from power for several hours.
• Booting directly into BIOS, without accessing hard drives, to see if Windows is causing the effect. (The effect persists.)
• Go to a computer repair company, but leave immediately after they tried to explain that the slow fade-in of the login screen is due to individual pixels being loaded from a fragmented harddrive.

It's super hard to debug this, as the problem only rarely happens, and while it happens any debugging tools that use time delays (countdowns, fade-in effects for their windows etc.) take forever to use. Also searching for this problem online is like looking for a needle in a haystack, as everything that pops up is about performance issues.

Again, performance is not the issue. Time itself is slow, otherwise the computer is fine.

What could be the cause of this? I fear this is unsolvable and I just have to replace the mainboard.

P.S. I have no idea what to tag this question with. Please feel free to edit my tags. Thanks!

Update

So, today the problem occured again (hence my motivation to post this question):

• I unplugged the computer before going to work.
• During lunch break I went home and tried turning the computer on, but the problem persisted.
• I tried multiple restarts, to no avail.
• I unplugged the computer again and went back to work.
• After coming back home from work in the evening, I tried again, no luck.
• I then removed the graphics card and plugged the monitor into the onboard HDMI slot, computer booted immediately, time speed was back to normal.
• I shut down the computer and plugged the graphics card back in, since then the problem didn't occur anymore.
• I ran some stress tests with OCCT and IntelProcessor Diagnostics Tool, everything looks perfectly fine.

Was this my graphics card? Was this coincidence? I don't know :(