Okay, I've figured out how although I'm not sure if this is a thorough solution.
1 Keys in Registry Editor
It seems that the registry key:
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\GameUX
is related to the Games Folder. Inside, the key with a GUID name ({s-1-5 etc...} on mine) contained all the custom added games. I checked which one has the steam game's name (Torchlight) and removed it.
2 Removed from AppData
I went to:
C:\Users\IvoryTower\AppData\Local\Microsoft\Windows\GameExplorer
and removed the folders which have the same GUID as the games you deleted from registry
I found this by searching the registry for Torchlight and then searching for the same in Windows Explorer..
This doesn't tell you how to remove it, but it does tell you how to add an actual working icon: http://forums.guru3d.com/showpost.php?s=380b0527659314a9c15d33d16f3aebbd&p=3380963&postcount=6
– SeinopSys – 2013-06-20T11:31:18.180