Scratchy sound with ALSA and USB Sound card

Question

Since frying my onboard soundcard, I'm trying to get a new USB sound card to work with ALSA.

Everything looks fine too:

[oliland@olidesktop ~]$ lsmod | grep '^snd'
snd_seq_dummy           1455  0 
snd_seq_oss            28052  0 
snd_seq_midi_event      5332  1 snd_seq_oss
snd_seq                48705  5 snd_seq_dummy,snd_seq_oss,snd_seq_midi_event
snd_usb_audio          84746  0 
snd_pcm_oss            37890  0 
snd_mixer_oss          14851  1 snd_pcm_oss
snd_pcm                71032  2 snd_usb_audio,snd_pcm_oss
snd_timer              18992  2 snd_seq,snd_pcm
snd_page_alloc          7017  1 snd_pcm
snd_hwdep               6134  1 snd_usb_audio
snd_usbmidi_lib        17889  1 snd_usb_audio
snd_rawmidi            18826  1 snd_usbmidi_lib
snd_seq_device          5100  4 snd_seq_dummy,snd_seq_oss,snd_seq,snd_rawmidi
snd                    55132  11 snd_seq_oss,snd_seq,snd_usb_audio,snd_pcm_oss,snd_mixer_oss,snd_pcm,snd_timer,snd_hwdep,snd_usbmidi_lib,snd_rawmidi,snd_seq_device

/proc/asound/version:
Advanced Linux Sound Architecture Driver Version 1.0.23.
/proc/asound/cards:
 0 [Device         ]: USB-Audio - USB Sound Device
                      USB Sound Device at usb-0000:00:1d.3-1, full speed
/proc/asound/devices:
  1:        : sequencer
  2: [ 0- 0]: digital audio playback
  3: [ 0- 0]: digital audio capture
  4: [ 0]   : control
 33:        : timer
/proc/asound/timers:
G0: system timer : 3333.333us (10000000 ticks)
P0-0-0: PCM playback 0-0-0 : SLAVE
P0-0-1: PCM capture 0-0-1 : SLAVE
/proc/asound/pcm:
00-00: USB Audio : USB Audio : playback 1 : capture 1

But it sounds terrible!

http://soundcloud.com/oli-2-2/alsa-problem-distorted-sound

It sounds like a tin can telephone. Does anyone have any ideas for why it sounds so terrible and how to fix it? All of the gains are set to 0, and my .asoundrc is blank. I've tested the USB card on Mac and Windows machines and it and the speakers are fine.