Possible to set per-session/context-sensitive key bindings in tmux?

Question

I do everything in tmux, so I have a stanza in my .bashrc to create/attach to a session whenever I enter a shell. Sometimes, I ssh into a machine that shares the same configuration. So I'm already in a tmux session on the local machine, and after ssh-ing into the remote, I'm in another tmux session on the remote machine.

I have C-a set as the prefix on both machines, so what winds up happening is that C-a sends the prefix locally, while C-a C-a sends the prefix into the remote machine's tmux session.

I'd like to set two different prefixes for the local and remote sessions (say, ` vs. C-a). But the remote machine is just downstairs, and I may want to walk down there to do something on it directly, in a brand new tmux session, using only the C-a prefix I'm accustomed to. Except since ` was set as a prefix in the ssh tmux session, that now applies to all tmux sessions, and now I have to hit backtick twice to enter it once on the command line.

Is there any way to set separate prefix (or key binding, generally) for different sessions in tmux? Or should I just suck it up and get used to hitting C-a twice?