In XOrg, can i switch keyboard-layout per application and still honor ~/.Xmodmap

Question

I'm using my ~/.Xmodmap to map: keycode 38 = a A adiaeresis Adiaeresis and similar to be able to type German umlauts (äöü) on a US-qwerty layout. For switching keyboard layouts between German qwertz and English qwerty, I use two shellscripts, one that executes xmodmap ~/.Xmodmap after setxkblayout-ing.

Now, I'd like to learn the neo layout, so I thought I'd best use a per-application keyboard-switcher. There are a few of them, XFCE4-XKB-Plugin is one example.

However, I'd like to execute my Xmodmap when switching back to English. Is there a way to do that, except for touching the source of said plugin or similar applications?

Thanks!