I would recommend against this as X11's LAN-oriented traffic will make it painful. You can easily run X11 applications on the remote end. However, unless you have extremely high bandwidth between both locations, you will have laggy response times. X11 is a synchronous protocol and every little event generates a stream of traffic. It is one of the driving reasons that there have been replacement protocols to deal with this, i.e. NX (NoMachine) or VNC.
A place where using X11 would make sense is on a local LAN, using a shared box for everyone to do development on. In this context, the security of the box is assured (or at least monitored), the latency is extremely low, and you are re-using resources efficiently. Having 2-3 people connect to a dev box and running jobs and editing and compiling and etc. on the box in this context is just fine. Not GREAT, but fine.
You will NEED to enable compression on your SSH connection to make this viable. You do NOT need to install the xorg video drivers. Installation through apt-get will most likely pull the minimum libraries required to make this work. So don't bother installing x.org stuff or anything else; just pull the tool that you need and if the package maintainer did their job, it will pull the rest.
A Note:
With regard to how you use it, I reserve judgement. I see people hating the question not so much because it's taboo (yes, it is) but because they are giving knee-jerk reactions. Example: running a server-based tool like gsmartmoncontrol, which is directly aimed at montoring drive health, is a strange but valid example. The tool is strictly meant for the server and only augments your command line experience (in this context, I'm thinking of a traditional file server). Running development tools on a production box is probably not the greatest idea. Hint: if you have a break-in event, you just gave your attackers plenty of tools to play with.