I'm trying to segment a few servers for a Rails project and part of that is entertaining the idea of isolating Resque workers to their own box. I already have Redis broken out, but is it common to put Resque workers elsewhere?
I'm going to make this a bit more generic. Is it common to put job processing workers on different machines than the application? Absolutely. This is something you will see in almost any job queue processing architecture.
Do they need to be tied to the application itself?
No. The application can be written to read live and pass updates off to the queue... and itself be incapable of handling the updates. They're just so supposed to done... [wistful look into the distance] in the cloud. Whether that's all on one machine or spread across a monstrous network doesn't matter with good separation design.
If it can be done, are there any commonly referenced best practices or writeups on the subject?
It feels wrong to answer this with a no... but I really don't know of any. Everybody faces different challenges in scaling. Being able to write something to be scalable is just a measure of experience. You can try reading stories of how others have scaled applications in the past and attempt to learn there lessons, but there is no catch-all "Scaling HOWTO".
Use loose coupling and good segmentation of processes (in the flow chart sense) so they can be broken off when possible. Don't over-think it.
If not, am I better off just throwing more RAM at the app servers running Resque workers?
As long as you can and nothing else is bottle-necking, go for it. Scaling is a cost / benefit assessment as much as anything else. Think carefully, though, about whether you'll hit the point where you're maxing out what you can put in the machine and suddenly it is looking very expensive to get a platform supporting more memory. If you don't think you'll get there, don't lose sleep over it.
