It sounds to like MySQL is having to create large in-memory temporary tables while processing an unoptimized query.
If it isn't enabled, enable the slow query log by adding the following to your my.cnf file:
log_slow_queries = /var/log/mysql/mysql-slow.log
long_query_time = 2
log-queries-not-using-indexes
Then take a look at the /var/log/mysql/mysql-slow.log file. You should run EXPLAIN on each query listed in the file to get an idea of what is happening when that query is executed. With that knowledge you can optimize the query for better performance.