Scanning through files is inevitable, there is no other way. Depending on your disk speed this time will be slower or faster.
Some usefull command you may try:
find /somedir -type f -size +10M
This will show you all files with size greater(the + sign) than 10M. You can use this only for files since directory's size is actually the directory's metadata(the file listing) and not the size of the files inside.
With find you can also find OLD files, which may be something you forgot a long time ago, for example
find /somedir -mtime +30
This will show files,links,dirs which were not modified for the last 30 days. You can also combine time + size like this:
find /somedir -type f -mtime +30 -size +100M
will show all files which are bigger than 100MiB and were not modified for more than 30 days
For directories use:
du -h -d1 /somedir
-h for human readable (kilo mega giga tera and so on)
-d1 - show only first level dirs, by default du will scan everything recursively, which is not very useful in our case.
if you want to sort omit the -h and add '|sort -n -k1'
alternatively you can do:
du -sh *
in the directory you are interested in