I am wanting to write a node application that looks up a bunch of blacklists from plain text files. Basically are all under directories representing different categories, and in each there is a file called domains which has a bunch of domain names separated by newline.
So my application will look at these files, and when I do an API call with a particular domain, it should look under that directory, find the domains file, and then determine whether or not the domain is listed in that domains file.
I wanted to do this efficiently, both memory-wise and computation-wise. I have been considering several approaches.
- First I tried just loading the whole file into a dictionary, with the domain as the key and a category ID as the value. This was really bad for memory, as some of these domains files were as large as 15 MB, and the resulting dictionary is something like 250 MB of memory due to the way lookup tables work.
- I considered just keeping a map of category ID to domains list, and then opening the file and searching, and keeping a local cache in case that domain is used again. However, this is really inefficient obviously from a disk perspective as you would be loading the entire file and searching through it when you do a lookup.
- I tried using redis to accomplish number 1 but it was even slower.
I am wondering if there is some library out there that maybe uses indexing, where a lot of the list is on disk, index makes it easy to jump to the right spot in the file if a disk lookup is necessary, and maybe then caching sections of the file in memory as opposed to the entire thing. It seems like there must be some library or application suited to this. Any ideas?
Thanks