I am looking for a tool that generates a set of word lists based on the words it finds on the pages of a particular web domain. This question has an answer that generates permuted word lists starting from existing ones. Other tools such crunch in backtrack/kali has a number of modes of generation but does not take into account an existing word list.
How can I generate passwords based on words on the web domain?
A Python solution
You can use Scrapy which will make most of the work for you. You then would just need to use Counter() to get the top words (if looking for frequency counts).
You could also use a more low-level approach with Beautiful Soup to get the top 5 words:
# coding=utf-8
import requests
import collections
from bs4 import BeautifulSoup
thesite = requests.get("http://www.lemonde.fr").text
soup = BeautifulSoup(thesite, 'html.parser')
thewords = soup.get_text().split()
print(collections.Counter(thewords).most_common(5))
Since the output is
[('de', 223), ('la', 154), (':', 123), ('{', 115), ('à', 84)]
you could look at setting a minimal length of a "word" (3 perhaps?)
UPDATE: the code for a sorted list of most common words with 3 or more letters
# coding=utf-8
import requests
import collections
from bs4 import BeautifulSoup
import operator
thesite = requests.get("http://www.lemonde.fr").text
soup = BeautifulSoup(thesite, 'html.parser')
thewords = soup.get_text().split()
# keep only words over 3 chars
thewords = {w: f for w, f in collections.Counter(thewords).items() if len(w) > 3}
topwords = sorted(thewords.items(), key=operator.itemgetter(1), reverse=True)
print(topwords)
How about using a tool called CeWL - Custom Word List generator?
CeWL is a ruby app which spiders a given url to a specified depth, optionally following external links, and returns a list of words which can then be used for password crackers such as John the Ripper.
