For bugbounty and penetration testing/redteaming, a headless web crawler is used

Lolcrawler is a headless browser crawler designed for BugBounty and RedTeaming.

Be aware that this code is really sluggish, but it should be able to run for several days and uncover some fascinating urls/paths.

BeautifulSoup, jsbeautifier, urlextract, and linkfinder.py are among the crawler’s methods for attempting to locate links and urls.

Due to the headless selenium browser, you’ll need to install Docker to run this crawler.

Users need to install docker to run this crawler due to the headless selenium browser.

Disclaimer: The intended use for the tool is strictly educational and should not be used for any other purpose.

Download Link:  https://github.com/jonaslejon/lolcrawler