image
GoSpider – Fast web spider written in Go Installation go get -u github.com/jaeles-project/gospider Features Fast web crawling Brute force and parse sitemap.xml Parse robots.txt Generate and verify link from JavaScript files Link Finder Find AWS-S3 from response source Find subdomains from response source Get URLs from Wayback Machine, Common Crawl, Virus Total, Alien Vault Format output easy to Grep Support Burp input Crawl multiple sites in parallel Random mobile/web User-Agent Showcases Usage Fast web spider written in Go – v1.1.0 by @theblackturtle Usage: gospider [flags] Flags: -s, –site string Site to crawl -S, –sites string Site list to crawl -p, –proxy string Proxy (Ex: http://127.0.0.1:8080) -o, –output string Output folder -u, –user-agent string User Agent to use web: random web user-agent mobi: random mobile user-agent or you can set your special user-agent (default "web") –cookie string Cookie to use (testA=a; testB=b) -H, –header stringArray Header to use (Use multiple flag to set multiple header) –burp string Load headers and cookie from burp raw http request –blacklist string Blacklist URL Regex -t, –threads int Number of threads (Run sites in parallel) (default 1) -c, –concurrent…

Source