pyWebCrawler

GitHub last commit GitHub release (latest by date)

pyCrawler is a Python-based web crawling tool designed to automatically extract URLs, subdomains, and JavaScript files from websites.
It provides a simple command-line interface for users to specify the target URL, recursion depth, number of threads, and output file.

Features

Extracts subdomains, links, and JavaScript files recursively.
Supports multithreaded crawling for improved performance.
Allows customization of recursion depth.
Provides an option to save the output to a specified file.

Installation

Clone the repository to your local machine.
Install the required dependencies using pip

git clone https://github.com/aashish36/pyWebCrawler.git

cd pyWebCrawler

pip install -r requirements.txt

Usage

Give url to tool that you want to crawl.
This python code will save the results of the analysis to a file named ‘pyWebCrawler.txt’.
Run the script using the following commands:

usage: webcrawlerMark-2.py [-h] -u URL [-d DEPTH] [-t THREADS] [-o OUTPUT]

pyCrawler is a tool designed to automatically extracting Urls, Subdomins and JS Files from websites.

options:
  -h, --help            show this help message and exit
  -u URL, --url URL     Specify the URL, provide it along http/https
  -d DEPTH, --depth DEPTH
                        Specify the recursion depth limit.
  -t THREADS, --threads THREADS
                        Specify the threads, default=100.
  -o OUTPUT, --output OUTPUT
                        Specify the file u want to save, default=pyWebCrawler.txt

Tool Output

Total Tool Output

██████╗░██╗░░░██╗░█████╗░██████╗░░█████╗░░██╗░░░░░░░██╗██╗░░░░░███████╗██████╗░
██╔══██╗╚██╗░██╔╝██╔══██╗██╔══██╗██╔══██╗░██║░░██╗░░██║██║░░░░░██╔════╝██╔══██╗
██████╔╝░╚████╔╝░██║░░╚═╝██████╔╝███████║░╚██╗████╗██╔╝██║░░░░░█████╗░░██████╔╝
██╔═══╝░░░╚██╔╝░░██║░░██╗██╔══██╗██╔══██║░░████╔═████║░██║░░░░░██╔══╝░░██╔══██╗
██║░░░░░░░░██║░░░╚█████╔╝██║░░██║██║░░██║░░╚██╔╝░╚██╔╝░███████╗███████╗██║░░██║
╚═╝░░░░░░░░╚═╝░░░░╚════╝░╚═╝░░╚═╝╚═╝░░╚═╝░░░╚═╝░░░╚═╝░░╚══════╝╚══════╝╚═╝░░╚═╝

               Author   : Aashish💕💕  

               Github   : https://github.com/aashish36

               pyCrawler is a tool designed to automatically extracting urls and Js from websites.

        
--------------------------------------------------------------------------------
Recursive PyCrawler starting at 30/11/2023 07:30:54
--------------------------------------------------------------------------------
[*] URL     : https://hackerone.com
[*] Max Depth : 1
[*] Threads : 100
--------------------------------------------------------------------------------
[+] Subdomains : https://www.hackerone.com/hacktivitycon
[+] Subdomains : https://hackerone.com/security?type=team
[+] Subdomains : https://www.hackerone.com/reports/6th-annual-hacker-powered-security-report
[+] Subdomains : https://www.hackerone.com/customer-hub/Hyatt
[+] Subdomains : https://hackerone.com/hacktivity
[+] Subdomains : https://www.hackerone.com/events
[+] Subdomains : https://www.hackerone.com/product/challenge
[+] Subdomains : https://www.hackerone.com/product/bug-bounty-platform
[+] Subdomains : https://www.hackerone.com/resources
[+] Subdomains : https://www.hackerone.com/partners/integrations
[+] Subdomains : https://hackerone.com/users/sign_in
[+] Subdomains : https://www.philvenables.com/post/bug-bounty-programs
[+] Subdomains : https://www.hackerone.com/product/overview
[+] Subdomains : https://docs.hackerone.com/
[+] Subdomains : https://www.twitter.com/Hacker0x01
[+] Subdomains : https://www.hackerone.com/hackers/hacker101
[+] Subdomains : https://www.hackerone.com/vulnerability-and-security-testing-blog
[+] Subdomains : https://www.hackerone.com/public-policy
[+] Subdomains : https://www.instagram.com/hacker0x01
[+] Subdomains : https://www.linkedin.com/company/hackerone
[+] Subdomains : https://www.hackerone.com/resources/customer-story/zebra-technologies-case-study
[+] Subdomains : https://www.hackerone.com/solutions/vulnerability-management-system
[+] Subdomains : https://www.hackerone.com/leadership
[+] Subdomains : https://hackerone.com/directory/programs?order_direction=DESC&order_field=resolved_report_count
[+] Subdomains : https://www.hackerone.com/customer-stories
[+] Subdomains : https://hackerone.com/leaderboard
[+] Subdomains : https://www.hackerone.com/product/attack-surface-management
[+] Subdomains : https://www.hackerone.com/ebooks/pentesting-matrix
[+] Subdomains : https://hackerone.com/leaderboard/all-time
[+] Subdomains : https://www.hackerone.com/reports/7th-annual-hacker-powered-security-report
[+] Subdomains : https://www.facebook.com/Hacker0x01
[+] Subdomains : https://hackerone.com/opportunities/all/search
[+] Subdomains : https://h1.community/events/#/list

[+] Links : https://hackerone.com/vulnerability-management/zoom-salesforce-ethical-hackers
[+] Links : https://hackerone.com/product/bug-bounty-platform
[+] Links : https://www.hackerone.com/hacktivitycon
[+] Links : https://hackerone.com/partners/integrations
[+] Links : https://hackerone.com/product/attack-surface-management
[+] Links : https://www.hackerone.com/resources
[+] Links : https://www.hackerone.com/partners/integrations
[+] Links : https://hackerone.com/trust
[+] Links : https://hackerone.com/culture-and-talent
[+] Links : https://hackerone.com/thought-leadership/generative-ai-security-predictions
[+] Links : https://www.hackerone.com/vulnerability-and-security-testing-blog
[+] Links : https://hackerone.com/services
[+] Links : https://www.hackerone.com/resources/customer-story/zebra-technologies-case-study
[+] Links : https://hackerone.com/
[+] Links : https://hackerone.com/three-stages-continuous-vulnerability-testing
[+] Links : https://hackerone.com/customer-hub/Nintendo
[+] Links : https://hackerone.com/product/code-security-audit
[+] Links : https://hackerone.com/from-the-ceo
[+] Links : https://hackerone.com/leadership
[+] Links : https://hackerone.com/knowledge-center/devsecops-quick-guide-process-tools-and-best-practices
[+] Links : https://hackerone.com/solutions/vulnerability-management-system
[+] Links : https://hackerone.com/knowledge-center/attack-surface-and-how-analyze-manage-and-reduce-it
[+] Links : https://hackerone.com/vulnerability-management/owasp-llm-vulnerabilities
[+] Links : https://hackerone.com/customer-hub/ATT
[+] Links : https://hackerone.com/solutions/attack-resistance-management
[+] Links : https://hackerone.com/company-news
[+] Links : https://hackerone.com/product/response-vulnerability-disclosure-program
[+] Links : https://hackerone.com/hacktivity
[+] Links : https://hackerone.com#main-content
[+] Links : https://hackerone.com/customer-stories
[+] Links : https://hackerone.com/knowledge-center/principles-threats-and-solutions
[+] Links : https://www.hackerone.com/hackers/hacker101
[+] Links : https://hackerone.com/security-compliance/new-sec-cybersercurity-rules
[+] Links : https://hackerone.com/press
[+] Links : https://hackerone.com/careers
[+] Links : https://hackerone.com/partners/aws
[+] Links : https://hackerone.com/knowledge-center/what-hacking-black-hat-white-hat-blue-hat-and-more
[+] Links : https://hackerone.com/knowledge-center/what-application-security-concepts-tools-best-practices
[+] Links : https://www.hackerone.com/leadership
[+] Links : https://hackerone.com/security-incident
[+] Links : https://hackerone.com/solutions/government
[+] Links : https://hackerone.com/press-archive
[+] Links : https://hackerone.com/vulnerability-and-security-testing-blog
[+] Links : https://hackerone.com/customer-hub/GM
[+] Links : https://hackerone.com/services-2
[+] Links : https://hackerone.com/customer-hub/Hyatt
[+] Links : https://h1.community/events/#/list
[+] Links : https://hackerone.com/knowledge-center/security-compliance-ten-regulations-and-four-tips-success
[+] Links : https://hackerone.com/knowledge-center/what-penetration-testing-how-does-it-work-step-step
[+] Links : https://hackerone.com/hackerone-go
[+] Links : https://hackerone.com/product/challenge
[+] Links : https://hackerone.com/terms
[+] Links : https://hackerone.com/solutions/financial-services
[+] Links : https://hackerone.com/hackerone-community-blog
[+] Links : https://hackerone.com/partners
[+] Links : https://www.hackerone.com/solutions/vulnerability-management-system
[+] Links : https://hackerone.com/knowledge-center/cloud-security-challenges-solutions-and-best-practices
[+] Links : https://hackerone.com/solutions/united-states-federal
[+] Links : https://hackerone.com/contact
[+] Links : https://hackerone.com/leaderboard
[+] Links : https://hackerone.com/product/pentest
[+] Links : https://hackerone.com/solutions/high-growth-companies
[+] Links : https://hackerone.com/solutions/application-security-testing-software
[+] Links : https://hackerone.com/events
[+] Links : https://hackerone.com/knowledge-center/16-types-cybersecurity-attacks-and-how-prevent-them
[+] Links : https://hackerone.com/opportunities/all/search
[+] Links : https://hackerone.com/product/overview
[+] Links : https://hackerone.com/solutions/continuous-security-testing
[+] Links : https://hackerone.com/customer-hub/Paypal
[+] Links : https://www.hackerone.com/events
[+] Links : https://hackerone.com/users/sign_in
[+] Links : https://hackerone.com/knowledge-center/beyond-owasp-top-ten-13-resources-boost-your-security
[+] Links : https://hackerone.com/solutions/cloud-security-solution
[+] Links : https://www.hackerone.com/public-policy
[+] Links : https://hackerone.com/security-compliance/nist-vdp-control
[+] Links : https://hackerone.com/product/insights
[+] Links : https://hackerone.com/knowledge-center/common-vulnerabilities-exposures-glossary-cve
[+] Links : https://hackerone.com/policies
[+] Links : https://hackerone.com/knowledge-center
[+] Links : https://hackerone.com/company
[+] Links : https://hackerone.com/hackers
[+] Links : https://hackerone.com/knowledge-center/website-testing-importance-techniques-5-tips-success
[+] Links : https://hackerone.com/privacy
[+] Links : https://hackerone.com/vulnerability-management/security-advisory-services-sdlc
[+] Links : https://hackerone.com/solutions/united-states-federal-old

[+] JS Files : /sites/default/files/js/js_EOrKavGmjAkpIaCW_cpGJ240OpVZev_5NI-WGIx5URg.js
[+] JS Files : https://cdn.optimizely.com/js/21892691969.js
[+] JS Files : /sites/default/files/google_tag/google_tag/google_tag.script.js?s3z4tf
[+] JS Files : https://consent.trustarc.com/notice?domain=hackerone.com&c=teconsent&js=nj&noticeType=bb&gtm=1
[+] JS Files : ////app-sj17.marketo.com/js/forms2/js/forms2.min.js

Contributing

Contributions are welcome!
If you have any suggestions, bug reports, or feature requests, please open an issue or submit a pull request.

This site is open source. Improve this page.