async-web-crawler

Project Overview

This project is a smart, asynchronous web crawler capable of crawling websites within specified domains.
It supports JavaScript-heavy pages using Selenium, respects robots.txt, handles retries, and converts pages to clean Markdown.
The crawler also generates a JSONL index, tracks failed URLs, and logs detailed crawl info. The crawler has been tested on WordPress, React, and Next.js websites.

Project Structure

/scraper/
├── crawler.py # Main crawler script
├── crawlLog.txt # Detailed log file
├── seeds.txt # Priority URLs to crawl
├── .gitignore
├── requirements.txt # Python dependencies
└── output/
    ├── MDs/ # Markdown files for each crawled page
    ├── index.jsonl # JSON index of all crawled pages
    └── failed_urls.txt # URLs that failed to crawl

HTML files and screenshots are generated during runtime but ignored in Git (optional).

---

Prerequisites

• Python 3.11+
• Chrome Browser installed
• ChromeDriver matching your Chrome version: https://chromedriver.chromium.org/downloads
• Install Python dependencies:

pip install -r requirements.txt

Configuration Variables

You can modify these at the top of crawler.py:

Variable	Description
CRAWL_DEPTH	Depth of crawl from each seed URL
ALLOWED_DOMAIN	List of allowed domains
PAGES_PER_SEED	Max pages to crawl per seed URL
MAX_PAGES	Total max pages across all seeds
BLOCKED_PAGES_FULL	Full URLs to block
BLOCK_PATTERNS	Regex patterns to block
POLITE_DELAY	Delay between requests per domain
ALLOW_INSECURE	Skip SSL verification if True

Running the Crawler from the repo

Option 1 : Using Python Virtual Environment

# clone repo
git clone <repo-url>
cd scraper

# create venv
python -m venv venv
source venv/bin/activate  # macOS/Linux
venv\Scripts\activate     # Windows

# install dependencies
pip install -r requirements.txt

# run crawler
python crawler.py

Cross-Platform Notes

• Works on Windows, macOS, Linux.
• Requires Python 3.11+ and Chrome + ChromeDriver.
• For Linux/macOS: make chromedriver executable.

chmod +x /path/to/chromedriver

• For Windows: put chromedriver.exe in a folder listed in your PATH.

Option 2: Using Docker (Recommended for Consistency)

Build Docker image

docker build -t web-crawler .

Run Docker container

docker run --rm -v $(pwd)/output:/app/output web-crawler

Notes:

• The Docker setup includes Python 3.11, all dependencies, and Chrome/ChromeDriver preconfigured.
• Output files are mounted to the local output/ folder for easy access.
• Cross-platform compatible: works on Windows, macOS, Linux without manual dependency setup.

Logging & Error Handling

• crawlLog.txt logs crawl progress, retries, and errors.
• Failed URLs (e.g., CAPTCHA-protected) are tracked in output/failed_urls.txt.
• Robots.txt compliance and polite delays are respected.
• JS-heavy pages are handled with Selenium (headless by default).

Demo Video

Click on the video to play it on youtube

Notes on Testing

Sites tested:

• https://jeevee.com – Next.js
• https://prettyclickcosmetics.com – WordPress
• https://www.tranquilityspa.com.np – React
• http://kiec.edu.np – was blocked due to bot protection (CAPTCHA).