Skip to content

Latest commit

 

History

History
13 lines (7 loc) · 387 Bytes

README.md

File metadata and controls

13 lines (7 loc) · 387 Bytes

link-scraper

Scrapes links (a["href"] tags) within a given webpage URL, specific domains / links can be filtered out.

Configuration support added:

URLs - A list of urls to scrape links from

Filters - A list of domains / words to filter when scraping links

UserAgent - Browser header to avoid being blocked by many websites

UpdateFrequency - Time between each link / scrape cycle