DeepCOFFEA-Crawler

Usage

Install docker
Build the container using make build
Edit the CRAWL_PARAMS in the Makefile to correctly reflect the login credentials to the server that will act as the proxy
- make sure tcpdump is installed and usable by your user account for the proxy server (nothing else is needed)
Run the crawler using make run

Name		Name	Last commit message	Last commit date
Latest commit History 17 Commits
crawler		crawler
Dockerfile		Dockerfile
Entrypoint.sh		Entrypoint.sh
Makefile		Makefile
README.md		README.md
config.ini		config.ini
majestic_million.csv		majestic_million.csv
requirements.txt		requirements.txt
setup.py		setup.py
top-1m.csv		top-1m.csv