This is a partner project of Legal Housing. Check there for a more complete description of the project motivation and goals.
This project implements:
-
A scraper that pulls Boston-area housing listings from Craigslist and stores them in a Postgres database.
-
A webserver that can dispatch the scraper on demand and return the listings in JSON format, with optional filtering.
- Python 3
- Postgres
(You should create a new Python virtual environment for use with this project, but that process is outside the scope of this document.)
- Flask - a lightweight web framework
- psycopg2 - a Postgres adapter
- Scrapy - a framework for webscraping
- schedule - a framework for job scheduling
You can install all of these dependencies by running
pip install -r Scrapple/requirements.txt, or by installing them
individually.
Scrapple is intended to use a Postgresql database the connection parameters must set by one of two methods, ether set up a environment parameter POSTGRES_URI witch is of the form "host=host port=9999 dbname=bd_name user=user_name password=secret_pw".
The parameter may also be set in data_factory_config.json by using the postgres_uri value. POSTGRES_URI tacks precedents over postgres_uri if both are defined.
To perform initial setup on the database, you need to execute the commands in Scrapple/
python3 create_listings.py
or alternatively
You may execute the commands in
Scrapple/Database/create_listings.sql. You can use psql. For instance:
psql <conn_str> -f Scrapple/Database/create_listings.sql
Where <conn_str> is any way of specifying the location, database, and
credentials to use: URI (postgresql://...), flags (--user, etc.), param
lists ("host=localhost port=5432 dbname=clsp user=clsp password=secret_pw") substituted your values for all.
In the Scrapple subdirectory, run:
FLASK_APP=./runserver.py FLASK_DEBUG=1 flask run
Omit FLASK_DEBUG=1 to disable automatic code reloading.
If you have Docker and Docker Compose installed, cd to the Scrapple
directory and run:
docker-compose up
To retrieve listings between a given date range use
http://<host>:5555/listings/get?dfrom=<str_from>&dto=<str_dto>&pagesize=<pagesize>
dfrom and dto describe the date range if dto is omitted it defaults to the current date
Two date formats are supported YYY-MM-DD and MM/DD/YYYY
pagesize gives the number of records to be returned starting with the oldest record if there are more records in the range than pagesize then only the first pagesize number of records will be returned. If pagesize is omitted it defaults to systems configurable page_maximum which is the supplied config file sets to 500. If pagesize exceeds page_maximum it's reset to page_maximum.
Example valid URIs:
http://localhost:5555/listings/get?dfrom=2017-01-01&dto=2018-01-01&pagesize=400
http://localhost:5555/listings/get?dfrom=2017-01-01
The endpoint returns a JSON in the form of a list of dictionaries, each dictionary describes a listing record.
The scraper runs by executing Scraper modules which target specific websites.
Currently there is only one scraper module which scrapes listings from craigslist
Scraper module run on a periodic schedule there can be one schedule per scraper module. To control the activity of schedules two endpoints are provided one to activate a schedule and one to remove it and terminate scraping.
Currently there is only one scraper craig_spyder.py and one schedule. To start scraping using the following POST command.
http://<host>:5555/start_spider_sch?scraper=craigslist
The schedule will run until the system is stopped. To stop it use the following POST command.
http://<host>:5555/stop_spider_sch?scraper=craigslist
The default schedule is set to 90 minutes, comments in data_manager_schedulable.py explain how to edit the scheduling.
The scheduling system is implemented in separate processes for each running scraper it is self-contained and runs within the Python flask server. If the flask server or the overall process running Python is terminated scheduling will halted.