Skip to content

Commit acb779a

Browse files
Merge pull request #190 from rustprooflabs/dev
Prep for 0.4.0 rc1
2 parents bc06a6a + 25736c5 commit acb779a

29 files changed

+609
-239
lines changed

Dockerfile

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -10,13 +10,18 @@ RUN apt-get update \
1010
libboost-filesystem-dev libexpat1-dev zlib1g-dev \
1111
libbz2-dev libpq-dev libproj-dev lua5.2 liblua5.2-dev \
1212
python3 python3-distutils python3-psycopg2 \
13-
curl \
13+
postgresql-server-dev-14 \
14+
curl luarocks \
1415
&& rm -rf /var/lib/apt/lists/*
1516

1617
RUN curl -o /tmp/get-pip.py https://bootstrap.pypa.io/get-pip.py \
1718
&& python3 /tmp/get-pip.py \
1819
&& rm /tmp/get-pip.py
1920

21+
RUN luarocks install inifile
22+
RUN luarocks install luasql-postgres PGSQL_INCDIR=/usr/include/postgresql/
23+
24+
2025
WORKDIR /tmp
2126
RUN git clone git://github.com/openstreetmap/osm2pgsql.git \
2227
&& mkdir osm2pgsql/build \
@@ -27,15 +32,15 @@ RUN git clone git://github.com/openstreetmap/osm2pgsql.git \
2732
&& apt remove -y \
2833
make cmake g++ \
2934
libexpat1-dev zlib1g-dev \
30-
libbz2-dev libpq-dev libproj-dev \
35+
libbz2-dev libproj-dev \
3136
curl \
3237
&& apt autoremove -y \
3338
&& cd /tmp && rm -r /tmp/osm2pgsql
3439

40+
3541
COPY ./sqitch.conf /etc/sqitch/sqitch.conf
3642

3743
WORKDIR /app
3844
COPY . ./
3945

40-
# --pre added to switch to psycopg3 during beta, remove after inital official release
41-
RUN pip install --pre -r requirements.txt
46+
RUN pip install -r requirements.txt

Makefile

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -49,7 +49,7 @@ build-run-docker: ## Builds and runs PgOSM Flex with D.C. test file
4949
-e POSTGRES_USER=postgres \
5050
-u $(CURRENT_UID):$(CURRENT_GID) \
5151
pgosm python3 docker/pgosm_flex.py \
52-
--layerset=run-all \
52+
--layerset=default \
5353
--ram=1 \
5454
--region=north-america/us \
5555
--subregion=district-of-columbia \

README.md

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,8 @@ mkdir ~/pgosm-data
5656
```
5757

5858
Set environment variables for the temporary Postgres connection in Docker.
59+
These are required for the Docker container to run.
60+
5961

6062
```bash
6163
export POSTGRES_USER=postgres
@@ -74,11 +76,10 @@ docker run --name pgosm -d --rm \
7476
```
7577

7678
Run the processing for the Washington D.C. The `docker/pgosm_flex.py` script
77-
requires three (3) parameters, typical use will use four (4) to include
79+
requires two (2) parameters, typical use will use three (3) to include
7880
the `--subregion`.
7981

8082

81-
* PgOSM-Flex layer set (`run-all`)
8283
* Total RAM for osm2pgsql, Postgres and OS (`8`)
8384
* Region (`north-america/us`)
8485
* Sub-region (`district-of-columbia`) (Optional)
@@ -87,10 +88,7 @@ the `--subregion`.
8788

8889
```bash
8990
docker exec -it \
90-
-e POSTGRES_PASSWORD=$POSTGRES_PASSWORD \
91-
-e POSTGRES_USER=$POSTGRES_USER \
9291
pgosm python3 docker/pgosm_flex.py \
93-
--layerset=run-all \
9492
--ram=8 \
9593
--region=north-america/us \
9694
--subregion=district-of-columbia
@@ -413,9 +411,12 @@ Current `JSONB` columns:
413411

414412
## Additional resources
415413

414+
See the listing of known
415+
[projects using PgOSM Flex](docs/PROJECTS.md).
416+
417+
416418
Blog posts covering various details and background information.
417419

418420
* [Better OpenStreetMap places in PostGIS](https://blog.rustprooflabs.com/2021/01/pgosm-flex-improved-openstreetmap-places-postgis)
419421
* [Improved OpenStreetMap data structure in PostGIS](https://blog.rustprooflabs.com/2021/01/postgis-openstreetmap-flex-structure)
420422
* [Hands on with osm2pgsql's new Flex output](https://blog.rustprooflabs.com/2020/12/osm2gpsql-flex-output-to-postgis).
421-

docker/db.py

Lines changed: 19 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -302,18 +302,20 @@ def get_db_conn(db_name):
302302
return conn
303303

304304

305-
def pgosm_after_import(layerset, paths):
306-
"""Runs post-processing SQL via psql.
305+
def pgosm_after_import(paths):
306+
"""Runs post-processing SQL via Lua script.
307+
308+
Layerset logic is established via environment variable, must happen
309+
before this step.
307310
308311
Parameters
309312
---------------------
310-
layerset : str
311-
312313
paths : dict
313314
"""
314315
LOGGER.info('Running post-processing...')
315-
conn_string = connection_string(db_name='pgosm')
316-
cmds = ['psql', '-d', conn_string, '-f', f'{layerset}.sql']
316+
317+
cmds = ['lua', 'run-sql.lua']
318+
317319
output = subprocess.run(cmds,
318320
text=True,
319321
capture_output=True,
@@ -336,10 +338,18 @@ def pgosm_nested_admin_polygons(paths):
336338
LOGGER.info('Building nested polygons... (this can take a while)')
337339
output = subprocess.run(cmds,
338340
text=True,
339-
capture_output=True,
340341
cwd=paths['flex_path'],
341-
check=True)
342-
LOGGER.info(f'Nested polygon output: \n {output.stderr}')
342+
check=False,
343+
stdout=subprocess.PIPE,
344+
stderr=subprocess.STDOUT)
345+
LOGGER.info(f'Nested polygon output: \n {output.stdout}')
346+
347+
348+
if output.returncode != 0:
349+
err_msg = f'Failed to build nested polygons. Return code: {output.returncode}'
350+
LOGGER.error(err_msg)
351+
sys.exit(f'{err_msg} - Check the log output for details.')
352+
343353

344354

345355
def rename_schema(schema_name):

docker/osm2pgsql_recommendation.py

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@
1010
LOGGER = logging.getLogger('pgosm-flex')
1111

1212

13-
def osm2pgsql_recommendation(ram, layerset, pbf_filename, out_path):
13+
def osm2pgsql_recommendation(ram, pbf_filename, out_path):
1414
"""Returns recommended osm2pgsql command.
1515
1616
Recommendation from API at https://osm2pgsql-tuner.com
@@ -29,7 +29,8 @@ def osm2pgsql_recommendation(ram, layerset, pbf_filename, out_path):
2929
osm2pgsql_cmd : str
3030
"""
3131
system_ram_gb = ram
32-
pgosm_layer_set = layerset
32+
# The layerset is now set via env var. This is used to set filename for osm2pgsql command
33+
pgosm_layer_set = 'run'
3334

3435
pbf_file = os.path.join(out_path, pbf_filename)
3536
osm_pbf_gb = os.path.getsize(pbf_file) / 1024 / 1024 / 1024

0 commit comments

Comments
 (0)