Merge changes - switch from all views to tables via centroid (#341)

* Add shop tests (#338)

Add tests for shop point/polygon counts. Includes moving test docs to main docs page, and overall improvements to the examples.

* Change "all" views to tables loaded via Flex style (#337)

* Changing vbuilding_all view to building_combined_point table

* Create PK, move helper into style folder

* Starting to convert vshop_all. Current version has test errors due to moving code into shop_helper. Need to add more tests on original to help find the cause of the problem with this change.

* Fix loading of shop polygons. Reset shop_polygon test output to actual.

* Add detailed POI tests

* Restructure POI loading in prep to migrate all view

* Moving from vpoi_all mat view to poi_combined_point table

* still failing on poi_combined_point

* Fixed import problem by setting up ID column properly

* Cleanup RE #320 (#340)

* Remove refresh of MV that no longer exists. Make comments on new tables more consistent

* Add missing comments. Install PgDD extension

Dockerfile

@@ -47,8 +47,9 @@ RUN git clone --depth 1 --branch $OSM2PGSQL_BRANCH https://github.com/openstreet
     && apt autoremove -y \
     && cd /tmp && rm -r /tmp/osm2pgsql
 
-
-COPY ./sqitch.conf /etc/sqitch/sqitch.conf
+RUN wget https://github.com/rustprooflabs/pgdd/releases/download/0.5.0/pgdd_0.5.0_postgis_pg15_amd64.deb \
+    && dpkg -i ./pgdd_0.5.0_postgis_pg15_amd64.deb \
+    && rm ./pgdd_0.5.0_postgis_pg15_amd64.deb
 
 WORKDIR /app
 COPY . ./

db/deploy/replication_functions.sql

@@ -1,5 +1,3 @@
--- Deploy pgosm-flex:002 to pg
--- requires: 001
 
 BEGIN;
 
@@ -22,7 +20,6 @@ CREATE OR REPLACE PROCEDURE osm.append_data_finish(skip_nested BOOLEAN = False)
  BEGIN
 
     REFRESH MATERIALIZED VIEW osm.vplace_polygon_subdivide;
-    REFRESH MATERIALIZED VIEW osm.vpoi_all;
 
     IF $1 = False THEN
         RAISE NOTICE 'Populating nested place table';

docs/src/tests.md

@@ -21,30 +21,185 @@ A simplified usage for quicker testing during development.
 make docker-exec-default unit-tests
 ```
 
-
-## Python unit tests
-
-The Python unit tests are under `pgosm-flex/docker/tests/`.  These tests use
-Python's `unittest` module.  The `make` process runs these using
-`coverage run ...` and `coverage report ...`.
-See the [Makefile](https://github.com/rustprooflabs/pgosm-flex/blob/main/Makefile)
-for exact implementation.
-
-
-## Data import tests
+## Data Tests
 
 Under `pgosm-flex/tests`.  The `run-output-tests.sh` script is ran by
 running `make`.  The script loops over the `.sql` scripts under
 `pgosm-flex/tests/sql/`, runs the queries via `psql` using
 `--no-psqlrc -tA` and compares the output from the query against the
 expected output saved under `pgosm-flex/tests/expected`.
+Running `make docker-exec-default unit-tests` should finish with this line
+reporting data tests completed successfully.
+
+
+```bash
+Data output tests completed successfully.
+```
+
+If something in the Lua styles or SQL post-processing changes, the intention is
+they will be reported by these tests.   Note the second line of this section
+reports the `docker exec` command to run in order to see the changes to the
+data tests.
+
+```bash
+FAILED TEST: sql/shop_polygon_osm_type_subtype_count.sql - See tmp/shop_polygon_osm_type_subtype_count.diff
+  docker exec -it pgosm /bin/bash -c "cat /app/tests/tmp/shop_polygon_osm_type_subtype_count.diff " 
+One or more data output tests failed.
+```
+
+The output from the `docker exec` command looks like the following.
+Note the `-` and `+` lines showing the count of records with
+`osm_type='shop'` and `osm_subtype='chemist'` changed from 1 to 2.
+
+
+```bash
+diff --git a/expected/shop_polygon_osm_type_subtype_count.out b/tmp/shop_polygon_osm_type_subtype_count.out
+index 75c16c3..2385d8d 100644
+--- a/expected/shop_polygon_osm_type_subtype_count.out
++++ b/tmp/shop_polygon_osm_type_subtype_count.out
+@@ -12,7 +12,7 @@ shop|books|1
+ shop|car|4
+ shop|car_parts|3
+ shop|car_repair|7
+-shop|chemist|1
++shop|chemist|2
+ shop|clothes|8
+ shop|convenience|35
+ shop|copyshop|1
+```
+
+
+### Add / Update Data Tests
+
+This section provides guidance to adding/updating data tests for PgOSM Flex.
+The SQL file to run is under `tests/sql/*.sql`, the expected results are saved
+under `tests/expected/*.out`.
+
+
+#### Load Test Data
+
+Load `data/district-of-columbia-2021-01-13.osm.pbf` with `run-all`
+before running these tests.
+
+
+> PBF sourced [from Geofabrik's download service](https://download.geofabrik.de/) on January 13, 2021.
+
+#### Craft Test Query
+
+Connect to the PgOSM Flex database with `data/district-of-columbia-2021-01-13.osm.pbf`
+loaded.  Write the query that provide results to test for.
+
+Important:  Results must be ordered using `COLLATE "C"` to ensure consistent
+ordering across systems.  For example it should be written like this:
+
+```sql
+SELECT osm_type COLLATE "C", COUNT(*)
+    FROM osm.amenity_point
+    GROUP BY osm_type COLLATE "C"
+    ORDER BY osm_type COLLATE "C"
+;
+```
+
+Not like this:
+
+```sql
+SELECT osm_type, COUNT(*)
+    FROM osm.amenity_point
+    GROUP BY osm_type
+    ORDER BY osm_type
+;
+```
 
 
 
+#### Create Expected Output
+
+
+To create new tests, or to update existing tests use `psql --no-psqlrc -tA <details>`.
+Example for amenity count of `osm_type`.
+
+Assuming [Quick Start](quick-start.md) instructions, set the env var for the Postgres
+connection first.
+
+```bash
+export PGOSM_CONN=postgresql://postgres:mysecretpassword@localhost:5433/pgosm
+```
+
+Run the query, save the output. 
+
+```bash
+psql --no-psqlrc -tA  \
+    -d $PGOSM_CONN \
+     -f sql/amenity_osm_type_count.sql \
+     > expected/amenity_osm_type_count.out
+```
+
+#### Validate New Tests
+
+Ensure the data tests work and are reported if values change via `make`.
+The best way to ensure the test is working is manually change one value in the
+generated `.out` file which should cause the following error message.
+Setting the `.out` data back to right should return the message back to successful.
+
+```bash
+FAILED TEST: sql/shop_polygon_osm_type_subtype_count.sql - See tmp/shop_polygon_osm_type_subtype_count.diff
+  docker exec -it pgosm /bin/bash -c "cat /app/tests/tmp/shop_polygon_osm_type_subtype_count.diff " 
+One or more data output tests failed.
+```
+
+
+
+----
+
+### Create PBFs for areas w/ Failures
+
+Identify a feature related to the issue and load small region around
+into JOSM (as if making an edit).
+
+Use JOSM's "Save As..." to save the `<region-failure-name>.osm` file.
+Use `osmium-tool` (https://osmcode.org/osmium-tool/manual.html)
+to convert to `.pbf` format.
+
+```bash
+osmium cat  <region-failure-name>.osm -o <region-failure-name>.osm.pbf
+mv <region-failure-name>.osm.pbf ~/git/pgosm-flex/tests/data/extra-regions/
+``` 
+
+
+### Test for import failures
+
+Test for specific regions that have had failures due to unusual
+tags and/or bugs in PgOSM-Flex.
+
+
+Run extra region load tests.
+
+
+
+```bash
+export PGOSM_CONN=pgosm_tests
+export PGOSM_CONN_PG=postgres
+./run-extra-loads.sh
+```
 
 > FIXME: At this time the `run-extra-loads.sh` script is not ran automatically.  There are not any usage notes covering those random side tests.
 
 
+
+## Python unit tests
+
+The Python unit tests are under `pgosm-flex/docker/tests/`.  These tests use
+Python's `unittest` module.  The `make` process runs these using
+`coverage run ...` and `coverage report ...`.
+See the [Makefile](https://github.com/rustprooflabs/pgosm-flex/blob/main/Makefile)
+for exact implementation.
+
+These unit tests cover specific logic and functionality to how PgOSM Flex's Python
+program runs.
+
+----
+
+
 ## What is not tested
 
 Functionality of `osm2pgsql-replication` to actually update data.  Challenge
@@ -71,6 +226,3 @@ refresh fails.
 2023-01-29 08:11:36,890:INFO:pgosm-flex:pgosm_flex:Skipping pg_dump
 2023-01-29 08:11:36,890:WARNING:pgosm-flex:pgosm_flex:PgOSM Flex completed with errors. Details in output
 ```
-
-
-

flex-config/layerset/basic.ini

@@ -1,10 +1,11 @@
 [layerset]
 amenity=false
-building=true
+building_combined_point=true
 landuse=true
 leisure=false
 natural=false
 place=true
 poi=false
+poi_combined_point=true
 road_major=true
 unitable=true

flex-config/layerset/default.ini

@@ -1,16 +1,19 @@
 [layerset]
 amenity=true
 building=true
+building_combined_point=false
 indoor=true
 infrastructure=true
 landuse=true
 leisure=true
 natural=true
 place=true
 poi=true
+poi_combined_point=false
 public_transport=true
 road=true
 shop=true
+shop_combined_point=false
 tags=true
 traffic=true
 water=true

flex-config/layerset/everything.ini

@@ -1,17 +1,20 @@
 [layerset]
 amenity=true
 building=true
+building_combined_point=true
 indoor=true
 infrastructure=true
 landuse=true
 leisure=true
 natural=true
 place=true
 poi=true
+poi_combined_point=true
 public_transport=true
 road=true
 road_major=true
 shop=true
+shop_combined_point=true
 tags=true
 traffic=true
 unitable=true

flex-config/layerset/minimal.ini

@@ -1,4 +1,4 @@
 [layerset]
 place=true
-poi=true
+poi_combined_point=true
 road_major=true

flex-config/run-sql.lua

@@ -13,10 +13,10 @@ else
     error('ENV VAR PGOSM_CONN must be set.')
 end
 
-layers = {'amenity', 'building', 'indoor', 'infrastructure', 'landuse'
-          , 'leisure'
+layers = {'amenity', 'building', 'building_combined_point', 'indoor'
+          , 'infrastructure', 'landuse', 'leisure'
           , 'natural', 'place', 'poi', 'public_transport'
-          , 'road', 'road_major', 'shop', 'tags'
+          , 'road', 'road_major', 'shop', 'shop_combined_point', 'tags'
           , 'traffic', 'unitable', 'water'}
 
 

flex-config/run.lua

@@ -12,6 +12,11 @@ if conf['layerset']['building'] then
     require "style.building"
 end
 
+if conf['layerset']['building_combined_point'] then
+    print('Including building_combined_point')
+    require "style.building_combined_point"
+end
+
 if conf['layerset']['indoor'] then
     print('Including indoor')
     require "style.indoor"
@@ -47,6 +52,11 @@ if conf['layerset']['poi'] then
     require "style.poi"
 end
 
+if conf['layerset']['poi_combined_point'] then
+    print('Including poi_combined_point')
+    require "style.poi_combined_point"
+end
+
 if conf['layerset']['public_transport'] then
     print('Including public_transport')
     require "style.public_transport"
@@ -67,6 +77,11 @@ if conf['layerset']['shop'] then
     require "style.shop"
 end
 
+if conf['layerset']['shop_combined_point'] then
+    print('Including shop_combined_point')
+    require "style.shop_combined_point"
+end
+
 if conf['layerset']['tags'] then
     print('Including tags')
     require "style.tags"

flex-config/sql/amenity.sql

@@ -58,3 +58,11 @@ COMMENT ON COLUMN osm.amenity_polygon.address IS 'Address combined from address
 COMMENT ON COLUMN osm.amenity_point.osm_subtype IS 'Further describes osm_type for amenities.';
 COMMENT ON COLUMN osm.amenity_line.osm_subtype IS 'Further describes osm_type for amenities.';
 COMMENT ON COLUMN osm.amenity_polygon.osm_subtype IS 'Further describes osm_type for amenities.';
+
+COMMENT ON COLUMN osm.amenity_point.wheelchair IS 'Indicates if feature is wheelchair accessible. Values:  yes, no, limited.  Per https://wiki.openstreetmap.org/wiki/Key:wheelchair';
+COMMENT ON COLUMN osm.amenity_line.wheelchair IS 'Indicates if feature is wheelchair accessible. Values:  yes, no, limited.  Per https://wiki.openstreetmap.org/wiki/Key:wheelchair';
+COMMENT ON COLUMN osm.amenity_polygon.wheelchair IS 'Indicates if feature is wheelchair accessible. Values:  yes, no, limited.  Per https://wiki.openstreetmap.org/wiki/Key:wheelchair';
+
+COMMENT ON COLUMN osm.amenity_point.wheelchair_desc IS 'Value from wheelchair:description Per https://wiki.openstreetmap.org/wiki/Key:wheelchair:description';
+COMMENT ON COLUMN osm.amenity_line.wheelchair_desc IS 'Value from wheelchair:description Per https://wiki.openstreetmap.org/wiki/Key:wheelchair:description';
+COMMENT ON COLUMN osm.amenity_polygon.wheelchair_desc IS 'Value from wheelchair:description Per https://wiki.openstreetmap.org/wiki/Key:wheelchair:description';