Output warnings in Latitude/Longtitude, not WebMercator

[stoecker]

I recently did an import with osm2pgsql and I got a new sort of errors like this:
NOTICE: Self-intersection at or near point -19412892.617835309 -20061906.389537029
NOTICE: Self-intersection at or near point -18940006.029517677 -20061906.389537029
NOTICE: Self-intersection at or near point -18981978.108431712 -20061906.389537029
NOTICE: Self-intersection at or near point -19463448.420231782 -20061906.389537029

That's not very helpful, as for an editor you need Latitude/Longitude to find the position.
Please output the warning in latitude/longitude instead.

Or as copyable string like https://www.openstreetmap.org/#map=14/30.52151244209846/120.72325756759668

[lonvia]

These notices are generated by Postgresql, nothing we can do about their format. You can safely ignore them as they are generated while repairing the self-intersections. The final database is intersection free.

We could silence postgres while doing the repair but that would mean that any other notices are not printed either.

[stoecker]

Are they directly from postgres or coming via osm2pgsql? Because then they could be transformed back to lat/lon. It is clear they can be ignored, but better would be to have a chance to fix the osm database. I always try to fix osm2pgsql import errors and others do the same. So for each import some errors vanish...

[lonvia]

It's Postgres printing the message all on its own. osm2pgsql never sees them. Note that these errors are all close to the poles and are a result of mercator projection and cutting geometries near the pole. They are not errors in the OSM data.

[kennykb]

It's possible to intercept those messages with PQNoticeReceiver, which turns off printing them and lets the client handle them.

[pnorman]

other notices are not printed either

We could turn up the min level to warning.

[Komzpa]

@stoecker you can get the warning in longitude/latitude if you do -E 4326 import.

[stoecker]

@stoecker you can get the warning in longitude/latitude if you do -E 4326 import.

That would change the database and thus is no option. The suggestion from kennykb sounds sane to me. I assume it also has other benefits when osm2pgsql knows what errors/warnings are issued.

[woodpeck]

That would change the database and thus is no option. The suggestion from kennykb sounds sane to me. I assume it also has other benefits when osm2pgsql knows what errors/warnings are issued.

Errors are returned to the calling code even without using the PQNoticeReceiver functionality, which is for notices and warnings only. AFAICS PQNoticeReceiver offers no elegant mechanism to access the coordinates in the text message; one would have to parse the messages with regular expressions to guess if they contain coordinates. This is feasible but not trivial, and at risk of breaking any time PostGIS changes its error message formatting.

[mmd-osm]

I'm not exactly sure where this notice is being triggered. Could this possibly originate in GEOS and triggered by ST_IsValid?

Maybe https://postgis.net/docs/ST_IsValidDetail.html could be useful here to get a bit more structured information ("Useful to substitute and improve the combination of ST_IsValid and ST_IsValidReason to generate a detailed report of invalid geometries.").

 gid  |      reason       |  location
------+-------------------+-------------
 5330 | Self-intersection | POINT(32 5)
 5340 | Self-intersection | POINT(42 5)
 5350 | Self-intersection | POINT(52 5)

[Komzpa]

It is triggered in GEOS during ST_IsValid. Please do not parse the text, as its whole existence is undocumented quirk.

If you want to use this for QA/QC, it would be best to import without removing invalid geometries, and then perform the ST_IsValidDetail on them. #27 has two patches about it not yet applied since 2015, anyone fancy a rebased pull request?

[lonvia]

Let me reiterate: the self intersections are not geometry errors in OSM data. They are an artefact of converting geometries from WSG84 to Mercator. They happen when geometries are cut close to the poles. For details see discussions in #793 and #787.

Errors in OSM geometries are filtered much earlier in the processing chain when creating the polygons with libosmium. Objects with bad geometries are never written into the output tables. There is no way to use osm2pgsql to find those errors. Please simply use OSM inspector for that purpose.

If the warning still bothers you, create a pull request that changes the warning level around the SQL command that sorts and validates the geometries, just as @pnorman suggested. See this code for an example how to do it.

[Komzpa]

#895 opens a possibility for another flag, --allow-invalid, that will not add trigger or perform checks. That will make import faster and let @stoecker have a look at the geometries by running ST_IsValidDetail and/or checking osm id of feature.