Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

77 weighted points #82

Open
wants to merge 7 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions examples/york_minimal/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
README_files/
input/north-yorkshire-latest.osm.pbf
input/inputs.html
150 changes: 150 additions & 0 deletions examples/york_minimal/README.md
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm hesitant to check both qmd and md into git, because the second is derived from the first, and it contains lots of log-spam. Looking at https://quarto.org/docs/publishing/github-pages.html, checking in both is option 1, right? Publishing to GH Pages is more work upfront, but feels like the better option. I can set that up after we get this first PR merged?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, happy to remove the .md. Glad I didn't check in the rest of the stuff that quarto outputs based on this comment!

Original file line number Diff line number Diff line change
@@ -0,0 +1,150 @@


The setup information is contained within the `setup.py` file, which
generates minimal input files.

``` bash
python setup.py
```

We’ll get a sample of 2 schools in York (York High School and Huntington
School) using the `osmextract` package.

``` r
library(osmextract)
```

Data (c) OpenStreetMap contributors, ODbL 1.0. https://www.openstreetmap.org/copyright.
Check the package website, https://docs.ropensci.org/osmextract/, for more details.

``` r
q = "SELECT * FROM multipolygons WHERE amenity='school'"
schools_york = osmextract::oe_get("York", query = q, extra_tags = "amenity")
```

No exact match found for place = York and provider = geofabrik. Best match is Corse.
Checking the other providers.

No exact match found in any OSM provider data. Searching for the location online.

The input place was matched with North Yorkshire.

The chosen file was already detected in the download directory. Skip downloading.

The corresponding gpkg file was already detected. Skip vectortranslate operations.

Reading query `SELECT * FROM multipolygons WHERE amenity='school''
from data source `/home/robin/data/osm/geofabrik_north-yorkshire-latest.gpkg'
using driver `GPKG'
Simple feature collection with 603 features and 25 fields
Geometry type: MULTIPOLYGON
Dimension: XY
Bounding box: xmin: -2.546044 ymin: 53.6425 xmax: -0.2912398 ymax: 54.61681
Geodetic CRS: WGS 84

``` r
# schools_york$name
destinations = dplyr::filter(
schools_york,
name %in% c("York High School", "Huntington School")
) |>
dplyr::select(name, everything())
destinations$name
```

[1] "York High School" "Huntington School"

``` r
# Remove columns that only contain NA:
destinations = destinations[, colSums(is.na(destinations)) < nrow(destinations)]
destinations = sf::st_centroid(destinations)
```

Warning: st_centroid assumes attributes are constant over geometries

``` r
sf::write_sf(destinations, "input/destinations.geojson", delete_dsn = TRUE)
```

We’ll also create a sample of subpoints in York, taking 3 random points
from each zone.

``` r
zones = sf::st_read("input/zones.geojson")
```

Reading layer `zones' from data source
`/home/robin/github/Urban-Analytics-Technology-Platform/od2net/examples/york_minimal/input/zones.geojson'
using driver `GeoJSON'
Simple feature collection with 3 features and 1 field
Geometry type: POLYGON
Dimension: XY
Bounding box: xmin: -1.146752 ymin: 53.92474 xmax: -1.025942 ymax: 54.01074
Geodetic CRS: WGS 84

``` r
set.seed(123)
subpoints = sf::st_sample(zones, size = rep(3, nrow(zones))) |>
sf::st_sf()
# Let's add provide the subpoints with values representing their importance:
subpoints$size = runif(nrow(subpoints), 1, 10) |>
round(1)
sf::write_sf(subpoints, "input/subpoints.geojson", delete_dsn = TRUE)
```

We can visualise these as follows:

``` python
import geopandas as gpd
import pandas as pd
zones = gpd.read_file("input/zones.geojson")
destinations = gpd.read_file("input/destinations.geojson")
subpoints = gpd.read_file("input/subpoints.geojson")
od = pd.read_csv("input/od.csv")
ax = zones.plot()
destinations.plot(ax=ax, color='red')
subpoints.plot(ax=ax, color='blue', markersize=subpoints['size'] * 3)
ax.set_title("Origins and Destinations")
```

![](README_files/figure-commonmark/origins_destinations_plot-1.png)

Let’s visualise the flows between the origins and destinations:

``` r
library(ggplot2)
od = readr::read_csv("input/od.csv")
```

Rows: 6 Columns: 3
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr (2): from, to
dbl (1): count

ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.

``` r
od_geo = od::od_to_sf(od, zones, destinations)
```

0 origins with no match in zone ids
0 destinations with no match in zone ids
points not in od data removed.

``` r
ggplot() +
geom_sf(data = zones, fill = "grey") +
geom_sf(data = subpoints, aes(size = size), color = "blue") +
geom_sf(data = destinations, color = "red") +
geom_sf(data = od_geo, aes(size = count), color = "black")
```

![](README_files/figure-commonmark/flows_plot-3.png)

We can then run the od2net command as follows:

``` bash
docker run -v $(pwd):/app ghcr.io/urban-analytics-technology-platform/od2net:main /app/config.json
```
96 changes: 96 additions & 0 deletions examples/york_minimal/README.qmd
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
---
format: gfm
---

The setup information is contained within the `setup.py` file, which generates minimal input files.

```{bash}
#| eval: false
python setup.py
```

We'll get a sample of 2 schools in York (York High School and Huntington School) using the `osmextract` package.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm fine showing examples like this here, but FWIW, something like Overpass can be quicker/more interactive. Ultimately a user guide could point to a variety of methods for generating inputs like this. https://overpass-turbo.eu/s/1RL9 as an example

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes Overpass is fine for this also. Happy for this to be tweaked, just using what I know best. I don't think it matters here but happy to use another tool to get the schools if there are ways of doing that with reproducible and readable code. Doesn't seem like a biggy or blocker at this stage but let me know if you'd like to see changes to this bit.


```{r}
library(osmextract)
q = "SELECT * FROM multipolygons WHERE amenity='school'"
schools_york = osmextract::oe_get("York", query = q, extra_tags = "amenity")
# schools_york$name
destinations = dplyr::filter(
schools_york,
name %in% c("York High School", "Huntington School")
) |>
dplyr::select(name, everything())
destinations$name
# Remove columns that only contain NA:
destinations = destinations[, colSums(is.na(destinations)) < nrow(destinations)]
destinations = sf::st_centroid(destinations)
sf::write_sf(destinations, "input/destinations.geojson", delete_dsn = TRUE)
```

We'll also create a sample of subpoints in York, taking 3 random points from each zone.

```{r}
zones = sf::st_read("input/zones.geojson")
set.seed(123)
subpoints = sf::st_sample(zones, size = rep(3, nrow(zones))) |>
sf::st_sf()
# Let's add provide the subpoints with values representing their importance:
subpoints$size = runif(nrow(subpoints), 1, 10) |>
round(1)
sf::write_sf(subpoints, "input/subpoints.geojson", delete_dsn = TRUE)
```

We can visualise these as follows:

```{python}
#| label: origins_destinations_plot
import geopandas as gpd
import pandas as pd
zones = gpd.read_file("input/zones.geojson")
destinations = gpd.read_file("input/destinations.geojson")
subpoints = gpd.read_file("input/subpoints.geojson")
od = pd.read_csv("input/od.csv")
ax = zones.plot()
destinations.plot(ax=ax, color='red')
subpoints.plot(ax=ax, color='blue', markersize=subpoints['size'] * 3)
ax.set_title("Origins and Destinations")
```

Let's visualise the flows between the origins and destinations:

```{r}
#| label: flows_plot
library(ggplot2)
od = readr::read_csv("input/od.csv")
od_geo = od::od_to_sf(od, zones, destinations)
ggplot() +
geom_sf(data = zones, fill = "grey") +
geom_sf(data = subpoints, aes(size = size), color = "blue") +
geom_sf(data = destinations, color = "red") +
geom_sf(data = od_geo, aes(size = count), color = "black")
```

```{r}
#| eval: false
#| echo: false
library(tmap)
tmap_mode("view")
m = qtm(zones) +
tm_shape(subpoints) +
tm_dots(size = "size", col = "blue") +
tm_shape(destinations) +
tm_dots(fill = "red", size = 5) +
tm_shape(od_geo) +
tm_lines(lwd = "count", col = "black", scale = 9)
tmap_save(m, "input/inputs.html")
browseURL("input/inputs.html")
```

We can then run the od2net command as follows:


```{bash}
#| eval: false
docker run -v $(pwd):/app ghcr.io/urban-analytics-technology-platform/od2net:main /app/config.json
```
22 changes: 22 additions & 0 deletions examples/york_minimal/config.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
{
"requests": {
"description": "Manually drawn zones and flows to schools, demonstrating weighted subpoints",
"pattern": {
"ZoneToPoint": {
"zones_path": "zones.geojson",
"destinations_path": "destinations.geojson",
"csv_path": "od.csv",
"origin_zone_centroid_fallback": false
}
},
"origins_path": "subpoints.geojson",
"destinations_path": "destinations.geojson"
},
"uptake": "Identity",
"cost": {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It'd be simpler to not use a custom external command for lts and cost in this example. That's causing the Docker issue, and other examples demonstrate how to do this already. How about following the Liverpool example for cost and lts? Do you want something like quiet routes for this, just distance / direct roues, or something else?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, simple is good, clearly just copy-pasted from previous York example. Can update that, good suggestion.

"ExternalCommand": "python3 cost.py"
},
"lts": {
"ExternalCommand": "python3 lts.py"
}
}
26 changes: 26 additions & 0 deletions examples/york_minimal/cost.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
import json
import sys

# Output a numeric edge cost
def calculate(edge):
tags = edge["osm_tags"]
length_meters = edge["length_meters"]
lts = edge["lts"]
nearby_amenities = edge["nearby_amenities"]
slope = edge["slope"]

# Return None to not use the edge at all

if tags["highway"] == "residential":
return [round(length_meters), round(length_meters)]
else:
# Strongly avoid non-residential roads
return [round(10 * length_meters), round(10 * length_meters)]


# Read an array of JSON dictionaries from STDIN
input_batch = json.loads(sys.stdin.read())
# Calculate an edge cost for each one
results = list(map(calculate, input_batch))
# Write a JSON array of the resulting numbers
print(json.dumps(results))
9 changes: 9 additions & 0 deletions examples/york_minimal/input/destinations.geojson
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
{
"type": "FeatureCollection",
"name": "destinations",
"crs": { "type": "name", "properties": { "name": "urn:ogc:def:crs:OGC:1.3:CRS84" } },
"features": [
{ "type": "Feature", "properties": { "name": "York High School", "osm_way_id": "29110900", "amenity": "school", "other_tags": "\"addr:city\"=>\"York\",\"addr:country\"=>\"GB\",\"addr:postcode\"=>\"YO24 3WZ\",\"addr:street\"=>\"Cornlands Road\",\"addr:suburb\"=>\"Acomb\",\"capacity\"=>\"991\",\"email\"=>\"[email protected]\",\"isced:level\"=>\"2\",\"max_age\"=>\"16\",\"min_age\"=>\"11\",\"phone\"=>\"+44 1904 555500\",\"ref:GB:uprn\"=>\"100052163540\",\"ref:edubase\"=>\"144652\",\"ref:edubase:group\"=>\"16073\",\"school:trust\"=>\"yes\",\"school:trust:name\"=>\"South Bank Multi Academy Trust\",\"school:trust:type\"=>\"multi_academy\",\"school:type\"=>\"academy\",\"website\"=>\"https://www.yorkhighschool.co.uk/\",\"wikidata\"=>\"Q8055461\",\"wikipedia\"=>\"en:York High School, York\"" }, "geometry": { "type": "Point", "coordinates": [ -1.128900930965928, 53.947613356147485 ] } },
{ "type": "Feature", "properties": { "name": "Huntington School", "osm_way_id": "122135723", "amenity": "school", "other_tags": "\"addr:country\"=>\"GB\",\"addr:postcode\"=>\"YO32 9WT\",\"addr:street\"=>\"Huntington Road\",\"capacity\"=>\"1545\",\"email\"=>\"[email protected]\",\"isced:level\"=>\"2;3\",\"max_age\"=>\"18\",\"min_age\"=>\"11\",\"phone\"=>\"+44 1904 752100\",\"ref:GB:uprn\"=>\"100052170371\",\"ref:edubase\"=>\"121673\",\"school:trust\"=>\"no\",\"school:type\"=>\"community\",\"website\"=>\"https://huntingtonschool.co.uk/\"" }, "geometry": { "type": "Point", "coordinates": [ -1.063642017028779, 53.990093958328686 ] } }
]
}
7 changes: 7 additions & 0 deletions examples/york_minimal/input/od.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
from,to,count
south,York High School,500
center,York High School,100
north,York High School,200
south,Huntington School,800
center,Huntington School,300
north,Huntington School,600
16 changes: 16 additions & 0 deletions examples/york_minimal/input/subpoints.geojson
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. I want to carve out really clear terminology in this project, and "subpoints" does not seem ideal. Since these are used as origins, what about origins.geojson?
  2. The property needs to be called weight, not size. I need to document this still

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No objection to origins.geojson, other than slight name clash with 'zones of origin': you can have zones in which flows originate and origins that specificy the exact points they must flow from. But as long as it's documented could well be the least bad options, I have no better suggestions, and happy with origins.geojson if you are. Subpoints comes from the jittering paper and it seems we're using the word in a similar way, what's the objection to subpoints out of interest?

👍 to clarify terms upfront and happy to adapt.

Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
{
"type": "FeatureCollection",
"name": "subpoints",
"crs": { "type": "name", "properties": { "name": "urn:ogc:def:crs:OGC:1.3:CRS84" } },
"features": [
{ "type": "Feature", "properties": { "size": 7.2 }, "geometry": { "type": "Point", "coordinates": [ -1.084948428827924, 53.959860019753116 ] } },
{ "type": "Feature", "properties": { "size": 8.2 }, "geometry": { "type": "Point", "coordinates": [ -1.065675923641903, 53.968255508127534 ] } },
{ "type": "Feature", "properties": { "size": 1.2 }, "geometry": { "type": "Point", "coordinates": [ -1.08027588725639, 53.960397590493173 ] } },
{ "type": "Feature", "properties": { "size": 5.3 }, "geometry": { "type": "Point", "coordinates": [ -1.048145775760174, 53.988466436780065 ] } },
{ "type": "Feature", "properties": { "size": 7.8 }, "geometry": { "type": "Point", "coordinates": [ -1.055372173407674, 54.009228087733909 ] } },
{ "type": "Feature", "properties": { "size": 2.9 }, "geometry": { "type": "Point", "coordinates": [ -1.077859414261401, 53.998822595720604 ] } },
{ "type": "Feature", "properties": { "size": 3.9 }, "geometry": { "type": "Point", "coordinates": [ -1.11062851182091, 53.93422678953317 ] } },
{ "type": "Feature", "properties": { "size": 3.1 }, "geometry": { "type": "Point", "coordinates": [ -1.107718347978171, 53.929568710302227 ] } },
{ "type": "Feature", "properties": { "size": 2.3 }, "geometry": { "type": "Point", "coordinates": [ -1.116778858632627, 53.956331735484653 ] } }
]
}
1 change: 1 addition & 0 deletions examples/york_minimal/input/zones.geojson
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"type":"FeatureCollection","features":[{"type":"Feature","properties":{"name":"center"},"geometry":{"coordinates":[[[-1.08285,53.970735],[-1.096017,53.958917],[-1.075591,53.947693],[-1.057528,53.967309],[-1.08285,53.970735]]],"type":"Polygon"}},{"type":"Feature","properties":{"name":"north"},"geometry":{"coordinates":[[[-1.094806,53.998733],[-1.068685,53.977605],[-1.025942,53.9965],[-1.056812,54.010736],[-1.094806,53.998733]]],"type":"Polygon"}},{"type":"Feature","properties":{"name":"south"},"geometry":{"coordinates":[[[-1.146752,53.957545],[-1.139312,53.924745],[-1.091661,53.9281],[-1.1067,53.956427],[-1.146752,53.957545]]],"type":"Polygon"}}]}
17 changes: 17 additions & 0 deletions examples/york_minimal/lts.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
import json
import sys

# Output 0 (not allowed), 1 (suitable for children), 2 (low stress), 3 (low stress), or 4 (high stress)
def calculate(tags):
if tags["highway"] == "residential":
return 2
else:
return 4


# Read an array of JSON dictionaries from STDIN
tags_batch = json.loads(sys.stdin.read())
# Calculate LTS for each one
lts_results = list(map(calculate, tags_batch))
# Write a JSON array of the resulting numbers
print(json.dumps(lts_results))
Loading