Reference data is missing countries

[zelima]

There are countries in scanned data that are not included in reference data for countries.
Currently they are added manually by me.

This should to be automated somehow

[zelima]

Examples

AQ,Antarctica,antarctica,Antarctica,Antarctica 
C,Unknown,unknown,, 
CF,Central African Republic,central-african-republic,, 
EU,Europe,europe,, 
FK,"Falkland Islands (Malvinas)",falkland-islands,Americas,Amricas 
GS,South Georgia and South Sandwich Islands,south-georgia-and-south-sandwich-islands,Americas,Americas 
GW,Guinea-Bissau,guinea-bissau,Africa,Africa 
IO,British Indian Ocean Territory,british-indian-ocean-territory,Asia,Asia
NF,Norfolk Island,norfolk-island,Oceania,Oceania 
NR,Nauru,nauru,Africa,Africa
NU,Niue,niue,Oceania,Oceania
PW,Palau,palau,Oceania,Oceania
TK,Tokelau,tokelau,Oceania,Oceania
TV,Tuvalu,tuvalu,Oceania,Oceania
UM,"U.S. Outlying Islands",us-outlying-slands,Americas,Americas
VA,Vatican City,vatican-city,Europe,Europe
WF,Wallis and Futuna Islands,Wallis-and-futuna-islands,Territory of FR,Europe
XY,Unknown XY,unknown-xy,,
YT,Mayotte,mayotte,Africa,Afcrica
ZZ,-Reserved AS-,reserved-as,,

[aaronkaplan]

I believe the country code -> country name mapping came from Atomatic. So, yes, please update as needed. Thanks.

[zelima]

cc @rufuspollock
Currently we use this dataset https://github.com/datasets/country-codes/ for getting data.

We need to think what to do if countries from scanned data not present in that dataset https://raw.githubusercontent.com/datasets/country-codes/master/data/country-codes.csv

[rufuspollock]

@zelima first of all some items are definitely not countries 😉 e.g. Antartica is a region or continent!

Some of the rest are places that are islands that are part of a larger geopolitical entity. E.g. Falkland Islands are legally a part of Britain (I believe!).

It is a "business" question as to how we treat these. @aaronkaplan any thoughts?

I believe the country code -> country name mapping came from Atomatic. So, yes, please update as needed. Thanks.

@aaronkaplan generally this is not a country -> code issue as an issue that codes are being assigned during the enrichment that don't exist as ISO 2 digit codes ...

[aaronkaplan]

On 27 Jan 2017, at 06:55, Rufus Pollock ***@***.***> wrote: @zelima first of all some items are definitely not countries 😉 e.g. Antartica is a region or continent! Some of the rest are places that are islands that are part of a larger geopolitical entity. E.g. Falkland Islands are legally a part of Britain (I believe!). It is a "business" question as to how we treat these. @aaronkaplan any thoughts?

The country codes that come out of the enrichment are "maxmind extended country codes". That means: A1 = satellite or so. So basically we have to stick with that.

…

I believe the country code -> country name mapping came from Atomatic. So, yes, please update as needed. Thanks. @aaronkaplan generally this is not a country -> code issue as an issue that codes are being assigned during the enrichment that don't exist as ISO 2 digit codes ... — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[zelima]

I understand that they are not countries, but still they come up in scanned data.

@rufuspollock @aaronkaplan I'm not quite clear about the solution. should we switch to other source? Eg: http://dev.maxmind.com/geoip/legacy/codes/iso3166/

[rufuspollock]

@zelima i think we need to create a reference dataset with these items. However, it will present problems if we ever want to do charting as most of the charts will want country ISO codes.

Need to think about a solution here and not immediately obvious.

[aaronkaplan]

I think ISO codes are nearly 100% identical to maxmind. Except for "GB" versus "UK" and the A1 and A2 satellite providers

…

--- Mobile

On 28 Jan 2017, at 12:29, Rufus Pollock ***@***.***> wrote: @zelima i think we need to create a reference dataset with these items. However, it will present problems if we ever want to do charting as most of the charts will want country ISO codes. Need to think about a solution here and not immediately obvious. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.