low-level-api-proposal.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Background\n",
    "\n",
    "**This notebook was authored when I was trying to figure out the structure of the XML so I could write the API**\n",
    "\n",
    "## XTbML\n",
    "\n",
    "An XML standard for representing mortality tables called XTbML was developed in a joint effort between the Technology Section of the SOA and the standards-setting body ACORD. For more information, read [this](https://www.soa.org/news-and-publications/newsletters/compact/2013/april/com-2013-iss47/tables-database-goes-xtbml/).\n",
    "\n",
    "## Low-level access\n",
    "\n",
    "Not all XTbML files follow the select and ultimate structure of some mortality tables. For example, table `2682` has three tables and each table has two axes. We cannot make strong assumptions about the structure of the tables and so a general, low-level access is provided that is guaranteed to work across all tables. A simplified high-level access is to be provided in the future for tables that are known to follow the select and ultimate structure."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## An Object-Oriented API\n",
    "\n",
    "We propose to construct an object-oriented API for accessing XTbML files. The API is designed to be easy to use and to be flexible enough to handle all the different types of XTbML files that appear in MORT. At the bottom of the following code block you will see an `XTbML` class that is used as the main entry point to the API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from dataclasses import dataclass\n",
    "from pandas import DataFrame\n",
    "from typing import List, Dict, Set\n",
    "\n",
    "@dataclass\n",
    "class AxisDef:\n",
    "    ScaleType: str\n",
    "    AxisName: str\n",
    "    MinScaleValue: int\n",
    "    MaxScaleValue: int\n",
    "    Increment: int\n",
    "\n",
    "@dataclass\n",
    "class MetaData:\n",
    "    ScalingFactor: float\n",
    "    DataType: str\n",
    "    Nation: str\n",
    "    TableDescription: str\n",
    "    AxisDefs: List[AxisDef]\n",
    "\n",
    "@dataclass\n",
    "class Table:\n",
    "    MetaData: MetaData\n",
    "    Values: DataFrame\n",
    "\n",
    "@dataclass\n",
    "class ContentClassification:\n",
    "    TableIdentity: str\n",
    "    ProviderDomain: str\n",
    "    ProviderName: str\n",
    "    TableReference: str\n",
    "    ContentType: str\n",
    "    TableName: str\n",
    "    TableDescription: str\n",
    "    Comments: str\n",
    "    KeyWords: List[str]\n",
    "\n",
    "@dataclass\n",
    "class XTbML:\n",
    "    ContentClassification: ContentClassification\n",
    "    Tables: List[Table]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Designing the API\n",
    "\n",
    "The [About](https://mort.soa.org/About.aspx) page for MORT has a link to an [XML Schema](https://mort.soa.org/XTbML2.7.01.xsd) that describes the structure of the XTbML files. In theory the XML schema is the specification for what XTbML files should look like, but in practice several of the features it describes are not implemented in the XTbML files. Because we want a minimal API to optimize the access patterns in Pymort, we take a more empirical approach so that we can only provide the features that are actually used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# imports\n",
    "from collections import Counter\n",
    "from glob import glob\n",
    "import xml.etree.ElementTree as ET\n",
    "from typing import Union\n",
    "from pprint import pprint\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we get the root element of each XTbML file, which is an `<XTbML>` element. This is a slower computation so we store the result and use it in later computations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "tablePaths = glob('pymort/archive_2021_Oct_17_051924/*')\n",
    "roots = [tree.getroot() for tree in [ET.parse(tablePath) for tablePath in tablePaths if tablePath.endswith('.xml')]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These are some utility functions used in the analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def getTagCounts(roots: List[ET.Element]) -> List['Counter']:\n",
    "    '''\n",
    "    Return dict[tag, count of elements with that tag] for all root XTbML elements\n",
    "    '''\n",
    "    return [Counter(element.tag for element in root.iter()) for root in roots]\n",
    "\n",
    "def getChildrenCounters(roots: List[ET.Element], path: str) -> List['Counter']:\n",
    "    '''\n",
    "    Return list of dict[tag, count of children with tag] for all elements at path from root\n",
    "    This is similar to getTagCounts, but not recursive (children only) and starting from a particular path.\n",
    "    '''\n",
    "    allElements = [element for root in roots for element in root.findall(path)]\n",
    "    childrenTags = [[child.tag for child in el] for el in allElements]\n",
    "    childrenTagCounts = [Counter(tags) for tags in childrenTags]\n",
    "    return childrenTagCounts\n",
    "\n",
    "@dataclass\n",
    "class CounterStats:\n",
    "    exists: int\n",
    "    counts: int\n",
    "    uniqueCounts: Set[int]\n",
    "    \n",
    "def getChildrenCounterStats(roots: List[ET.Element], path: str) -> Dict[str, CounterStats]:\n",
    "    '''\n",
    "    Return dict[tag, CounterStats] for all elements at path from root\n",
    "    CounterStats contains summary statistics. We can see if the elements at path always have a tag for example, or what the multiplicities of tag are.\n",
    "    '''\n",
    "    counterStats: Dict[str, CounterStats] = {}\n",
    "    childrenCounters = getChildrenCounters(roots, path)\n",
    "    for counter in childrenCounters:\n",
    "        for key in counter.keys():\n",
    "            if not key in counterStats:\n",
    "                counterStats[key] = CounterStats(exists=0, counts=0, uniqueCounts=set())\n",
    "            counterStats[key].exists += 1\n",
    "            counterStats[key].counts += counter[key]\n",
    "            counterStats[key].uniqueCounts.add(counter[key])\n",
    "    return counterStats\n",
    "\n",
    "def childrenStatsPrint(path: str):\n",
    "    '''\n",
    "    Print summary statistics for all elements at path from root\n",
    "    '''\n",
    "    print(path)\n",
    "    pprint(getChildrenCounterStats(roots, path))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## An unused part of the XML schema\n",
    "\n",
    "The `AxisDef` element provides a concrete example of a feature of the XML schema that is not implemented in the XTbML files. Here is the definition of `AxisDef`:\n",
    "\n",
    "```xml\n",
    "<xsd:element name=\"AxisDef\">\n",
    "    <xsd:annotation>\n",
    "        <xsd:documentation>Definition of each Axis</xsd:documentation>\n",
    "    </xsd:annotation>\n",
    "    <xsd:complexType>\n",
    "        <xsd:sequence>\n",
    "            <xsd:element ref=\"ScaleType\" minOccurs=\"0\"/>\n",
    "            <xsd:element ref=\"ScaleSubType\" minOccurs=\"0\"/>\n",
    "            <xsd:element ref=\"AxisName\" minOccurs=\"0\"/>\n",
    "            <xsd:choice>\n",
    "                <xsd:sequence>\n",
    "                    <xsd:element ref=\"MinScaleDate\"/>\n",
    "                    <xsd:element ref=\"MaxScaleDate\" minOccurs=\"0\"/>\n",
    "                </xsd:sequence>\n",
    "                <xsd:sequence>\n",
    "                    <xsd:element ref=\"MinScaleValue\"/>\n",
    "                    <xsd:element ref=\"MaxScaleValue\" minOccurs=\"0\"/>\n",
    "                </xsd:sequence>\n",
    "            </xsd:choice>\n",
    "            <xsd:element ref=\"Increment\" minOccurs=\"0\"/>\n",
    "            <xsd:element ref=\"Mode\" minOccurs=\"0\"/>\n",
    "            <xsd:element ref=\"Continuous\" minOccurs=\"0\"/>\n",
    "        </xsd:sequence>\n",
    "        <xsd:attribute name=\"id\" type=\"xsd:string\"/>\n",
    "    </xsd:complexType>\n",
    "</xsd:element>\n",
    "```\n",
    "\n",
    "An investigation of the child elements of `AxisDef` shows that there are always 5 child tags of `AxisDef` and that they always appear exactly once."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "./Table/MetaData/AxisDef\n",
      "{'AxisName': CounterStats(exists=5364, counts=5364, uniqueCounts={1}),\n",
      " 'Increment': CounterStats(exists=5364, counts=5364, uniqueCounts={1}),\n",
      " 'MaxScaleValue': CounterStats(exists=5364, counts=5364, uniqueCounts={1}),\n",
      " 'MinScaleValue': CounterStats(exists=5364, counts=5364, uniqueCounts={1}),\n",
      " 'ScaleType': CounterStats(exists=5364, counts=5364, uniqueCounts={1})}\n"
     ]
    }
   ],
   "source": [
    "childrenStatsPrint(\"./Table/MetaData/AxisDef\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is possible that the XML schema could be more strict than it currently is by guaranteeing that the child tags of `AxisDef` always appear exactly once and by removing unused tags.\n",
    "\n",
    "```xml\n",
    "<xsd:element name=\"AxisDef\">\n",
    "    <xsd:annotation>\n",
    "        <xsd:documentation>Definition of each Axis</xsd:documentation>\n",
    "    </xsd:annotation>\n",
    "    <xsd:complexType>\n",
    "        <xsd:sequence>\n",
    "            <xsd:element ref=\"ScaleType\"/>\n",
    "            <xsd:element ref=\"AxisName\"/>\n",
    "            <xsd:element ref=\"MinScaleValue\"/>\n",
    "            <xsd:element ref=\"MaxScaleValue\"/>\n",
    "            <xsd:element ref=\"Increment\"/>\n",
    "        </xsd:sequence>\n",
    "        <xsd:attribute name=\"id\" type=\"xsd:string\"/>\n",
    "    </xsd:complexType>\n",
    "</xsd:element>\n",
    "```\n",
    "\n",
    "## Unused children of ContentClassification\n",
    "\n",
    "The following child elements of `ContentClassification` were found to not be in use:\n",
    "\n",
    "* `Extension`\n",
    "* `ContentClassificationKey`\n",
    "* `EffDate`\n",
    "* `TableURL`\n",
    "\n",
    "## Wrapping up\n",
    "\n",
    "Basically the approach is to just look at the structure of all the XTbML files and see what tags have what children, and if the children always appear exactly once. That is how the object-oriented API is designed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ".\n",
      "{'ContentClassification': CounterStats(exists=3012, counts=3012, uniqueCounts={1}),\n",
      " 'Table': CounterStats(exists=3012, counts=4483, uniqueCounts={1, 2, 3, 4, 5, 6, 7, 44, 55, 28, 29})}\n",
      "./ContentClassification\n",
      "{'Comments': CounterStats(exists=3012, counts=3012, uniqueCounts={1}),\n",
      " 'ContentType': CounterStats(exists=3012, counts=3012, uniqueCounts={1}),\n",
      " 'KeyWord': CounterStats(exists=3012, counts=9086, uniqueCounts={2, 3, 4, 5}),\n",
      " 'ProviderDomain': CounterStats(exists=3012, counts=3012, uniqueCounts={1}),\n",
      " 'ProviderName': CounterStats(exists=3012, counts=3012, uniqueCounts={1}),\n",
      " 'TableDescription': CounterStats(exists=3012, counts=3012, uniqueCounts={1}),\n",
      " 'TableIdentity': CounterStats(exists=3012, counts=3012, uniqueCounts={1}),\n",
      " 'TableName': CounterStats(exists=3012, counts=3012, uniqueCounts={1}),\n",
      " 'TableReference': CounterStats(exists=3012, counts=3012, uniqueCounts={1})}\n",
      "./Table\n",
      "{'MetaData': CounterStats(exists=4483, counts=4483, uniqueCounts={1}),\n",
      " 'Values': CounterStats(exists=4483, counts=4483, uniqueCounts={1})}\n",
      "./Table/MetaData\n",
      "{'AxisDef': CounterStats(exists=4483, counts=5364, uniqueCounts={1, 2}),\n",
      " 'DataType': CounterStats(exists=4483, counts=4483, uniqueCounts={1}),\n",
      " 'Nation': CounterStats(exists=4483, counts=4483, uniqueCounts={1}),\n",
      " 'ScalingFactor': CounterStats(exists=4483, counts=4483, uniqueCounts={1}),\n",
      " 'TableDescription': CounterStats(exists=4483, counts=4483, uniqueCounts={1})}\n",
      "./Table/MetaData/AxisDef\n",
      "{'AxisName': CounterStats(exists=5364, counts=5364, uniqueCounts={1}),\n",
      " 'Increment': CounterStats(exists=5364, counts=5364, uniqueCounts={1}),\n",
      " 'MaxScaleValue': CounterStats(exists=5364, counts=5364, uniqueCounts={1}),\n",
      " 'MinScaleValue': CounterStats(exists=5364, counts=5364, uniqueCounts={1}),\n",
      " 'ScaleType': CounterStats(exists=5364, counts=5364, uniqueCounts={1})}\n",
      "./Table/Values/Axis/Axis\n",
      "{'Y': CounterStats(exists=49218, counts=1487580, uniqueCounts={1, 2, 3, 5, 8, 9, 10, 11, 12, 14, 15, 25, 31, 43, 46, 48, 80, 81, 82, 83, 84, 85, 86, 106, 108, 120})}\n"
     ]
    }
   ],
   "source": [
    "childrenStatsPrint(\".\")\n",
    "childrenStatsPrint(\"./ContentClassification\")\n",
    "childrenStatsPrint(\"./Table\")\n",
    "childrenStatsPrint(\"./Table/MetaData\")\n",
    "childrenStatsPrint(\"./Table/MetaData/AxisDef\")\n",
    "childrenStatsPrint(\"./Table/Values/Axis/Axis\")"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "ca30357621e784c1753d2fc18f21cac83bb234d084e93385854d0a3052d917e8"
  },
  "kernelspec": {
   "display_name": "Python 3.8.6 64-bit ('pymort-r50VwxHF-py3.8': poetry)",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.6"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}