component_check_relevant_all_registry.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "## Check the MVP status of the Data registry publications\n",
        "\n",
        "Use this notebook to check which publications in the Data Registry pass the MVP Relevant and Active criteria, for example, for updating the MEL1 tracker upon OCP Rapid Reflection meetings."
      ],
      "metadata": {
        "collapsed": false
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": [
        "# @title Get all the publications from the registry { display-mode: \"form\" }\n",
        "\n",
        "publications = get_publications()"
      ],
      "metadata": {
        "collapsed": false
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Check for non-frozen publications whose latest data has not been updated in the previous four calendar quarters\n",
        "\n",
        "From the list, check also the \"last_retrieved\" and \"update_frequency\" columns. If the data is not being retrieved, check the publication log in the Data Registry to check if there is a problem with either a job or the source data itself."
      ],
      "metadata": {
        "collapsed": false
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": [
        "non_frozen_publications = list(filter(lambda item: not item[\"frozen\"] and item[\"date_to\"], publications))\n",
        "today = datetime.now(tz=timezone.utc)\n",
        "past_year = today - relativedelta(years=1)\n",
        "lapsed_publications = list(\n",
        "    filter(\n",
        "        lambda item: datetime.strptime(item[\"date_to\"], \"%Y-%m-%d\").astimezone(timezone.utc) < past_year,\n",
        "        non_frozen_publications,\n",
        "    )\n",
        ")\n",
        "lapsed_publications_table = pd.DataFrame(lapsed_publications)\n",
        "lapsed_publications_table"
      ],
      "metadata": {
        "collapsed": false
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Check non-relevant publications\n",
        "Check which active publications pass and not pass the \"Relevant\" criterion."
      ],
      "metadata": {
        "collapsed": false
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": [
        "results = []\n",
        "active_publications = [item for item in non_frozen_publications if item not in lapsed_publications]\n",
        "for publication in active_publications:\n",
        "    year = publication[\"date_to\"][:4]\n",
        "    if int(year) > today.year:\n",
        "        year = today.year\n",
        "    file_name = download_file(publication, year)\n",
        "    field_table = cardinal_calculate_coverage(file_name)\n",
        "    fields_list = field_table.iloc[:, 0].tolist()\n",
        "    relevant, relevant_table = is_relevant(fields_list)\n",
        "    relevant_table[\"publisher\"] = publication[\"label\"]\n",
        "    relevant_table[\"relevant\"] = relevant\n",
        "    results.append(relevant_table)"
      ],
      "metadata": {
        "collapsed": false
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Filter the non-relevant ones"
      ],
      "metadata": {
        "collapsed": false
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": [
        "result = pd.concat(results)\n",
        "not_relevant_publishers = result[~result[\"relevant\"]]\n",
        "non_relevant_rules = (\n",
        "    not_relevant_publishers[not_relevant_publishers[\"possible_to_calculate\"] == \"No\"]\n",
        "    .groupby(\"publisher\")\n",
        "    .apply(lambda x: \", \".join(x[\"rule\"].astype(str) + \": \" + x[\"missing_fields\"].astype(str)))\n",
        "    .reset_index()\n",
        "    .rename(columns={0: \"failed rules\"})\n",
        ")"
      ],
      "metadata": {
        "collapsed": false
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Check the results"
      ],
      "metadata": {
        "collapsed": false
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": [
        "non_relevant_rules"
      ],
      "metadata": {
        "collapsed": false
      }
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 2
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython2",
      "version": "2.7.6"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}