images-to-voice-with-ai.ipynb

{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"source":"<a href=\"https://www.kaggle.com/code/sharanharsoor/images-to-voice-with-ai?scriptVersionId=143440146\" target=\"_blank\"><img align=\"left\" alt=\"Kaggle\" title=\"Open in Kaggle\" src=\"https://kaggle.com/static/images/open-in-kaggle.svg\"></a>","metadata":{},"cell_type":"markdown"},{"cell_type":"markdown","source":"# Introduction: \n\nIn this evolving era of AI, we witness an array of tools lending a helping hand to humanity in various capacities. Within this article, we'll explore the process of transforming an image into speech. The constructed model comprehends the contents of the image, extracting pertinent details and converting them into text, which is then transformed into audible speech. This technology holds great potential, particularly in aiding the visually impaired - offering them a virtual 'eye'. Moreover, it can also be employed while driving, enabling your device to verbally convey the surroundings for enhanced safety.\nAt a higher level, the process of generating captions from images is facilitated by encoder-decoder models, specifically those integrating convolutional neural networks (CNN) and recurrent neural networks (RNN). The encoder segment applies various convolutions, max pooling, and fully connected layers to process the input image, ultimately producing a feature vector. <br>\n\n![image.png](attachment:2cb6964a-a00d-419f-9d74-63ffb5f8b63b.png) <br>","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"},"attachments":{"2cb6964a-a00d-419f-9d74-63ffb5f8b63b.png":{"image/png":"iVBORw0KGgoAAAANSUhEUgAABRkAAACpCAYAAACvdDpUAAAEDmlDQ1BrQ0dDb2xvclNwYWNlR2VuZXJpY1JHQgAAOI2NVV1oHFUUPpu5syskzoPUpqaSDv41lLRsUtGE2uj+ZbNt3CyTbLRBkMns3Z1pJjPj/KRpKT4UQRDBqOCT4P9bwSchaqvtiy2itFCiBIMo+ND6R6HSFwnruTOzu5O4a73L3PnmnO9+595z7t4LkLgsW5beJQIsGq4t5dPis8fmxMQ6dMF90A190C0rjpUqlSYBG+PCv9rt7yDG3tf2t/f/Z+uuUEcBiN2F2Kw4yiLiZQD+FcWyXYAEQfvICddi+AnEO2ycIOISw7UAVxieD/Cyz5mRMohfRSwoqoz+xNuIB+cj9loEB3Pw2448NaitKSLLRck2q5pOI9O9g/t/tkXda8Tbg0+PszB9FN8DuPaXKnKW4YcQn1Xk3HSIry5ps8UQ/2W5aQnxIwBdu7yFcgrxPsRjVXu8HOh0qao30cArp9SZZxDfg3h1wTzKxu5E/LUxX5wKdX5SnAzmDx4A4OIqLbB69yMesE1pKojLjVdoNsfyiPi45hZmAn3uLWdpOtfQOaVmikEs7ovj8hFWpz7EV6mel0L9Xy23FMYlPYZenAx0yDB1/PX6dledmQjikjkXCxqMJS9WtfFCyH9XtSekEF+2dH+P4tzITduTygGfv58a5VCTH5PtXD7EFZiNyUDBhHnsFTBgE0SQIA9pfFtgo6cKGuhooeilaKH41eDs38Ip+f4At1Rq/sjr6NEwQqb/I/DQqsLvaFUjvAx+eWirddAJZnAj1DFJL0mSg/gcIpPkMBkhoyCSJ8lTZIxk0TpKDjXHliJzZPO50dR5ASNSnzeLvIvod0HG/mdkmOC0z8VKnzcQ2M/Yz2vKldduXjp9bleLu0ZWn7vWc+l0JGcaai10yNrUnXLP/8Jf59ewX+c3Wgz+B34Df+vbVrc16zTMVgp9um9bxEfzPU5kPqUtVWxhs6OiWTVW+gIfywB9uXi7CGcGW/zk98k/kmvJ95IfJn/j3uQ+4c5zn3Kfcd+AyF3gLnJfcl9xH3OfR2rUee80a+6vo7EK5mmXUdyfQlrYLTwoZIU9wsPCZEtP6BWGhAlhL3p2N6sTjRdduwbHsG9kq32sgBepc+xurLPW4T9URpYGJ3ym4+8zA05u44QjST8ZIoVtu3qE7fWmdn5LPdqvgcZz8Ww8BWJ8X3w0PhQ/wnCDGd+LvlHs8dRy6bLLDuKMaZ20tZrqisPJ5ONiCq8yKhYM5cCgKOu66Lsc0aYOtZdo5QCwezI4wm9J/v0X23mlZXOfBjj8Jzv3WrY5D+CsA9D7aMs2gGfjve8ArD6mePZSeCfEYt8CONWDw8FXTxrPqx/r9Vt4biXeANh8vV7/+/16ffMD1N8AuKD/A/8leAvFY9bLAAAAOGVYSWZNTQAqAAAACAABh2kABAAAAAEAAAAaAAAAAAACoAIABAAAAAEAAAUZoAMABAAAAAEAAACpAAAAAI8G3JcAAD8ZSURBVHgB7d0LlBXVvefxHYN31kUDPQtNlKtAHtd49YLtW0d5mGiIyNOr8YUPXHKJA5FoJFwGRsChRwkmBgNRo8sXPjA6AQE1RhN56KiJj7ZZeo15NWDUXGVNg8Jda+RO5vx2+z/uPl3n9Dl9XlWnvtuF3X1O1a5dn121a9e/dlV96q+Z5EgIIIAAAggggAACCCCAAAIIIIAAAggggEAvBfbq5XzMhgACCCCAAAIIIIAAAggggAACCCCAAAIIeAGCjGwICCCAAAIIIIAAAggggAACCCCAAAIIIFCWAEHGsviYGQEEEEAAAQQQQAABBBBAAAEEEEAAAQQIMrINIIAAAggggAACCCCAAAIIIIAAAggggEBZAgQZy+JjZgQQQAABBBBAAAEEEEAAAQQQQAABBBAgyMg2gAACCCCAAAIIIIAAAggggAACCCCAAAJlCRBkLIuPmRFAAAEEEEAAAQQQQAABBBBAAAEEEECAICPbAAIIIIAAAggggAACCCCAAAIIIIAAAgiUJUCQsSw+ZkYAAQQQQAABBBBAAAEEEEAAAQQQQAABgoxsAwgggAACCCCAAAIIIIAAAggggAACCCBQlgBBxrL4mBkBBBBAAAEEEEAAAQQQQAABBBBAAAEECDKyDSCAAAIIIIAAAggggAACCCCAAAIIIIBAWQIEGcviY2YEEEAAAQQQQAABBBBAAAEEEEAAAQQQIMjINoAAAggggAACCCCAAAIIIIAAAggggAACZQkQZCyLj5kRQAABBBBAAAEEEEAAAQQQQAABBBBAgCAj2wACCCCAAAIIIIAAAggggAACCCCAAAIIlCVAkLEsPmZGAAEEEEAAAQQQQAABBBBAAAEEEEAAAYKMbAMIIIAAAggggAACCCCAAAIIIIAAAgggUJYAQcay+JgZAQQQQAABBBBAAAEEEEAAAQQQQAABBAgysg0ggAACCCCAAAIIIIAAAggggAACCCCAQFkCBBnL4mNmBBBAAAEEEEAAAQQQQAABBBBAAAEEECDIyDaAAAIIIIAAAggggAACCCCAAAIIIIAAAmUJEGQsi4+ZEUAAAQQQQAABBBBAAAEEEEAAAQQQQIAgI9sAAggggAACCCCAAAIIIIAAAggggAACCJQl0KesuYuYedeu3e7Z515wO3d+UMTUTIJAaQLHHnOkGzzo4NJmYuqGEaB9aZiqjOWK0L7EslpqVijal5pRp3JBtC+prHZWGgEEEOgmQH+jGwkfVFCgHv2NT/01kyq4DtmsPtqzxy1ctMQ9snat69+vyTU3D81+xy8IVEKgo2OHe7WtzWc1csRwd+OSFrd3n6rHzStRdPIoU4D2pUxAZu9RgPalR6KGnYD2pWGrNjYrRvsSm6qgIAgggEDdBOhv1I0+NQuuV3+jKkFG7TBTps5w7e1b3a3Lf+AOP+zQ1FQkK1pbAW1rL73c6pbdfLtf8J23LSPQWNsqqPnSaF9qTp7aBdK+pK/qaV/SV+f1WmPal3rJs1wEEECg/gL0N+pfB2kpQT36G1UJMs64crZT1FQjy/bfb0Ba6o/1rKPAe+9vd5POvtCNGjncLVowp44lYdHVFqB9qbYw+ecK0L7kijTu37QvjVu3cV0z2pe41gzlQgABBKonQH+jerbkHC1Qy/5GVV78smHjJjfj8ssIMEbXL59WQUDBbI2a1e35eq4FqXEFaF8at27juma0L3GtmcqXi/al8qbkWFiA9qWwD98igAACjShAf6MRazXe61TL/kbFg4xbtm7zukcf1RxvZUrXcAK6LV/P/9SLhkiNKUD70pj1moS1on1JQi2VV0bal/L8mLv3ArQvvbdjTgQQQCBpAvQ3klZjjVPeWvU3Kh5k/M2Lr7gjhg3juXiNsy0mak30gqGt2/6cqDJT2OIFaF+Kt2LKygvQvlTeNE450r7EqTbSVxbal/TVOWuMAALpFKC/kc56j8ta16K/UfEgo/CamvrHxZBypEygedhQ19q2OWVrna7VpX1JV33HaW1pX+JUG9UpC+1LdVzJtWcB2peejZgCAQQQaBQB+huNUpPJW49a9DeqEmRMHjUlbhSBpv79GmVVWA8EEIiZAO1LzCqE4iDQQAK0Lw1UmawKAggggAACMRWoRX+DIGNMK59iIYAAAggggAACCCCAAAIIIIAAAgggkBQBgoxJqSnKiQACCCCAAAIIIIAAAggggAACCCCAQEwFCDLGtGIoFgIIIIAAAggggAACCCCAAAIIIIAAAkkRIMiYlJqinAgggAACCCCAAAIIIIAAAggggAACCMRUgCBjTCuGYiGAAAIIIIAAAggggAACCCCAAAIIIJAUAYKMSakpyokAAggggAACCCCAAAIIIIAAAggggEBMBQgyxrRiKBYCCCCAAAIIIIAAAggggAACCCCAAAJJESDImJSaopwIIIAAAggggAACCCCAAAIIIIAAAgjEVIAgY0wrhmIhgAACCCCAAAIIIIAAAggggAACCCCQFAGCjEmpKcqJAAIIIIAAAggggAACCCCAAAIIIIBATAUIMsa0YigWAggggAACCCCAAAIIIIAAAggggAACSRHok5SCUk4EEEAAAQQQQAABBBBAAIF4CXy0Z497++13uhTqNy++0uXvqD86duz0Hzf17xf1dZfPjj3myC5/Dxx4oNu7D6eyXVD4AwEEEIiBAC1zDCqBIiCAAAIIIIAAAggggAACcRLYtWu3e3/7dvfOu39xb731ti/a+k3P+p/t7Vvclq1b8xb3iGHDXFNT/7zfl/JFR8cOt7Dl+ryzDB40yA0ZMth/P2r4Sf7nQQcNdAce8Dm334ABbp99+uadly8QQAABBCorQJCxsp7khgACCCCAAAIIIIAAAggkQsACiRp5qJGFrW2bXW4AsX+/JtfcPNSvT/OwoU4jD/ud8XX35UO+lF3HwYMOzv5ei1+2bN2WXcxv3/y927nzA19+C4K2tm52O3Z2ZKexQKSVXyMjCUBmefgFAQQQqJgAQcaKUZIRAggggAACCCCAAAIIIBBPAQXmFJBre+31TCBxq9uwcVO2oBaE00hACyD27dvX7b/fgOw0cfolDGqGv+eW8b33t7vdu3f79VYgUkFIBVHDkZEjRwzPjIQc5IYdfpgPnBbKLzd//kYAAQQQ6CpAkLGrB38hgAACCCCAAAIIIIAAAokWUHDtD3/8k3vmuRecRvW92tbm1ycMJk4+72z3xS98PraBxEpUQGeQdICzwOFZZ47PZmtGuhVcwcf16zdlbwHX7d4avXnyicc3vFEWhF8QQACBCggQZKwAIlkggAACCCCAAAIIIIAAAvUSUMDsldY2t/GZ5936DZv8rcJ2m/PEcWNcy8K5jpeldK0dBSB9EPI45yz4aC+x0e3jCjyuXv1o1nLUyOFuxMknuCObhzV0YLarEn8hgAACpQkQZCzNi6kRQAABBBBAAAEEEEAAgboKKBj20sutfqRiGAibOPEMd8Piaxl918va0RurNepR/yzwaCMeNSr02pYl2aCjrDXS8eijmnnTdS+9mQ0BBBpPgCBj49Upa4QAAggggAACCCCAAAINJmCjFe+578Hs7c8Txo1z18ydxei6Kta1jXg84bhj3NUzpzurB40anXr5FX7Jur36ogvOoR6qWA9kjQACyRAgyJiMeqKUCCCAAAIIIIAAAgggkDIBBbQeWfu4W71mnX9eoJ6pOHH8WDdn1rfd4YcdmjKNeKyugo5fO/UU/2/RgjnutdffcM+98KK7afmtXepowrjTua06HlVGKRBAoIYCBBlriM2iEEAAAQQQQAABBBBAAIFCArt27XbPZm7NtRGLFlgkaFVIrX7fKdirf5dNmexHOVpQeOmyHzsb4XhS5rbqffbpW79CsmQEEECgRgIEGWsEzWIQQAABBBBAAAEEEEAAgXwCGhH3wE9XZUYurnV6acslF53vblzSwmi4fGAx/FyjHBVsDAOO9hxH3dp+3jcmMQI1hvVGkRBAoHICBBkrZ0lOCCCAAAIIIIAAAggggEDRAnqByyNrHnN3rbjf32qrQNTKFXcQiCpaML4ThgFHCyCfe+GlmZfKDHKXXHi+mzB+DC+MiW/1UTIEEOilAEHGXsIxGwIIIIAAAggggAACCCDQGwHdEn3z7Xe6u++5z49a/Pa3vulOH30qt9T2BjMB8+h2aj2/cc6sme7xJ55yP/zRLW5hy/Xu4osucJdfNoV6T0AdUkQEEChOgCBjcU5MhQACCCCAAAIIIIAAAgiUJbBl6zZ32x33+luiNaLttptvcnprMSkdAnou41lnjvf/nv/1i27RdTf4QLNGsE69dHJmlOPB6YBgLRFAoGEF9mrYNWPFEEAAAQQQQAABBBBAAIEYCGjk4rwF17mxk85xHTs6/C3R61atJMAYg7qpVxEUXNY2oNvjtU1o29A2om2FhAACCCRVgJGMSa05yo0AAggggAACCCCAAAKxFghvi9bIxXWrHmS0WqxrrPaF063Uy25cnHkm5zY3feYsd8KIU7mNuvbVwBIRQKBCAoxkrBAk2SCAAAIIIIAAAggggAACJvDwz9a40WPPcuvXb/K3RWvUGrfDmg4/cwW0bWgb0S302ma07WgbIiGAAAJJEmAkY5Jqi7IigAACCCCAAAIIIIBArAVsRFpHx06nF7rwFuFYV1fsCqfbqFc9lHluZ+at43pBjN48vnzpEgLUsaspCoQAAlECjGSMUuEzBBBAAAEEEEAAAQQQQKAEgY/27HG333mvf7bekCGD3RPrHvYv+Ni7D+M6SmBk0oyAthm9IEbbkLYlPa9R25a2MRICCCAQZwGOeHGuHcqGAAIIIIAAAggggAACsRd47/3tbsrU6U6jF/UiDz1nj4RAuQJ6G7We1/ja62+4adOvcqvXrHN33rbc7b/fgHKzZn4EEECgKgKMZKwKK5kigAACCCCAAAIIIIBAGgR+8dTT7iujx/kRZxp5RoAxDbVe23XUNmWjGrWtaZsjIYAAAnEUYCRjHGuFMiGAAAIIIIAAAggggEDsBW5Yutzdfc99bv7cf/G3t8a+wBQwsQI2qlEvg/nO7Ln+DdRXz5ye2PWh4Agg0JgCBBkbs15ZKwQQQAABBBBAAAEEEKiSgJ6Nt3DRErd+wyZuj66SMdlGC+hZjf9w6CH+9umO/7PTzZ83yz/DMXpqPkUAAQRqK8Dt0rX1ZmkIIIAAAggggAACCCCQYAEFGKdMneEDjKseWsHt0Qmuy6QWXbdPa9tTkFvbIi+ESWpNUm4EGk+AIGPj1SlrhAACCCCAAAIIIIAAAlUQsABje/tWH+ThBRxVQCbLogS07SnQqG2RQGNRZEyEAAI1ECDIWANkFoEAAggggAACCCCAAALJFrAAo9ZCwR0CjMmuz0YovQUatS4EGhuhRlkHBJIvQJAx+XXIGiCAAAIIIIAAAggggECVBfQMRo0au3XZDwgwVtma7IsXUKBR26S2TW2jJAQQQKCeAgQZ66nPshFAAAEEEEAAAQQQQCD2Anqj7yNr17r77r7V6S2/JATiJKBtUtumtlFtqyQEEECgXgIEGeslz3IRQAABBBBAAAEEEEAg9gKvvf6GW9hyvbvt5pvc4EEHx768FDCdAto2tY1qW9U2S0IAAQTqIUCQsR7qLBMBBBBAAAEEEEAAAQRiL6DnME6bfpW7+KIL3AnHHRP78lLA4gU04k//du3aXfxMMZ9S26i2VW2zvHE65pVF8RBoUIE+DbperBYCCCCAQBUF7Fac00efym1jVXQmawTSKKAT/sefeModdNBAgjpp3ABits6PrHnMl2jm9Gk1K9ntd97rWts2l7S8ZTcuLml6JnZ+xJ8cjj3myIbqy2hbXb36Uadt96wzx1PVCKRKYMvWbe43L77SpQ8R9VmqUGq8sgQZewCfceVsP8Xk886mo9uDFV8jgIBzz//6RXfvAw8VTTF/7ncT+fB43Yqj1Ggd86IrjgkRqLHAe+9vz5wQf6/bUpuHDXX/ePihbujhhzXMSfL727Wu17uRI4bT9+pW43xQSwEFvLUtfn9xi9u7T+1OmxRg3LBxUy1XteCyfvHU027Noz93SQ1kWt8st881f+6/+PXeb8CAguuftC+1rV4zd5b7zuy5jovBSau9ximvAntLblzmV6iWsRQFGHP7EFGfNY50/NakdkfL+K17USWyA/yo4Sc5d1xRszARAgikWOCtt94u6cRg95UzMlqN1blNcfWz6ghUTWD37t2RbYv1U7Tg/v2a3CUXne8umzK5auUgYwTSJHDz7XdmnsE4yH3t1FNqutoKhnX2Dz5Z7EOr1ri777nPHTFsmGtZOPeTL2rwmwKMYVtTg0VWdBHPPPeCL39un6uRR/lpm9W2q2346pnTK+pJZggUI6A2y9qN9vYtbt2qlcXMVpVpdGeELlzqwiyp+gIEGatvzBIQQCCFAsWeBAwceGAKdVhlBBAoR2Ddqgezs3/44S73+JO/dOvXb3Jbtm51S5f92K3f+Iy787ZlNR15lS0QvyDQIAJ6np2CeitX3FHzNdp/P1187HoBcsjBnS+caWrqX/DlMxo99M67f/Fl/uIXPt/tbgmNztRoYaWol9iE32uEn6a1QIHyVlLfpZSRnRqJrQslPc2nZW9+7XX3mX33dYcc8qXIZVj5VDa9UVn19Oabv3d/fvsdd2TzsC7rq+/eznze2rrZl/u3menC8hdaH1uORkD9w6GHuM9+dv8uefuMMv/LXTcrzwcfflj3kdiLWxa4cy+81On26VLqy9aNnwj0VkD7j9pPJV0AVf9E+1tUm9PbZZQyn55VyjN1SxErb1qCjOX5Zee2A1HUjqODT9+//duibmPSzte3b9/Ig1h2YR//YgfOqGXmTmt/24Gw2GXYfPxEAIHSBHo6CSgmN9vHe+qUF5NXOI3lq8+skx5+H/W72qZip9X81ibq91LaKM33N//pb+gMC46EQB6B3H3q8MMO9SNVdEvg1MuvcK+2tblJZ092D957R96+h51cl7Jf5ylOt4+tr6EvcsvabeLMB9ZeFDOtzd/b8qtsnQEcy4mfCEQLvPRyqz851v6VhKR94oKLp7kdOzu6FFcn+PfdfWt2X9z97//uxk46x0+jAGq4fuofnDP5Uh8Q0MtDFNi0x6NoBptPFzp62l+1X0+bcZVvj6xAGll3523Lu+2Duh372pYl3co+Ydw4N3PGP3eZXs9rVZlmzvivrmPnjmwgw5ahi7x2kUUBRiuzvtftw0pWfvvO/vZfZv6nZ2Lqgk1uUt43LmnpUh49xkJBWOVho01tvnqPLlfdqgzalgmwWK3wsxYC2k+VtA83/ed+fj/V/lGNUbW96UPkGvTm3Cg3D/7+RIAg4ycWRf+mg7gOShpyOytzq+P0mbP8wdgy0PM9NPxeL0b44Y9uyR4wbfrcg7I9J8SuEiofHRAmTjzDXXzBuV0OZPpOHeQpU6d3WaZ24PnzZrmjjh+hSbIHT/9H5n86eNuB1T7TT3UguLoVivB7NQVee/2NLp3Zai4raXlbu6Jyv/zCRnflrLnZkQP6TPv41EsnR3bqb1i6vEsnO18nXm2H8lUAIkxqb25d/oPIulH7tOi6G7LtjeUdzh/+rjrWGw1zT3LU/qljHl5J1zNv1e6pzWzfti27Drmd/TB/fkcgnwDti/MnkbfdfJMPNGrUQO5tcupEL1y0xD2ydm0XxkJtQFSfI1/fIV+gIGr/VwF0YqAy2mgHfaa27rxvTNKv3ZLKv3T5rV2mt4n0zLzcW1qHHv1f/NfPb3wqG+xQWZL6XDlbV372LFCJ9kDPV1ZfPAkp7EMoEDZx3Bhf7LtW3O+P3zpvsWOrguw67ipQp+P100+uyR6b716x0k+vY73ODxSk0/5u+2ixzzBUuzHp7At9X0Dtix7j0L5lm297vjJ6nH/Gpe2vagfsHEXLOvnE451ub/YvLvm4rVq0YE63arjrnvu75L96zTpfdvVxpkyd4e698xZ/cdTWVRkofwVOdXElXwr7VFZ2e0Zm5wWcC7uYWT4W4JXdJRee71avfcz3txSsPPH4YyL7WDZvNX9qG9a2TJCxMsratosdPFSZJSYzF7U9SmPHnOY0olptiP5FxR3C9mvzS/+72wrb+YK1YTZBVB9CFx+a+vezSbI/FZex5zSGfQDlMXvewi7nXJpJ+/4Ni69lv8kKlvbLXqVNztShgIbe64CijrwOWtoYlbQB6wqYfirpOyWdTGv6MKnDfPXsa/x36hToQKgOtk7QtSMuXfaTcHLfIddBW8tUUmdZ/3TCoANqVNJOZQdvLUM7n5VJy8g3X1RefIZAOQK6ZUMnfXP++//wL0jR9k/qLqB9Uu2F9m21B0rax3VBI0zy04FX+7GSptU+rvZBnXh7A7S+00FUbYcFGJW3tQNqb1Q3OikL08mnjMkGK5SvplfeyicqaX7lo/z8CUqmrVF7o6T10ciqqDpfv+nZ7DpoWo20JiFQqgDtS6eYTiSt3dBJuiXte6ecNj4bYNQ06nNoX7U2QBcVwqR92voc6uOoDdD0anO0P+tky5L1NZSXkk2r37X/a9nh/q+8Txhxanbft+nV1s2eu0CzuY6OHf6n/U9to7V3asNUfv1UUj9Hfa+opECmtX1N/Tv7alHT8VnjCJTbHuiYqe329NO+mggUe7mCjtUaxafBDvq36qF7s+2BTaMV0nfad7S/6sKDkvZnG723fOkSH3jUwIizJ33ydmLLV7cpF0oaDKG8tV8riKnnxCpQqH1WSfurtQc6wVdSeTTKSW2Yfj6x7mH/udoE1Uduys1fz3uz/LW/a31UTpXZktZFf+crv5ZjbYz6L1Z2BSU6Axyd7aW9cdzy1U+VRyNDVQ4tQ0FOa5/0SIt6JW3D2pajDOtVpiQvd8PGZ/2x65/OvdgP4sG1e23q+G6xiqOPavYDptQ2KT2debRLb5MeXWBJ/ZWwD6H9Vf0TtWEa5FVMUhs0euxZfv/Q9JpfbZaS9mfdGRKeS/kv+F9RAgQZi2KKnkgbX3PzUKcr5DoYPvP0Y/4EX1NrA7eDk77TyCQlzRN2ynVFXp9po9bBSAclHYR/9UTnKIPcA+sDP/2Zn16dfR3IdNDTP41caG/vDDxqOTZaUjuP7WjaabQMHehVJh0slY8OxGoMSAjUSmDdY4/7hlsjb2f9t/kEHHPgOzo6/P6pfTtsD3TADtsPnRio46gDt/ZnTat9XO2Bki50WOdHJ9nW1mha5a12QO2XBSTsxF7z6uCt6dVGKD/la9Or3YtKNr/KoxMbtTX6p/ZMbZzKH9W50DqofdJ0uoLJrYxRunxWrADti8s82Pxwz6V92JJOiu1v7dNqL9Tn0AmxXQzQqGVLajvsooFO3HWy3dl3WJndn+1t12FfQ+2J9YuUt5aldkTLDk/Mr1vyQ78otRe509vJiQUGNaHaJPtbeaoNU/n1U6MYldT3ssCF/+Dj/ylooD6T+mJab1J6BHrbHtgzC8NbieOspuOokl4IE94xoN91F4RSbqBp8aL5ft/UuYbOA3Sng5L2dzuP8B+U+D/tg7YP646ssDwTxo/x+WsftqR9WMd/lSdMCgSq7VB6NjOyMSpdftmULvmrTbCkYFCpyW7x1Hzqv4Rll0nzEUf4LHVxNDepn5O7vfgXh2Ym1DNz65WsTLZN16scjbbcN3/3O/fdOdcQcIyo2Ad+usp/qr697UMXXdD5iIabMrGPSiTrr1ifQ/ur9Tmsr9PTcqxfZOc6mt/OdezcyOIoPeXF910FCDJ29Sj5r8unTulyNSw8+b74wk8OrNrBLIIfHvR08NUJv56bFCadZEcdWHUrgJKGvttBQ3/rqt+okZ1X8/W3JZ3Qa0dTXtppwqSDpd0GYo1B+D2/I1ALgZ8/8aT75vQr/a3+/5z5qdvtLDBWi+VXaxnqzGvUZqF/+a6OTRw/tksHX+2BOq9KYfvR+uqr/rM5s77dZXq1BzpJ0D89e0nJrszrFp7w5EGdeDsB0UmBbllQ0q01SmrTlJ8lTT/5vLPtz+xPzWcnFfYsJPtS5ddylfJ1LnSiQHDRxPhZKYFGbV968jn2mCOzk9g+rVv3lBRQDPdpfab+ilLYBrRv6bxwqf6DTtztREHTaZST2hc7gba+hr7To1vCUUJh/8ROzBWEsICh2q/c6a2/pPwsWZukjn9u+XXbpfWZoi5kaDSR+kzhOli+/EyPQCntgV6olJRk+7jK++QvN/iRN+pf2L9w9E8YaNJ+p0elKOmCgvZJ7XthoM5/WeL/9BIWS7nHde2Dyl/7cLg/ajqVTX1AK3c4Mnnnzg8syy4/w7bDvohqP+y7Yn9anyt3eruAYy+SCb8fNar7eZheGKNk/aNw+lr/nqRtutY2vV3ef/y///Cz5gYctQ2HgwJ6m38S59M5nC5cKIWjoE/5eP8I+xm9Xb/wnEPnMGE7oLYl3/6buzwLIOpxDmG/QvnpWbBKiqOEbWxuHvwdLdAn+mM+LVZAbz4Lk735TQe48OCpafQiiNzUefAd4IMq2oDDjkBTUz+/YYcHVjtI2XLC/EacfEJ2p1bDprxt3iFDNIqoM3gQztPUr7NMHTs+Ge0Qfs/vCNRCwA7Szz3/gvt1ZrSK/j7k7//eTbvsEndS5tk84cGjFuWpxDJ0whtedIjK86CDBkZ97J/dk/vFkCGDu3RSdRC39kBvPMxN4UlCOLLHOrzh9OEbrnM7oc3Duo9aPPCAz4Wz+9/tjY1abz3DKTf16/cZ/5GVOfxe7WUS6zhcB36Pr0Ajti+laNuzxyyop2cVRfUH1CnX/ql+iC5E/Osbb/rFRLVj+j68WGF9De3/uX0fZaITc5102GirsI3Yd999uq2OlmnltS/t1mk9QD6q/JpH+VtZbD79jLowEn7P7+kRKLY90PZvt7omScdudy62zAq+az1t39RIyHKTtR3F+ml/nju/pds+31M58uVv51utba+VHDC1CyE2YjG3DNaHihopFXVuFtW+5eZZi79lpXoJB6jUYrlpWoa1LQo4Lrpuib+b6MADDnDfOOvMzB1Dp6fmQrqNBlafIuwnqG+gi4TqC1TyBTDWxwm3tdxzpvC78Hfbj//x8O4v9wovkOgcJ1yXMA9+jxYgyBjtUvSnUZ1pzWwHuJ4yUqAg6mGjPc0XjlKwab8cBDx379azSwY4O1iqs25vULPpw59RV+TC75P0u40gi3pwbJLWI61lDQ/SV8+e5/6a+U8Bx/O+8U/ugwSNLNAJr24B6k0qphMYjkYID4RRywtP6IcM7hwRGU6ndkzBAR1srRPa3r7FTxL18OSoA62d2CuPQm1NuFz7vdj20qav50/al3rql7/sRmlfepIIL1gqgB9eaLDnRfeUh/Ufoi405M6rk3mlqICkPrcTc/2em6Lak6iTdQs6alS2jczOzUt/t2/rfkE16sJI1Lz1/oz2pbY1UKg9+PM779a2MBVamm5DLrS9hxcVtUgNStB2Z+m2O+6t2CMFij23sBemqAwaIa1zHJVTfRM9G9oCAVZG+2kXHuzv3J826jD380J/a1DGhswTrvINvvhzxEXUQvnF6bv/+b0f+MAX50fVrxVrW95591237Me3+kd5KOB41pkT3KQJY6tfgDouwUYHKtCnEZ1RScfwqBfARE0b9dk77/4l+3HUIAXdZRG2a9mJg1/CkaZ6MU1U0iAI9T3sHCdqGj6LFiDIGO1Ss0/PmXxpdjSSnlsQDivOfWt1WKjc0Ubhd+Hv9oBzXU2w2xXD7xv5d3urZCOvI+uGQKkCun066oBcSj6FbmdXsPLb3/pmKdk5deqTlmhfklZj6SqvBf1spE94QVQjCQqdfOdexOzY2fXlK1GSGl1YKH3w4YeFvi7pO62T3aYdNWO+EeJR08b1M9qXuNZMPMsVviztM/vuW/SIG1180AtalPQYBb2tWaOMdGeUvfm5N2tsbYiCg1pG2P7k5qcTfQsi6jwovAsj/C53Pv1tFx5yvys2uJk7n/62Cxz58ti67c9+tkrckh21/Gp+9tFH/9dnT/tSTeV0523Pc5eCgnyFAn16tEk57Uwh6Y4dOwt97b/TG8It2eAs+9t+6hn5pN4JEGTsnVtF5tLB024d1MEqfGZieCtk1MJstFH4XThywT63W5Q6OnZ2OXDb9434UycgvR1B1ogecVqnYjo2n97r03lvl853RSxO69jbskSN5imUV3h7gG41KjR/+N2//dt73W7ZUHtjnXw7ObBbDdojRgWFoyitjHY7tPLRg90LnVTYPPbTOvX2d5x/0r7Et3ZoXzrrRn2L9Rs6RyZFjUJUvyA8kc9XozYSIN/Jdjif7cP5pn3rrbf95LoIkZui2i8bRRlOq31PJyy6eFpM+cN5k/I77UvlaqoS7cHvfv+HyhWoijnpbga7G+G5F17sdkusAn06oc99/MzdK1b68xCdg+ilCbplUG9TvbZlSbdprfg9BQ01XThaUs9nDO/OUPv0ldHjfHZ62csrrW2WtR/ZlP0j80v4Xfh5+HtU+2H9md5ccLB5lIf6RrkXZVvbNvvFDxk8OCxGIn7XKNFGbTtrWQE6F+npjgA7l0nb7dL27GS1KRPHjYmsFh3fdSzXM9qLCTKqzckNVoajtQvtp5EF+PjDcN/Odzu0xWmsXSiUH991FeDFL109avrXH/74p+zy9KKEMG1+7fXwz+zv9iDTqBP/qE55eOKvA3tu0mdRn+dOx98IVFNAB2OlE0843n3vumv9m0b/18q7/cEnPAhUswxJy1sudsJuz0O0ddABWbcY6SRLD1FXsmntWUk2rX62f/yCB/1uIyJsFHT41np9rxR1QePI5mGdX2b+Hz703T5UO6OTARICtRZIU/ui/WzS2Rf6iwba5+2FLjK3kTdRfQV9r7fLqrNuyTrVUaOFNFpB7YvaGSWb1k7MLQ/7acu026nDIER425NNb49rsL/109okC6CG3+l3tS9h+XO/528EJFBse6Bb/HNPbOMsaHcQaDSigiC2L6g/oLdGf2f2XDdtxlXZVdD+Ys9vvHFJ59vZ9eIDBbq1H+tRTpa0v1ofInxDvH2f+1MXGTUqUWna9Kuyt0yqTEuX/cR/rhHVCo7ayyD04Usvt/rvNN0NS5f7MttyrQ3xEwT/011fao+UtE6Tp3TeSaH5jj6qOTulnT/pWXCFUvjSCN1tFuatMtk2YS/MK5RXnL5TuQs9tiJOZU1qWaxt0SOe5s2Z5d+Y/otHf+YD+D091iip6xyWW/ut7R96oZsC2lH/7C3yCuDZeUHuwIkwX2sXws/sXEWfRQ18sHKE80T9bu1CW0Tcxcqm+cKgZlQ+fNZdgCBjd5OafTL08MOyy7KTcnUG1Dmw17JrgvDAesX0aX4ePctAb13T9HYwjtqhdPC2HWhhy/eyO7My0Xw6GdEVxfANbn4B/A+BGgl8ffRp7pblN7qXX9jofpL5qataBBaLw7eTCo06CEd56iRAJwna9+0qoU2rZ6VoWrUdSupAz567wP+ukwvrCI0dc5r/TO2KOtaa3tone96Kn+Dj/2k+za903ZIfZjvm+lvz6SRHz2oM2xoLGmgaEgLVEEhD+6L9S/vxvAXX+eO59n0lvTU2HFE84/LL/Ofap8M2QB/qYoTeLjt67FnZ4ER4sj3jytnZ/oP6DtZHsXYlnFbBCeuca1oty0Y42gtYVC5rL66efU22vVCQtHNZnW+29gX++H92Uq/1U5ukvC0pQKr25YQRp2aXbd/xEwETKKU9iMsLO6zsPf3UHQQWINQoK+0LCrgddfwIf+Kv/oANaFCboecgKml0mx339bcCAArQWTuhz7S/2r6uvHWBIexzaJrcdPllU/yFDe2vmkcXJFQme+vsnFkz/SzKWwFHJY2iHDvpXD+dznP0+RPrHvbfqTzKIxwYoXXSnVqaT9+pDbCLIjcsvrZL+7e4ZYHPR/n2VP7lS5d4AwVBwrw1r5ICqOEdIv7DBPwvadt0AkizFy0UWAwHSSi4Fu5XSViXcsv4wE9/5rPQfhmOXs7NV+d4dtHTgv7hZ0/+ckP2HEXH9mU3356bhbe1PHShQdMpqe+hPkSxad6cq/2kq1c/2qVfpPyW3Ng5AEzLSeL+XqxBtabjdulqyRaRr3YoHUB1wFXnXp0DdcR1QNYGrQOiPteBVTvM/Lnf9Vf89J0OoroCqSuWdkLx/cUt/qqfFm0Rfh28w3yUl+2UdiBWGXSbBAmBWgmMHXN6Zts/3V9l1jbaiEn7mjqyPSXt9729vV8nFavXPubbA3Xi71pxv+9wW5tgB0+VIXdaBQr1Bnu7FUAnFWpjLCloYG2NOtY6ACtZ+9Te3vm7//Dj/2n+SZk2TG2LOubqaOi2a1koKb9wZJU9zuHj2fmBQEUE0tC+FGpbtN/pJDm3U6x9Wu2N9ke1F/qnvzVq0NoBBSbDizy5/QebXxWlNkPtiqXcaa39sO81r8pg6fKpU3xZ1KaovQint76RTaufWh+d3Ks9sn+55ddLL3LXO8yD39Mn0Nv2wEbW6GSz0AlzLUU1YljbfNRjENSXUl9CwT8NTtD5hO5E0PR69MHpo0/NBt1067RGFUc9ekD7vwJ0uu1R+WgfV94KmuhZr/ZCFBu9nG/9lY+Cmkszt0SqLOoXaB+/6IJzut2KPX/eLP8cyHvue9BPp/1fz4XUQAktW+c3ax79ebdFqX+hfsfd96106zPrpLZE66u2JbfO9LfaD7s7w8qv6ZXsvEm/qw1Z9dAKP+qyfcsWXybrz0TlbfVheSoPS8rXlmGf1fqnBWBsm6718ht1eQosTrvskm7bc6Oub0/rpZiEUjHvgNBFTx33dSy3F8CMGnGy39csvmHnKGo3wv6BlUOjJRUnUf9FP7WPWl9G+5yde9j0UT/Dcx31iaLOjVoWzo2alc96EGjMs/seVjpOX+tK3pDBB7vVa9b5nUE7yCUXne9PxHVgDQ+IKrc+Cw/aOljqYKxRR+HIyPDqiQ6seu6JblHQbUYWXNQOqI5HeJIQJxvK0ngCK1fc0a3j13hrWf4aaYROeKKfL0drDxYuWuL3bR1cdeKvduP0077axdqm1TOY1N5oWnXINf2okcOd2qLcZaqtsbxtWrU3mlajlXIP4Gp3NOrguiVLs+XRctQ5UOdBAUaVg4RANQTS2r5oH1bAQCe6gw7+u+yJeZSxghAatWgn89qH1e+IajM0v/oP61Y96K/oa1r9s0CBHpEQ7s/W19BdE5rO+hqaXs9mUpAiTJpeQUGNUtC0FoRQW3HaV0dmRzyF8+jZ1WrbNFpa02s5Ptj5cVAiDGKG8/F7+gTKbQ90PFQ/+fEnf9nlWFpPSW3fPW3jdntioXLqDge7yyFqunzLWbRgTtTkeT9T+xA+bz7fhJquUJkKfad+h5ZRzHKipsl3kVf5Fru+hQZqKJ98y8jnUenPtQ1rW87t41V6OWnJb+SIk9zzo5/CM6hw3U2h8wQlXdDoKelRBjp2ax57AYzOEfRcWF3g0LFdSX2Tiy84N3NR9HvdssztQ9j5hi5kfPmQL2Xz6DZjzgf33nmLv8uq2HOjnNn5M4/Ap/6aSXm+69XHdgWt3g1qrwpf55mKDSzkK6auVFkkf92qlfkm87cYNeqBhu0vb7U3xBfUb+FqLKUN0e1SSmGQoFDupeRt+WgZxeZv88T5J9tfnGun/LKlsX5L3UdLmb6UaVV7pU6veXrTLmm+OKY0bn9xrIewTDpx1u38zzz9WPgxv9dZQPuKjcTmfLPnytCt5Bqd2lOAuuecmKJSArT3hSVL7Q9oeqVKnHOUuuzCaxLPb2ux/fFMxhjVfTGBPwUSdet07nNJtBq6UqU0KnN7QaFUzHIKzc93CCAQT4FS9m0diEs5GJeSt+mUkr/Nw08EEKidQKn7aCnTlzKt1rjU6TVPb9olzUdCoBgBjbbRSBu73bSYeZgGgTgJaNvVNhy+BCdO5aMsCEQJlNof0PSlzhO1XH1WqXzy5Z+WzwkyJqymNTR4yJBB/oChFykoEq0rrXq4s55roKRhxSQEEEAAAQQQQAABBBDonYBONnW7nr0crXe5MBcC9RPQtqttmMBJ/eqAJSOQRgEejpXAWrdniiioaM880mro2QYaDq/nf5AQQAABBBBAAAEEEECg9wJ6S7L623qWaaHnGPZ+CcxZqsCxxxzp34od9aKVUvNq5Om1zeo5ddqGSQgggEAtBQgy1lK7gstSoPHsSePdhx/ucv/6xptOB1oNhedKVQWRyQoBBBBAAAEEEEAgtQK6JX/+3H9x17YsKfhSpdQC1WHF9fZn3iJfGF7PldM2q22Xx0oUtuJbBBCovAC3S1fetGY56gCr26f1Jjk9zJcAY83oWRACCCCAAAIIIIBACgQmjB/j13Lp8ltTsLasYiMI2LZq224jrBPrgAACyREgyJicuqKkCCCAAAIIIIAAAgggUEMBXcS/dfkP/G3Teg46CYE4C2gb1S3+2mYZgBLnmqJsCDSuAEHGxq1b1gwBBBBAAAEEEEAAAQTKFNCdQ7r1dOrlV2Sec7etzNyYHYHqCGjb1DaqbVXbLAkBBBCohwBBxnqos0wEEEAAAQQQQAABBBBIjIAeTzRh3Dh3wcXT3K5duxNTbgqaDgFtk9o2tY1qWyUhgAAC9RIgyFgveZaLAAIIIIAAAggggAACiRGYP2+WGzJkkJs24yr33vvbE1NuCtrYAtoWtU1q29Q2SkIAAQTqKUCQsZ76LBsBBBBAAAEEEEAAAQQSIaBn3N152zJf1klnX0igMRG11tiFVIBR26KStk2ew9jY9c3aIZAEAYKMSaglyogAAggggAACCCCAAAJ1F7BAo0aNEWise3WkugAWYNS2SIAx1ZsCK49ArAQIMsaqOigMAggggAACCCCAAAIIxFnAAo2jRg73gcbXXn8jzsWlbA0ooG1OQW5tgwQYG7CCWSUEEixAkDHBlUfREUAAAQQQQAABBBBAoPYCCjQuWjDHTZx4hjv3wkvdwz9bU/tCsMRUCmhb0zanbU/bILdIp3IzYKURiK1An9iWjIIhgAACCCCAAAIIIIAAAjEWuHrmdDfs8MPcd2bPdes3PesWL5rv9tmnb4xLTNGSKqA3SM+et9Bt2LjJfX9xi/vaqackdVUoNwIINLAAIxkbuHJZNQQQQAABBBBAAAEEEKiugII9v3pirWtv3+JGjz3Lcft0db3TmLu2KW1b2sa0rRFgTONWwDojkAwBgozJqCdKiQACCCCAAAIIIIAAAjEV2H+/AW7VQ/e6Sy4639/KOuPK2U4jz0gIlCOgbUjbkm6P1ralbUzbGgkBBBCIqwBBxrjWDOVCAAEEEEAAAQQQQACBxAjo2XiXTZns1q16MDuqUc/P+2jPnsSsAwWNh4C2GW07NnpR25S2LZ6/GI/6oRQIIJBfgCBjfhu+QQABBBBAAAEEEEAAAQRKEhg86OBMoHGl+/a3vul++KNbMm8Bnuye//WLJeXBxOkV0LaibUbbjrYhbUvapkgIIIBAEgQIMiahligjAggggAACCCCAAAIIJErgrDPHuyfWPexGjRrupl5+hRs76Vy3Zeu2RK0Dha2dgLYNbSPaVrTNaNvRNkRCAAEEkiRAkDFJtUVZEUAAAQQQQAABBBBAIDECetO03kD9/ManXPMRR2SCSOf4Z+zxcpjEVGHVC6ptQc9d1LahbUTbirYZ3lJedXoWgAACVRAgyFgFVLJEAAEEEEAAAQQQQAABBExAAaNFC+b45zU29W/yL/LQqDVuozah9P1U3Wsb0EtdtE3ouYvaRggupm9bYI0RaCQBgoyNVJusCwIIIIAAAggggAACCMRWQM/WUyBJo9XsNuqTTxnjX/LB26hjW20VK5jqWC90UZ3bbdHaFrRN8NzFijGTEQII1FGgTx2XzaIRQAABBBBAAAEEEEAAgdQJ2G3UM6dPc4+seczdteJ+t7Dlejdh3Dh33jcmucMPOzR1Jo28wrol+oGfrnKPrF2bCSYO8i90mTB+DG+LbuRKZ90QSKkAQcaUVjyrjQACCCCAAAIIIIAAAvUV2LtPH/9yD73gwwJRun22f78md8lF52eCjqe7/fcbUN9CsvReCbz3/vZMUPFxd9c997sdOzt8AHnlijsIIPdKk5kQQCApAgQZk1JTlBMBBBBAAAEEEEAAAQQaVkCjF3Xb7JxZM92zz73g7rnvQbd02Y/9yLeJ48cScExAzVtgcfWadZk3iW91Rwwb5q6ZO8uddOLxPGsxAfVHERFAoHwBgozlG5IDAggggAACCCCAAAIIIFARAd1K/bVTT/H/wqBVGHA88fhjGBFXEe3yM9EI1OdeeNFZYFG3QxMULt+VHBBAIJkCBBmTWW+UGgEEEEAAAQQQQAABBBpcQLdKXzZlsv+ngOMrrW3ZEY5adT3DccTJJ7gjm4dxW3WNtgWrh43PPO+fsajFasTiFZnna1IPNaoEFoMAArEVIMgY26qhYAgggAACCCCAAAIIIIBAp4ACjjbC8aM9e9xLL7e6ZzK3VV/bssQ/80/PcZw48Qx3cubW3C9+4fMEHSu04Sio+Ic//slbr179aBfr226+yR19VDMvcKmQNdkggEDyBQgyJr8OWQMEEEAAAQQQQAABBBBIkYBeGHPCccf4f1fPnO7C0XVXz74mGwhrbh7qRg0/yR17zJFu4MADCYb1sI0oePv22++437z4ilu/6VnX2ro5azlq5HD/fEVGK/aAyNcIIJBqAYKMqa5+Vh4BBBBAAAEEEEAAAQSSLhCOcnRujg86ZkffrX3MLWy53q+inhc4ZMhgH3g86KCBqR7xaCMU33rrbR9QbG/f4l/WIijd/qwA7eTzzk61UdL3C8qPAAK1FyDIWHtzlogAAggggAACCCCAAAIIVE1AQUf902hHS1u2bnO/ffP3ru21131QbcPGTfaVf4O1BR/79fuM+/IhX3J9+/ZN/C3XCiTu3r3br/fOnR90CyYKYOSI4ZnA6yA3/oyv+/UePOjgrAu/IIAAAgiUJkCQsTQvpkYAAQQQQAABBBBAAAEEEieg4Jn+6bmOlnbt2u3e377d3x7csWNnZBBOz3rUqD6l5mFDXVP/fs4CkZZPrQNzCphaUuBUAUSVv7Vts//YbnO2aWwEp8pvt4/vN2CA05u8SQgggAAClRMgyFg5S3JCAAEEEEAAAQQQQAABBBIjoCCb/kUFCS0A+c67f3G6pVhJzylUCm8t9h/k/E+3Gzc19c/5tHd/dnTscK+2teWd2QKImkABRCXd5nzgAZ9zBBI9B/9DAAEEaiZAkLFm1CwIAQQQQAABBBBAAAEEEEiGQJcA5HGdZT7rzPHdCm8vSwm/0ItTekoaeaikkZE9pZaFc7tMwktsunDwBwIIIBAbAYKMsakKCoIAAggggAACCCCAAAIIJEtAb7rOHQmZ+3ey1ojSIoAAAgj0VmCv3s7IfAgggAACCCCAAAIIIIAAAggggAACCCCAgAQIMrIdIIAAAggggAACCCCAAAIIIIAAAggggEBZAgQZy+JjZgQQQAABBBBAAAEEEEAAAQQQQAABBBAgyMg2gAACCCCAAAIIIIAAAggggAACCCCAAAJlCRBkLIuPmRFAAAEEEEAAAQQQQAABBBBAAAEEEECAICPbAAIIIIAAAggggAACCCCAAAIIIIAAAgiUJUCQsSw+ZkYAAQQQQAABBBBAAAEEEEAAAQQQQAABgoxsAwgggAACCCCAAAIIIIAAAggggAACCCBQlgBBxrL4mBkBBBBAAAEEEEAAAQQQQAABBBBAAAEECDKyDSCAAAIIIIAAAggggAACCCCAAAIIIIBAWQIEGcviY2YEEEAAAQQQQAABBBBAAAEEEEAAAQQQIMjINtBQAh07djbU+rAyCCAQHwHal/jUBSVBoNEEaF8arUZZHwQQQAABBOInUIv+BkHG+NU7JSpDYP3GZ1zzsKFl5MCsCCCAQLQA7Uu0C58igED5ArQv5RuSAwIIIIAAAggUFqhFf6PiQcZ+/T7jWls3F14zvkWgCgIf7dnjXm1rc6d9dWQVcifLOAjQvsShFtJZBtqXxq932pfGr+O4riHtS1xrhnIhgAAClRegv1F5U3IsTqBW/Y2KBxlPOvF4t2Nnh3vt9TeKW1OmQqBCAi+93OpzGjzo4ArlSDZxE6B9iVuNpKc8tC+NX9e0L41fx3FdQ9qXuNYM5UIAAQQqL0B/o/Km5FicQK36GxUPMu6zT183Ydw4N236Ve6997cXt7ZMhUCZAtrWlt18uxs5YniZOTF7nAVoX+JcO41bNtqXxq3bcM1oX0INfq+VAO1LraRZDgIIIBAPAfob8aiHtJWilv2NTy/IpEoDDz/5RLdh07PuiSd/5QYOPMB97nOfdZ/eq+LxzEoXm/wSKqBRs5Mv+aZraurvbv7R99nWElqPxRab9qVYKaarhADtSyUUk5MH7Uty6qoRSkr70gi1yDoggAACpQvQ3yjdjDl6L1Dr/san/ppJvS9u/jl1v/eVs+a6DRs3+YmOGDbMjRpxsmvq3y//THyDQJECeitSa9tm//xP3Z6v0bPz581ye/fpU2QOTJZkAdqXJNde/MtO+xL/OqpmCWlfqqlL3rQvbAMIIIAAAhKgv8F2UE2BevY3qhZkDMG2bN3mnvzlBh8UCj/ndwTKEdBbpAcd/HdOz7XQsHNSOgVoX9JZ79Vea9qXagsnI3/al2TUU9JKSfuStBqjvAgggEB1BehvVNc3rbnXq79RkyBjWiuV9UYAAQQQQAABBBBAAAEEEEAAAQQQQCANAjwoMQ21zDoigAACCCCAAAIIIIAAAggggAACCCBQRQGCjFXEJWsEEEAAAQQQQAABBBBAAAEEEEAAAQTSIECQMQ21zDoigAACCCCAAAIIIIAAAggggAACCCBQRQGCjFXEJWsEEEAAAQQQQAABBBBAAAEEEEAAAQTSIECQMQ21zDoigAACCCCAAAIIIIAAAggggAACCCBQRQGCjFXEJWsEEEAAAQQQQAABBBBAAAEEEEAAAQTSIECQMQ21zDoigAACCCCAAAIIIIAAAggggAACCCBQRQGCjFXEJWsEEEAAAQQQQAABBBBAAAEEEEAAAQTSIECQMQ21zDoigAACCCCAAAIIIIAAAggggAACCCBQRQGCjFXEJWsEEEAAAQQQQAABBBBAAAEEEEAAAQTSIECQMQ21zDoigAACCCCAAAIIIIAAAggggAACCCBQRQGCjFXEJWsEEEAAAQQQQAABBBBAAAEEEEAAAQTSIECQMQ21zDoigAACCCCAAAIIIIAAAggggAACCCBQRQGCjFXEJWsEEEAAAQQQQAABBBBAAAEEEEAAAQTSIECQMQ21zDoigAACCCCAAAIIIIAAAggggAACCCBQRQGCjFXEJWsEEEAAAQQQQAABBBBAAAEEEEAAAQTSIECQMQ21zDoigAACCCCAAAIIIIAAAggggAACCCBQRQGCjFXEJWsEEEAAAQQQQAABBBBAAAEEEEAAAQTSIECQMQ21zDoigAACCCCAAAIIIIAAAggggAACCCBQRQGCjFXEJWsEEEAAAQQQQAABBBBAAAEEEEAAAQTSIECQMQ21zDoigAACCCCAAAIIIIAAAggggAACCCBQRYH/Dweeta05LMjCAAAAAElFTkSuQmCC"}}},{"cell_type":"code","source":"# Import all the required libraries\n\nfrom __future__ import print_function\nimport tensorflow as tf\nfrom tensorflow.keras.datasets import cifar10\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.layers import Dense, Dropout, Activation, Flatten\nfrom tensorflow.keras.layers import BatchNormalization\nfrom tensorflow.keras.layers import Conv2D, MaxPooling2D\nfrom keras.losses import SparseCategoricalCrossentropy, CategoricalCrossentropy\nfrom tensorflow.keras.preprocessing.text import Tokenizer\nfrom tensorflow.keras.preprocessing.sequence import pad_sequences\nfrom tensorflow.keras.regularizers import l2\nfrom tensorflow.keras.applications.inception_v3 import preprocess_input\nfrom tensorflow.keras.preprocessing.image import load_img, img_to_array\nfrom sklearn.model_selection import train_test_split\nimport matplotlib.image as mpimg\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport pandas as pd\nimport os\nimport cv2\nimport matplotlib.pyplot as plt\nimport glob\nimport cv2\nimport random as rd\nimport copy\nimport random\nfrom PIL import Image\nfrom tqdm.notebook import tqdm\nfrom collections import Counter\nfrom wordcloud import WordCloud\nimport spacy\nimport pickle \n\n%matplotlib inline\n","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-09-18T14:45:19.105546Z","iopub.execute_input":"2023-09-18T14:45:19.106171Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"! pip install pyttsx3","metadata":{"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"! sudo apt install espeak -y","metadata":{"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def convert_to_audio(caption):\n    engine = pyttsx3.init()\n    engine.say(caption)\n    engine.runAndWait()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd.set_option('display.max_rows', None)\npd.set_option('display.max_columns', None)\npd.set_option('display.width', 2000)\npd.set_option('display.float_format', '{:20,.2f}'.format)\npd.set_option('display.max_colwidth', None)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"directory = '/kaggle/working/checkpoints/'\nif not os.path.exists(directory):\n    os.makedirs(directory)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"! cp -rf /kaggle/input/flickr8k/ /kaggle/working/","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"! ls /kaggle/working/","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"physical_devices = tf.config.experimental.list_physical_devices('GPU')\n# Define a distribution strategy\nstrategy = tf.distribute.MirroredStrategy()\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 1. Data understanding\nThe dataset utilized in this context is the publicly accessible Flickr 8K dataset, encompassing 8000 images capturing diverse scenarios and events. Each data entry is meticulously annotated with five captions. A significant advantage of employing this dataset lies in its compact size, enabling users to engage with it even on less powerful, low-end PCs.","metadata":{"execution":{"iopub.status.busy":"2023-09-01T14:02:07.190383Z","iopub.execute_input":"2023-09-01T14:02:07.19163Z","iopub.status.idle":"2023-09-01T14:02:07.198643Z","shell.execute_reply.started":"2023-09-01T14:02:07.191588Z","shell.execute_reply":"2023-09-01T14:02:07.197213Z"}}},{"cell_type":"code","source":"images='/kaggle/working/flickr8k/Images'\n\nall_imgs = glob.glob(images + '/*.jpg',recursive=True)\nprint(\"The total images present in the dataset: {}\".format(len(all_imgs)))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# taking 6 random images and display the same with the file name.\n\nrandom.shuffle(all_imgs)\nselected_imgs = all_imgs[:6]\n\n# Create a subplot for each image\nfig, axes = plt.subplots(2, 3, figsize=(12, 8))\n\nfor i, ax in enumerate(axes.flat):\n    img_path = selected_imgs[i]\n    img = Image.open(img_path)\n    ax.imshow(img)\n    ax.axis('off')\n    \n    file_name = img_path.split('/')[-1] \n    ax.set_title(file_name)\n\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def load_doc(filename):\n    try:\n        with open(filename, 'r') as file:\n            next(file)\n            captions_doc = file.read()\n        return captions_doc\n    except FileNotFoundError:\n        print(f\"File '{filename}' not found.\")\n        return None\n\n\n# Specify the correct file path\nfile_path = '/kaggle/working/flickr8k/captions.txt'\ndoc = load_doc(file_path)\nprint(doc[:300])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(len(doc))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# mapping image to captions\n# A single image have multiple description, in the mapping a single image file name with multiple descriptions are mapped\n# for eg: '1000268201_693b08cb0e': ['A child in a pink dress is climbing up a set of stairs in an entry way .',\n#  'A girl going into a wooden building .',\n# 'A little girl climbing into a wooden playhouse .',\n#  'A little girl climbing the stairs to her playhouse .',\n#  'A little girl in a pink dress going into a wooden cabin .'],\n# print(len(mapping))  8091\n# print(len(img_id_list)) 40455\n# print(len(caption_list)) 40455\n\nmapping = {}\nimg_id_list = []\nimage_path_list = []\ncaption_list = []\n\nfor line in tqdm(doc.split(\"\\n\")):\n    #split the line with comma\n    tokens = line.split(',')\n    if len(line)<2:\n        continue\n        \n    img_id , caption = tokens[0] , tokens[1:] \n    image_path_list.append('/kaggle/working/flickr8k/Images/' + img_id)\n    # remove extensions from img id\n    img_id = img_id.split('.')[0]\n    img_id_list.append(img_id)\n    caption_list.append(caption)\n    \n    # caption list to string\n    caption = \" \".join(caption)\n    \n    if img_id not in mapping:\n        mapping[img_id] = []\n    \n    #store the captions\n    mapping[img_id].append(caption)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"caption","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(len(mapping))\nprint(len(img_id_list))\nprint(len(caption_list))\nprint(len(image_path_list))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"random.shuffle(all_imgs)\nselected_imgs = all_imgs[:6]\n\n# Create a subplot for each image\nfig, axes = plt.subplots(2, 3, figsize=(12, 8))\n\nfor i, ax in enumerate(axes.flat):\n    img_path = selected_imgs[i]\n    img = Image.open(img_path)\n    ax.imshow(img)\n    ax.axis('off')\n    \n    # Extract the file name from the path and display it below the image\n    file_name = img_path.split('/')[-1]  # You may need to adjust the path separator if necessary\n    file_name = os.path.splitext(file_name)[0]\n    image_txt = mapping[file_name][0]\n    ax.set_title(image_txt)\n\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# this is for reference.\n\nall_img_id= img_id_list\nall_img_vector= image_path_list\nannotations= caption_list\n\ndf = pd.DataFrame(list(zip(all_img_id, all_img_vector, annotations)),columns =['ID','Path', 'Captions']) \n\ndf.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.isnull().sum()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#  Captions string is inside a list, removing the list.\n\ndf['Captions'] = df['Captions'].apply(lambda x: ' '.join(x))\ndf.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.isnull().sum()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Adding start and end tokens to each sentence. Later this will help in prediction. \n\ndef add_tokens(captions):\n    for i in range(len(captions)):\n        # Take one caption at a time\n        caption = captions[i]\n        # Preprocessing steps\n        # Add unique start and end tokens to the caption\n        caption = '<start> ' + ' '.join([word for word in caption.split() if len(word) > 1]) + ' <end>'\n        captions[i] = caption\n    return captions","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Create a list which contains all the captions\nannotations= list(df['Captions'])\n#add the <start> & <end> token to all those captions as well\nannotations = add_tokens(annotations)\n\n#Create a list which contains all the path to the images\nall_img_path = image_path_list\n\nprint(\"Total captions present in the dataset: \"+ str(len(annotations)))\nprint(\"Total images present in the dataset: \" + str(len(all_img_path)))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['Captions'] = annotations\ndf.head(5)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.isnull().sum()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create the vocabulary & the counter for the captions\n\nvocabulary= [word for sentence in annotations for word in sentence.split()]\n\nval_count=Counter(vocabulary)\nval_count.most_common(5)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(len(val_count))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Visualise the top 30 occuring words in the captions\n\ntop_30_words = val_count.most_common(30)\nwords, counts = zip(*top_30_words)\n\n# Create a bar chart to visualize the top 30 occurring words\nplt.figure(figsize=(12, 6))\nplt.bar(words, counts)\nplt.xlabel('Words')\nplt.ylabel('Count')\nplt.title('Top 30 Occurring Words in Sentences')\nplt.xticks(rotation=90)\nplt.tight_layout()\n\n# Show the plot\nplt.show()\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 2. Pre-Processing the captions\nPre-processing in NLP/Deep Learning readies text data for effective analysis. It involves tokenization, breaking text into units like words, essential for models to understand structure. Uncommon words are replaced with \"UNK\" to simplify vocabulary and save memory. Creating word-to-index and index-to-word mappings enables numerical processing. Padding sequences to a uniform length ensures compatibility with neural networks. These steps collectively transform the raw text into a format that models can learn from, setting the foundation for various NLP tasks while optimizing memory usage and computational resources.\n\n1.Create the tokenized vectors by tokenizing the captions fore ex :split them using spaces & other filters. \nThis gives us a vocabulary of all of the unique words in the data. Keep the total vocaublary to top 5,000 words for saving memory.\n\n2.Replace all other words with the unknown token \"UNK\" .\n\n3.Create word-to-index and index-to-word mappings.\n\n4.Pad all sequences to be the same length as the longest one.","metadata":{"execution":{"iopub.status.busy":"2023-09-02T06:47:42.317179Z"}}},{"cell_type":"code","source":"# create the tokenizer\n# using preprocessing.text.Tokenizer from tenrfloe to tokenize the words. It will create words (english) to numeric representation.\n# filter_chars arguments to Tokenizer makes sure to exculde them during tokenization.\n# ovv_token (out-of-vocabulary): tokenize a new text that contains words not present in the learned vocabulary words to \"<unk>\"\n# Eg: english sentence : ['<start> child in pink dress is climbing up set of stairs in an entry way <end>'] tokenized to \n# [2, 43, 4, 91, 173, 7, 120, 51, 394, 12, 395, 4, 28, 1, 671, 3]\n\ntop_word_count = 5000\n\nfilter_chars = '!\"#$%&()*+.,-/:;=?@[\\]^_`{|}~ '\ntokenizer = tf.keras.preprocessing.text.Tokenizer(num_words=top_word_count,\n                                                  oov_token=\"<unk>\",\n                                                  filters=filter_chars)\ntokenizer.fit_on_texts(annotations)\ntrain_seqs = tokenizer.texts_to_sequences(annotations)\nprint(train_seqs[:5])\nprint(annotations[:5])\nprint(f'OOV Token: {tokenizer.oov_token}')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create word-to-index and index-to-word mappings.\n# word_index: contains words to numeric representation.[eg : 'the': 5, 'is': 7]\n# index_word: contains numeric to words representation.[eg : 5: 'the', 7: 'is']\n\nword_index = tokenizer.word_index\nindex_word = tokenizer.index_word\n\n# Add Pad Token\ntokenizer.word_index['<pad>'] = 0\ntokenizer.index_word[0] = '<pad>'\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"! ls /kaggle/working/","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## Save Tokenizer,in case if we want to read it back from disk.\nimport pickle\n\nwith open('/kaggle/working/checkpoints/tokenizer.pkl',\"wb\") as f:\n    pickle.dump(tokenizer,f)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"i = 0\nfor k, v in index_word.items():\n    print(k, v)\n    if(i ==8):\n        break\n    else:\n        i += 1\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"i = 0\nfor k, v in word_index.items():\n    print(k, v)\n    if(i ==8):\n        break\n    else:\n        i += 1\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create a word count of your tokenizer to visulize the Top 30 occuring words after text processing\n\nword_counts = {word: count for word, count in tokenizer.word_counts.items() if word != \"<unk>\"}\n\n# Sort the word counts in descending order\nsorted_word_counts = dict(sorted(word_counts.items(), key=lambda item: item[1], reverse=True))\n\n# Extract the top 30 words and their counts\ntop_30_words = list(sorted_word_counts.items())[:30]\n\n# Extract words and counts from the top 30\nwords, counts = zip(*top_30_words)\n\n# Create a bar chart to visualize the top 30 occurring words\nplt.figure(figsize=(12, 6))\nplt.bar(words, counts)\nplt.xlabel('Words')\nplt.ylabel('Count')\nplt.title('Top 30 Occurring Words after Text Processing')\nplt.xticks(rotation=90)\nplt.tight_layout()\n\n# Show the plot\nplt.show()\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Pad each vector to the max_length of the captions ^ store it to a vairable\n# max length of a caption is 35 and append the word <pad> if any of the senetence is less than 30. \n# keras.preprocessing.sequence.pad_sequences from tensorflow would help to do that.\n\nmax_sequence_len = max(len(t) for t in train_seqs)\nprint(f'Max Sequence Length of the Texts is: {max_sequence_len}')\n\ncap_vector = tf.keras.preprocessing.sequence.pad_sequences(train_seqs, padding='post', maxlen=max_sequence_len)\nprint(\"The shape of Caption vector is :\" + str(cap_vector.shape))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# printing all details about captions that are performed so far.\n\nprint(\"Vocabulary size:\", len(val_count))\nprint(\"Word-to-index mapping:\", len(word_index))\nprint(\"Index-to-word mapping:\", len(index_word))\nprint(\"len Captions (padded):\\n\", len(cap_vector))\nprint(\"len[0] Captions (padded):\\n\", len(cap_vector[0]))\nprint(\" 1st Captions (padded):\\n\", cap_vector[0])\nprint(\"Captions (padded):\\n\", cap_vector)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3. Image Feature Vector Construction:\n\nThe objective is to extract image features (weights) from a pre-trained model, which will then serve as input for the encoder in an RNN model tasked with generating captions. Specifically, the inception_v3 pre-trained model is employed for this purpose. Since pre-trained models have specific requirements regarding image size and shape, this code segment handles the necessary pre-processing steps to meet those specifications.\n\n1.Resize them into the shape of (299, 299)\n\n3.Normalize the image within the range of -1 to 1, such that it is in correct format for InceptionV3. ","metadata":{}},{"cell_type":"code","source":"# for capturing image features, we will use the pre-trained model inception_v3. this model expects input images to be in the share (299, 299) and\n# image value range from -1 to +1. the same operation is performed in the below function preprocess_image.\n# Size of image before preprocessing: (500, 375, 3)\n# Size of image after preprocessing: (299, 299, 3)\n# Note : preprocess_image function returns the resized image along with it's path. \n \n# write your code here to create the dataset consisting of image paths\n\n#image_paths = df['Path'].values\n#image_paths_dataset = tf.data.Dataset.from_tensor_slices(image_paths)\n\ndef preprocess_image(image_path):\n    img = tf.io.read_file(image_path)\n    img = tf.image.decode_jpeg(img, channels=3)\n    img = tf.image.resize(img, (299, 299))\n    img = tf.keras.applications.inception_v3.preprocess_input(img)\n    return img, image_path\n\n# Check the preprocessing Logic\nprint(f'Size of image before preprocessing: {mpimg.imread(all_img_vector[0]).shape}')\nprint(f'Size of image after preprocessing: {preprocess_image(all_img_vector[0])[0].shape}')\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# write your code here for applying the function to the image path dataset, such that the transformed dataset should contain images & their path\n# we need to pre-process all the images before feeding them to inception_v3 model and also create a dataset out of the same.\n# to create a dataset tensorflow function from_tensor_slices would do that and to speed up the process parallelly. \n# with batch(64), the groups consecutive elements of the dataset into batches of size 64.\n\nunique_img_vector = sorted (set(all_img_vector))\nimage_dataset = tf.data.Dataset.from_tensor_slices(unique_img_vector)\nimage_dataset = image_dataset.map(preprocess_image, num_parallel_calls=tf.data.experimental.AUTOTUNE) \\\n                .batch(64)\nimage_dataset\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Load the pretrained Imagenet weights of Inception net V3","metadata":{}},{"cell_type":"code","source":"# we will load the InceptionV3 model include_top=False means that we're excluding the fully connected layers at the top of the network, which are typically used for classification. \n# weights='imagenet' specifies that the model should be loaded with pre-trained ImageNet weights.\n# image_model.input gives you the input tensor of the InceptionV3 model. This will be a placeholder for the input images.\n# image_model.layers[-1] retrieves the last layer of the InceptionV3 model, which in this case is the last convolutional layer.\n# .output retrieves the output tensor of that layer. This tensor will contain features extracted from the images. The shape of the output of this layer is 8x8x2048.\n# Use a function to extract the features of each image in the train & test dataset such that the shape of each image should be (batch_size, 8*8, 2048)\n# tf.keras.Model(new_input, hidden_layer): This line creates a new model, which is a modified version of the original InceptionV3 model.\n# It takes the same input as the original model but outputs the tensor from the last convolutional layer.\n# the new model code can also be written as tf.keras.Model(image_model.input, image_model.layers[-1].output)\n# This new model is designed for feature extraction. It will take an image as input and output the features extracted by the InceptionV3 model.\n\nimage_model = tf.keras.applications.InceptionV3(include_top=False, weights='imagenet')\n\nnew_input = image_model.input #write code here to get the input of the image_model\nhidden_layer = image_model.layers[-1].output #write code here to get the output of the image_model\n\nimage_features_extract_model = tf.keras.Model(new_input, hidden_layer) #build the final model using both input & output layer\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# image_features_extract_model.summary()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# write the code to apply the feature_extraction model to your earlier created dataset which contained images & their respective paths\n# Once the features are created, you need to reshape them such that feature shape is in order of (batch_size, 8*8, 2048)\n# This line reshapes the batch_features tensor. It changes the shape to have the same number of samples in the batch, but the second dimension is \n# flattened. The new shape is (batch_size, -1, num_features), where batch_size is the number of images in the batch, \n# and num_features is the number of extracted features.\n# The use of -1 in the second dimension allows TensorFlow to automatically calculate the size of that dimension \n# based on the other dimensions, ensuring that the reshaping is compatible.\n# This reshaping would help in further feeding it into a next model for processing and predicting the text. \n# the image 1000268201_693b08cb0e.jpg  is converted to numpy feature store to 1000268201_693b08cb0e.jpg.npy. Same is done to all the images.\n# Overall, this code processes batches of preprocessed images, extracts features using the modified InceptionV3 model, and saves\n# these features as NumPy arrays with filenames corresponding to the original image files. This can be useful for later stages of a \n# machine learning pipeline, such as training a model using these features.\n\nfor img, path in tqdm(image_dataset):\n    batch_features = image_features_extract_model(img)\n    batch_features = tf.reshape(batch_features,\n                              (batch_features.shape[0], -1, batch_features.shape[3]))\n    \n    \n    for bf, p in zip(batch_features, path):\n        path_of_feature = p.numpy().decode(\"utf-8\")\n        np.save(path_of_feature, bf.numpy())\n","metadata":{"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# !ls /kaggle/working/flickr8k/Images","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 4. Dataset creation\nThe division of training and testing data has been completed, which includes paths to the images in the training set and their associated captions. Previously, we stored the features of each image in a numpy array. To facilitate further model training, we need to retrieve these stored features and convert them into a TensorFlow dataset. The 'map_func' function accomplishes this by loading the previously saved features from the disk and returning the image tensor along with its caption. The 'gen_dataset' function utilizes 'map_func' to create a dataset. The output shape of the image features in the dataset is (64, 64, 2048) with a batch size of 64, while the captions have a shape of (64, 35).","metadata":{}},{"cell_type":"code","source":"path_train, path_test, cap_train, cap_test = train_test_split(all_img_vector, cap_vector, test_size=0.2, random_state=42)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Training data for images: \" + str(len(path_train)))\nprint(\"Testing data for images: \" + str(len(path_test)))\nprint(\"Training data for Captions: \" + str(len(cap_train)))\nprint(\"Testing data for Captions: \" + str(len(cap_test)))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Define a function 'map_func' that takes an image path and caption as inputs.\n# It decodes the image path using UTF-8, appends '.npy', and loads it as an image tensor.\n# The function then returns a tuple containing the image tensor and the original caption.\n\ndef map_func(image, caption):\n    img_tensor = np.load(image.decode('utf-8')+'.npy')\n    return img_tensor,caption","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"the devide of train and test is done and that contains path for the images of the train and captions associated with them. \nearlier we have stored the features of each image in the numpy array. For further training of the model, we need to fetch these stored feature into \ntensorflow dataset. For that the fucntion 'map_func' loads the previously saved features from disk and returns the image tensor and the caption.\ngen_dataset is a function that creates a dataset by using the map_func. \nThe shape of the output of dataset of image features is (64, 64, 2048) (batch_size is 64)\ncaptions shape (64, 35)\n","metadata":{}},{"cell_type":"code","source":"# create a builder function to create dataset which takes in the image path & captions as input\n# This function should transform the created dataset(img_path,cap) to (features,cap) using the map_func created earlier\n# # This function should transform the created dataset(img_path,cap) to \n# (features,cap) using the map_func created earlier\n\ndef gen_dataset(image, caption, batch_size = 32, buffer_size=1000):\n    \n    dataset = tf.data.Dataset.from_tensor_slices((image, caption))\n\n    # Use map to load the numpy files in parallel\n    dataset = dataset.map(lambda item1, item2: tf.numpy_function(\n            map_func, [item1, item2], [tf.float32, tf.int32]),\n            num_parallel_calls=tf.data.AUTOTUNE)\n\n    # Shuffle and batch\n    dataset = dataset.shuffle(buffer_size).batch(batch_size)\n    dataset = dataset.prefetch(buffer_size=tf.data.AUTOTUNE)\n    \n    return dataset","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"BATCH_SIZE = 64\n\ntrain_dataset = gen_dataset(path_train, cap_train, BATCH_SIZE)\ntest_dataset = gen_dataset(path_test, cap_test, BATCH_SIZE)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_img_batch, sample_cap_batch = next(iter(train_dataset))\nprint(sample_img_batch.shape)  # (batch_size, 8*8, 2048)\nprint(sample_cap_batch.shape) # (batch_size,max_len)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# sample_img_batch","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 5. Model Architecture:\nThe CNN-based encoder generates a fixed feature vector, representing the input image's encoded information. This feature vector remains constant across all timestamps. It's combined with the decoder's hidden state to form the context vector in the attention model.\n<br>\nOn the decoder side, the input sequence is pre-processed and standardized to have equal sequence lengths. This processed sequence is then passed through an embedding layer to create embedded vectors. These embeddings are concatenated with the context vector from the attention model and fed into a concatenation layer.\n<br>\nThe output of this concatenation layer serves as input for the GRU, a type of RNN. The GRU produces an output and a hidden state, which are used at the subsequent timestep. A dense layer follows the GRU, performing a linear transformation on the previous output. This yields a list of probability values for all words in the vocabulary, with the word having the highest probability chosen as the prediction for that timestep.\n","metadata":{}},{"cell_type":"code","source":"embedding_dim = 256 \nunits = 512\nvocab_size = 5001 #top 5,000 words +1\ntrain_num_steps = len(path_train) // BATCH_SIZE\ntest_num_steps = len(path_test) // BATCH_SIZE\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 6. Model Building:","metadata":{}},{"cell_type":"markdown","source":"### Encoder","metadata":{}},{"cell_type":"code","source":"# this is for the encoder class where input features extracted from the image i.e 64(8*8)*2048 is fed to dense layer. the dense layer mapping \n# will be from 2048 to 256. \n# output of each image will be 64(8*8) * 256 (with batch size of 64, so output is 64*64*256).\n\nclass Encoder(tf.keras.Model):\n    def __init__(self,embed_dim = 256):\n        super(Encoder, self).__init__()\n        self.dense = tf.keras.layers.Dense(embed_dim) # build your Dense layer with relu activation\n        \n    def call(self, features):\n        features =  self.dense(features) # extract the features from the image shape: (batch, 8*8, embed_dim)\n        features = tf.nn.relu(features)\n        return features","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"encoder = Encoder(embedding_dim)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Attention model","metadata":{}},{"cell_type":"code","source":"# attention is called from decoder.\n\nclass Attention_model(tf.keras.Model):\n    def __init__(self, units):\n        super(Attention_model, self).__init__()\n        self.W1 = tf.keras.layers.Dense(units) #build your Dense layer, number of units is 512\n        self.W2 = tf.keras.layers.Dense(units) #build your Dense layer, number of units is 512\n        self.V = tf.keras.layers.Dense(1) #build your final Dense layer with unit 1\n        self.units=units\n        \n#features.shape (64, 64, 256)\n#hidden.shape (64, 512)\n#hidden_with_time_axis.shape (64, 1, 512)\n#attention_hidden_layer.shape (64, 64, 512)\n#score.shape (64, 64, 1)\n#attention_weights.shape (64, 64, 1)\n#1 context_vector.shape (64, 64, 256)\n#2 context_vector.shape (64, 256)\n\n\n    def call(self, features, hidden):\n        #print(\"features.shape\",features.shape)\n        #print(\"hidden.shape\",hidden.shape)\n\n        # feature variable is nothing but output from encoder with shape batch_size, 8*8, embedding_dim. (64*64*256)\n        # hidden variable output init state from decoder hidden_size (64 * 512)\n        hidden_with_time_axis =  tf.expand_dims(hidden,1) # Expand the hidden shape to : (batch_size, 1, hidden_size)\n        #print(\"hidden_with_time_axis.shape\",hidden_with_time_axis.shape)\n        # This line expands the dimensions of the hidden tensor. It adds an extra dimension at index 1. This is done to match the shape of the features for subsequent operations.\n        # attention_hidden_layer shape == (batch_size, 64, units)\n        attention_hidden_layer = (tf.nn.tanh(self.W1(features) +\n                                            self.W2(hidden_with_time_axis))) # [(64, 1, 512) + (64, 64, 512)]  =  (64, 64, 512)\n        #print(\"attention_hidden_layer.shape\",attention_hidden_layer.shape)\n        score = self.V(attention_hidden_layer) # build your score funciton to shape: (batch_size, 8*8, units) (64, 64, 1)\n        #print(\"score.shape\",score.shape)\n        attention_weights =  tf.nn.softmax(score, axis=1) # extract your attention weights with shape: (batch_size, 8*8, 1) (64, 64, 1)\n        #print(\"attention_weights.shape\",attention_weights.shape)\n        context_vector =  attention_weights * features # shape: context vector (batch_size, 8*8,embedding_dim) (64*64*1)*(64, 64, 256) = (64, 64, 256)\n        #print(\"1 context_vector.shape\",context_vector.shape)\n        context_vector =  tf.reduce_sum(context_vector, axis=1) # reduce the shape to (batch_size, embedding_dim) (64, 256)\n        #print(\"2 context_vector.shape\",context_vector.shape)\n\n        return context_vector, attention_weights","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"how attention_weights are used? Answer here","metadata":{}},{"cell_type":"markdown","source":"### Decoder","metadata":{}},{"cell_type":"code","source":"# Decoder model\n\nclass Decoder(tf.keras.Model):\n    def __init__(self, embed_dim, units, vocab_size):\n        super(Decoder, self).__init__()\n        self.units=units\n        self.attention = Attention_model(self.units) #iniitalise your Attention model with units\n        self.embed = tf.keras.layers.Embedding(vocab_size, embedding_dim)#build your Embedding layer\n        self.gru = tf.keras.layers.GRU(self.units,return_sequences=True,return_state=True,recurrent_initializer='glorot_uniform')\n        self.d1 = tf.keras.layers.Dense(self.units)#build your Dense layer\n        self.d2 = tf.keras.layers.Dense(vocab_size)#build your Dense layer\n        \n    '''\nx.shape (64, 1)\nfeatures.shape (64, 64, 256)\nhidden.shape (64, 512)\nembed.shape 1 (64, 1, 256)\nembed.shape 2 (64, 1, 512)\noutput.shape  (64, 1, 512)\nstate.shape   (64, 512)\noutput.shape 1   (64, 1, 512)\noutput.shape 2  (64, 512)\noutput.shape 3  (64, 5001)\n    '''\n    def call(self, x, features, hidden):\n        #print(\"x.shape\",x.shape)\n        #print(\"features.shape\",features.shape)\n        #print(\"hidden.shape\",hidden.shape)\n\n        context_vector, attention_weights = self.attention(features, hidden) # create your context vector & attention weights from attention model\n        embed =  self.embed(x)# embed your input to shape: (batch_size, 1, embedding_dim)\n        #print(\"embed.shape 1\",embed.shape)\n        embed =  tf.concat([tf.expand_dims(context_vector, 1), embed], axis=-1) # Concatenate your input with the context vector from attention layer. Shape: (batch_size, 1, embedding_dim + embedding_dim)\n        #print(\"embed.shape 2\",embed.shape)\n        output,state = self.gru(embed)# Extract the output & hidden state from GRU layer. Output shape : (batch_size, max_length, hidden_size)\n        #print(\"output.shape \",output.shape)\n        #print(\"state.shape  \",state.shape)\n\n        output = self.d1(output)\n        #print(\"output.shape 1  \",output.shape)\n        output = tf.reshape(output, (-1, output.shape[2])) # shape : (batch_size * max_length, hidden_size)\n        #print(\"output.shape 2 \",output.shape)\n        output = self.d2(output) # shape : (batch_size * max_length, vocab_size)\n        #print(\"output.shape 3 \",output.shape)\n        return output, state, attention_weights\n    \n    def init_state(self, batch_size):\n        return tf.zeros((batch_size, self.units))\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"decoder=Decoder(embedding_dim, units, vocab_size)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# predictions from Decoder is equal to size of vocabulary (later highest valued word can be taken from list).\n\nfeatures=encoder(sample_img_batch)\n\nhidden = decoder.init_state(batch_size = sample_cap_batch.shape[0])\ndec_input = tf.expand_dims([tokenizer.word_index['<start>']] * sample_cap_batch.shape[0], 1) # you just send <start> keyword to start the prediction\n# after multiplying with batch, so entire batch size can be sent.\n\npredictions, hidden_out, attention_weights= decoder(dec_input, features, hidden)\nprint('Feature shape from Encoder: {}'.format(features.shape)) #(batch, 8*8, embed_dim)\nprint('Predcitions shape from Decoder: {}'.format(predictions.shape)) #(batch,vocab_size)\nprint('Attention weights shape from Decoder: {}'.format(attention_weights.shape)) #(batch, 8*8, embed_dim)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 7. Model Training.","metadata":{}},{"cell_type":"code","source":"optimizer = tf.keras.optimizers.Adam() #define the optimizer\nloss_object = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True, reduction='none')#define your loss object","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def loss_function(real, pred):\n    mask = tf.math.logical_not(tf.math.equal(real, 0))\n    loss_ = loss_object(real, pred)\n\n    mask = tf.cast(mask, dtype=loss_.dtype)\n    loss_ *= mask\n\n    return tf.reduce_mean(loss_)\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"checkpoint_path = os.path.join(\"checkpoints\",\"train\")\nckpt = tf.train.Checkpoint(encoder=encoder,\n                           decoder=decoder,\n                           optimizer = optimizer)\nckpt_manager = tf.train.CheckpointManager(ckpt, checkpoint_path, max_to_keep=5)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"start_epoch = 0\nif ckpt_manager.latest_checkpoint:\n    start_epoch = int(ckpt_manager.latest_checkpoint.split('-')[-1])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* Teacher forcing is a technique where the target/real word is passed as the next input to the decoder instead of previous prediciton.","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n\n    @tf.function\n    def train_step(img_tensor, target):\n        loss = 0\n        hidden = decoder.init_state(batch_size=target.shape[0])\n        dec_input = tf.expand_dims([tokenizer.word_index['<start>']] * target.shape[0], 1)\n\n        with tf.GradientTape() as tape:\n            #write your code here to do the training steps\n            features = encoder(img_tensor)\n            for i in range(1,target.shape[1]): # we want to predict till we reach the same number of words as target has. \n                predictions,hidden,_ = decoder(dec_input, features, hidden)\n                loss += loss_function(target[:,i], predictions)\n                # using teacher forcing\n                dec_input = tf.expand_dims(target[:, i], 1)\n\n            avg_loss = (loss / int(target.shape[1]))\n            trainable_variables = encoder.trainable_variables + decoder.trainable_variables\n            gradients = tape.gradient(loss, trainable_variables)\n            optimizer.apply_gradients(zip(gradients, trainable_variables))\n            '''\n            These lines perform backpropagation to update the model's weights.\n\n            It first concatenates the trainable variables of both the encoder and decoder.\n\n            Then, it computes the gradients of the loss with respect to these variables.\n\n            Finally, it applies the computed gradients using an optimizer.\n\n            '''\n\n        return loss, avg_loss","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* While creating the test step for your model, you will pass your previous prediciton as the next input to the decoder.","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n\n    @tf.function\n    def test_step(img_tensor, target):\n        loss = 0\n        hidden = decoder.init_state(batch_size=target.shape[0])\n        dec_input = tf.expand_dims([tokenizer.word_index['<start>']] * target.shape[0], 1)\n\n        with tf.GradientTape() as tape:\n        #write your code here to do the training steps\n            features = encoder(img_tensor)\n            for i in range(1,target.shape[1]):\n                predictions,hidden,_ = decoder(dec_input,features,hidden)\n                loss += loss_function(target[:,i], predictions)\n                predicted_id = tf.argmax(predictions,1)\n                dec_input = tf.expand_dims(predicted_id, 1)\n\n        avg_loss = (loss / int(target.shape[1]))\n\n        return loss, avg_loss\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def test_loss_cal(test_dataset):\n    total_loss = 0\n\n    for (batch,(img_tensor,target)) in enumerate(test_dataset):\n        batch_loss,t_loss = test_step(img_tensor,target)\n        total_loss += t_loss\n    return total_loss/test_num_steps\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Train Loop","metadata":{}},{"cell_type":"code","source":"import time\nloss_plot = []\ntest_loss_plot = []\nEPOCHS = 15\n\nbest_test_loss=100\nfor epoch in tqdm(range(EPOCHS)):\n    start = time.time()\n    total_loss = 0\n\n    for (batch, (img_tensor, target)) in enumerate(train_dataset):\n        batch_loss, t_loss = train_step(img_tensor, target)\n        total_loss += t_loss\n        avg_train_loss=total_loss / train_num_steps\n\n    loss_plot.append(avg_train_loss)    \n    test_loss = test_loss_cal(test_dataset)\n    test_loss_plot.append(test_loss)\n\n    print ('For epoch: {}, the train loss is {:.3f}, & test loss is {:.3f}'.format(epoch+1,avg_train_loss,test_loss))\n    print ('Time taken for 1 epoch {} sec\\n'.format(time.time() - start))\n\n    if test_loss < best_test_loss:\n        print('Test loss has been reduced from %.3f to %.3f' % (best_test_loss, test_loss))\n        best_test_loss = test_loss\n        ckpt_manager.save()\n","metadata":{"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.plot(loss_plot)\nplt.plot(test_loss_plot)\nplt.xlabel('Epochs')\nplt.ylabel('Loss')\nplt.title('Loss Plot')\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### NOTE: \nThe presence of teacher forcing leads to a distinction between the train and test steps, resulting in a scenario where the train loss is decreasing while the test loss is not being reduced. It should be noted that in this context, a direct comparison between train and test results is not applicable, as the approaches used for each are distinct. This does not imply overfitting of the model","metadata":{}},{"cell_type":"markdown","source":"# 8.  Model Evaluation\nFor this project, we will employ two methods to assess the model: greedy search and beam search. We will conduct testing with sample data and evaluate using the BLEU score.\nGreedy Search:\nGreedy search is a decoding algorithm commonly used in Natural Language Processing (NLP) for generating sequences, such as sentences or translations.\nIt works by selecting the most probable next word at each step of sequence generation based on the probability distribution predicted by the model.\nGreedy search is computationally efficient as it only considers the most likely option at each step. However, it may lead to suboptimal results since it doesn't consider future context.\n\nBeam Search:\nBeam search is another decoding algorithm used in NLP, especially in tasks like machine translation or text generation.\nUnlike greedy search, beam search keeps track of multiple candidate sequences (known as the \"beam width\") at each step. It explores a broader range of possibilities.\nAt each step, it scores and keeps the top candidates based on their likelihood. This allows beam search to consider a wider context and potentially find better sequences compared to greedy search.\n\nBLEU Score:\nBLEU (Bilingual Evaluation Understudy) is a metric used to evaluate the quality of machine-translated text. It can also be used for other NLP tasks like text summarization and generation.\nBLEU measures the similarity between a generated text and one or more reference texts (human-generated). It does this by comparing n-grams (sequences of n words) in the generated and reference texts.\nThe BLEU score ranges from 0 to 1, where higher scores indicate better agreement with the reference texts. However, it's important to note that BLEU is not always a perfect indicator of the quality of the generated text, and it has its limitations.","metadata":{}},{"cell_type":"markdown","source":"### Greedy Search","metadata":{}},{"cell_type":"code","source":"attention_features_shape = 64\n\ndef evaluate(image):\n    attention_plot = np.zeros((max_sequence_len, attention_features_shape))\n\n    hidden = decoder.init_state(batch_size=1)\n\n    temp_input = tf.expand_dims(preprocess_image(image)[0], 0) #process the input image to desired format before extracting features\n    img_tensor_val = image_features_extract_model(temp_input)# Extract features using our feature extraction model\n    img_tensor_val = tf.reshape(img_tensor_val, (img_tensor_val.shape[0], -1, img_tensor_val.shape[3]))\n\n    features = encoder(img_tensor_val)# extract the features by passing the input to encoder\n\n    dec_input = tf.expand_dims([tokenizer.word_index['<start>']], 0)\n    result = []\n\n    for i in range(max_sequence_len):\n        predictions, hidden, attention_weights = decoder(dec_input,\n                                                         features,\n                                                         hidden)# get the output from decoder\n\n        attention_plot[i] = tf.reshape(attention_weights, (-1, )).numpy()\n\n        predicted_id = tf.random.categorical(predictions, 1)[0][0].numpy() # extract the predicted id(embedded value) which carries the max value\n        result.append(tokenizer.index_word[predicted_id])\n        #map the id to the word from tokenizer and append the value to the result list\n\n        if tokenizer.index_word[predicted_id] == '<end>':\n            return result, attention_plot, predictions\n\n        dec_input = tf.expand_dims([predicted_id], 0)\n\n    attention_plot = attention_plot[:len(result), :]\n    return result, attention_plot, predictions\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Beam Search","metadata":{}},{"cell_type":"code","source":"def beam_evaluate(image, beam_index = 3):\n\n    #write your code to evaluate the result using beam search\n    start = [tokenizer.word_index['<start>']]\n    result = [[start, 0.0]]\n\n    attention_plot = np.zeros((max_sequence_len, attention_features_shape))\n\n    hidden = decoder.init_state(batch_size=1)\n\n    temp_input = tf.expand_dims(preprocess_image(image)[0], 0)\n    img_tensor_val = image_features_extract_model(temp_input)\n    img_tensor_val = tf.reshape(img_tensor_val, (img_tensor_val.shape[0], -1, img_tensor_val.shape[3]))\n\n    features = encoder(img_tensor_val)\n\n    dec_input = tf.expand_dims([tokenizer.word_index['<start>']], 0)\n\n    while len(result[0][0]) < max_sequence_len:\n        temp = []\n        for i, s in enumerate(result):\n            predictions, hidden, attention_weights = decoder(dec_input, features, hidden)\n            attention_plot[i] = tf.reshape(attention_weights, (-1, )).numpy()\n            word_preds = np.argsort(predictions[0])[-beam_index:]\n\n            for w in word_preds:\n                next_cap, prob = s[0][:], s[1]\n                next_cap.append(w)\n\n                prob += np.log(predictions[0][w])\n\n                temp.append([next_cap, prob])\n        result = temp\n        result = sorted(result, reverse=False, key=lambda l: l[1])\n        result = result[-beam_index:]\n\n\n        predicted_id = result[-1]\n        pred_list = predicted_id[0]\n\n        prd_id = pred_list[-1]\n        if(prd_id!=3):\n            dec_input = tf.expand_dims([prd_id], 0)  \n        else:\n            break\n\n\n    result2 = result[-1][0]\n\n    intermediate_caption = [tokenizer.index_word[i] for i in result2]\n    final_caption = []\n    for i in intermediate_caption:\n        if i != '<end>':\n            final_caption.append(i)\n\n        else:\n            break\n\n    attention_plot = attention_plot[:len(result), :]\n    final_caption = ' '.join(final_caption[1:])\n    return final_caption\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_image = \"/kaggle/working/flickr8k/Images/3106883334_419f3fb16f.jpg\"\ncaptions=beam_evaluate(test_image)\nprint(captions)\nImage.open(test_image)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plot_attmap(caption, weights, image):\n\n    fig = plt.figure(figsize=(10, 10))\n    temp_img = np.array(Image.open(image))\n    \n    len_cap = len(caption)\n    for cap in range(len_cap):\n        weights_img = np.reshape(weights[cap], (8,8)) # This line reshapes the attention weights associated with the current word. It is assumed that each set of weights corresponds to an 8x8 grid. as we have maintained 8*8 since the start.\n        weights_img = np.array(Image.fromarray(weights_img).resize((224, 224), Image.LANCZOS)) #This resizes the attention weights image to be compatible with the size of the original image. It uses the Lanczos resampling method for high-quality resizing.\n        \n        ax = fig.add_subplot(len_cap//2, len_cap//2, cap+1)\n        ax.set_title(caption[cap], fontsize=15)\n        \n        img=ax.imshow(temp_img)\n        \n        ax.imshow(weights_img, cmap='gist_heat', alpha=0.6,extent=img.get_extent())\n        ax.axis('off')\n    plt.subplots_adjust(hspace=0.2, wspace=0.2)\n    plt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from nltk.translate.bleu_score import sentence_bleu","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def filt_text(text):\n    filt=['<start>','<unk>','<end>'] \n    temp= text.split()\n    [temp.remove(j) for k in filt for j in temp if k==j]\n    text=' '.join(temp)\n    return text","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"captions=beam_evaluate(test_image)\nprint(captions)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def test_caption_generation(img_test):\n\n\n    rid = np.random.randint(0, len(img_test))\n    test_image = img_test[rid]\n    #test_image = './images/413231421_43833a11f5.jpg'\n    #real_caption = '<start> black dog is digging in the snow <end>'\n\n    real_caption = ' '.join(\n        tokenizer.index_word[i] for i in cap_test[rid] if i not in [0]\n    )\n\n    result, attention_plot, pred_test = evaluate(test_image)\n\n\n    real_caption=filt_text(real_caption)      \n\n\n    pred_caption=' '.join(result).rsplit(' ', 1)[0]\n\n    real_appn = [real_caption.split()]\n    reference = real_appn\n    candidate = pred_caption.split()\n\n    print ('Real Caption:', real_caption)\n    print ('Prediction Caption:', pred_caption)\n    \n    # to audio\n    #convert_to_audio(pred_caption)\n    \n    score1 = sentence_bleu(reference, candidate, weights=(1,0,0,0))\n    score2 = sentence_bleu(reference, candidate, weights=(0,1,0,0))\n    score3 = sentence_bleu(reference, candidate, weights=(0,0,1,0))\n    score4 = sentence_bleu(reference, candidate, weights=(0,0,0,1))\n    print(\"\\nBELU score: \")\n    print(f\"Individual 1-gram: {score1*100}\")\n    print(f\"Individual 2-gram: {score2*100}\")\n    print(f\"Individual 3-gram: {score3*100}\")\n    print(f\"Individual 4-gram: {score4*100}\")\n\n    plot_attmap(result, attention_plot, test_image)\n\n\n    Image.open(test_image)\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_caption_generation(path_test)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def Prediction_Unknown_data(test_image):\n    #Testing on test image\n    openImg = test_image\n    result, attention_plot,pred_test = evaluate(test_image)\n    pred_caption=' '.join(result).rsplit(' ', 1)[0]\n\n    candidate = pred_caption.split()\n\n    print ('Prediction Caption:', pred_caption)\n    print ('')\n    newsize = (800, 800)\n    im = Image.open(openImg).resize(newsize)\n    width, height = im.size\n    print(width,height)\n    div=3\n    if width > 3000:\n        div=10\n    im = im.resize((width//div, height//div))\n    \n    return im\n    \n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Prediction_Unknown_data(path_test[30])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# end","metadata":{}}]}