language_detection.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "language detection.ipynb",
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "**Langauge detection with the help of multinomial Naive Bayes**\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "dRRAZQ-e6WZi"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "*Importing dataset from the github repository*\n",
        "\n",
        "**Dataset components** \n",
        "- contains data about 22 popular languages and contains 1000 sentences in each of the languages\n",
        "\n"
      ],
      "metadata": {
        "id": "uiX8MvY36kNs"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-Gew63Fd0chb",
        "outputId": "49d4eae0-5f1b-489f-f9e3-83a98c018ffe"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "                                                Text  language\n",
            "0  klement gottwaldi surnukeha palsameeriti ning ...  Estonian\n",
            "1  sebes joseph pereira thomas  på eng the jesuit...   Swedish\n",
            "2  ถนนเจริญกรุง อักษรโรมัน thanon charoen krung เ...      Thai\n",
            "3  விசாகப்பட்டினம் தமிழ்ச்சங்கத்தை இந்துப் பத்திர...     Tamil\n",
            "4  de spons behoort tot het geslacht haliclona en...     Dutch\n"
          ]
        }
      ],
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "from sklearn.feature_extraction.text import CountVectorizer\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.naive_bayes import MultinomialNB\n",
        "data = pd.read_csv(\"https://raw.githubusercontent.com/amankharwal/Website-data/master/dataset.csv\")\n",
        "print(data.head())"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**checking for null values in dataset** \n"
      ],
      "metadata": {
        "id": "VJFNOQrJ7W4S"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "data.isnull().sum()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uSjICsuz3PUm",
        "outputId": "a9154212-3188-4480-bef0-9be00e6bbeea"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Text        0\n",
              "language    0\n",
              "dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**languages containing in dataset** "
      ],
      "metadata": {
        "id": "cXm1ahVU7aE6"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "data[\"language\"].value_counts()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "eEB_0APr5ejn",
        "outputId": "7d099860-70da-4962-a9c7-3faf17fe578c"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Estonian      1000\n",
              "Swedish       1000\n",
              "English       1000\n",
              "Russian       1000\n",
              "Romanian      1000\n",
              "Persian       1000\n",
              "Pushto        1000\n",
              "Spanish       1000\n",
              "Hindi         1000\n",
              "Korean        1000\n",
              "Chinese       1000\n",
              "French        1000\n",
              "Portugese     1000\n",
              "Indonesian    1000\n",
              "Urdu          1000\n",
              "Latin         1000\n",
              "Turkish       1000\n",
              "Japanese      1000\n",
              "Dutch         1000\n",
              "Tamil         1000\n",
              "Thai          1000\n",
              "Arabic        1000\n",
              "Name: language, dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Splitting dataset into training and test sets**"
      ],
      "metadata": {
        "id": "3vjAJCZK7hIq"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "x = np.array(data[\"Text\"])\n",
        "y = np.array(data[\"language\"])\n",
        "\n",
        "cv = CountVectorizer()\n",
        "X = cv.fit_transform(x)\n",
        "X_train, X_test, y_train, y_test = train_test_split(X, y, \n",
        "                                                    test_size=0.33, \n",
        "                                                    random_state=42)"
      ],
      "metadata": {
        "id": "YuQyOmE_5lJ8"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# **Using Multinomial Naïve**"
      ],
      "metadata": {
        "id": "HfcAipxw72Pc"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "As this is a problem of multiclass classification, so I will be using the Multinomial Naïve Bayes algorithm to train the language detection model as this algorithm always performs very well on the problems based on multiclass classification"
      ],
      "metadata": {
        "id": "1MFdLe2-7_0D"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "model = MultinomialNB()\n",
        "model.fit(X_train,y_train)\n",
        "model.score(X_test,y_test)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "T7p7BBQt5m-4",
        "outputId": "a749522c-ec06-4672-e029-b87593f68fbb"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.953168044077135"
            ]
          },
          "metadata": {},
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Testing of the model**"
      ],
      "metadata": {
        "id": "yNoMmVxK8OGL"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "user = input(\"Enter a Text: \")\n",
        "data = cv.transform([user]).toarray()\n",
        "output = model.predict(data)\n",
        "print(output)"
      ],
      "metadata": {
        "id": "8wEnCq7p5o6c",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e1052da8-2dff-4461-ddc3-7ecfa636510e"
      },
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Enter a Text: तुम क्या कर रहे हो\n",
            "['Hindi']\n"
          ]
        }
      ]
    }
  ]
}