Text_Processing_StopWords.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyOogzdzi4Qssk8pkVXZ7rK6"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xk3Ub_J-kb9Z",
        "outputId": "9fbd9df2-c5ec-4cf0-8f93-7ac3e988d62c"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: nltk in /usr/local/lib/python3.10/dist-packages (3.8.1)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.10/dist-packages (from nltk) (8.1.7)\n",
            "Requirement already satisfied: joblib in /usr/local/lib/python3.10/dist-packages (from nltk) (1.4.2)\n",
            "Requirement already satisfied: regex>=2021.8.3 in /usr/local/lib/python3.10/dist-packages (from nltk) (2024.5.15)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from nltk) (4.66.5)\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Package punkt is already up-to-date!\n",
            "[nltk_data] Downloading package wordnet to /root/nltk_data...\n",
            "[nltk_data]   Package wordnet is already up-to-date!\n",
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Package stopwords is already up-to-date!\n"
          ]
        }
      ],
      "source": [
        "\n",
        "!pip install nltk\n",
        "import nltk\n",
        "nltk.download('punkt')\n",
        "nltk.download('wordnet')\n",
        "stopword=nltk.download('stopwords')\n",
        "from nltk.stem import PorterStemmer,SnowballStemmer,RegexpStemmer\n",
        "porter_stemmer= PorterStemmer()\n",
        "from nltk.stem import WordNetLemmatizer\n",
        "lemmatizer=WordNetLemmatizer()\n",
        "from nltk.tokenize import sent_tokenize,word_tokenize\n",
        "from nltk.corpus import stopwords"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "paragraph=\"\"\"\n",
        "A paragraph is a series of sentences that are organized and coherent, and are all related to a single topic. Almost every piece of writing you do that is longer than a few sentences should be organized into paragraphs. This is because paragraphs show a reader where the subdivisions of an essay begin and end, and thus help the reader see the organization of the essay and grasp its main points.\n",
        "\n",
        "Paragraphs can contain many different kinds of information. A paragraph could contain a series of brief examples or a single long illustration of a general point. It might describe a place, character, or process; narrate a series of events; compare or contrast two or more things; classify items into categories; or describe causes and effects. Regardless of the kind of information they contain, all paragraphs share certain characteristics. One of the most important of these is a topic sentence.\n",
        "\n",
        "TOPIC SENTENCES\n",
        "A well-organized paragraph supports or develops a single controlling idea, which is expressed in a sentence called the topic sentence. A topic sentence has several important functions: it substantiates or supports an essay’s thesis statement; it unifies the content of a paragraph and directs the order of the sentences; and it advises the reader of the subject to be discussed and how the paragraph will discuss it. Readers generally look to the first few sentences in a paragraph to determine the subject and perspective of the paragraph. That’s why it’s often best to put the topic sentence at the very beginning of the paragraph. In some cases, however, it’s more effective to place another sentence before the topic sentence—for example, a sentence linking the current paragraph to the previous one, or one providing background information.\n",
        "\n",
        "Although most paragraphs should have a topic sentence, there are a few situations when a paragraph might not need a topic sentence. For example, you might be able to omit a topic sentence in a paragraph that narrates a series of events, if a paragraph continues developing an idea that you introduced (with a topic sentence) in the previous paragraph, or if all the sentences and details in a paragraph clearly refer—perhaps indirectly—to a main point. The vast majority of your paragraphs, however, should have a topic sentence.\n",
        "\n",
        "PARAGRAPH STRUCTURE\n",
        "Most paragraphs in an essay have a three-part structure—introduction, body, and conclusion. You can see this structure in paragraphs whether they are narrating, describing, comparing, contrasting, or analyzing information. Each part of the paragraph plays an important role in communicating your meaning to your reader.\n",
        "\n",
        "Introduction: the first section of a paragraph; should include the topic sentence and any other sentences at the beginning of the paragraph that give background information or provide a transition.\n",
        "\n",
        "Body: follows the introduction; discusses the controlling idea, using facts, arguments, analysis, examples, and other information.\n",
        "\n",
        "Conclusion: the final section; summarizes the connections between the information discussed in the body of the paragraph and the paragraph’s controlling idea.\n",
        "\n",
        "The following paragraph illustrates this pattern of organization. In this paragraph the topic sentence and concluding sentence (CAPITALIZED) both help the reader keep the paragraph’s main point in mind.\n",
        "\n",
        "SCIENTISTS HAVE LEARNED TO SUPPLEMENT THE SENSE OF SIGHT IN NUMEROUS WAYS. In front of the tiny pupil of the eye they put, on Mount Palomar, a great monocle 200 inches in diameter, and with it see 2000 times farther into the depths of space. Or they look through a small pair of lenses arranged as a microscope into a drop of water or blood, and magnify by as much as 2000 diameters the living creatures there, many of which are among man’s most dangerous enemies. Or, if we want to see distant happenings on earth, they use some of the previously wasted electromagnetic waves to carry television images which they re-create as light by whipping tiny crystals on a screen with electrons in a vacuum. Or they can bring happenings of long ago and far away as colored motion pictures, by arranging silver atoms and color-absorbing molecules to force light waves into the patterns of original reality. Or if we want to see into the center of a steel casting or the chest of an injured child, they send the information on a beam of penetrating short-wave X rays, and then convert it back into images we can see on a screen or photograph. THUS ALMOST EVERY TYPE OF ELECTROMAGNETIC RADIATION YET DISCOVERED HAS BEEN USED TO EXTEND OUR SENSE OF SIGHT IN SOME WAY.\n",
        "\n",
        "George Harrison, “Faith and the Scientist”\n",
        "\n",
        "\"\"\"\n"
      ],
      "metadata": {
        "id": "_DgXBOIck0LO"
      },
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "stopwords.words(\"english\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "collapsed": true,
        "id": "1cgVlCUYmOIi",
        "outputId": "3adffe86-4a0f-4b67-8f6b-eb5d0133de5c"
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['i',\n",
              " 'me',\n",
              " 'my',\n",
              " 'myself',\n",
              " 'we',\n",
              " 'our',\n",
              " 'ours',\n",
              " 'ourselves',\n",
              " 'you',\n",
              " \"you're\",\n",
              " \"you've\",\n",
              " \"you'll\",\n",
              " \"you'd\",\n",
              " 'your',\n",
              " 'yours',\n",
              " 'yourself',\n",
              " 'yourselves',\n",
              " 'he',\n",
              " 'him',\n",
              " 'his',\n",
              " 'himself',\n",
              " 'she',\n",
              " \"she's\",\n",
              " 'her',\n",
              " 'hers',\n",
              " 'herself',\n",
              " 'it',\n",
              " \"it's\",\n",
              " 'its',\n",
              " 'itself',\n",
              " 'they',\n",
              " 'them',\n",
              " 'their',\n",
              " 'theirs',\n",
              " 'themselves',\n",
              " 'what',\n",
              " 'which',\n",
              " 'who',\n",
              " 'whom',\n",
              " 'this',\n",
              " 'that',\n",
              " \"that'll\",\n",
              " 'these',\n",
              " 'those',\n",
              " 'am',\n",
              " 'is',\n",
              " 'are',\n",
              " 'was',\n",
              " 'were',\n",
              " 'be',\n",
              " 'been',\n",
              " 'being',\n",
              " 'have',\n",
              " 'has',\n",
              " 'had',\n",
              " 'having',\n",
              " 'do',\n",
              " 'does',\n",
              " 'did',\n",
              " 'doing',\n",
              " 'a',\n",
              " 'an',\n",
              " 'the',\n",
              " 'and',\n",
              " 'but',\n",
              " 'if',\n",
              " 'or',\n",
              " 'because',\n",
              " 'as',\n",
              " 'until',\n",
              " 'while',\n",
              " 'of',\n",
              " 'at',\n",
              " 'by',\n",
              " 'for',\n",
              " 'with',\n",
              " 'about',\n",
              " 'against',\n",
              " 'between',\n",
              " 'into',\n",
              " 'through',\n",
              " 'during',\n",
              " 'before',\n",
              " 'after',\n",
              " 'above',\n",
              " 'below',\n",
              " 'to',\n",
              " 'from',\n",
              " 'up',\n",
              " 'down',\n",
              " 'in',\n",
              " 'out',\n",
              " 'on',\n",
              " 'off',\n",
              " 'over',\n",
              " 'under',\n",
              " 'again',\n",
              " 'further',\n",
              " 'then',\n",
              " 'once',\n",
              " 'here',\n",
              " 'there',\n",
              " 'when',\n",
              " 'where',\n",
              " 'why',\n",
              " 'how',\n",
              " 'all',\n",
              " 'any',\n",
              " 'both',\n",
              " 'each',\n",
              " 'few',\n",
              " 'more',\n",
              " 'most',\n",
              " 'other',\n",
              " 'some',\n",
              " 'such',\n",
              " 'no',\n",
              " 'nor',\n",
              " 'not',\n",
              " 'only',\n",
              " 'own',\n",
              " 'same',\n",
              " 'so',\n",
              " 'than',\n",
              " 'too',\n",
              " 'very',\n",
              " 's',\n",
              " 't',\n",
              " 'can',\n",
              " 'will',\n",
              " 'just',\n",
              " 'don',\n",
              " \"don't\",\n",
              " 'should',\n",
              " \"should've\",\n",
              " 'now',\n",
              " 'd',\n",
              " 'll',\n",
              " 'm',\n",
              " 'o',\n",
              " 're',\n",
              " 've',\n",
              " 'y',\n",
              " 'ain',\n",
              " 'aren',\n",
              " \"aren't\",\n",
              " 'couldn',\n",
              " \"couldn't\",\n",
              " 'didn',\n",
              " \"didn't\",\n",
              " 'doesn',\n",
              " \"doesn't\",\n",
              " 'hadn',\n",
              " \"hadn't\",\n",
              " 'hasn',\n",
              " \"hasn't\",\n",
              " 'haven',\n",
              " \"haven't\",\n",
              " 'isn',\n",
              " \"isn't\",\n",
              " 'ma',\n",
              " 'mightn',\n",
              " \"mightn't\",\n",
              " 'mustn',\n",
              " \"mustn't\",\n",
              " 'needn',\n",
              " \"needn't\",\n",
              " 'shan',\n",
              " \"shan't\",\n",
              " 'shouldn',\n",
              " \"shouldn't\",\n",
              " 'wasn',\n",
              " \"wasn't\",\n",
              " 'weren',\n",
              " \"weren't\",\n",
              " 'won',\n",
              " \"won't\",\n",
              " 'wouldn',\n",
              " \"wouldn't\"]"
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "sentences=sent_tokenize(paragraph)"
      ],
      "metadata": {
        "id": "rvSInjKin9E0"
      },
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "sentences"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "sOygzcRSoBj5",
        "outputId": "608030a0-f07c-48d7-edbd-abcc46adf6f5"
      },
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[' \\nA paragraph is a series of sentences that are organized and coherent, and are all related to a single topic.',\n",
              " 'Almost every piece of writing you do that is longer than a few sentences should be organized into paragraphs.',\n",
              " 'This is because paragraphs show a reader where the subdivisions of an essay begin and end, and thus help the reader see the organization of the essay and grasp its main points.',\n",
              " 'Paragraphs can contain many different kinds of information.',\n",
              " 'A paragraph could contain a series of brief examples or a single long illustration of a general point.',\n",
              " 'It might describe a place, character, or process; narrate a series of events; compare or contrast two or more things; classify items into categories; or describe causes and effects.',\n",
              " 'Regardless of the kind of information they contain, all paragraphs share certain characteristics.',\n",
              " 'One of the most important of these is a topic sentence.',\n",
              " 'TOPIC SENTENCES\\nA well-organized paragraph supports or develops a single controlling idea, which is expressed in a sentence called the topic sentence.',\n",
              " 'A topic sentence has several important functions: it substantiates or supports an essay’s thesis statement; it unifies the content of a paragraph and directs the order of the sentences; and it advises the reader of the subject to be discussed and how the paragraph will discuss it.',\n",
              " 'Readers generally look to the first few sentences in a paragraph to determine the subject and perspective of the paragraph.',\n",
              " 'That’s why it’s often best to put the topic sentence at the very beginning of the paragraph.',\n",
              " 'In some cases, however, it’s more effective to place another sentence before the topic sentence—for example, a sentence linking the current paragraph to the previous one, or one providing background information.',\n",
              " 'Although most paragraphs should have a topic sentence, there are a few situations when a paragraph might not need a topic sentence.',\n",
              " 'For example, you might be able to omit a topic sentence in a paragraph that narrates a series of events, if a paragraph continues developing an idea that you introduced (with a topic sentence) in the previous paragraph, or if all the sentences and details in a paragraph clearly refer—perhaps indirectly—to a main point.',\n",
              " 'The vast majority of your paragraphs, however, should have a topic sentence.',\n",
              " 'PARAGRAPH STRUCTURE\\nMost paragraphs in an essay have a three-part structure—introduction, body, and conclusion.',\n",
              " 'You can see this structure in paragraphs whether they are narrating, describing, comparing, contrasting, or analyzing information.',\n",
              " 'Each part of the paragraph plays an important role in communicating your meaning to your reader.',\n",
              " 'Introduction: the first section of a paragraph; should include the topic sentence and any other sentences at the beginning of the paragraph that give background information or provide a transition.',\n",
              " 'Body: follows the introduction; discusses the controlling idea, using facts, arguments, analysis, examples, and other information.',\n",
              " 'Conclusion: the final section; summarizes the connections between the information discussed in the body of the paragraph and the paragraph’s controlling idea.',\n",
              " 'The following paragraph illustrates this pattern of organization.',\n",
              " 'In this paragraph the topic sentence and concluding sentence (CAPITALIZED) both help the reader keep the paragraph’s main point in mind.',\n",
              " 'SCIENTISTS HAVE LEARNED TO SUPPLEMENT THE SENSE OF SIGHT IN NUMEROUS WAYS.',\n",
              " 'In front of the tiny pupil of the eye they put, on Mount Palomar, a great monocle 200 inches in diameter, and with it see 2000 times farther into the depths of space.',\n",
              " 'Or they look through a small pair of lenses arranged as a microscope into a drop of water or blood, and magnify by as much as 2000 diameters the living creatures there, many of which are among man’s most dangerous enemies.',\n",
              " 'Or, if we want to see distant happenings on earth, they use some of the previously wasted electromagnetic waves to carry television images which they re-create as light by whipping tiny crystals on a screen with electrons in a vacuum.',\n",
              " 'Or they can bring happenings of long ago and far away as colored motion pictures, by arranging silver atoms and color-absorbing molecules to force light waves into the patterns of original reality.',\n",
              " 'Or if we want to see into the center of a steel casting or the chest of an injured child, they send the information on a beam of penetrating short-wave X rays, and then convert it back into images we can see on a screen or photograph.',\n",
              " 'THUS ALMOST EVERY TYPE OF ELECTROMAGNETIC RADIATION YET DISCOVERED HAS BEEN USED TO EXTEND OUR SENSE OF SIGHT IN SOME WAY.',\n",
              " 'George Harrison, “Faith and the Scientist”']"
            ]
          },
          "metadata": {},
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "for i in range(len(sentences)):\n",
        "  words=word_tokenize(sentences[i].lower())\n",
        "  words=[lemmatizer.lemmatize(word,pos='v') for word in words if word not in set(stopwords.words(\"english\"))]\n",
        "  sentences[i]=' '.join(words)"
      ],
      "metadata": {
        "id": "W_nWOqhjnpkp"
      },
      "execution_count": 22,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "sentences"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ECIK8Gt9qmrx",
        "outputId": "e3df254e-5aa3-4722-da24-5ba179ce055c"
      },
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['paragraph series sentences organized coherent , related single topic .',\n",
              " 'almost every piece writing longer sentences organized paragraphs .',\n",
              " 'paragraphs show reader subdivisions essay begin end , thus help reader see organization essay grasp main points .',\n",
              " 'paragraphs contain many different kinds information .',\n",
              " 'paragraph could contain series brief examples single long illustration general point .',\n",
              " 'might describe place , character , process ; narrate series events ; compare contrast two things ; classify items categories ; describe causes effects .',\n",
              " 'regardless kind information contain , paragraphs share certain characteristics .',\n",
              " 'one important topic sentence .',\n",
              " 'topic sentences well-organized paragraph supports develops single controlling idea , expressed sentence called topic sentence .',\n",
              " 'topic sentence several important functions : substantiates supports essay ’ thesis statement ; unifies content paragraph directs order sentences ; advises reader subject discussed paragraph discuss .',\n",
              " 'readers generally look first sentences paragraph determine subject perspective paragraph .',\n",
              " '’ ’ often best put topic sentence beginning paragraph .',\n",
              " 'cases , however , ’ effective place another sentence topic sentence—for example , sentence linking current paragraph previous one , one providing background information .',\n",
              " 'although paragraphs topic sentence , situations paragraph might need topic sentence .',\n",
              " 'example , might able omit topic sentence paragraph narrates series events , paragraph continues developing idea introduced ( topic sentence ) previous paragraph , sentences details paragraph clearly refer—perhaps indirectly—to main point .',\n",
              " 'vast majority paragraphs , however , topic sentence .',\n",
              " 'paragraph structure paragraphs essay three-part structure—introduction , body , conclusion .',\n",
              " 'see structure paragraphs whether narrating , describing , comparing , contrasting , analyzing information .',\n",
              " 'part paragraph plays important role communicating meaning reader .',\n",
              " 'introduction : first section paragraph ; include topic sentence sentences beginning paragraph give background information provide transition .',\n",
              " 'body : follows introduction ; discusses controlling idea , using facts , arguments , analysis , examples , information .',\n",
              " 'conclusion : final section ; summarizes connections information discussed body paragraph paragraph ’ controlling idea .',\n",
              " 'following paragraph illustrates pattern organization .',\n",
              " 'paragraph topic sentence concluding sentence ( capitalized ) help reader keep paragraph ’ main point mind .',\n",
              " 'scientists learned supplement sense sight numerous ways .',\n",
              " 'front tiny pupil eye put , mount palomar , great monocle 200 inches diameter , see 2000 times farther depths space .',\n",
              " 'look small pair lenses arranged microscope drop water blood , magnify much 2000 diameters living creatures , many among man ’ dangerous enemies .',\n",
              " ', want see distant happenings earth , use previously wasted electromagnetic waves carry television images re-create light whipping tiny crystals screen electrons vacuum .',\n",
              " 'bring happenings long ago far away colored motion pictures , arranging silver atoms color-absorbing molecules force light waves patterns original reality .',\n",
              " 'want see center steel casting chest injured child , send information beam penetrating short-wave x rays , convert back images see screen photograph .',\n",
              " 'thus almost every type electromagnetic radiation yet discovered used extend sense sight way .',\n",
              " 'george harrison , “ faith scientist ”']"
            ]
          },
          "metadata": {},
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "sentences"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "cB0ZbVmRqrev",
        "outputId": "64bbab18-6fa9-4d7d-c8e2-d0f962abcbe3"
      },
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['paragraph series sentence organize coherent , relate single topic .',\n",
              " 'almost every piece write longer sentence organize paragraph .',\n",
              " 'paragraph show reader subdivision essay begin end , thus help reader see organization essay grasp main point .',\n",
              " 'paragraph contain many different kind information .',\n",
              " 'paragraph could contain series brief example single long illustration general point .',\n",
              " 'might describe place , character , process ; narrate series event ; compare contrast two thing ; classify item category ; describe cause effect .',\n",
              " 'regardless kind information contain , paragraph share certain characteristic .',\n",
              " 'one important topic sentence .',\n",
              " 'topic sentence well-organized paragraph support develop single control idea , express sentence call topic sentence .',\n",
              " 'topic sentence several important function : substantiate support essay ’ thesis statement ; unify content paragraph direct order sentence ; advise reader subject discuss paragraph discus .',\n",
              " 'reader generally look first sentence paragraph determine subject perspective paragraph .',\n",
              " '’ ’ often best put topic sentence begin paragraph .',\n",
              " 'case , however , ’ effective place another sentence topic sentence—for example , sentence link current paragraph previous one , one provide background information .',\n",
              " 'although paragraph topic sentence , situation paragraph might need topic sentence .',\n",
              " 'example , might able omit topic sentence paragraph narrate series event , paragraph continue develop idea introduce ( topic sentence ) previous paragraph , sentence detail paragraph clearly refer—perhaps indirectly—to main point .',\n",
              " 'vast majority paragraph , however , topic sentence .',\n",
              " 'paragraph structure paragraph essay three-part structure—introduction , body , conclusion .',\n",
              " 'see structure paragraph whether narrate , describe , compare , contrast , analyze information .',\n",
              " 'part paragraph play important role communicate mean reader .',\n",
              " 'introduction : first section paragraph ; include topic sentence sentence begin paragraph give background information provide transition .',\n",
              " 'body : follow introduction ; discus control idea , use fact , argument , analysis , example , information .',\n",
              " 'conclusion : final section ; summarize connection information discuss body paragraph paragraph ’ control idea .',\n",
              " 'follow paragraph illustrate pattern organization .',\n",
              " 'paragraph topic sentence conclude sentence ( capitalize ) help reader keep paragraph ’ main point mind .',\n",
              " 'scientist learn supplement sense sight numerous way .',\n",
              " 'front tiny pupil eye put , mount palomar , great monocle 200 inch diameter , see 2000 time farther depth space .',\n",
              " 'look small pair lens arrange microscope drop water blood , magnify much 2000 diameter live creature , many among man ’ dangerous enemy .',\n",
              " ', want see distant happen earth , use previously waste electromagnetic wave carry television image re-create light whip tiny crystal screen electron vacuum .',\n",
              " 'bring happen long ago far away color motion picture , arrange silver atom color-absorbing molecule force light wave pattern original reality .',\n",
              " 'want see center steel cast chest injure child , send information beam penetrate short-wave x ray , convert back image see screen photograph .',\n",
              " 'thus almost every type electromagnetic radiation yet discover use extend sense sight way .',\n",
              " 'george harrison , “ faith scientist ”']"
            ]
          },
          "metadata": {},
          "execution_count": 23
        }
      ]
    }
  ]
}