2_2형태소_분석.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "2.2형태소 분석.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "mount_file_id": "1vhuAlTcrUcbqrLAYo5jOot-kV57AWtM2",
      "authorship_tag": "ABX9TyM1hD3RAXgbKwOKWcX2PeiN"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_YUNU9Kh1gcg"
      },
      "source": [
        "# 형태소 분석\n",
        "- 품질 좋은 임베딩을 만들기 위해서는 문장이나 단어의 경계를 컴퓨터에 알려줘야 한다.\n",
        "- 형태소를 잘 나누지 않으면 단어집합의 크기가 커지고, 연산의 비효율 발생\n",
        "- 한국어는 조사와 어미가 발달한 교착어->형태소 분석이 중요하다.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NslmUORklFin"
      },
      "source": [
        "root_path = '/content/drive/MyDrive/2021-1/AI데이터활용교재개발/code'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nI1lxk7f1yw-"
      },
      "source": [
        "## 지도학습 기반 형태소 분석\n",
        " - konlpy 사용하기\n",
        " - 사용자 기반 사전"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f92CMLEY4v1J"
      },
      "source": [
        "### konlpy와 mecab을 사용하기 위한 패키지 설치"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B9DsfJKz1uZp"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cFLxsVQ60g5z"
      },
      "source": [
        "! pip install konlpy\n",
        "\n",
        "# google colab에서 mecab을 사용하기 편하게 만들어주는 shell 파일도 실행시켜 줍니다.\n",
        "! git clone https://github.com/SOMJANG/Mecab-ko-for-Google-Colab.git\n",
        "# ! cd ./Mecab-ko-for-Google-Colab\n",
        "# ! bash ./Mecab-ko-for-Google-Colab/install_mecab-ko_on_colab190912.sh\n",
        "! bash ./Mecab-ko-for-Google-Colab/install_mecab-ko_on_colab_light_210108.sh\n",
        "\n",
        "# shell 파일 출처: https://somjang.tistory.com/entry/Google-Colab에서-Mecab-koMecab-ko-dic-쉽게-사용하기 [솜씨좋은장씨]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Yp4WuqAG49jE"
      },
      "source": [
        "### konlpy 를 사용한 품사 태깅 예시 코드"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BeD8cQIgytFX",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "1db999c5-15ed-430f-fa5f-6886483c1f3e"
      },
      "source": [
        "# NameError: name 'Tagger' is not defined 오류 발생 시 런타임을 재실행 해주세요(ctrl + M)\n",
        "\n",
        "from konlpy.tag import Okt, Komoran, Mecab, Hannanum, Kkma \n",
        "\n",
        "# tokenizer 사용 방법\n",
        "# tokenizer = Komoran()\n",
        "# tokenizer = Okt()\n",
        "# tokenizer = Hannanum()\n",
        "# tokenizer = Kkma()\n",
        "from konlpy.tag import Mecab\n",
        "tokenizer = Mecab()\n",
        "\n",
        "\n",
        "print(tokenizer.morphs(\"아버지가방에들어가신다\"))\n",
        "print(tokenizer.pos(\"아버지가방에들어가신다\"))\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['아버지', '가', '방', '에', '들어가', '신다']\n",
            "[('아버지', 'NNG'), ('가', 'JKS'), ('방', 'NNG'), ('에', 'JKB'), ('들어가', 'VV'), ('신다', 'EP+EC')]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WjM-aVJ2SRmK"
      },
      "source": [
        "import re\n",
        "def tokenize(corpus_fname, output_fname):\n",
        "    tokenizer = Mecab()\n",
        "\n",
        "    with open(corpus_fname, 'r', encoding='utf-8') as f1, \\\n",
        "            open(output_fname, 'w', encoding='utf-8') as f2:\n",
        "        for line in f1:\n",
        "            sentence = line.replace('\\n', '').strip()\n",
        "            tokens = tokenizer.morphs(sentence)\n",
        "            tokenized_sent = ' '.join(post_processing(tokens))\n",
        "            f2.writelines(tokenized_sent + '\\n')\n",
        "\n",
        "def post_processing(tokens):\n",
        "    results = []\n",
        "    for token in tokens:\n",
        "        # 숫자에 공백을 주어서 띄우기\n",
        "        processed_token = [el for el in re.sub(r\"(\\d)\", r\" \\1 \", token).split(\" \") if len(el) > 0]\n",
        "        results.extend(processed_token)\n",
        "    return results\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "l-dGQWBfSohw"
      },
      "source": [
        "print(\"wiki\")\n",
        "corpus_fname = root_path + '/data/processed/processed_wiki_ko.txt'\n",
        "output_fname = root_path + '/data/processed/tokenized/wiki_ko_mecab.txt'\n",
        "tokenize(corpus_fname, output_fname)\n",
        "\n",
        "print(\"naver\")\n",
        "corpus_fname = root_path + '/data/processed/processed_ratings.txt'\n",
        "output_fname = root_path + '/data/processed/tokenized/ratings_mecab.txt'\n",
        "tokenize(corpus_fname, output_fname)\n",
        "\n",
        "print(\"koquard\")\n",
        "corpus_fname = root_path + '/data/processed/processed_korquad.txt'\n",
        "output_fname = root_path + '/data/processed/tokenized/korquad_mecab.txt'\n",
        "tokenize(corpus_fname, output_fname)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SCWYptKqIJ0A"
      },
      "source": [
        "명사만 형태소로 저장하는 파일 생성"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-zqT-9zxIICR"
      },
      "source": [
        "def noun_tokenize(corpus_fname, output_fname):\n",
        "    tokenizer = Mecab()\n",
        "\n",
        "    with open(corpus_fname, 'r', encoding='utf-8') as f1, \\\n",
        "            open(output_fname, 'w', encoding='utf-8') as f2:\n",
        "        for line in f1:\n",
        "            sentence = line.replace('\\n', '').strip()\n",
        "            tokens = tokenizer.nouns(sentence)\n",
        "            long_tokens = []\n",
        "            for t in tokens:\n",
        "              if len(t)>1:\n",
        "                long_tokens.append(t)\n",
        "            if len(long_tokens) >0: \n",
        "              tokenized_sent = ' '.join(post_processing(long_tokens))\n",
        "              f2.writelines(tokenized_sent + '\\n')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Y8CANML5INKO",
        "outputId": "fa147b19-0edd-439f-b8d1-fc59d9409fc1"
      },
      "source": [
        "print(\"naver\")\n",
        "corpus_fname = root_path + '/data/processed/processed_ratings.txt'\n",
        "output_fname = root_path + '/data/processed/tokenized/ratings_mecab_noun.txt'\n",
        "noun_tokenize(corpus_fname, output_fname)\n",
        "\n",
        "print(\"koquard\")\n",
        "corpus_fname = root_path + '/data/processed/processed_korquad.txt'\n",
        "output_fname = root_path + '/data/processed/tokenized/korquad_mecab_noun.txt'\n",
        "noun_tokenize(corpus_fname, output_fname)\n",
        "\n",
        "# print(\"wiki\")\n",
        "# corpus_fname = root_path + '/data/processed/processed_wiki_ko.txt'\n",
        "# output_fname = root_path + '/data/processed/tokenized/wiki_ko_mecab_noun.txt'\n",
        "# noun_tokenize(corpus_fname, output_fname)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "naver\n",
            "koquard\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AaVWX7Yj9vA0"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xb95TgVi5Q6I"
      },
      "source": [
        "### 사용자 기반 사전 추가(Komoran)\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0EXyPngooAAY"
      },
      "source": [
        "tokenizer = Komoran(userdic = root_path + '/user_dic.txt')\n",
        "print(tokenizer.pos(\"바람과 함께 사라지다는 정말 명작이야\"))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ks6DHdqi5Xs9"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "njcimYb490QO"
      },
      "source": [
        "## 비 지도학습 기반 형태소 분석\n",
        "- soynlp\n",
        "- bert(bpe)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Bb4shyqr5T48"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q3LgVXFy-tc8"
      },
      "source": [
        "\n",
        "# ! pip install soynlp\n",
        "from soynlp.word import WordExtractor\n",
        "import math\n",
        "from soynlp.tokenizer import LTokenizer\n",
        "from soynlp.normalizer import *"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UgPj35aZ-_Me"
      },
      "source": [
        "def compute_soy_word_score(corpus_fname, model_fname):\n",
        "    # corpus읽어서 sentences에 넣음\n",
        "    sentences = [sent.strip() for sent in open(corpus_fname, 'r',  encoding='UTF8').readlines()]\n",
        "\n",
        "    #학습 파라미터 설정\n",
        "    word_extractor = WordExtractor(min_frequency=100,\n",
        "                                   min_cohesion_forward=0.05,\n",
        "                                   min_right_branching_entropy=0.0\n",
        "                                   )\n",
        "    #학습\n",
        "    word_extractor.train(sentences)\n",
        "\n",
        "    #모델 저장\n",
        "    word_extractor.save(model_fname)\n",
        "\n",
        "    \n",
        "corpus_fname = root_path + \"/data/processed/processed_korquad.txt\"\n",
        "model_fname = root_path + \"/data/processed/soyword.model\"\n",
        "\n",
        "compute_soy_word_score(corpus_fname, model_fname)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2hMnYppaBEGW"
      },
      "source": [
        "\n",
        "def soy_tokenize(corpus_fname, model_fname):\n",
        "  # 학습환경과 동일하게 word_extractor 객체 생성\n",
        "    word_extractor = WordExtractor(min_frequency=100,\n",
        "                                   min_cohesion_forward=0.05,\n",
        "                                   min_right_branching_entropy=0.0\n",
        "                                   )\n",
        "    \n",
        "    #학습된 모델 로드\n",
        "    word_extractor.load(model_fname)\n",
        "\n",
        "    #저장된 점수 로드\n",
        "    scores = word_extractor.word_scores()\n",
        "    scores = {key:(scores[key].cohesion_forward * math.exp(scores[key].right_branching_entropy)) for key in scores.keys()}\n",
        "    tokenizer = LTokenizer(scores=scores)\n",
        "\n",
        "    # 테스트\n",
        "    tokens = tokenizer.tokenize('패턴을 스스로 학습한다')\n",
        "    print(tokens)\n",
        "\n",
        "soy_tokenize(corpus_fname, model_fname)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "05Ldw9PELYrw"
      },
      "source": [
        "soynlp space교정"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rfs_i1PzLata"
      },
      "source": [
        "!pip install soyspacing\n",
        "from soyspacing.countbase import CountSpace\n",
        "\n",
        "corpus_fname =  root_path + '/data/processed/processed_ratings.txt'\n",
        "model_fname = root_path + '/data/processed/space-correct.model'\n",
        "\n",
        "model = CountSpace()\n",
        "model.train(corpus_fname)\n",
        "model.save_model(model_fname, json_format = False)\n",
        "\n",
        "# 모델을 로드하는 부분. 사실 위에서 한번 학습 시킨 것이므로 로드 할 필요는 없지만 나중에 필요할 경우를 위해서 넣어 둠\n",
        "model.load_model(model_fname, json_format = False)\n",
        "print(model.correct(\"어릴때보고다시봐도재미있다\"))\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oWyDKl_pD0Sy"
      },
      "source": [
        "### sentence piece + bert"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nT8c3QpmDzzp"
      },
      "source": [
        "! pip install sentencepiece\n",
        "import sentencepiece as spm"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RG0H6ReqKdv5"
      },
      "source": [
        "sentece piece Train 후 bert모델에 넣을 수 있게 형태 변형"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dSLu9MXUFo3m"
      },
      "source": [
        "in_f= root_path + '/data/processed/processed_wiki_ko.txt '\n",
        "vocab_f = root_path +'/data/processed/bert.vocab'\n",
        "\n",
        "def make_bert_vocab(input_fname, output_fname):\n",
        "\n",
        "  # sentence piece 학습\n",
        "    train = '--input=' + input_fname + ' --model_prefix=sentpiece --vocab_size=5000 --model_type=bpe --character_coverage=0.9995'\n",
        "    spm.SentencePieceTrainer.Train(train)\n",
        "\n",
        "    #[PAD][UNK] 등 bert 모델에서 쓰는 토큰을 넣어 형태 수정\n",
        "    with open('sentpiece.vocab', 'r', encoding='utf-8') as f1, \\\n",
        "            open(output_fname, 'w', encoding='utf-8') as f2:\n",
        "        f2.writelines(\"[PAD]\\n[UNK]\\n[CLS]\\n[SEP]\\n[MASK]\\n\")\n",
        "        for line in f1:\n",
        "            word = line.replace('\\n', '').split('\\t')[0]\n",
        "            if not word or word in [\"▁\", \"<unk>\", \"<s>\", \"</s>\"]:\n",
        "                continue\n",
        "            if word[0] == '▁':\n",
        "                word = word.replace('▁', '')\n",
        "            else:\n",
        "                word = '##' + word\n",
        "            f2.writelines(word + \"\\n\")\n",
        "\n",
        "make_bert_vocab(in_f, vocab_f)\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qJ5wnTVIKnlt"
      },
      "source": [
        "bert 모델 test"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vHbuZoieKcuu"
      },
      "source": [
        "# !pip install transformers\n",
        "import transformers\n",
        "vocab_fname = root_path + \"/data/processed/bert.vocab\"\n",
        "vocab_fname = root_path + \"/data/processed/bert.vocab\"\n",
        "tokenizer =transformers.BertTokenizer(vocab_file = vocab_fname, do_lower_case = False)\n",
        "\n",
        "print(tokenizer.tokenize(\"동해물과 백두산이 마르고 닳도록 하느님이 보우하사 우리나라 만세\"))\n"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}