Skip to content

Adding a bug fix to train and infer model as tokenizer.convert_tokens _to_string expects list[strings] instead of strings #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
arunjose696 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Adding a bug fix to train and infer model as tokenizer.convert_tokens _to_string expects list[strings] instead of strings #2

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ see `README_tokenizer.md` for further information.
#### Simple use

```python
from convlm.tokenizer import SpokenDialogTokenizer
from turngpt.tokenizer import SpokenDialogTokenizer

pretrained_model_name_or_path="microsoft/DialoGPT-small"
tokenizer = SpokenDialogTokenizer(pretrained_model_name_or_path)
Expand Down Expand Up @@ -110,7 +110,7 @@ An un-trained TurnGPT model, loads pre-trained weights by default, and includes

```python
from argparse import ArgumentParser
from convlm.turngpt import TurnGPT
from turngpt import TurnGPT

parser = ArgumentParser()
parser = TurnGPT.add_model_specific_args(parser)
Expand Down
2 changes: 1 addition & 1 deletion turngpt/model.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ def idx_to_string(self, idx):
idx = idx.item()
s = self.tokenizer.convert_ids_to_tokens(idx)
s = self.tokenizer.convert_tokens_to_string(
s.strip()
[s.strip()]
) # remove prefix space/symbol
return s

Expand Down
2 changes: 1 addition & 1 deletion turngpt/train.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@

import pytorch_lightning as pl

from datasets_turntaking import DialogTextDM
from datasets_turntaking import ConversationalDM as DialogTextDM
from turngpt.model import TurnGPT, TurnGPTWandbCallbacks


Expand Down