Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DRY Cleaning #105

Open
michelole opened this issue Apr 23, 2019 · 1 comment
Open

DRY Cleaning #105

michelole opened this issue Apr 23, 2019 · 1 comment
Labels
enhancement New feature or request

Comments

@michelole
Copy link
Member

michelole commented Apr 23, 2019 •
edited
Loading

We do cleaning in several places in DataUtilities:

  • getSentences() (called from SentenceIterator and getTokens)
  • processTextReduced() (called from CharacterTrigram, deprecated)
  • cleanText() (called from Patient.getCleanedText(), used only by RBC)
  • LSTMClassifier.initializeTruncateLength (removed in d524666)

DRY
@michelole michelole added the enhancement New feature or request label Apr 23, 2019
@michelole michelole added the P1 Higher priority issues, a SHOULD label Jun 4, 2019
This was referenced Jun 5, 2019
Closed
Merged
@michelole
Copy link
Member Author

We should

  • Call clean outside of getSentences (e.g. getTokens) so that SalienceAnalyzer can analyze dirty sentences (and thus show eventual overfitting to garbage).
  • NN: call clean and/or lowercase only in InputRepresentation
  • Replace the call to getTokens in TokenIterator to tokenize

@michelole michelole removed the P1 Higher priority issues, a SHOULD label Jun 11, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@michelole