Refactor NN code #111
Merged
Refactor NN code #111
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Introduce the new interface `InputRepresentation` to separate logic of input representation (e.g. word embeddings, character trigrams) from iterators and classifiers. This allows new combinations required as part of bst-mug#107 and bst-mug#110. Move data-dependent methods such as `initializeTruncateLength` and `loadFeaturesForNarrative` to the iterators. Remove public and duplicate attributes to reduce complexity.
This allows other combinations as required by bst-mug#107 and bst-mug#110.
So that it can be reused by other NN tests.
Pull-up `loadFeaturesForNarrative` with a call to an abstract method `getUnits`.
Make code closer to the original example and to the `SentenceIterator`.
Both `SentenceIterator` (`calculateMaxSentences`) and `TokenIterator` (`initializeTruncateLength`) implemented a method to calculate the longest sequence in training data and used it to initialize `truncateLength`. Now that we have `getUnits`, DRY and pull up this method. Note that `TokenIterator` method cleaned the text before tokenization, but such cleaning was not done at training/test time and was therefore a bug. Not also `TokenIterator` calculated in the same method token/type coverage, which was used only for debugging before introduction of `fasttext`, that returns an approximate vector for out-of-dictionary tokens.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This refactor NN code so that pieces of code can more easily be reused in #107 and #110.