Dev.ap/text pf updates expanded #627

roedoejet · 2025-01-20T22:11:17Z

PR Goal?

This PR improves the phonological feature calculation by:

increasing the number of features to include more punctuation (getting rid of BB and SB in favour of colon, semi-colon, and period)
getting rid of custom tone feature calculation and just using Panphon's calculation using tone-bars. Note: this makes us require tone bars and not accents, but it also fixes Improve tone feature calculation for phonological features #357
we also add phonological feature calculation for common text encoding special tokens like [MASK] and [UNK]

Fixes?

#357

Feedback sought?

Sanity + high-level comments. low-level code comments are also welcome but less of a priority.

Priority?

medium

Tests added?

I've added some tests and updated existing ones

How to test?

A little bit tricky to test, but maybe inspecting some of the doctests will help. The main work is just analysing the updated code in text/features.py

Confidence?

medium (I think this is an improvement, and it is what we use in PF-BERT, but it's a big set of interrelated problems so I would appreciate the feedback)

Version change?

Yes, this is a breaking change.

Related PRs?

EveryVoiceTTS/DeepForcedAligner#34
EveryVoiceTTS/FastSpeech2_lightning#109

semanticdiff-com · 2025-01-20T22:11:19Z

Review changes with

Changed Files

File	Status
everyvoice/config/text_config.py	42% smaller
everyvoice/tests/test_model.py	33% smaller
everyvoice/text/features.py	17% smaller
everyvoice/text/text_processor.py	10% smaller
everyvoice/tests/test_text.py	6% smaller
everyvoice/.schema/everyvoice-aligner-0.3.json	0% smaller
everyvoice/.schema/everyvoice-shared-text-0.3.json	0% smaller
everyvoice/.schema/everyvoice-text-to-spec-0.3.json	0% smaller
everyvoice/.schema/everyvoice-text-to-wav-0.3.json	0% smaller
everyvoice/model/aligner/DeepForcedAligner	0% smaller
everyvoice/model/feature_prediction/FastSpeech2_lightning	0% smaller
everyvoice/text/utils.py	0% smaller

roedoejet · 2025-01-20T22:15:13Z

whoops - looks like some tests aren't passing - I'll investigate, but it should still be ready for review

roedoejet · 2025-01-20T23:51:03Z

@joanise - I don't understand why the doctests are failling here, they're not failing on my machine, do you have any ideas?

codecov · 2025-01-21T20:40:06Z

Codecov Report

Attention: Patch coverage is 78.04878% with 18 lines in your changes missing coverage. Please review.

Project coverage is 76.21%. Comparing base (a6829b4) to head (bc4dac0).
Report is 3 commits behind head on main.

Files with missing lines	Patch %	Lines
everyvoice/text/features.py	74.57%	11 Missing and 4 partials ⚠️
everyvoice/text/utils.py	75.00%	1 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #627      +/-   ##
==========================================
- Coverage   76.24%   76.21%   -0.04%     
==========================================
  Files          47       47              
  Lines        3490     3536      +46     
  Branches      481      493      +12     
==========================================
+ Hits         2661     2695      +34     
- Misses        726      734       +8     
- Partials      103      107       +4

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

github-actions · 2025-01-21T20:40:28Z

CLI load time: 0:00.27
Pull Request HEAD: bc4dac05489793e659d5c702b453cb9ca34766b7
Imports that take more than 0.1 s:
import time: self [us] | cumulative | imported package
import time:      1044 |     102595 |     typer.main
import time:       283 |     122714 |   typer
import time:      7901 |     202274 | everyvoice.cli

joanise · 2025-01-21T21:38:25Z

We didn't publish 0.3.0 yet, so this PR should update the 0.3 schemas, not create the 0.4 schemas.

joanise

sorry, a bunch of comments suggesting changes.

everyvoice/_version.py

everyvoice/text/features.py

joanise · 2025-01-21T22:08:16Z

everyvoice/text/features.py

+                punctuation_features.append([1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])
            elif char == self.punctuation_hash["question_symbols"]:
-                punctuation_features.append([0, 1, 0, 0, 0, 0, 0, 0])
-            elif char == self.punctuation_hash["big_breaks"]:
-                punctuation_features.append([0, 0, 1, 0, 0, 0, 0, 0])
-            elif char == self.punctuation_hash["small_breaks"]:
-                punctuation_features.append([0, 0, 0, 1, 0, 0, 0, 0])
+                punctuation_features.append([0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])
+            elif char == self.punctuation_hash["periods"]:
+                punctuation_features.append([0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0])
+            elif char == self.punctuation_hash["colons"]:
+                punctuation_features.append([0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0])
+            elif char == self.punctuation_hash["semi_colons"]:
+                punctuation_features.append([0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0])
+            elif char == self.punctuation_hash["commas"]:
+                punctuation_features.append([0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0])
+            elif char == self.punctuation_hash["hyphens"]:
+                punctuation_features.append([0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0])
            elif char == self.punctuation_hash["quotemarks"]:
-                punctuation_features.append([0, 0, 0, 0, 1, 0, 0, 0])
-            elif char == self.punctuation_hash["ellipsis"]:
-                punctuation_features.append([0, 0, 0, 0, 0, 1, 0, 0])
+                punctuation_features.append([0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0])
+            elif char == self.punctuation_hash["parentheses"]:
+                punctuation_features.append([0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0])
+            elif char == self.punctuation_hash["ellipses"]:
+                punctuation_features.append([0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0])
            elif char == self.punctuation_hash["exclamations"]:
-                punctuation_features.append([0, 0, 0, 0, 0, 0, 1, 0])
+                punctuation_features.append([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0])
            elif char in self.config.symbols.silence:
-                punctuation_features.append([0, 0, 0, 0, 0, 0, 0, 1])
+                punctuation_features.append([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1])
            else:
-                punctuation_features.append([0, 0, 0, 0, 0, 0, 0, 0])
+                punctuation_features.append([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])


I'm really not a fan of this long if/elif list. Can we not construct a dict from punc hash value to these one-hot vectors with a loop, and do a dict look up for the punc entries, followed by the last elif and else?
I mean, this code works, so you don't have to change it, but it would be nicer to refactor it.

Hm, I kind of like just being able to see the full embedding like this - let's talk at the meeting today

pyproject.toml

joanise

I haven't been able to test yet, just tiny comments.

everyvoice/text/features.py

everyvoice/config/text_config.py

joanise

Since this PR breaks fs2 and hfgl models, it will need to include targeted messages when a user tries to load a model created with an older version of EV that's not compatible with the current punctuation lists.

includes features for stress and special tokens

since panphon already deals with them also adds parentheses support in pf encoding

joanise

I think this generally looks good, we're getting close to be able to merge. Just a couple small comments in the related PRs.

roedoejet requested review from SamuelLarkin, joanise and MENGZHEGENG and removed request for SamuelLarkin January 20, 2025 22:13

joanise reviewed Jan 21, 2025

View reviewed changes

roedoejet requested a review from joanise February 10, 2025 18:40

joanise reviewed Feb 10, 2025

View reviewed changes

everyvoice/text/features.py Show resolved Hide resolved

everyvoice/config/text_config.py Outdated Show resolved Hide resolved

joanise requested changes Feb 11, 2025

View reviewed changes

roedoejet added 11 commits February 12, 2025 13:08

feat(text): upgrade phonological feature encoding

ddc480f

includes features for stress and special tokens

refactor: remove tone features

cb168a3

since panphon already deals with them also adds parentheses support in pf encoding

feat: expand punctuation beyond small and big breaks

4d91fc7

refactor: remove unused normalize and denormalize methods

0d58e20

fix(tests): fix text processing tests

e0b2d4f

chore: update schema and version to 0.4.0

3e99c57

refactor: pin panphon to 0.20.0

d1520fb

refactor: revert to version 0.3.0

11ca732

refactor: move punctuation hash to default constant

2575cb0

fix: fix typo

577ba61

refactor: move symbol sorter to utils

78dc32c

roedoejet force-pushed the dev.ap/text-pf-updates-expanded branch from cbcf71a to 78dc32c Compare February 12, 2025 21:13

roedoejet added 2 commits February 12, 2025 14:30

fix: update tests for new checkpoint loading behaviour

4bfa1af

chore: update submodule to include fix for loading checkpoints

bc4dac0

roedoejet mentioned this pull request Feb 12, 2025

chore: update model due to incompatibility with new text processing EveryVoiceTTS/DeepForcedAligner#34

Open

roedoejet mentioned this pull request Feb 12, 2025

fix: attempt automatic embedding table update for previous models EveryVoiceTTS/FastSpeech2_lightning#109

Open

roedoejet requested a review from joanise February 12, 2025 22:33

joanise reviewed Feb 13, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dev.ap/text pf updates expanded #627

Dev.ap/text pf updates expanded #627

roedoejet commented Jan 20, 2025 •

edited

Loading

semanticdiff-com bot commented Jan 20, 2025 •

edited

Loading

roedoejet commented Jan 20, 2025

roedoejet commented Jan 20, 2025

codecov bot commented Jan 21, 2025 •

edited

Loading

github-actions bot commented Jan 21, 2025 •

edited

Loading

joanise commented Jan 21, 2025

joanise left a comment

joanise Jan 21, 2025

roedoejet Feb 10, 2025

joanise left a comment

joanise left a comment

joanise left a comment

Dev.ap/text pf updates expanded #627

Are you sure you want to change the base?

Dev.ap/text pf updates expanded #627

Conversation

roedoejet commented Jan 20, 2025 • edited Loading

PR Goal?

Fixes?

Feedback sought?

Priority?

Tests added?

How to test?

Confidence?

Version change?

Related PRs?

semanticdiff-com bot commented Jan 20, 2025 • edited Loading

roedoejet commented Jan 20, 2025

roedoejet commented Jan 20, 2025

codecov bot commented Jan 21, 2025 • edited Loading

Codecov Report

github-actions bot commented Jan 21, 2025 • edited Loading

joanise commented Jan 21, 2025

joanise left a comment

Choose a reason for hiding this comment

joanise Jan 21, 2025

Choose a reason for hiding this comment

roedoejet Feb 10, 2025

Choose a reason for hiding this comment

joanise left a comment

Choose a reason for hiding this comment

joanise left a comment

Choose a reason for hiding this comment

joanise left a comment

Choose a reason for hiding this comment

roedoejet commented Jan 20, 2025 •

edited

Loading

semanticdiff-com bot commented Jan 20, 2025 •

edited

Loading

codecov bot commented Jan 21, 2025 •

edited

Loading

github-actions bot commented Jan 21, 2025 •

edited

Loading