nil pointer dereference error from tokenizer library from within fastembed-go

[alkuma]

I faced a nil pointer dereference error in tokenizer library when it was being used in a downstream library called fastembed-go and have reported the issue at Anush008/fastembed-go#13

Cross posting here in case this is a known issue with a known workaround.

[alkuma]

The error message / stack trace I got seems to indicate that this is a duplicate of #37 which has been fixed in June 2024. Let me investigate some more, hopefully this item can then be closed as a duplicate

[alkuma]

Since there was a similar issue reported (and closed via a code change / PR) on the tokenizer side I just forked both tokenizer and fastemebed-go and published the latest master / main branch and used them as dependency, and the error is gone.

Perhaps all that's needed to be done is to publish the latest versions of both?

[sugarme]

@alkuma ,
Thanks for your report.