TotalTokens Count after initialization should count instructions tokens.

[manvkaur]

In the current implementation for TotalTokens in ChatCompletion.

1. Initialise the chat state with PUT method

   PUT http://localhost:7071/api/chats/test123
   Content-Type: application/json
   
   {
     "instructions": "You are a helpful chatbot. In all your English responses, speak as if you are Shakespeare."
   }
   

2. Retrieve the chat state with GET method (TotalTokens is 0) because nothing has been posted to OpenAI yet and it has not returned consumed tokens.
   GET http://localhost:7071/api/chats/test123?timestampUTC=2023-08-10T07:51:10Z

Behavior:
Actual: The response returns totalTokens to be 0.
Expected: it should count the instructions tokens.

[manvkaur]

OpenAI provides a Python library called tiktoken that can be used to count tokens in a text string according to OpenAI's tokenization rules. Unfortunately, since C# is not directly supported by OpenAI, won't be able to use the tiktoken library directly in C#; will need custom implementation.

[cgillum]

It looks like there are some 3rd party MIT-licensed libraries we could consider:

• https://github.com/dmitry-brazhenko/SharpToken
• https://github.com/aiqinxuancai/TiktokenSharp

[manvkaur]

https://github.com/microsoft/Tokenizer