How to enhance the accuracy of results obtained from various sources using Flowise

[trepont]

How to train the system to get accuracy results from the system from different sources using Flowise components
I have uploaded a collection of 10 PDF files, each containing over 500 pages, with the intention of extracting answers for specific questions.

.

[tristinbaileel]

I'm new to this so please correct me if I'm wrong in any areas and apologies if you already know this but the quality/accuracy of my responses were impacted the most when changing parameters like chunk and batch size, here's an overview of everything I've learned
Accuracy: The quality or accuracy of the responses generated by the AI system depends on various factors. When you're using the Load Existing Index method, the AI is retrieving data from a pre-existing index in Pinecone's vector store. The relevance and accuracy of its responses will depend on the quality of the information already stored in the index. If the index contains high-quality, relevant data, then the AI system will likely generate accurate and relevant responses. Conversely, if the index contains outdated or irrelevant information, then the accuracy of the responses may suffer.
In contrast, when you're using the Upsert Document method, you're inputting new information into the system. The quality of the responses in this case depends more directly on the quality of the new data you're inputting. If the document you're inserting contains accurate, detailed, and relevant information, then the AI system can generate high-quality responses based on this information.
In both cases, the accuracy of the responses also depends on the effectiveness of the language model (like GPT-3.5 Turbo) and the quality of the embeddings generated by the model like Ada.
Token Limitations and Rate Limits: The models have token and request limitations, defined as TPM (Tokens Per Minute) and RPM (Requests Per Minute). Ada model has an RPM of 3000 and a TPM of 250,000, whereas the GPT-3.5 Turbo has an RPM of 3,500 and a TPM of 180,000. These limits determine how many requests you can make per minute and how many tokens you can process per minute, respectively.
Batch Size: The batch size refers to the number of data samples (in this case, text inputs) processed together. Larger batch sizes allow for faster processing but may lead to increased latency. The optimal batch size depends on the specific requirements of your use case, the computational resources available, and the model's rate limits.
Chunk Size: When splitting text into chunks, the size of these chunks (measured in tokens) is an important consideration. For the Ada model with a context window of 8192 tokens, the chunk size for the text splitter should ideally be around 8000-8100 tokens. If a chunk exceeds the model's context window size, the context gets truncated, which could lead to loss of information and reduced quality of the embeddings.
Embeddings: The Ada model transforms text into high-dimensional vectors or embeddings, which are then stored in Pinecone's vector store. The quality of these embeddings is crucial for the performance of the retrieval system. The Ada model produces 1536-dimensional embeddings, which are smaller and more efficient than previous versions.
Strip New Lines, Timeout: 'Strip New Lines' is an option in the Ada model which, when turned on, removes new line characters from the input text. 'Timeout' is the maximum time to wait for the model to generate embeddings.