feature: evals for mcp endpoints

[andrewxhill]

just an mvp. but

pnpm eval

some things not quite working

[github-actions]

MCP Tool Evaluation Results

Running evals file: /home/runner/work/trading-simulator-mcp/trading-simulator-mcp/src/evals/evals.ts
Using server file: /home/runner/work/trading-simulator-mcp/trading-simulator-mcp/src/index.ts
Running all evaluations...

Running Get Balances Evaluation...
Running Get Portfolio Evaluation...
Running Get Trades Evaluation...
Running Get Price Evaluation...
Running Get Token Info Evaluation...
Running Get Price History Evaluation...
Running Execute Trade Evaluation...
Running Get Quote Evaluation...
Running Solana Token Evaluation...
Running Get Competition Status Evaluation...
Running Get Leaderboard Evaluation...
Running Trading Workflow Evaluation...

Evaluation Results:

Get Balances Evaluation:
{
"accuracy": 5,
"completeness": 5,
"relevance": 5,
"clarity": 5,
"reasoning": 5,
"overall_comments": "The LLM's response is accurate, relevant, and clear. It correctly states that it cannot provide the requested information due to a network error. The response is complete in the sense that it addresses the user's query to the best of its ability under the circumstances. The reasoning is sound as it explains the reason for the inability to provide the requested information."
}

Get Portfolio Evaluation:
{
"accuracy": 5,
"completeness": 1,
"relevance": 3,
"clarity": 5,
"reasoning": 2,
"overall_comments": "The answer is accurate in the sense that it doesn't provide false information, but it doesn't provide the requested information either. The completeness of the answer is low because it doesn't provide the portfolio information, token valuations, or total value. The relevance is moderate because it addresses the issue of accessing the portfolio but doesn't provide the specific information requested. The clarity is high because the message is clear and easy to understand. The reasoning is low because it doesn't provide a reason for the network error or suggest a solution beyond trying again later."
}

Get Trades Evaluation:
{
"accuracy": 5,
"completeness": 1,
"relevance": 5,
"clarity": 5,
"reasoning": 1,
"overall_comments": "The answer is accurate and relevant to the question, as it pertains to the user's request for information about their recent trades. It is also clear and easy to understand. However, it does not fully address the question, as it does not provide any information about the user's recent trades or filter for the EVM blockchain. The answer also does not show logical thinking or provide evidence or rationale, as it simply states that there was a network error without providing any further explanation or potential solutions."
}

Get Price Evaluation:
{
"accuracy": 5,
"completeness": 2,
"relevance": 5,
"clarity": 5,
"reasoning": 3,
"overall_comments": "The answer is accurate and relevant to the question, and it is also clear and easy to understand. However, it does not fully address the question because it does not provide the current price of WETH on Ethereum, which is the main point of the question. The reasoning is average because while it provides a reason for not being able to answer the question, it does not provide any evidence or rationale related to the actual price of WETH."
}

Get Token Info Evaluation:
{
"accuracy": 5,
"completeness": 1,
"relevance": 1,
"clarity": 5,
"reasoning": 1,
"overall_comments": "The answer is accurate in the sense that it does not provide any false information. However, it does not address the question at all, hence the low scores for completeness and relevance. The answer is clear and easy to understand, but it does not show any logical thinking or provide any evidence or rationale for the information requested. The main weakness of the answer is its lack of responsiveness to the question."
}

Get Price History Evaluation:
{
"accuracy": 5,
"completeness": 1,
"relevance": 5,
"clarity": 5,
"reasoning": 1,
"overall_comments": "The answer is accurate and relevant to the question, and it is also clear and easy to understand. However, it does not provide the requested information about the price history of WETH, making it incomplete. Additionally, there is no reasoning or evidence provided to support the claim of a network error."
}

Execute Trade Evaluation:
{
"accuracy": 1,
"completeness": 1,
"relevance": 1,
"clarity": 5,
"reasoning": 1,
"overall_comments": "The LLM's response is clear and easy to understand, but it does not accurately or completely answer the question. The user asked for a trade execution, but the LLM responded with a network error message, which is not relevant to the question. The LLM does not provide any reasoning or evidence to support its response."
}

Get Quote Evaluation:
{
"accuracy": 5,
"completeness": 5,
"relevance": 5,
"clarity": 5,
"reasoning": 5,
"overall_comments": "The LLM's response is accurate, relevant, and clear. It directly addresses the user's request and provides a logical reason for not being able to fulfill it at the moment. The response is also well-structured and easy to understand. However, it's important to note that the LLM's ability to provide a quote for the requested trade could not be evaluated due to the reported network error."
}

Solana Token Evaluation:
{
"accuracy": 5,
"completeness": 3,
"relevance": 5,
"clarity": 5,
"reasoning": 3,
"overall_comments": "The LLM's response is accurate and relevant to the question asked. It is also clear and easy to understand. However, it does not fully address the question as it does not provide the current price of SOL on the Solana blockchain. The reasoning is also somewhat lacking as it does not provide a clear explanation for the network error. Overall, the response is satisfactory but could be improved by providing more information."
}

Get Competition Status Evaluation:
{
"accuracy": 5,
"completeness": 1,
"relevance": 1,
"clarity": 5,
"reasoning": 1,
"overall_comments": "The answer is accurate in the sense that it does not provide any false information. However, it does not address the question at all, hence the low scores for completeness and relevance. The answer is clear and easy to understand, but it does not show any logical thinking or provide any evidence or rationale related to the question asked. It seems like a generic error message rather than a response to the specific question."
}

Get Leaderboard Evaluation:
{
"accuracy": 5,
"completeness": 1,
"relevance": 5,
"clarity": 5,
"reasoning": 1,
"overall_comments": "The LLM's response is accurate and relevant to the question, as it acknowledges that it cannot provide the requested information due to a network error. However, it does not provide any information about the trading competition leaderboard or the top positions and their portfolio values, making it incomplete. The reasoning score is low because the LLM does not provide any evidence or rationale beyond the network error. The response is clear and easy to understand."
}

Trading Workflow Evaluation:
{
"accuracy": 5,
"completeness": 3,
"relevance": 5,
"clarity": 5,
"reasoning": 4,
"overall_comments": "The LLM's response is accurate and relevant to the user's request. It clearly communicates the issue of a network error preventing the completion of the tasks. However, it does not fully address the user's request as it does not provide the portfolio value or the quote for trading 0.01 WETH to USDC due to the network error. The reasoning is sound as it suggests trying again later when the network issue has been resolved."
}