API issues with CRFM stanford

[SaeedSubayyal]

Hi, I tried running it but it always keep showing issues with crfm and requires its api key.txt. I checked over the internet its a stanford project and require personal api to acess. Therefore, I cant run the project on colabs even though I follow all the directions mentioned by you in the repository. Also I have a question why not use llama 3.1 instead of chatgpt 4 as it can save money and probably is more efficient.

[SaeedSubayyal]

try:
from helm.common.authentication import Authentication
from helm.common.request import Request, RequestResult
from helm.proxy.accounts import Account
from helm.proxy.services.remote_service import RemoteService
# setup CRFM API
auth = Authentication(api_key=open("crfm_api_key.txt").read().strip())
service = RemoteService("https://crfm-models.stanford.edu")
account: Account = service.get_account(auth)
except Exception as e:
print(e)
print("Could not load CRFM API key crfm_api_key.txt.")

[guosyjlu]

Hi,

1. Currently, this repository is based on OpenAI API. The other LLMs are implemented by the original base repository. If you still find issues for using OpenAI LLMs, you can delete these irrelevant codes.
2. Yes, of course! Note that Llama 3 was not released yet when we built this project. We do test the performance of Llama3-70b-chat and find it can indeed achieve desired success rate. If you want to use Llama3-70b-chat as your base LLM, you can use vllm framework to simulate OpenAI-alike API service.