Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error handling for aws bedrock and openai response #133

Open
yuzisun opened this issue Jan 19, 2025 · 0 comments · May be fixed by #152
Open

Error handling for aws bedrock and openai response #133

yuzisun opened this issue Jan 19, 2025 · 0 comments · May be fixed by #152
Assignees
Labels
bug Something isn't working
Milestone

Comments

@yuzisun
Copy link
Contributor

yuzisun commented Jan 19, 2025

  • Add logic to handle error response from upstream aws bedrock or openai, currently it is assuming the response is 200 and fails on response body unmarshalling.
  • Add tests for error responses from upstream
time=2025-01-19T18:34:38.104Z level=INFO msg="starting external processor" version=8056bb8bcfe484ca6f6007b021ea69448b3073f4
time=2025-01-19T18:34:38.104Z level=INFO msg="loading a new config" path=/etc/ai-gateway/extproc/extproc-config.yaml
time=2025-01-19T18:34:38.104Z level=INFO msg="start watching the config file" path=/etc/ai-gateway/extproc/extproc-config.yaml interval=5s
time=2025-01-19T18:34:42.422Z level=DEBUG msg="request headers processing" request_headers="headers:{key:\":authority\"  raw_value:\"localhost:8080\"}  headers:{key:\":path\"  raw_value:\"/v1/chat/completions\"}  headers:{key:\":method\"  raw_value:\"POST\"}  headers:{key:\":scheme\"  raw_value:\"http\"}  headers:{key:\"user-agent\"  raw_value:\"curl/8.7.1\"}  headers:{key:\"accept\"  raw_value:\"*/*\"}  headers:{key:\"content-type\"  raw_value:\"application/json\"}  headers:{key:\"x-ai-gateway-llm-backend\"  raw_value:\"kserve-llama-backend\"}  headers:{key:\"content-length\"  raw_value:\"113\"}  headers:{key:\"x-forwarded-for\"  raw_value:\"10.244.1.40\"}  headers:{key:\"x-forwarded-proto\"  raw_value:\"http\"}  headers:{key:\"x-envoy-external-address\"  raw_value:\"127.0.0.1\"}  headers:{key:\"x-request-id\"  raw_value:\"41605e8c-7ef8-4fbc-b1f9-a8f01740746e\"}"
time=2025-01-19T18:34:42.422Z level=DEBUG msg="request headers processed" response=request_headers:{}
time=2025-01-19T18:34:42.422Z level=DEBUG msg="request body processing" request="request_body:{body:\"{\\\"model\\\": \\\"dsp.llama-3.3-70b-instruct\\\", \\\"prompt\\\": \\\"Write a poem about colors\\\", \\\"stream\\\":false, \\\"max_tokens\\\": 100}\"  end_of_stream:true}  metadata_context:{}"
time=2025-01-19T18:34:42.422Z level=INFO msg="Processing request" path=/v1/chat/completions model=dsp.llama-3.3-70b-instruct
time=2025-01-19T18:34:42.422Z level=INFO msg="Selected backend" backend=kserve-llama-backend.ai-gateway
time=2025-01-19T18:34:42.422Z level=DEBUG msg="request body processed" response="request_body:{response:{header_mutation:{set_headers:{header:{key:\"x-envoy-ai-gateway-model\"  raw_value:\"dsp.llama-3.3-70b-instruct\"}}  set_headers:{header:{key:\"x-envoy-ai-gateway-selected-backend\"  raw_value:\"kserve-llama-backend.ai-gateway\"}}}  clear_route_cache:true}}"
time=2025-01-19T18:34:57.425Z level=DEBUG msg="response headers processing" response_headers="headers:{key:\":status\"  raw_value:\"504\"}  headers:{key:\"content-length\"  raw_value:\"24\"}  headers:{key:\"content-type\"  raw_value:\"text/plain\"}"
time=2025-01-19T18:34:57.425Z level=ERROR msg="upstream returned non-200 response"
time=2025-01-19T18:34:57.425Z level=DEBUG msg="response headers processed" response=response_headers:{}
time=2025-01-19T18:34:57.426Z level=DEBUG msg="response body processing" request="response_body:{body:\"upstream request timeout\"  end_of_stream:true}  metadata_context:{}"
time=2025-01-19T18:34:57.426Z level=DEBUG msg="response body processed" response=<nil>
time=2025-01-19T18:34:57.426Z level=ERROR msg="cannot process request" error="cannot process response body: failed to transform response: failed to unmarshal body: invalid character 'u' looking for beginning of value"
@yuzisun yuzisun added the bug Something isn't working label Jan 19, 2025
@yuzisun yuzisun added this to the v.0.1.0 milestone Jan 19, 2025
@yuzisun yuzisun self-assigned this Jan 20, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant