Should we change 500 to another status code when the error comes from the dataset?

[severo]

See #1661 for example.

Same for the "retry later" error: is 500 the most appropriate status code?

[severo]

Also: many 501 errors should be 500 (or something else). For example, if the cache directory has not been initialized before job compute, it's a server internal error (we should have initialized it). It's not that we did not implement something.

https://github.com/huggingface/datasets-server/blob/4c189b1cfb51dd19d7c448478d51d0dcf2db86d1/libs/libcommon/src/libcommon/exceptions.py#L145

[severo]

see also #998. Copying relevant comments here:

Context @severo's comment #995 (comment)
For now we support only error 500 INTERNAL_SERVER_ERROR and 501 NOT_IMPLEMENTED error as part of custom Exception classes in worker project.
But this could be misunderstood since we manage those error in purpose and they are not a result of another request or dependent module.
We should evaluate each case and change to an appropiate HTTPStatus code.

It's a bit weird to return INTERNAL_SERVER_ERROR because we do it on purpose. Not sure which code we should return (theoretically it could be a 30x code because another endpoint has the response, but we only manage OK and error responses for now).
But it's not the only place where the return code is weird, I think. We might want to review all of them.