httpclient: add support for tls.Config.GetClientCertificate

[clwluvw]

What this PR does / why we need it:

Expose tls.Config.GetClientCertificate to allow supplying a callback that loads the client certificate and key on each request. This enables mTLS setups where certificates are rotated automatically without restarting the application.

Inspired by the etcd approach: etcd-io/etcd#7829

Which issue(s) this PR fixes:

Needed for: grafana/grafana#113982
Related to grafana/grafana#44296

[CLAassistant]

All committers have signed the CLA.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[andresmgot]

Hello,

Thank you for the contribution. Unfortunately this goes against plugin best practices: Plugins should not use the local file system. This is a security measure to avoid plugins accessing restricted files from the system or other plugin files.

The recommended approach is still using a provisioning file, which could read files from the system, even though that requires a restart of the pod running the plugin. This shouldn't have any impact or downtime if served in HA.

[clwluvw]

@andresmgot - thanks for the feedback. i'm still little confused about it.
The goal here isn’t avoiding downtime, but avoiding the need for scheduled restarts every time mTLS client certificates rotate.
What confuses me is that Grafana already allows reading certificate files during provisioning, and the SDK itself already loads TLS certs from disk using tls.LoadX509KeyPair. The patch doesn’t expand access beyond what the SDK already does today, it only reloads the same files using the same API, but on demand instead of once at startup. Since the paths are controlled and only used for mTLS, I don’t see how this introduces a new security issue.
I mean we perhaps should be able to find a way to sort this out, and i don't think this practice is really a problem to not have it supported.

[andresmgot]

Let me try to clarify.

What confuses me is that Grafana already allows reading certificate files during provisioning,

Yes, the provisioning system is part of Grafana core, where the code that's being executed is controlled and verified that it works as expected. In the case of plugins, we don't own the code and therefore we need to be more careful of what's being allowed.

and the SDK itself already loads TLS certs from disk using tls.LoadX509KeyPair.

I'm not sure if you mean this middleware. As you can see, the struct has those file paths deprecated and has not been removed (yet) to avoid breaking changes.

The patch doesn’t expand access beyond what the SDK already does today, it only reloads the same files using the same API, but on demand instead of once at startup. Since the paths are controlled and only used for mTLS, I don’t see how this introduces a new security issue.

This patch introduces a new set of paths for local files and that's what is not allowed. Apart from the security implications (plugins could try to read unexpected files from those locations), plugins should not assume any state of the machine they are running since that may change over time and datasources will not.

Hope that makes sense.

[tuunit]

@andresmgot understood that you don't want filesystem access through plugins. Do you have another idea in mind on how to realise mTLS cert rotation in a generic way to be supported by any kind of datasource?

[andresmgot]

See this suggestion: grafana/grafana#113982 (comment)