Steeltoe redis connector failing for SSL, even though the default value for SSL is false

[sajanep]

Describe the bug

I have a .net core3.1 application using steeltoe 3.1.1 packages. I do have below code in Startup.cs

services.AddRedisConnectionMultiplexer(configuration);

I do have below packages in .csproj

<PackageReference Include="Microsoft.Extensions.Caching.Memory" Version="3.1.21" />
<PackageReference Include="Newtonsoft.Json" Version="12.0.3" />
<PackageReference Include="StackExchange.Redis" Version="2.2.88" />
<PackageReference Include="Steeltoe.Connector.ConnectorCore" Version="3.1.1" />
<PackageReference Include="Steeltoe.Extensions.Logging.DynamicSerilogCore" Version="3.1.1" />
<PackageReference Include="Steeltoe.Management.CloudFoundryCore" Version="3.1.1" />
<PackageReference Include="Steeltoe.Extensions.Configuration.CloudFoundryCore" Version="3.1.1" />

Whenever I try to connect to Redis via below lines

public class RedisCache
    {
        private readonly IConnectionMultiplexer _conn;
       
        public RedisCache(IConnectionMultiplexer conn) 
        {
            this._conn = conn;
        }
        
public bool PutItem<T>(string key, T value, TimeSpan expiry, bool overwrite = true)
        {
            bool isAdded = false;
            IDatabase database = _conn.GetDatabase();
            if (!database.KeyExists(key) || overwrite)
            {
                var jsonString = JsonHelper.Serialize(value);
                database.StringSet(key, jsonString, expiry);
                isAdded = true;
            }
            return isAdded;
        }
}

Below exception is getting generated in vmw apps manager console

2021-12-01T19:03:17.263+05:30 [APP/PROC/WEB/0] [OUT] System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
2021-12-01T19:03:17.263+05:30 [APP/PROC/WEB/0] [OUT] ---> StackExchange.Redis.RedisConnectionException: It was not possible to connect to the redis server(s). There was an authentication failure; check that passwords (or client certificates) are configured correctly.
2021-12-01T19:03:17.263+05:30 [APP/PROC/WEB/0] [OUT] at StackExchange.Redis.ConnectionMultiplexer.ConnectImpl(ConfigurationOptions configuration, TextWriter log) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 1195
2021-12-01T19:03:17.263+05:30 [APP/PROC/WEB/0] [OUT] at StackExchange.Redis.ConnectionMultiplexer.Connect(ConfigurationOptions configuration, TextWriter log) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 1061
2021-12-01T19:03:17.263+05:30 [APP/PROC/WEB/0] [OUT] --- End of inner exception stack trace ---

Steeltoe documentation says that ssl is false by default https://docs.steeltoe.io/api/v3/connectors/redis.html, we are not sure why this exception is thrown then. This is noted after the platform upgrade https://docs.pivotal.io/redis/2-4/release.html. After this upgrade, the env. variable of the application has TLS ports too like below.

"my.redis": [
        {
          "label": "my.redis",
          "provider": null,
          "plan": "cache-medium",
          "name": "MyRedis",
          "tags": [
            "redis",
            "pivotal",
            "on-demand"
          ],
          "instance_guid": "*****",
          "instance_name": "MyRedis",
          "binding_guid": ****",
          "binding_name": null,
          "credentials": {
            "host": "q-s0.redis-instance.service-network.service-instance-eb72c7c5--4866-a44f-.bosh",
            "password": "*****",
            "port": 6379,
            "tls_port": 16379,
            "tls_versions": [
              "tlsv1.2",
              "tlsv1.3"
            ]
          },

Any idea how to solve this?

Environment (please complete the following information):

• Steeltoe Version : 3.1.1
• Platform: VMW Tanzu Application Service

Screenshots

If applicable, add screenshots to help explain your problem.

Additional context or links

Add any other context about the problem here.

[TimHess]

This line is probably where the SSL decision is being made. Connectors in general expect configuration in service bindings from the platform to override other settings. Any idea why there would be a TLS port specified if it isn't going to work? I'd expect the better approach would be to use the secured connection that seems to available if that's possible

[shouah]

@TimHess
the root cause is that the Redis server uses internally-issued TLS certificates that the client application doesn't trust, therefore the application developer needs to be able to initialize the trusted CA store before Steeltoe attempts to establish the TLS connection. Alternatively, Steeltoe needs to automatically pick up the various Cloud Foundry' system Root CAs stored in the PATH specified by the CF_SYSTEM_CERT_PATH environment variable -- when it is running inside a CF container.

This issue is visible because of the rollout sequence:

1. we enable TLS on the Redis server-side
2. this exposes the TLS port in the service definition
3. applications, under certain restart scenario are seeing the TLS port and tries to use (as the case here)
4. application developers have not yet been able to make the code change to configure the trusted CA and therefore their apps are failing

the expectation is that because "ssl=false" under default condition, then Steeeltoe must not use the TLS port until the application developers have made the code change to configure the trusted CA.

[TimHess]

Platform certificates should be automatically installed into the trust store, it sounds like there might be a configuration issue with the platform.
Could you try deploying this application to check where in the cert chain validation is failing? If you call the endpoint at [Route("/remote/{host}/{port=443}")] you should get some detailed information about what's wrong and potentially your platform team can fix it

[shouah]

Agreed on that point, but it's specific to scenario where the application being hosted inside the platform.

When the application is running outside, like on a developer workstation or outside of the platform like a traditional IaaS then they don't have "automatic" access to the internal CAs of the platform so the issue is still applicable.

[TimHess]

I understand what you're saying, but that goes against the design intent for service bindings to take priority here.

For the scenario you're describing, it seems that the credentials are being manually provided, in which case omitting the TLS properties should result in a working solution (assuming you have a way to resolve .bosh hostnames from outside of the platform).

In order to change the behavior we'd likely need to add a new property like DisableSsl (or similar) that forces secure connections be disabled and apply it in this method but I'm not sure that's really the best idea here

[TimHess]

As a follow-up, if it is possible to disable certificate validation on the Redis client, it would likely be a useful feature to investigate and would make sense to add to Steeltoe. We will try to take a look in the near future, but adding this option would also be a good opportunity for community contribution