WIP Password auth design doc #32005

jubrad · 2025-03-25T05:28:12Z

Motivation

Tips for reviewer

Checklist

This PR has adequate test coverage / QA involvement has been duly considered. (trigger-ci for additional test/nightly runs)
This PR has an associated up-to-date design doc, is a design doc (template), or is sufficiently small to not require a design.
If this PR evolves an existing $T ⇔ Proto$T mapping (possibly in a backwards-incompatible way), then it is tagged with a T-proto label.
If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label (example).
If this PR includes major user-facing behavior changes, I have pinged the relevant PM to schedule a changelog post.

WIP password auth design doc

doc/developer/design/20250321_password_authentication.md

alex-hunt-materialize · 2025-03-25T11:21:05Z

doc/developer/design/20250321_password_authentication.md

+## Open questions
+1. **Where exactly are passwords stored**: While it makes sense to store passwords in the catalog, I have not yet identified exactly where they should be stored. Should it go alongside `mz_roles`, or in a new `mz_auth_id` table. The latter makes sense as it could store password metadata, and might let us store multiple passwords down the road.
+
+2. **Audit Logging and Brute Force Detection**: Do we need to log all failed password attempts and the IPs of those attempts? Do we need to take action to lock an IP or User when an attempt threshold is met?


Any action taken should be user configurable. We don't want to allow DOS attacks by a user just spamming incorrect passwords, for example.

Agree with @alex-hunt-materialize - there's too much risk to availability from DOS or automatic retries with an incorrect password.
Our default behavior should log all login attempts, and we should aim to replicate PostgreSQL log messages. This will allow customers to send the data to their SIEM and/or use fail2ban or another tool to act on any failures.
We'd need to add a way for customers to lock users with fail2ban or a script, but I think we can leave that out for now unless it is very simple to implement.

here's some discussion on how to configure fail2ban w/ Postgres, for example
https://gist.github.com/rc9000/fd1be13b5c8820f63d982d0bf8154db1

doc/developer/design/20250321_password_authentication.md

SangJunBak · 2025-03-26T15:46:52Z

Regarding closing off the internal port for Cloud in the cloud sync:

We should consider Teleport too since i think it uses some internal port?
Are we able to easily disable this for the emulator?

jubrad · 2025-03-26T15:59:26Z

Option A) - preferred
We should consider adding confimap syncing as a self-managed replacement for LD for system vars. If we do this we can probably drop the need for mz_system (in self-managed) and just close off the internal port for self-managed, perhaps not even start it up.

Option B) - backup

We need to evaluate other mz_system capabilities and ensure that it is not needed for any other purposes.
If mz_system is needed then we may need to optionally allow external port login for mz_system with a password.

alex-hunt-materialize · 2025-03-26T16:02:20Z

We should consider Teleport too since i think it uses some internal port?

Teleport prefers mutual TLS, and we've actually gone out of our way to disable it. It's been on the roadmap for a long time to get that working, but never got to it.

DAlperin · 2025-03-26T17:58:38Z

Here are my thoughts on implementation:

A system table for storing sensitive role data like the (hashed and salted) password of a role
- In the future we could transition this to secrets once those are supported for self managed
We will expand the adapter::Client surface area to expose an API to the protocol layer which can be used to exchange a username and password for a role
- when running in cloud we will continue to use frontegg and this codepath will be unexercised
we will do the same adapter call or equivalent in tower middlewares for the http endpoint

The rest of it as laid out in the doc makes sense to me

jubrad · 2025-03-27T15:32:53Z

On system users... We might be able to pass in a bootstrap user that is also considered internal/system. This user should be considered superuser and internal user and it's password should come from a k8s secret. The k8s secret passed in can be used as it's password secrets reference in the catalog.

SangJunBak · 2025-03-27T17:55:38Z

doc/developer/design/20250321_password_authentication.md

+
+### 4. Configurable admin system login:
+
+Passwords for `mz_system` and `mz_support` roles will be settable via environment variables `MZ_MZ_<USER>_EXTERNAL_LOGIN_PASSWORD`. Orchestratord should provide a set of parameters to set these variables via a Kubernetes secret. Additionally, We should enable login of system users through the external ports when they have external login passwords set. Login through the external port must not be possible unless this flag is set, this logic should not rely on whether the internal user has a password.


Nit: Should be MZ_<USER>_EXTERNAL_LOGIN_PASSWORD given mz_system would give MZ_MZ_MZ_SYSTEM_EXTERNAL_LOGIN_PASSWORD

Maybe the user should be last? MZ_EXTERNAL_LOGIN_PASSWORD_<USER>

That also gives a consistent prefix when grepping.

Sounds good!

EXTERNAL_LOGIN_PASSWORD_<USER>

with a env variable of

MZ_EXTERNAL_LOGIN_PASSWORD_<USER>

SangJunBak · 2025-03-27T18:09:19Z

Still not clear to me the flow of setting up auth. In my head, it looks like:

As they initially set up via the helm chart, orchestratord will have the env variable to set the password of the mz_system user. This is probably configurable via the helm chart?
Once the user deploys the materialize CR, orchestratord will pass that env variable in order to start the environment with an mz_system type user
The user can now login via that user and add more roles

Does that sound correct? Then we just make it a limitation that a user can't use the Console unless there's a registered role. In the Console, we'll most likely need the following information

Is (password) auth enabled?
Does there exist an authenticated user?
But I'm also confused how the Console might receive this information if we can't query the database to begin with.

Should also consider the migration path from self managed customers without auth to customers with auth.

alex-hunt-materialize · 2025-03-27T17:53:31Z

doc/developer/design/20250321_password_authentication.md

+
+**Password Rotation or Expiration**: While not currently in scope, the design should allow for additions in the future, such as password rotation and expiration.
+
+**Abuse Detection**: Action should be taken action when brute force attempts are detected, either by locking the user or blocking the requesting client IP.


If we implement this, it should be optional. We don't want to enable DOS attacks just by spamming broken passwords at us.

We should write logs and let customers use fail2ban or whatever solutions they use elsewhere.

alex-hunt-materialize · 2025-03-27T17:57:18Z

doc/developer/design/20250321_password_authentication.md

+
+-   **Password Dating**: The creation date of passwords will be recorded along with the hashed and salted password. When the password changes a creation date should be reset. The creation date of passwords should be queryable.
+
+-   **OPTIONAL: Password Versioning**: Updates to the password hashing mechanisms may be required. As long as we are receiving passwords in plain text we should be able to take a validated password and replace the existing a new securely hashed value. This may require prefixing the password with data about the hash algorithm or parameters.


This is trivial so we should just do it non-optionally. We don't have to implement any replacement now, just store the metadata about the algorithm and parameters.

EDIT: To clarify, the trivial part is to store things like the algorithm used and parameters like number of iterations, not implementing the full replacement process.

alex-hunt-materialize · 2025-03-27T18:03:09Z

doc/developer/design/20250321_password_authentication.md

+
+### 4. Configurable admin system login:
+
+Passwords for `mz_system` and `mz_support` roles will be settable via environment variables `MZ_MZ_<USER>_EXTERNAL_LOGIN_PASSWORD`. Orchestratord should provide a set of parameters to set these variables via a Kubernetes secret. Additionally, We should enable login of system users through the external ports when they have external login passwords set. Login through the external port must not be possible unless this flag is set, this logic should not rely on whether the internal user has a password.


This may need an updated Materialize CRD to add a new field for the name of the secret.

alex-hunt-materialize · 2025-03-27T18:09:23Z

doc/developer/design/20250321_password_authentication.md

+
+3. **Store Passwords in K8s Secrets**: It would be viable to store passwords in Kubernetes secrets.
+
+-   **Reasons Not Chosen**: This may create an untenable number of Kubernetes secrets.


I thought we just discussed this as what we were going to do for our initial implementation. This is also just a detail of the implementation (where we store things).

Yeah I think this is from the original pass. Let's remove

After thinking about this some more and talking it over, I think we should just store passwords in the catalog. This is considered user metadata, it seems like it should be relatively simple to keep there may simplify security reviews.

see:
https://materializeinc.slack.com/archives/C07PN7KSB0T/p1743173985961509

alex-hunt-materialize · 2025-03-27T18:24:29Z

@SangJunBak +1 to everything you said.

But I'm also confused how the Console might receive this information if we can't query the database to begin with.

Maybe we can pass that to the console also? Maybe through an env var? That gets a bit tricky, since I think we update the balancers and console immediately upon getting an updated Materialize, while only updating environmentd if we trigger a rollout. Perhaps this logic needs to change, but that may be difficult too.

jubrad · 2025-03-27T18:54:00Z

Still not clear to me the flow of setting up auth. In my head, it looks like:

As they initially set up via the helm chart, orchestratord will have the env variable to set the password of the mz_system user. This is probably configurable via the helm chart?

Once the user deploys the materialize CR, orchestratord will pass that env variable in order to start the environment with an mz_system type user

The user can now login via that user and add more roles

Does that sound correct? Then we just make it a limitation that a user can't use the Console unless they have . In the Console, we'll most likely need the following information

Is (password) auth enabled?

Does there exist an authenticated user?
But I'm also confused how the Console might receive this information if we can't query the database to begin with.

Should also consider the migration path from self managed customers without auth to customers with auth.

Your initial flow is correct!

Then we just make it a limitation that a user can't use the Console unless they have .

Unless they have what?

The standard auth for postgres has login and non login users see pg_authid. For logins, one can first check that the role allows login then compare passwords. This would be done fully on the server side. I imagine we want to enable rolcanlogin if a password is set, and by that just check to see password.is_some() or something.

SangJunBak · 2025-03-27T20:04:46Z

@jubrad

Unless they have what?
Ah my bad. Unless there's a registered role! Edited.

The standard auth for postgres has login and non login users see pg_authid.

But I mean in order for the Console to query pg_authid via the HTTP API, wouldn't they need to be authenticated?

SangJunBak · 2025-03-27T20:06:48Z

Maybe we can pass that to the console also? Maybe through an env var? That gets a bit tricky, since I think we update the balancers and console immediately upon getting an updated Materialize

@alex-hunt-materialize and with the current design, an env variable won't do much good given the Console's already built by the time orchestratord installs the console pod. There's currently no way for orchestratord to edit build variables for the Console, which I think we should fix!

jubrad · 2025-03-27T20:45:43Z

Unless they have what?
Ah my bad. Unless there's a registered role! Edited.

The standard auth for postgres has login and non login users see pg_authid.

But I mean in order for the Console to query pg_authid via the HTTP API, wouldn't they need to be authenticated?

Console shouldn't need to query pg_authid. It should just need to attempt to connect using username and password. If the role isn't a login role or there's a password mismatch that should all be handled server side. To determine if auth is enabled it could just attempt to run a query and, on failed login, route to the login box, or we can install a configurable endpoint on NGINX that console can hit to tell it what kind of auth it should attempt to use.

SangJunBak

LGTM on my end! Thanks for writing all this out!

SangJunBak · 2025-03-28T19:39:09Z

doc/developer/design/20250321_password_authentication.md

@@ -127,7 +130,7 @@ The authentication mechanism for an environment must be configurable. A new flag

 ### Note on Console builds:

-We currently re-use the same Console build for the emulator and self-managed. For practical purposes we can build both with password auth enabled with no option to disable. 
+Console does currently do not support runtime or startup configuration. Configuration is handled only at build time. To resolve this we should add a `config.json` or `config.js` file which can be mounted directly into the Nginx container assets. This file should come from a materialize-console config map which must be setup by Orchestratord.  We will also need changes to the console to support reading in configuration from this map. The initial config value here should be `authentication_type: password`, in cloud we should use `authentication_type: frontegg` or `authentication_type: jwt`. The console build process can still be used to set default values for this config file. 


in the case where someone wants to bump the Console version or if we ever need to reinstall the console service, we can most likely just swap out the config file given the Console will get it on runtime.

We'll most likely just block the UI until the config map is loaded in (i.e. via a loading screen). This is fine given loading it in should be really quick.

jasonhernandez

My "hard requirements"

Set the minimum iteration count >=200,000.
Do not allow lockouts by default.
Log successful and failed login attempts by default.

I've made some additional comments and recommendations for discussion / consideration. Thanks for writing this all up and the consideration in this design.

jasonhernandez · 2025-04-03T00:32:41Z

doc/developer/design/20250321_password_authentication.md

+   a. The coordinator layer will check the password against the stored hash. If the password is correct, the user will be authenticated.
+   b. If the password is incorrect, the user will be denied access.
+
+3. **OPTIONAL: Book-Keeping**: Login and failed login requests should be tracked. Metrics should be created to monitor potential security events.


We should log successful and failed login attempts, as closely to what Postgres does as possible.

jasonhernandez · 2025-04-03T00:33:27Z

doc/developer/design/20250321_password_authentication.md

+
+**Password Rotation or Expiration**: While not currently in scope, the design should allow for additions in the future, such as password rotation and expiration.
+
+**Abuse Detection**: Action should be taken action when brute force attempts are detected, either by locking the user or blocking the requesting client IP.


We should write logs and let customers use fail2ban or whatever solutions they use elsewhere.

jasonhernandez · 2025-04-03T02:02:09Z

doc/developer/design/20250321_password_authentication.md

+
+### 3. Hashed Password Storage
+
+-   **Secure Hashing**: For future compatibility with SCRAM, passwords should be hashed using `scram-sha-256` encryption. The RFC for [SCRAM](https://datatracker.ietf.org/doc/html/rfc5802) dictates an iteration count of at least 4096. However, the RFC for SCRAM-SHA-256 dictates that "[the hash iteration-count should be such that a modern machine will take 0.1 seconds to perform the complete algorithm](https://www.rfc-editor.org/rfc/rfc7677)". This also dictates at least 4096 with a recommendation of 15000. We should aim for 15000 if it does not negatively impact connection times or load.


This RFC is 10 years old.
I ran a quick benchmark on two computers:
t4g.nano - 200,000 iterations in 0.100s
Macbook Pro M3 Max - 450,000 iterations in 0.102s

I recommend we plan on a minimum of 400,000 iterations for now. I would also like us to store the iteration count in a column in the database. If practical, it would be very nice to randomly generate an iteration count for each password, perhaps in the range of 400,000-450,000 iterations. This would significantly reduce the probability of birthday attacks and the usefulness of rainbow tables. We will want to track the iteration count for passwords regardless, because we may need to increase the number of iterations as computers get faster.

I will try and do some more research and might adjust my recommended iteration count.

(note: edited to increase iterations given the 2021 draft RFC https://www.ietf.org/archive/id/draft-ietf-kitten-password-storage-07.txt)

This (expired) draft RFC from 2021 recommends a minimum of 310,000 iterations.
https://www.ietf.org/archive/id/draft-ietf-kitten-password-storage-07.txt

We should also specify a length on salts. We should use at least 32 bytes of cryptographically random data for salts.

jasonhernandez · 2025-04-03T02:28:41Z

doc/developer/design/20250321_password_authentication.md

+
+3. **Store Passwords in K8s Secrets**: It would be viable to store passwords in Kubernetes secrets.
+
+-   **Reasons Not Chosen**: This may create an untenable number of Kubernetes secrets, and may cause issues with passing security reviews.


Using Kubernetes secrets also creates some additional complexity for backup / restore / migration. There's an advantage to keeping the same mental model as Postgres where possible.

jasonhernandez · 2025-04-03T02:44:17Z

doc/developer/design/20250321_password_authentication.md

+
+2. **Audit Logging and Brute Force Detection**: Do we need to log all failed password attempts and the IPs of those attempts? Do we need to take action to lock an IP or User when an attempt threshold is met?
+
+3. **Password Strength Validation**: Should Materialize include built-in password strength validation, or should it be handled outside the system (e.g., via Kubernetes or other security tools)?


I'm not sure if this is necessary yet. If we included the requirements, we would want to include an override setting for developer mode.
If we were to implement this, I would suggest copying the behavior of Postgres passwordcheck without cracklib + requiring an entropy score of 3+ from zxcvbn.
https://crates.io/crates/zxcvbn

Docs on Postgres passwordcheck
https://www.postgresql.org/docs/current/passwordcheck.html
https://github.com/postgres/postgres/blob/master/contrib/passwordcheck/passwordcheck.c

password-auth-design-doc

ce40d72

WIP password auth design doc

alex-hunt-materialize reviewed Mar 25, 2025

View reviewed changes

jubrad commented Mar 25, 2025

View reviewed changes

doc/developer/design/20250321_password_authentication.md Outdated Show resolved Hide resolved

DAlperin and others added 2 commits March 27, 2025 13:11

update implementation details

1896787

Add some better takes on internal user login

a595c81

jubrad requested a review from alex-hunt-materialize March 27, 2025 17:42

SangJunBak reviewed Mar 27, 2025

View reviewed changes

alex-hunt-materialize reviewed Mar 27, 2025

View reviewed changes

jubrad added 2 commits March 28, 2025 13:20

add some clarification on pw storage and http auth

2f2eec3

add blurb about console startup config

2c5754d

jubrad force-pushed the password-auth-design-doc branch from 291b450 to 2c5754d Compare March 28, 2025 18:52

SangJunBak approved these changes Mar 28, 2025

View reviewed changes

SangJunBak reviewed Mar 28, 2025

View reviewed changes

add note about emulator auth and mz_system password setting

bba9cfc

jubrad marked this pull request as ready for review April 1, 2025 01:00

jasonhernandez requested changes Apr 3, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP Password auth design doc #32005

WIP Password auth design doc #32005

jubrad commented Mar 25, 2025

alex-hunt-materialize Mar 25, 2025

jasonhernandez Apr 2, 2025

SangJunBak commented Mar 26, 2025 •

edited

Loading

jubrad commented Mar 26, 2025

alex-hunt-materialize commented Mar 26, 2025

DAlperin commented Mar 26, 2025

jubrad commented Mar 27, 2025

SangJunBak Mar 27, 2025

alex-hunt-materialize Mar 27, 2025 •

edited

Loading

jubrad Mar 27, 2025

SangJunBak commented Mar 27, 2025 •

edited

Loading

alex-hunt-materialize Mar 27, 2025

jasonhernandez Apr 3, 2025

alex-hunt-materialize Mar 27, 2025 •

edited

Loading

alex-hunt-materialize Mar 27, 2025

alex-hunt-materialize Mar 27, 2025

DAlperin Mar 27, 2025

jubrad Mar 28, 2025

alex-hunt-materialize commented Mar 27, 2025

jubrad commented Mar 27, 2025 •

edited

Loading

SangJunBak commented Mar 27, 2025

SangJunBak commented Mar 27, 2025

jubrad commented Mar 27, 2025

SangJunBak left a comment

SangJunBak Mar 28, 2025 •

edited

Loading

SangJunBak Mar 28, 2025

jasonhernandez left a comment

jasonhernandez Apr 3, 2025

jasonhernandez Apr 3, 2025

jasonhernandez Apr 3, 2025 •

edited

Loading

jasonhernandez Apr 3, 2025

jasonhernandez Apr 3, 2025

jasonhernandez Apr 3, 2025


		### 4. Configurable admin system login:

		Passwords for `mz_system` and `mz_support` roles will be settable via environment variables `MZ_MZ_<USER>_EXTERNAL_LOGIN_PASSWORD`. Orchestratord should provide a set of parameters to set these variables via a Kubernetes secret. Additionally, We should enable login of system users through the external ports when they have external login passwords set. Login through the external port must not be possible unless this flag is set, this logic should not rely on whether the internal user has a password.


		Password Rotation or Expiration: While not currently in scope, the design should allow for additions in the future, such as password rotation and expiration.

		Abuse Detection: Action should be taken action when brute force attempts are detected, either by locking the user or blocking the requesting client IP.


		- Password Dating: The creation date of passwords will be recorded along with the hashed and salted password. When the password changes a creation date should be reset. The creation date of passwords should be queryable.

		- OPTIONAL: Password Versioning: Updates to the password hashing mechanisms may be required. As long as we are receiving passwords in plain text we should be able to take a validated password and replace the existing a new securely hashed value. This may require prefixing the password with data about the hash algorithm or parameters.


		3. Store Passwords in K8s Secrets: It would be viable to store passwords in Kubernetes secrets.

		- Reasons Not Chosen: This may create an untenable number of Kubernetes secrets.


		### 3. Hashed Password Storage

		- Secure Hashing: For future compatibility with SCRAM, passwords should be hashed using `scram-sha-256` encryption. The RFC for [SCRAM](https://datatracker.ietf.org/doc/html/rfc5802) dictates an iteration count of at least 4096. However, the RFC for SCRAM-SHA-256 dictates that "[the hash iteration-count should be such that a modern machine will take 0.1 seconds to perform the complete algorithm](https://www.rfc-editor.org/rfc/rfc7677)". This also dictates at least 4096 with a recommendation of 15000. We should aim for 15000 if it does not negatively impact connection times or load.


		2. Audit Logging and Brute Force Detection: Do we need to log all failed password attempts and the IPs of those attempts? Do we need to take action to lock an IP or User when an attempt threshold is met?

		3. Password Strength Validation: Should Materialize include built-in password strength validation, or should it be handled outside the system (e.g., via Kubernetes or other security tools)?

WIP Password auth design doc #32005

Are you sure you want to change the base?

WIP Password auth design doc #32005

Conversation

jubrad commented Mar 25, 2025

Motivation

Tips for reviewer

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SangJunBak commented Mar 26, 2025 • edited Loading

jubrad commented Mar 26, 2025

alex-hunt-materialize commented Mar 26, 2025

DAlperin commented Mar 26, 2025

jubrad commented Mar 27, 2025

Choose a reason for hiding this comment

alex-hunt-materialize Mar 27, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SangJunBak commented Mar 27, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alex-hunt-materialize Mar 27, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alex-hunt-materialize commented Mar 27, 2025

jubrad commented Mar 27, 2025 • edited Loading

SangJunBak commented Mar 27, 2025

SangJunBak commented Mar 27, 2025

jubrad commented Mar 27, 2025

SangJunBak left a comment

Choose a reason for hiding this comment

SangJunBak Mar 28, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jasonhernandez left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jasonhernandez Apr 3, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SangJunBak commented Mar 26, 2025 •

edited

Loading

alex-hunt-materialize Mar 27, 2025 •

edited

Loading

SangJunBak commented Mar 27, 2025 •

edited

Loading

alex-hunt-materialize Mar 27, 2025 •

edited

Loading

jubrad commented Mar 27, 2025 •

edited

Loading

SangJunBak Mar 28, 2025 •

edited

Loading

jasonhernandez Apr 3, 2025 •

edited

Loading