RFC: Env variables management

[witoszekdev]

Each FE project in Saleor has different way of handling env variables and validating them. Each has some upsides and downsides, we should rethink our approach so that our apps are less error-prone to user error (invalid env vars, missing configuration, etc.).

Current usage

Saleor Core

Described on docs.saleor.io.

For macOS/Linux: we recommend using dotenv to set env variables. We'll skip how validation is done because core is written in Python.

Saleor Dashboard

Saleor Dashboard doesn't use any package for loading environment variables. Instead, loadEnv from Vite is used to load every env variable:
https://github.com/saleor/saleor-dashboard/blob/main/vite.config.js#L13

After that, we select a handful of env vars to be loaded into process.env: https://github.com/saleor/saleor-dashboard/blob/main/vite.config.js#L130:L140

This is done due to security concerns.
Internally, Vite uses dotenv to load environment variables from .env files. Previously, we've recommended using direnv to load the environment variables.

React Storefront

React Storefront is a monorepo, so configuration will be different from a single-repo project.

For entire monorepo it uses env-cmd in package.json scripts to load environment variables from .env and .env.local files: https://github.com/saleor/react-storefront/blob/canary/package.json#L17

Those files are located at the root of monorepo.

Each project can be also run independently, it can have its own .env and .env.local files that reference environment variables from the root of monorepo (e.g. in ./.env we declare variable TEST we can reuse that variable in ./app/my-app/.env like that: OTHER_ENV=$TEST). Those projects can be run from their own directory, for example when running tests. Because of that, we've developed a special package (env-vars) that loads env variables in correct order: https://github.com/saleor/react-storefront/tree/canary/packages/env-vars

env-vars uses internally dotenv, dotenv-expand and find-config: https://github.com/saleor/react-storefront/blob/canary/packages/env-vars/package.json

Saleor App Template

It uses default Next.js behavior for loading environment variables.

TL;DR: It loads envs from .env.local, but only envs that start with NEXT_PUBLIC_ to the frontend (by default it's only visible in Node.js environment - aka API Routes, getStaticProps, etc.)

Main idea

• Environment variables can be defined in .env and .env.local files. .env is commited into the repository
  • For monorepo: apps that require additional variables can have their own .env files, they should load environment variables from the root of monorepo as well (like we do in react-storefront right now)
• Environment variables should be validated if they are correct (they exist, have a value from a set, etc.) if not app build should fail
• We don't want to load every environment variable
• Server env variables shouldn't be exposed on the frontend
• Optional: Users should be notified on .env file update that they need to restart dev server in order to see changes / re-build the project

Validation solutions

zod schema

Inspired by create-t3-app.

We declare a schema for server and frontend environment variables using zod. See example

export const serverSchema = z.object({
  NODE_ENV: z.enum(["development", "test", "production"]),
});

export const serverEnv = {
  NODE_ENV: process.env.NODE_ENV,
};

export const clientSchema = z.object({
  // NEXT_PUBLIC_CLIENTVAR: z.string(),
});

export const clientEnv = {
  // NEXT_PUBLIC_CLIENTVAR: process.env.NEXT_PUBLIC_CLIENTVAR,
};

We create two files with logic for loading backend and frontend env variables. They can be included for example in next.config.js to be validated when a project is stared. Validation can also be skipped when a special env is passed (useful for Docker):

/**
 * Run `build` or `dev` with `SKIP_ENV_VALIDATION` to skip env validation.
 * This is especially useful for Docker builds.
 */
!process.env.SKIP_ENV_VALIDATION && (await import("./src/env/server.mjs"));

To use the env variable you import it from the file that sets them up:

import { env } from "../../env/server.mjs";

Sum-up

• ✅ Type-safe (environment variables have a declared type)
• ✅ Expressive validation thanks to zod, we can have many custom rules for validation
• ❌ We need to write our custom code for throwing errors when env variables don't match the schema
• ❌ We need to import env variables from one file

Use env-vars package

Similar to zod, except it can be used as a direct replacement from using process.env. It also doesn't require additional files for set-up:

const PASSWORD = env.get('DB_PASSWORD')
  // Throws an error if the DB_PASSWORD variable is not set (optional)
  .required()
  // Decode DB_PASSWORD from base64 to a utf8 string (optional)
  .convertFromBase64()
  // Call asString (or other APIs) to get the variable value (required)
  .asString();

// Read in a port (checks that PORT is in the range 0 to 65535)
// Alternatively, use a default value of 5432 if PORT is not defined
const PORT = env.get('PORT').default('5432').asPortNumber()

It has many built-in validators for most env vars use cases, but a custom accessor can also be written.

Sum-up

• ✅ Type-safe (validators narrow down the type of env variables)
• ✅ Even more validators specific to env vars (e.g. checking ports)
• ✅ Can be used as a direct replacement for process.env
• 💡 Nice to have: has built-in logger for displaying every loaded env var
• ❌ Doesn't have a separation from server and client environment variables like zod

Env loading solutions

Custom env vars package

We could re-use the package from react-storefront that loads the env variables in correct order.

• ✅ Has everything we need in a monorepo
• ❌ More maintenance burden

Using plain dotenv

This doesn't require any changes, we keep on using dotenv either via directly including this package or by a pre-build solution in one of our tools (e.g. Next.js, Vite)

• ✅ No work to do 🥳, no maintance burden
• ✅ Uses a standard dotenv package
• ❌ Might not work correctly in a monorepo

Not using any package for loading env vars

This is something we already do in core. You could argue that the job of loading environment variables should be on user, because it depends based on their environment (e.g. Windows vs Docker vs macOS).

To make this easier, we could recommend our users to use the direnv package that loads environment variables from .env or .envrc file. It uses a standard way of our uses shell for loading env variables (e.g. export XYZ=123 in bash/zsh).

• ✅ Uses OS standard way of loading env vars
• ✅ Tooling can be decided by the user to best suit their needs
• ❌ Might not be an expected behavior by most FE developers (.env is loaded automatically, vs it requires installing an external software and running direnv allow on every change)
• ❌ Requires to run a special command on each .env file change
• ❌ Doesn't solve loading of env vars in monorepo - env vars can only be specified in one file at the root of monorepo

[typeofweb]

Validation solutions

We declare a schema for server and frontend environment variables using zod

As we're already using yup and we're planning on using json-schema for webhooks validation, I'm not sure if adding another tool for validation is a good idea 🗡️

Use env-vars package
Similar to zod, except it can be used as a direct replacement from using process.env. It also doesn't require additional files for set-up:

const PASSWORD = env.get('DB_PASSWORD')
  // Throws an error if the DB_PASSWORD variable is not set (optional)
  .required()
  // Decode DB_PASSWORD from base64 to a utf8 string (optional)
  .convertFromBase64()
  // Call asString (or other APIs) to get the variable value (required)
  .asString();

// Read in a port (checks that PORT is in the range 0 to 65535)
// Alternatively, use a default value of 5432 if PORT is not defined
const PORT = env.get('PORT').default('5432').asPortNumber()

⬆️ this example looks nice but it'll only work with Node.js. Not with Vite. Not with Next.js. Not with CRA.

The issue is that on the frontend you need to explicitly provide process.env.X literally so it's statically analyzable and bundlers can find & replace these references with correct values. env-var with Next.js is a bit more explicit:

import { from } from 'env-var';

const { get } = from({
  SOME_VALUE: process.env.NEXT_PUBLIC_SOME_VALUE,
  PORT: process.env.NEXT_PUBLIC_PORT
})

const SOME_VALUE = get('SOME_VALUE')
  .required()
  .convertFromBase64()
  .asString();

const PORT = get('PORT')
  .default('5432')
  .asPortNumber();

It's still quite good and I 👍 this.

Env loading solutions

Using plain dotenv

We should also use dotenv-expand if we have to load env vars manually. It takes care of variables inside variables (Yo Dawg, I heard you like variables so I put variable in a variable so you can expand while you expand!)

We also want to load multiple files in a specific order – similar to what Next.js does.

[krzysztofzuraw]

Thanks for the RFC! It looks great and I have a couple of comments.

Firstly, we have to consider who will be using our dashboard. Currently, I think there are at least 3 groups:

• developers
• external people who want to self-host the dashboard
• testers

With this in mind, I’m wondering if:

• How will the new solution work with docker and dynamic env management during runtime? See more about how it can be done today at https://github.com/saleor/saleor-dashboard/blob/main/docs/docker.md
• How can it be used with cypress by QA and preferably it should be the same way as developers are loading their envs?

Other than that I think I quite like an env-var library.