Support keyring encrypted credential storage

[lmmx]

Outline

• Setting env. variables for auth config is awkward, so python-dotenv can be used to load them from a .env text file instead

• Keyrings can be preferable to plain text secret storage for security, and Python has a keyring module [source]

  These recommended keyring backends are supported:

  • macOS Keychain
  • Freedesktop Secret Service supports many DE including GNOME (requires secretstorage)
  • KDE4 & KDE5 KWallet (requires dbus)
  • Windows Credential Locker

• This practice is used by other well known tools such as the GitHub CLI tool gh and twine [source]

  Where Twine gets configuration and credentials
  A user can set the repository URL, username, and/or password via command line, .pypirc files, environment variables, and keyring.

• This could be made an optional dependency if desired (perhaps to keep package size minimal/consistent).

Impact

When I pip install keyring on Linux after first installing pydantic and pydantic-settings the additional dependencies are:

Installing collected packages: zipp, pycparser, more-itertools, jeepney, jaraco.classes, importlib-metadata, cffi, cryptography, SecretStorage, keyring

Usage

Once installed, secret access is achieved like so:

import keyring
my_secret = keyring.get_password("MY_SECRET", "secret_username")

The gh tool sets the username to an empty string, indicating that it's used as a simple key-value secret store.

You can also access specific keyrings, also known as 'collections' (for instance if you wanted to have different applications using different keys with the same name, say a different API key for different services). For reference

Proposed implementation

Essentially we are replacing os.environ.get(validation_alias) for keyring.get_password(validation_alias)

In this library, both environment variables and .env configured variables are loaded into the env_vars attribute.

EnvSettingsSource calls _load_env_vars() at initialisation:

pydantic-settings/pydantic_settings/sources.py

Lines 376 to 381 in 5933ea6

	self.env_vars = self._load_env_vars()
	def _load_env_vars(self) -> Mapping[str, str | None]:
	if self.case_sensitive:
	return os.environ
	return {k.lower(): v for k, v in os.environ.items()}

DotEnvSettingsSource subclasses EnvSettingsSource and overrides the _load_env_vars() method

pydantic-settings/pydantic_settings/sources.py

Lines 571 to 590 in 5933ea6

	def _load_env_vars(self) -> Mapping[str, str | None]:
	return self._read_env_files(self.case_sensitive)
	def _read_env_files(self, case_sensitive: bool) -> Mapping[str, str | None]:
	env_files = self.env_file
	if env_files is None:
	return {}
	if isinstance(env_files, (str, os.PathLike)):
	env_files = [env_files]
	dotenv_vars: dict[str, str | None] = {}
	for env_file in env_files:
	env_path = Path(env_file).expanduser()
	if env_path.is_file():
	dotenv_vars.update(
	read_env_file(env_path, encoding=self.env_file_encoding, case_sensitive=case_sensitive)
	)
	return dotenv_vars

I would have this work similarly to .env handling with a subclass exposing a custom way to load env vars.

We can enumerate all keys (as bytes) via:

all_items = keyring.core.get_keyring().get_preferred_collection().get_all_items()
keyring_vars: dict[str, str] = {
    item.get_attributes()["service"]: item.get_secret().decode()
    for item in all_items
}

(In real code you'd have to have some error handling in case the 3 chained methods error!)

I think default conversion of bytes to str type would be reasonable here?

Proof of concept

The attached PR supplies a working implementation of this feature on Linux, using the SecretStorage backend.

(keyringsettings) louis 🚶 ~/dev/testing/pydantic-settings $ keyring set MY_SECRET_KEY ''
Password for '' in 'MY_SECRET_KEY': 
(keyringsettings) louis 🚶 ~/dev/testing/pydantic-settings $ keyring get MY_SECRET_KEY ''
abc

from pydantic_settings import BaseSettings, SettingsConfigDict


class Settings(BaseSettings):
    MY_SECRET_KEY: str
    model_config = SettingsConfigDict(extra="ignore")


s = Settings()
print(s)

(keyringsettings) louis 🚶 ~/dev/testing/pydantic-settings $ python keyring_demo.py 
MY_SECRET_KEY='abc'

This is overridden by setting an environment variable.

(keyringsettings) louis 🚶 ~/dev/testing/pydantic-settings $ export MY_SECRET_KEY="foo"
(keyringsettings) louis 🚶 ~/dev/testing/pydantic-settings $ python keyring_demo.py 
MY_SECRET_KEY='foo'
(keyringsettings) louis 🚶 ~/dev/testing/pydantic-settings $ unset MY_SECRET_KEY
(keyringsettings) louis 🚶 ~/dev/testing/pydantic-settings $ python keyring_demo.py 
MY_SECRET_KEY='abc'

Selected Assignee: @dmontagu

[dmontagu]

Since @hramezani already replied on the PR I'll assign him here, but in general I think it makes sense for us to support/maintain this if you are willing to provide an initial implementation. (And it seems like you are — thank you!)

[ELC]

Are there any updates on this feature?

[lmmx]

Are there any updates on this feature?

Happy annual kanban refresh to all who celebrate: I've put it back onto the TODO list 🫣 ⏳