Skip to content

managing osmo configs with k8s configmap #822

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
vvnpn-nv wants to merge 24 commits into
base: main
Choose a base branch
from
Open

managing osmo configs with k8s configmap #822

Show file tree
Hide file tree
Changes from 14 commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
48a9a5d
feat: add ConfigMap-sourced dynamic configuration loader
vvnpn-nv Mar 26, 2026
fe33738
fix: address bugs and code quality issues in configmap_loader
vvnpn-nv Mar 26, 2026
ec56c17
test: add comprehensive tests for ConfigMap config loader
vvnpn-nv Mar 26, 2026
67ce147
fix: address review findings, add config watcher, fix test failures
vvnpn-nv Apr 1, 2026
2ddcda3
simplify: consolidate singleton config functions, fix advisory lock
vvnpn-nv Apr 1, 2026
71bb8a6
feat: add drift reconciliation for configmap-mode configs
vvnpn-nv Apr 2, 2026
d0db789
feat: add UX improvements — managed_by visibility, immediate re-apply…
vvnpn-nv Apr 3, 2026
7c8d00f
feat: reject CLI writes to configmap-managed configs with 409 Conflict
vvnpn-nv Apr 3, 2026
1a9c887
refactor: extract configmap_guard module, fix circular import, cover …
vvnpn-nv Apr 6, 2026
230f8b3
chore: remove debug print, dead code, stale comment
vvnpn-nv Apr 6, 2026
e493ec7
fix: rollback guard referenced removed configmap_loader.reject_if_man…
vvnpn-nv Apr 6, 2026
975635e
docs: add design doc for ConfigMap-sourced dynamic configuration
vvnpn-nv Apr 7, 2026
b2f0cee
fix: guard watcher startup to avoid service boot abort
vvnpn-nv Apr 7, 2026
452c4bb
feat: extend secret file references to all config credential fields
vvnpn-nv Apr 7, 2026
42a2253
feat: auto-detect Docker registry .dockerconfigjson secret format
vvnpn-nv Apr 7, 2026
a11c049
refactor: redesign ConfigMap config to in-memory + watchdog + global …
vvnpn-nv Apr 7, 2026
25dcec2
docs: rewrite design doc for v2 architecture
vvnpn-nv Apr 7, 2026
2890e97
feat: authz_sidecar reads roles from ConfigMap file instead of DB
vvnpn-nv Apr 8, 2026
fa3705b
feat: authz_sidecar uses --roles-file when dynamicConfig enabled
vvnpn-nv Apr 8, 2026
5f98c4b
feat: move product defaults to chart values, auto-derive service_base…
vvnpn-nv Apr 8, 2026
bc5bd00
feat: add default backend and pool to chart defaults
vvnpn-nv Apr 8, 2026
670fb18
docs: update design doc for v3 — authz file-backed, chart defaults
vvnpn-nv Apr 8, 2026
c0caafe
refactor: eliminate remaining DB deps in ConfigMap mode
vvnpn-nv Apr 8, 2026
321ddae
chore: fix medium review findings — remove stale docs and dead code
vvnpn-nv Apr 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
105 changes: 105 additions & 0 deletions deployments/charts/service/templates/_helpers.tpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -144,6 +144,111 @@ Extra sidecars helper
{{- end }}
{{- end }}

{{/*
Transform secretName to secret_file in a dict.
Expects a dict context. If the dict has a "secretName" key, replaces it with
"secret_file: /etc/osmo/secrets/<secretName>/cred.yaml".
*/}}
{{- define "osmo.resolve-secret-name" -}}
{{- if .secretName -}}
{{- $secretName := .secretName -}}
{{- $_ := set . "secret_file" (printf "/etc/osmo/secrets/%s/cred.yaml" $secretName) -}}
{{- $_ := unset . "secretName" -}}
{{- end -}}
{{- end -}}

{{/*
Recursively walk a config dict and resolve secretName references at any level.
Since Go templates lack true recursion via 'template' with dynamic context mutation,
we handle this with explicit depth traversal up to 4 levels deep.
Expects a dict as context.
*/}}
{{- define "osmo.resolve-secret-names-in-config" -}}
{{- range $key, $value := . }}
{{- if kindIs "map" $value }}
{{- if hasKey $value "secretName" }}
{{- $secretName := $value.secretName }}
{{- $_ := set $value "secret_file" (printf "/etc/osmo/secrets/%s/cred.yaml" $secretName) }}
{{- $_ := unset $value "secretName" }}
{{- else }}
{{/* Recurse one level deeper */}}
{{- range $k2, $v2 := $value }}
{{- if kindIs "map" $v2 }}
{{- if hasKey $v2 "secretName" }}
{{- $sn := $v2.secretName }}
{{- $_ := set $v2 "secret_file" (printf "/etc/osmo/secrets/%s/cred.yaml" $sn) }}
{{- $_ := unset $v2 "secretName" }}
{{- else }}
{{/* Recurse another level deeper */}}
{{- range $k3, $v3 := $v2 }}
{{- if kindIs "map" $v3 }}
{{- if hasKey $v3 "secretName" }}
{{- $sn := $v3.secretName }}
{{- $_ := set $v3 "secret_file" (printf "/etc/osmo/secrets/%s/cred.yaml" $sn) }}
{{- $_ := unset $v3 "secretName" }}
{{- else }}
{{/* One more level */}}
{{- range $k4, $v4 := $v3 }}
{{- if kindIs "map" $v4 }}
{{- if hasKey $v4 "secretName" }}
{{- $sn := $v4.secretName }}
{{- $_ := set $v4 "secret_file" (printf "/etc/osmo/secrets/%s/cred.yaml" $sn) }}
{{- $_ := unset $v4 "secretName" }}
{{- end }}
{{- end }}
{{- end }}
{{- end }}
{{- end }}
{{- end }}
{{- end }}
{{- end }}
{{- end }}
{{- end }}
{{- end }}
{{- end }}
{{- end -}}

{{/*
Collect all secretName values from a config dict (up to 4 levels deep).
Returns a comma-separated string of unique secret names.
Expects a dict as context.
*/}}
{{- define "osmo.collect-secret-names" -}}
{{- $secrets := dict -}}
{{- range $key, $value := . }}
{{- if kindIs "map" $value }}
{{- if hasKey $value "secretName" }}
{{- $_ := set $secrets $value.secretName "1" }}
{{- else }}
{{- range $k2, $v2 := $value }}
{{- if kindIs "map" $v2 }}
{{- if hasKey $v2 "secretName" }}
{{- $_ := set $secrets $v2.secretName "1" }}
{{- else }}
{{- range $k3, $v3 := $v2 }}
{{- if kindIs "map" $v3 }}
{{- if hasKey $v3 "secretName" }}
{{- $_ := set $secrets $v3.secretName "1" }}
{{- else }}
{{- range $k4, $v4 := $v3 }}
{{- if kindIs "map" $v4 }}
{{- if hasKey $v4 "secretName" }}
{{- $_ := set $secrets $v4.secretName "1" }}
{{- end }}
{{- end }}
{{- end }}
{{- end }}
{{- end }}
{{- end }}
{{- end }}
{{- end }}
{{- end }}
{{- end }}
{{- end }}
{{- end }}
{{- keys $secrets | sortAlpha | join "," -}}
Comment on lines +162 to +259
Copy link
Copy Markdown

@coderabbitai coderabbitai bot Apr 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Map-only traversal misses valid secretName locations.

These helpers stop after four map levels and never descend into lists. A secretName under something like foo[0].credentials or any fifth-level object will be skipped, so the rendered config keeps the unresolved reference and the later secret-mount generation never sees that Secret name. That breaks the PR’s “any nesting level” contract.

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@deployments/charts/service/templates/_helpers.tpl` around lines 160 - 249,
The helpers osmo.resolve-secret-names-in-config and osmo.collect-secret-names
only traverse maps up to four levels and ignore slices, so secretName entries in
lists or deeper levels are missed; replace the fixed-depth loops with a
genuinely recursive helper (e.g., define "osmo.walk-config") that accepts a node
and a secrets accumulator/map and: if kindIs "map" check hasKey "secretName"
then set/unset as before and recurse over map values; if kindIs "slice"/"list"
range over items and recursively call the walker for each element; use this same
walker from osmo.resolve-secret-names-in-config to set secret_file and unset
secretName and from osmo.collect-secret-names to populate the $secrets dict,
then emit keys $secrets | sortAlpha | join "," as before (retain use of
set/unset/keys/sortAlpha/join).

{{- end -}}

{{/*
Extra configmaps helper
*/}}
Expand Down
106 changes: 105 additions & 1 deletion deployments/charts/service/templates/api-service.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,9 @@ spec:
{{- end }}
annotations:
checksum/envoy-config: {{ .Values.sidecars.envoy | toYaml | sha256sum }}
{{- if .Values.services.dynamicConfig.enabled }}
checksum/dynamic-config: {{ .Values.services.dynamicConfig | toYaml | sha256sum }}
{{- end }}
{{- include "osmo.extra-annotations" .Values.services.service | nindent 8 }}
spec:
{{- with .Values.services.service.hostAliases }}
Expand Down Expand Up @@ -149,6 +152,10 @@ spec:
- --default_admin_username
- {{ .Values.services.defaultAdmin.username | quote }}
{{- end }}
{{- if .Values.services.dynamicConfig.enabled }}
- --dynamic_config_file
- /etc/osmo/dynamic-config/config.yaml
{{- end }}
{{- range $arg := .Values.services.service.extraArgs }}
- {{ $arg | quote }}
{{- end }}
Expand Down Expand Up @@ -192,14 +199,62 @@ spec:
ports:
- name: metrics
containerPort: 9464
{{- if or .Values.services.configFile.enabled .Values.global.logs.enabled .Values.services.service.extraVolumeMounts }}
{{- if or .Values.services.configFile.enabled .Values.global.logs.enabled .Values.services.dynamicConfig.enabled .Values.services.service.extraVolumeMounts }}
volumeMounts:
{{- end }}
{{- if .Values.services.configFile.enabled}}
- mountPath: {{ .Values.services.configFile.path }}
name: mek-volume
subPath: mek.yaml
{{- end }}
{{- if .Values.services.dynamicConfig.enabled }}
- name: dynamic-config
mountPath: /etc/osmo/dynamic-config
readOnly: true
{{- range $name, $data := .Values.services.dynamicConfig.secrets }}
- name: dynamic-secret-{{ $name }}
mountPath: /etc/osmo/secrets/{{ $name }}.yaml
subPath: {{ $name }}.yaml
readOnly: true
{{- end }}
{{- if .Values.services.dynamicConfig.dataset }}
{{- range $bucketName, $bucketConfig := .Values.services.dynamicConfig.dataset.config.buckets }}
{{- if and (kindIs "map" $bucketConfig) $bucketConfig.default_credential }}
{{- if $bucketConfig.default_credential.credentialSecretName }}
- name: {{ $bucketConfig.default_credential.credentialSecretName }}
mountPath: /etc/osmo/secrets/{{ $bucketConfig.default_credential.credentialSecretName }}
readOnly: true
{{- end }}
{{- end }}
{{- end }}
{{- end }}
{{- $allSecretNames := dict }}
{{- if .Values.services.dynamicConfig.workflow }}
{{- if .Values.services.dynamicConfig.workflow.config }}
{{- $wfSecrets := include "osmo.collect-secret-names" .Values.services.dynamicConfig.workflow.config }}
{{- range (splitList "," $wfSecrets) }}
{{- if . }}
{{- $_ := set $allSecretNames . "1" }}
{{- end }}
{{- end }}
{{- end }}
{{- end }}
{{- if .Values.services.dynamicConfig.service }}
{{- if .Values.services.dynamicConfig.service.config }}
{{- $svcSecrets := include "osmo.collect-secret-names" .Values.services.dynamicConfig.service.config }}
{{- range (splitList "," $svcSecrets) }}
{{- if . }}
{{- $_ := set $allSecretNames . "1" }}
{{- end }}
{{- end }}
{{- end }}
{{- end }}
{{- range $name, $_ := $allSecretNames }}
- name: dynamic-secret-ref-{{ $name }}
mountPath: /etc/osmo/secrets/{{ $name }}
readOnly: true
{{- end }}
{{- end }}
{{- if .Values.global.logs.enabled }}
- name: logs
mountPath: /logs
Expand Down Expand Up @@ -262,6 +317,55 @@ spec:
name: mek-config
name: mek-volume
{{- end}}
{{- if .Values.services.dynamicConfig.enabled }}
- name: dynamic-config
configMap:
name: {{ .Values.services.service.serviceName }}-dynamic-config
{{- range $name, $data := .Values.services.dynamicConfig.secrets }}
- name: dynamic-secret-{{ $name }}
secret:
secretName: {{ $.Values.services.service.serviceName }}-dynamic-secret-{{ $name }}
{{- end }}
{{- if .Values.services.dynamicConfig.dataset }}
{{- range $bucketName, $bucketConfig := .Values.services.dynamicConfig.dataset.config.buckets }}
{{- if and (kindIs "map" $bucketConfig) $bucketConfig.default_credential }}
{{- if $bucketConfig.default_credential.credentialSecretName }}
- name: {{ $bucketConfig.default_credential.credentialSecretName }}
secret:
secretName: {{ $bucketConfig.default_credential.credentialSecretName }}
optional: true
{{- end }}
{{- end }}
{{- end }}
{{- end }}
{{- $allSecretNames := dict }}
{{- if .Values.services.dynamicConfig.workflow }}
{{- if .Values.services.dynamicConfig.workflow.config }}
{{- $wfSecrets := include "osmo.collect-secret-names" .Values.services.dynamicConfig.workflow.config }}
{{- range (splitList "," $wfSecrets) }}
{{- if . }}
{{- $_ := set $allSecretNames . "1" }}
{{- end }}
{{- end }}
{{- end }}
{{- end }}
{{- if .Values.services.dynamicConfig.service }}
{{- if .Values.services.dynamicConfig.service.config }}
{{- $svcSecrets := include "osmo.collect-secret-names" .Values.services.dynamicConfig.service.config }}
{{- range (splitList "," $svcSecrets) }}
{{- if . }}
{{- $_ := set $allSecretNames . "1" }}
{{- end }}
{{- end }}
{{- end }}
{{- end }}
{{- range $name, $_ := $allSecretNames }}
- name: dynamic-secret-ref-{{ $name }}
secret:
secretName: {{ $name }}
optional: true
{{- end }}
{{- end }}
---
{{- if .Values.sidecars.envoy.enabled }}

Expand Down
30 changes: 30 additions & 0 deletions deployments/charts/service/templates/dynamic-config-secrets.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# SPDX-License-Identifier: Apache-2.0
{{- if and .Values.services.dynamicConfig.enabled .Values.services.dynamicConfig.secrets }}
{{- range $name, $data := .Values.services.dynamicConfig.secrets }}
---
apiVersion: v1
kind: Secret
metadata:
name: {{ $.Values.services.service.serviceName }}-dynamic-secret-{{ $name }}
labels:
app: {{ $.Values.services.service.serviceName }}
type: Opaque
stringData:
{{ $name }}.yaml: |
{{- toYaml $data | nindent 4 }}
{{- end }}
{{- end }}
87 changes: 87 additions & 0 deletions deployments/charts/service/templates/dynamic-config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# SPDX-License-Identifier: Apache-2.0
{{- if .Values.services.dynamicConfig.enabled }}
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ .Values.services.service.serviceName }}-dynamic-config
labels:
app: {{ .Values.services.service.serviceName }}
data:
config.yaml: |
managed_configs:
{{- $dc := .Values.services.dynamicConfig }}
{{- if $dc.service }}
service:
{{- $service := deepCopy $dc.service }}
{{- if $service.config }}
{{- include "osmo.resolve-secret-names-in-config" $service.config }}
{{- end }}
{{- toYaml $service | nindent 8 }}
{{- end }}
{{- if $dc.workflow }}
workflow:
{{- $workflow := deepCopy $dc.workflow }}
{{- if $workflow.config }}
{{- include "osmo.resolve-secret-names-in-config" $workflow.config }}
{{- end }}
{{- toYaml $workflow | nindent 8 }}
{{- end }}
{{- if $dc.dataset }}
dataset:
{{- $dataset := deepCopy $dc.dataset }}
{{- if $dataset.config }}
{{- range $bucketName, $bucketConfig := $dataset.config.buckets }}
{{- if and (kindIs "map" $bucketConfig) $bucketConfig.default_credential }}
{{- if $bucketConfig.default_credential.credentialSecretName }}
{{- $secretName := $bucketConfig.default_credential.credentialSecretName }}
{{- $_ := set $bucketConfig.default_credential "secret_file" (printf "/etc/osmo/secrets/%s/cred.yaml" $secretName) }}
{{- $_ := unset $bucketConfig.default_credential "credentialSecretName" }}
{{- end }}
{{- end }}
{{- end }}
{{- end }}
{{- toYaml $dataset | nindent 8 }}
{{- end }}
{{- if $dc.pools }}
pools:
{{- toYaml $dc.pools | nindent 8 }}
{{- end }}
{{- if $dc.podTemplates }}
pod_templates:
{{- toYaml $dc.podTemplates | nindent 8 }}
{{- end }}
{{- if $dc.resourceValidations }}
resource_validations:
{{- toYaml $dc.resourceValidations | nindent 8 }}
{{- end }}
{{- if $dc.backends }}
backends:
{{- toYaml $dc.backends | nindent 8 }}
{{- end }}
{{- if $dc.backendTests }}
backend_tests:
{{- toYaml $dc.backendTests | nindent 8 }}
{{- end }}
{{- if $dc.groupTemplates }}
group_templates:
{{- toYaml $dc.groupTemplates | nindent 8 }}
{{- end }}
{{- if $dc.roles }}
roles:
{{- toYaml $dc.roles | nindent 8 }}
{{- end }}
{{- end }}
Loading