Fix SHC appframework bundle-push loop in FIPS mode by dpericaxon · Pull Request #1672 · splunk/splunk-operator

dpericaxon · 2026-01-22T14:03:36Z

Description

This PR fixes a retry storm / “app update loop” seen when Splunk Operator (v3.0.0) manages an SHC with App Framework enabled and Splunk is running in FIPS mode. In this scenario, the SHC bundle push status file can contain benign FIPS/TLS banner output (and sometimes ConfDeploymentException: ... already running) which was previously treated as a hard failure. The operator would reset the bundle push state back to Pending and immediately re-trigger the bundle push, causing collisions and preventing SHC app updates from completing.

This change makes the status parsing more robust by treating known-benign output as “still in progress” rather than a failure.

Key Changes

pkg/splunk/enterprise/afwscheduler.go
- Use normalized status output (strip benign FIPS banner + TLS hostname warning) when determining completion.
- Treat banner-only output as in-progress (do not reset bundle state to Pending).
- Treat ConfDeploymentException: Can't start deployment job as one is already running! as in-progress (do not reset bundle state to Pending / do not re-trigger).
- Preserve existing behavior for true errors (non-empty output that is not success and not the known benign cases).
pkg/splunk/enterprise/afwscheduler_test.go
- Add unit test coverage for:
  - FIPS banner-only status output ⇒ remains BundlePushInProgress
  - FIPS banner + “already running” ⇒ remains BundlePushInProgress

Testing and Verification

Unit tests:
- go test ./pkg/splunk/enterprise -run '^TestSHCRunPlaybook$' -v
Behavioral verification (manual):
- Verified the operator no longer resets bundle push state to Pending when the status output contains only the FIPS/TLS banner or the “already running” ConfDeploymentException.
- Verified SHC bundle push progresses without a re-trigger storm and SHC app updates can complete.

Related Issues

GitHub: splunk/splunk-operator issue App Framework: #1611 (FIPS / SHC bundle push loop)

PR Checklist

Code changes adhere to the project's coding standards.
Relevant unit and integration tests are included.
Documentation has been updated accordingly.
All tests pass locally.
The PR description follows the project's guidelines.

- Treat FIPS banner-only status output as in-progress - Treat ConfDeploymentException 'already running' as in-progress - Add unit tests

rlieberman-splunk · 2026-01-22T20:11:20Z

pkg/splunk/enterprise/afwscheduler.go

+			continue
+		}
+
+		// FIPS banner emitted by Splunk CLI in FIPS-enabled environments


Is there a list of benign banners somewhere that we can add to the checks here? I'm worried if we merge this it will only cover a specific scenario with hard coded values.

If we don't find out, then as of now, we can just make a list of such banners (as of now 1-element), use this list in the code to validate the output and once we learn more, we can expand the list with new banners.

@rlieberman-splunk @kasiakoziol is this an action item on your side to dig into this? For now I Implemented a benignLinePrefixes list in normalizeSHCBundlePushStatusOutput() (currently includes the known FIPS banner and the TLS hostname validation warning). This keeps the logic extensible and we can expand the list as additional benign lines are identified.Implemented a benignLinePrefixes list in normalizeSHCBundlePushStatusOutput() (currently includes the known FIPS banner and the TLS hostname validation warning). This keeps the logic extensible and we can expand the list as additional benign lines are identified.

kasiakoziol · 2026-01-23T13:14:07Z

pkg/splunk/enterprise/afwscheduler.go

+	normalizedOut := normalizeSHCBundlePushStatusOutput(stdOut)
+
 	// Check if we did not get the desired output in the status file. There can be 2 scenarios -
 	// 1. stdOut is empty, which means bundle push is still in progress


Please update the comment for consistency - stdOut to normalizedOut

Done! Updated the comment to reference normalizedOut for consistency with the normalized parsing logic.

kasiakoziol · 2026-01-23T13:32:27Z

Did you perform manual tests with the fix on both FIPS and non-FIPS enabled clusters?

github-actions · 2026-01-23T14:35:27Z

CLA Assistant Lite bot CLA Assistant Lite bot All contributors have signed the COC ✍️ ✅

dpericaxon · 2026-01-23T14:49:48Z

I have read the Code of Conduct and I hereby accept the Terms

dpericaxon · 2026-01-23T14:51:45Z

recheck

dpericaxon · 2026-01-23T14:52:49Z

I have read the CLA Document and I hereby sign the CLA

kubabuczak · 2026-01-23T14:58:00Z

I have read the CLA Document and I hereby accept the Terms

@dpericaxon you need to post comment with "I have read the CLA Document and I hereby sign the CLA" - you hae slightly different wording

dpericaxon · 2026-01-23T15:04:44Z

Did you perform manual tests with the fix on both FIPS and non-FIPS enabled clusters?

I did it for FIPs clusters. Do you perhaps have a non-fips cluster to test in quickly? If not I can see what I can come up with.

dpericaxon · 2026-04-01T15:41:22Z

Hey @kubabuczak and @gabrielm-splunk I noticed that a new release got cut but the change above didn't make it in. Isn't this effecting other customers running on fips pods?

kubabuczak · 2026-04-03T12:40:47Z

Calling @rlieberman-splunk , as she is working on FIPS certification for this release

rlieberman-splunk · 2026-04-07T13:30:19Z

Hi @dpericaxon , I have opened a PR with similar changes and tested them on our FIPS enabled environment. These changes will be included in the next release.

Fix SHC appframework bundle-push loop in FIPS mode

83288f4

- Treat FIPS banner-only status output as in-progress - Treat ConfDeploymentException 'already running' as in-progress - Add unit tests

dpericaxon mentioned this pull request Jan 22, 2026

Fix SHC appframework bundle-push loop in FIPS mode #1669

Closed

5 tasks

rlieberman-splunk reviewed Jan 22, 2026

View reviewed changes

kasiakoziol reviewed Jan 23, 2026

View reviewed changes

kasiakoziol mentioned this pull request Jan 23, 2026

App Framework: #1611

Open

Normalize SHC bundle-push status output using a benign banner list

bd9a89a

rlieberman-splunk closed this Apr 7, 2026

github-actions bot locked and limited conversation to collaborators Apr 7, 2026

Conversation

dpericaxon commented Jan 22, 2026

Description

Key Changes

Testing and Verification

Related Issues

PR Checklist

Uh oh!

rlieberman-splunk Jan 22, 2026

Choose a reason for hiding this comment

Uh oh!

kasiakoziol Jan 23, 2026

Choose a reason for hiding this comment

Uh oh!

dpericaxon Jan 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kasiakoziol Jan 23, 2026

Choose a reason for hiding this comment

Uh oh!

dpericaxon Jan 23, 2026

Choose a reason for hiding this comment

Uh oh!

kasiakoziol commented Jan 23, 2026

Uh oh!

github-actions bot commented Jan 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dpericaxon commented Jan 23, 2026

Uh oh!

dpericaxon commented Jan 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dpericaxon commented Jan 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kubabuczak commented Jan 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dpericaxon commented Jan 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dpericaxon commented Apr 1, 2026

Uh oh!

kubabuczak commented Apr 3, 2026

Uh oh!

rlieberman-splunk commented Apr 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

dpericaxon Jan 23, 2026 •

edited

Loading

github-actions bot commented Jan 23, 2026 •

edited

Loading

dpericaxon commented Jan 23, 2026 •

edited

Loading

dpericaxon commented Jan 23, 2026 •

edited

Loading

kubabuczak commented Jan 23, 2026 •

edited

Loading

dpericaxon commented Jan 23, 2026 •

edited

Loading