App Framework:

[skanchi-apptio]

Please select the type of request

Bug

Tell us more

Describe the request
When AppFramework is enabled on a Search Head Cluster (SHC) in Splunk Operator v3.0.0, the operator continuously retries SHC bundle pushes, leading to a permanent “Bundle Push In Progress → Pending” loop. This prevents normal SHC operation

Expected behavior

• When we deploy first time, bundle push need to happen from SHC deployer and need to finish clean
• The bundle push shouldn't happen unless there is a new app is pushed or existing app hash changed in appRepo.

Splunk setup on K8S
• Splunk Operator v3.0.0
• Search Head Cluster (SHC) with 3 nodes
• FIPS mode enabled on Splunk pods

Reproduction/Testing steps

appRepo:
  appSources:
    - location: shc-apps/
      name: shcApps
      scope: cluster
  appsRepoPollIntervalSeconds: 600
  defaults:
    scope: local
    volumeName: splunk-apps-repo-volume
  volumes:
    - endpoint: https://s3.us-west-1.amazonaws.com
      name: splunk-apps-repo-volume
      path: splunk-apps/
      provider: aws
      region: us-west-1
      storageType: s3

Logs from controller-manager

splunk-operator-controller-manager manager [timestamp] INFO shcPlaybookContext.triggerBundlePush() Triggering bundle push {"controller": "searchheadcluster", "controllerGroup": "enterprise.splunk.com", "controllerKind": "SearchHeadCluster", "SearchHeadCluster": {"name":"shc"}, "command": "/opt/splunk/bin/splunk apply shcluster-bundle -target <SHC_CAPTAIN_URL> -auth admin:<password> --answer-yes -push-default-apps true &> /operator-staging/appframework/.shcluster_bundle_status.txt &"}
splunk-operator-controller-manager manager [timestamp] INFO setBundlePushState Setting the bundle push state {"controller": "searchheadcluster", "old state": "Bundle Push Pending", "new state": "Bundle Push In Progress"}
splunk-operator-controller-manager manager [timestamp] ERROR isBundlePushComplete SHC Bundle push status file reported an error while applying bundle {"controller": "searchheadcluster", "error": "FIPS provider enabled. ConfDeploymentException: Can't start deployment job as one is already running!"}
splunk-operator-controller-manager manager [timestamp] INFO setBundlePushState Setting the bundle push state {"controller": "searchheadcluster", "old state": "Bundle Push In Progress", "new state": "Bundle Push Pending"}
splunk-operator-controller-manager manager [timestamp] ERROR runPlaybook there was an error in SHC bundle push, will retry again {"controller": "searchheadcluster", "error": "FIPS provider enabled. ConfDeploymentException: Can't start deployment job as one is already running!"}

K8s environment
• K8s version: 1.30.14
• Cluster: EKS (AWS)
• Operator deployed via standard Deployment

Proposed changes(optional)
• Introduce a variable to disable even if AppFramework is defined, or
• Provide a clear way to disable AppFramework at SHC creation without causing bundle push loops.

[kasiakoziol]

Hello
It looks like there is a PR that might fix the problem: #1672. It is under review.
Best regards,
Kasia