App Framework: 2.5.2 can't install packages from splunkbase, placed in s3 bucket

[yaroslav-nakonechnikov]

Please select the type of request

Bug

Tell us more

Describe the request

2024-03-19T12:18:30.383243044Z  INFO    getAppTopFolderFromPackage      Pod exec result {"controller": "standalone", "controllerGroup": "enterprise.splunk.com", "controllerKind": "Standalone", "Standalone": {"name":"c-32002","namespace":"splunk-operator"}, "namespace": "splunk-operator", "name": "c-32002", "reconcileID": "3e10b2cf-bdee-4f96-9617-bb719c6aa14b", "name": "c-32002", "namespace": "splunk-operator", "appPkgPathOnPod": "/operator-staging/appframework/base/splunk-datasets-add-on_10.tgz_0a66ca605275663dbe276ef2a8975fae", "stdOut": "splunk_datasets_addon\n"}
2024-03-19T12:18:30.383248468Z  ERROR   installApp      local scoped app package install failed while getting name of installed app     {"controller": "standalone", "controllerGroup": "enterprise.splunk.com", "controllerKind": "Standalone", "Standalone":{"name":"c-32002","namespace":"splunk-operator"}, "namespace": "splunk-operator", "name": "c-32002", "reconcileID": "3e10b2cf-bdee-4f96-9617-bb719c6aa14b", "name": "c-32002", "namespace": "splunk-operator", "pod": "splunk-c-32002-standalone-0", "app name": "splunk-datasets-add-on_10.tgz", "error": "could not get installed app name stdOut: splunk_datasets_addon\n, stdErr: tar: Ignoring unknown extended header keyword 'LIBARCHIVE.creationtime'\ntar: Ignoring unknown extended header keyword 'SCHILY.dev'\ntar: Ignoring unknown extended header keyword 'SCHILY.ino'\ntar: Ignoring unknown extended header keyword 'SCHILY.nlink'\ntar: Ignoring unknown extended header keyword 'LIBARCHIVE.creationtime'\ntar: Ignoring unknown extended header keyword 'SCHILY.dev'\ntar: Ignoring unknown extended header keyword 'SCHILY.ino'\ntar: Ignoring unknown extended header keyword 'SCHILY.nlink'\ntar: Ignoring unknown extended header keyword 'SCHILY.dev'\ntar: Ignoring unknown extended header keyword 'SCHILY.ino'\ntar: Ignoring unknown extended header keyword 'SCHILY.nlink'\n, command: tar tf /operator-staging/appframework/base/splunk-datasets-add-on_10.tgz_0a66ca605275663dbe276ef2a8975fae|head -1|cut -d/ -f1"}
github.com/splunk/splunk-operator/pkg/splunk/enterprise.installApp
        /workspace/pkg/splunk/enterprise/afwscheduler.go:724
github.com/splunk/splunk-operator/pkg/splunk/enterprise.(*localScopePlaybookContext).runPlaybook
        /workspace/pkg/splunk/enterprise/afwscheduler.go:873
2024-03-19T12:18:30.383269293Z  ERROR   localScopePlaybookContext.runPlaybook   app package installation error  {"controller": "standalone", "controllerGroup": "enterprise.splunk.com", "controllerKind": "Standalone", "Standalone": {"name":"c-32002","namespace":"splunk-operator"}, "namespace": "splunk-operator", "name": "c-32002", "reconcileID": "3e10b2cf-bdee-4f96-9617-bb719c6aa14b", "name": "c-32002", "namespace": "splunk-operator", "pod": "splunk-c-32002-standalone-0", "app name": "splunk-datasets-add-on_10.tgz", "error": "could not get installed app name stdOut: splunk_datasets_addon\n, stdErr: tar: Ignoring unknown extended header keyword 'LIBARCHIVE.creationtime'\ntar: Ignoring unknown extended header keyword 'SCHILY.dev'\ntar: Ignoring unknown extended header keyword 'SCHILY.ino'\ntar: Ignoring unknown extended header keyword 'SCHILY.nlink'\ntar: Ignoring unknown extended header keyword 'LIBARCHIVE.creationtime'\ntar: Ignoring unknown extended header keyword 'SCHILY.dev'\ntar: Ignoring unknown extended header keyword 'SCHILY.ino'\ntar: Ignoring unknown extended header keyword 'SCHILY.nlink'\ntar: Ignoring unknown extended header keyword 'SCHILY.dev'\ntar:Ignoring unknown extended header keyword 'SCHILY.ino'\ntar: Ignoring unknown extended header keyword 'SCHILY.nlink'\n, command: tar tf /operator-staging/appframework/base/splunk-datasets-add-on_10.tgz_0a66ca605275663dbe276ef2a8975fae|head -1|cut -d/ -f1"}
github.com/splunk/splunk-operator/pkg/splunk/enterprise.(*localScopePlaybookContext).runPlaybook
        /workspace/pkg/splunk/enterprise/afwscheduler.go:875

splunk-datasets-add-on_10.tgz is https://splunkbase.splunk.com/app/3245

Expected behavior
it works :)

[yaroslav-nakonechnikov]

and additional:

2024-03-19T12:38:37.599778114Z  ERROR   afwSchedulerEntry       unable to create directory on splunk pod        {"controller": "standalone", "controllerGroup": "enterprise.splunk.com", "controllerKind": "Standalone", "Standalone": {"name":"c-32002","namespace":"splunk-operator"}, "namespace": "splunk-operator", "name": "c-32002", "reconcileID": "94ff6daa-6472-4d1c-bc83-31051dc9bcfb", "name": "c-32002", "namespace": "splunk-operator", "error": "unable to create directory on Pod at path=/operator-staging/appframework/custom. stdout: , stdErr: mkdir: cannot create directory '/operator-staging': Permission denied\n, err: command terminated with exit code 1"}
github.com/splunk/splunk-operator/pkg/splunk/enterprise.afwSchedulerEntry
        /workspace/pkg/splunk/enterprise/afwscheduler.go:2159
github.com/splunk/splunk-operator/pkg/splunk/enterprise.handleAppFrameworkActivity
        /workspace/pkg/splunk/enterprise/util.go:1967
github.com/splunk/splunk-operator/pkg/splunk/enterprise.ApplyStandalone
        /workspace/pkg/splunk/enterprise/standalone.go:231
github.com/splunk/splunk-operator/controllers.glob..func8
        /workspace/controllers/standalone_controller.go:119
github.com/splunk/splunk-operator/controllers.(*StandaloneReconciler).Reconcile
        /workspace/controllers/standalone_controller.go:109
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235

[akondur]

Hi @yaroslav-nakonechnikov , the issue has been fixed in the MR here.

Can you please try and let us know if the solution works?

[yaroslav-nakonechnikov]

@akondur yes, just after there will be new release it would be possible to test it.

[vivekr-splunk]

@yaroslav-nakonechnikov can you recheck this issue with 2.6.1 release.

[yaroslav-nakonechnikov]

2024-10-30T12:20:14.717768035Z  ERROR   getAppTopFolderFromPackage      could not get installed app name        {"controller": "licensemanager", "controllerGroup": "enterprise.splunk.com", "controllerKind": "LicenseManager", "LicenseManager": {"name":"43345","namespace":"splunk-operator"}, "namespace": "splunk-operator", "name": "43345", "reconcileID": "f879f0b2-9ba8-4798-8543-a4fa4e665fbf", "name": "43345", "namespace": "splunk-operator", "appPkgPathOnPod": "/operator-staging/appframework/config-explorer/splunk-datasets-add-on_10.tgz_b9da596a72847b26df597fc2ef69fd3a", "stdOut": "splunk_datasets_addon\n", "stdErr": "tar: Ignoring unknown extended header keyword 'LIBARCHIVE.creationtime'\ntar: Ignoring unknown extended header keyword 'SCHILY.dev'\ntar: Ignoring unknown extended header keyword 'SCHILY.ino'\ntar: Ignoring unknown extended header keyword 'SCHILY.nlink'\ntar: Ignoring unknown extended header keyword 'LIBARCHIVE.creationtime'\ntar: Ignoring unknown extended header keyword 'SCHILY.dev'\ntar: Ignoring unknown extended headerkeyword 'SCHILY.ino'\ntar: Ignoring unknown extended header keyword 'SCHILY.nlink'\n", "command": "tar tf /operator-staging/appframework/config-explorer/splunk-datasets-add-on_10.tgz_b9da596a72847b26df597fc2ef69fd3a|head -1|cut -d/ -f1", "appPkgPathOnPod": "/operator-staging/appframework/config-explorer/splunk-datasets-add-on_10.tgz_b9da596a72847b26df597fc2ef69fd3a"}
github.com/splunk/splunk-operator/pkg/splunk/enterprise.getAppTopFolderFromPackage
        /workspace/pkg/splunk/enterprise/afwscheduler.go:825
github.com/splunk/splunk-operator/pkg/splunk/enterprise.installApp
        /workspace/pkg/splunk/enterprise/afwscheduler.go:722
github.com/splunk/splunk-operator/pkg/splunk/enterprise.(*localScopePlaybookContext).runPlaybook
        /workspace/pkg/splunk/enterprise/afwscheduler.go:882

[yaroslav-nakonechnikov]

and more - it is not updating already installed apps:

2024-10-30T12:20:18.903835117Z  INFO    installApp      Not reinstalling app as it is already installed.        {"controller": "licensemanager", "controllerGroup": "enterprise.splunk.com", "controllerKind": "LicenseManager", "LicenseManager": {"name":"43345","namespace":"splunk-operator"}, "namespace": "splunk-operator", "name": "43345", "reconcileID": "f879f0b2-9ba8-4798-8543-a4fa4e665fbf", "name": "43345", "namespace": "splunk-operator", "pod": "splunk-43345-license-manager-0", "app name": "splunk-app-for-lookup-file-editing_404.tgz"}

[rlieberman-splunk]

@yaroslav-nakonechnikov the error you posted on October 30th should now allow the app to continue installation. It is logged because there are unknown extender header keywords in the tar file. Can you please check for a later log that says app pkg installed on all the pods? That will verify that the app was correctly installed. We can change this error log to a warning to make it more obvious that this is not a failed installation.

Edit: You can also exec into the pod and check in /opt/splunk/etc/apps and verify that you see splunk_datasets_addon created there to make sure it is installed.

[yaroslav-nakonechnikov]

ok, will check on 2.7.0

[michal-tatusko-splunk]

Hello @yaroslav-nakonechnikov,
How are you?

Did you have a chance to recheck that? Could you please let us know if it's fixed for you so we can close the issue?