Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for GPU monitoring #1601

Merged
merged 14 commits into from
Jan 30, 2025
Merged

Add support for GPU monitoring #1601

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
9bc1a53
Add support for GPU feature
gjulianm Dec 26, 2024
dd0dd9c
Add tests for runtime class changes
gjulianm Jan 7, 2025
7d40ddb
Documentation
gjulianm Jan 8, 2025
da4ab24
Update docs
gjulianm Jan 9, 2025
f5b5e32
Update api/datadoghq/v2alpha1/datadogagent_types.go
gjulianm Jan 24, 2025
dbf0019
Remove debug changes
gjulianm Jan 24, 2025
0c78f33
Move const variables to gpu package
gjulianm Jan 24, 2025
3fcc699
GPUMonitoringType -> GPUIDType
gjulianm Jan 24, 2025
76a3ac3
Rename gpuMonitoringFeature to gpuFeature
gjulianm Jan 24, 2025
8d4c14e
Apply suggestion
gjulianm Jan 24, 2025
9706adc
defaultGPURuntimeClass
gjulianm Jan 30, 2025
cc00090
Rename GPUMonitoringConfig
gjulianm Jan 30, 2025
f2ac105
Fix comment
gjulianm Jan 30, 2025
e44a167
Generate code
gjulianm Jan 30, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions api/datadoghq/v2alpha1/const.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,9 @@ const (
KubeServicesAndEndpointsListeners = "kube_services kube_endpoints"
EndpointsChecksConfigProvider = "endpointschecks"
ClusterAndEndpointsConfigProviders = "clusterchecks endpointschecks"

// DefaultGPUMonitoringRuntimeClass default runtime class for GPU pods
DefaultGPUMonitoringRuntimeClass = "nvidia"
gjulianm marked this conversation as resolved.
Show resolved Hide resolved
gjulianm marked this conversation as resolved.
Show resolved Hide resolved
)

// Labels
Expand Down Expand Up @@ -201,6 +204,10 @@ const (
FIPSProxyCustomConfigFileName = "datadog-fips-proxy.cfg"
FIPSProxyCustomConfigMapName = "%s-fips-config"
FIPSProxyCustomConfigMountPath = "/etc/datadog-fips-proxy/datadog-fips-proxy.cfg"

NVIDIADevicesMountPath = "/var/run/nvidia-container-devices/all"
NVIDIADevicesVolumeName = "nvidia-devices"
DevNullPath = "/dev/null" // used to mount the NVIDIADevicesHostPath to /dev/null in the container, it's just used as a "signal" to the nvidia runtime to use the nvidia devices
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since these (including line 83) only seem to be used in the feature/gpu/feature.go code, can they be placed within the feature/gpu directory? e.g. https://github.com/DataDog/datadog-operator/blob/main/internal/controller/datadogagent/feature/kubernetesstatecore/const.go

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved back to gpu/const.go

)

// Field paths
Expand Down
16 changes: 16 additions & 0 deletions api/datadoghq/v2alpha1/datadogagent_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,8 @@ type DatadogFeatures struct {
SBOM *SBOMFeatureConfig `json:"sbom,omitempty"`
// ServiceDiscovery
ServiceDiscovery *ServiceDiscoveryFeatureConfig `json:"serviceDiscovery,omitempty"`
// GPU monitoring
GPUMonitoring *GPUMonitoringFeatureConfig `json:"gpu,omitempty"`
gjulianm marked this conversation as resolved.
Show resolved Hide resolved

// Cluster-level features

Expand Down Expand Up @@ -498,6 +500,20 @@ type ServiceDiscoveryFeatureConfig struct {
Enabled *bool `json:"enabled,omitempty"`
}

// GPUMonitoringFeatureConfig contains the GPU monitoring configuration.
type GPUMonitoringFeatureConfig struct {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// GPUMonitoringFeatureConfig contains the GPU monitoring configuration.
type GPUMonitoringFeatureConfig struct {
// GPUFeatureConfig contains the GPU monitoring configuration.
type GPUFeatureConfig struct {

// Enabled enables GPU monitoring.
// Default: false
// +optional
Enabled *bool `json:"enabled,omitempty"`

// PodRuntimeClassName specifies the runtime class name required for the GPU monitoring feature.
// If left empty, the runtime class is not set.
// Default: nvidia
gjulianm marked this conversation as resolved.
Show resolved Hide resolved
// +optional
PodRuntimeClassName *string `json:"requiredRuntimeClassName"`
}

// DogstatsdFeatureConfig contains the Dogstatsd configuration parameters.
// +k8s:openapi-gen=true
type DogstatsdFeatureConfig struct {
Expand Down
30 changes: 30 additions & 0 deletions api/datadoghq/v2alpha1/zz_generated.deepcopy.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 7 additions & 1 deletion api/datadoghq/v2alpha1/zz_generated.openapi.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

30 changes: 30 additions & 0 deletions config/crd/bases/v1/datadoghq.com_datadogagents.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1019,6 +1019,21 @@ spec:
Default: false
type: boolean
type: object
gpu:
description: GPU monitoring
properties:
enabled:
description: |-
Enabled enables GPU monitoring.
Default: false
type: boolean
requiredRuntimeClassName:
description: |-
PodRuntimeClassName specifies the runtime class name required for the GPU monitoring feature.
If left empty, the runtime class is not set.
Default: nvidia
type: string
type: object
helmCheck:
description: HelmCheck configuration.
properties:
Expand Down Expand Up @@ -7883,6 +7898,21 @@ spec:
Default: false
type: boolean
type: object
gpu:
description: GPU monitoring
properties:
enabled:
description: |-
Enabled enables GPU monitoring.
Default: false
type: boolean
requiredRuntimeClassName:
description: |-
PodRuntimeClassName specifies the runtime class name required for the GPU monitoring feature.
If left empty, the runtime class is not set.
Default: nvidia
type: string
type: object
helmCheck:
description: HelmCheck configuration.
properties:
Expand Down
30 changes: 30 additions & 0 deletions config/crd/bases/v1/datadoghq.com_datadogagents_v2alpha1.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1065,6 +1065,21 @@
},
"type": "object"
},
"gpu": {
"additionalProperties": false,
"description": "GPU monitoring",
"properties": {
"enabled": {
"description": "Enabled enables GPU monitoring.\nDefault: false",
"type": "boolean"
},
"requiredRuntimeClassName": {
"description": "PodRuntimeClassName specifies the runtime class name required for the GPU monitoring feature.\nIf left empty, the runtime class is not set.\nDefault: nvidia",
"type": "string"
}
},
"type": "object"
},
"helmCheck": {
"additionalProperties": false,
"description": "HelmCheck configuration.",
Expand Down Expand Up @@ -7871,6 +7886,21 @@
},
"type": "object"
},
"gpu": {
"additionalProperties": false,
"description": "GPU monitoring",
"properties": {
"enabled": {
"description": "Enabled enables GPU monitoring.\nDefault: false",
"type": "boolean"
},
"requiredRuntimeClassName": {
"description": "PodRuntimeClassName specifies the runtime class name required for the GPU monitoring feature.\nIf left empty, the runtime class is not set.\nDefault: nvidia",
"type": "string"
}
},
"type": "object"
},
"helmCheck": {
"additionalProperties": false,
"description": "HelmCheck configuration.",
Expand Down
4 changes: 2 additions & 2 deletions config/manager/kustomization.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ resources:
- manager.yaml
images:
- name: controller
newName: gcr.io/datadoghq/operator
newTag: 1.11.1
newName: 601427279990.dkr.ecr.us-east-1.amazonaws.com/guillermo.julian/sandbox
newTag: operator
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mind undoing these changes?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops, sorry. Removed!

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
2 changes: 2 additions & 0 deletions docs/configuration.v2alpha1.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,8 @@ spec:
| features.externalMetricsServer.registerAPIService | RegisterAPIService registers the External Metrics endpoint as an APIService Default: true |
| features.externalMetricsServer.useDatadogMetrics | UseDatadogMetrics enables usage of the DatadogMetrics CRD (allowing one to scale on arbitrary Datadog metric queries). Default: true |
| features.externalMetricsServer.wpaController | WPAController enables the informer and controller of the Watermark Pod Autoscaler. NOTE: The Watermark Pod Autoscaler controller needs to be installed. See also: https://github.com/DataDog/watermarkpodautoscaler. Default: false |
| features.gpu.enabled | Enables GPU monitoring. Default: false |
| features.gpu.requiredRuntimeClassName | PodRuntimeClassName specifies the runtime class name required for the GPU monitoring feature. If left empty, the runtime class is not set. Default: nvidia |
| features.helmCheck.collectEvents | CollectEvents set to `true` enables event collection in the Helm check (Requires Agent 7.36.0+ and Cluster Agent 1.20.0+) Default: false |
| features.helmCheck.enabled | Enables the Helm check. Default: false |
| features.helmCheck.valuesAsTags | ValuesAsTags collects Helm values from a release and uses them as tags (Requires Agent and Cluster Agent 7.40.0+). Default: {} |
Expand Down
2 changes: 2 additions & 0 deletions examples/datadogagent/datadog-agent-all.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,8 @@ spec:
enabled: true
serviceDiscovery:
enabled: true
gpu:
enabled: true
eventCollection:
collectKubernetesEvents: true
orchestratorExplorer:
Expand Down
1 change: 1 addition & 0 deletions internal/controller/datadogagent/controller.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ import (
_ "github.com/DataDog/datadog-operator/internal/controller/datadogagent/feature/enabledefault"
_ "github.com/DataDog/datadog-operator/internal/controller/datadogagent/feature/eventcollection"
_ "github.com/DataDog/datadog-operator/internal/controller/datadogagent/feature/externalmetrics"
_ "github.com/DataDog/datadog-operator/internal/controller/datadogagent/feature/gpu"
_ "github.com/DataDog/datadog-operator/internal/controller/datadogagent/feature/helmcheck"
_ "github.com/DataDog/datadog-operator/internal/controller/datadogagent/feature/kubernetesstatecore"
_ "github.com/DataDog/datadog-operator/internal/controller/datadogagent/feature/livecontainer"
Expand Down
8 changes: 8 additions & 0 deletions internal/controller/datadogagent/defaults/datadogagent_default.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,8 @@ const (

defaultEBPFCheckEnabled bool = false

defaultGPUMonitoringEnabled bool = false

defaultServiceDiscoveryEnabled bool = false

defaultAPMEnabled bool = true
Expand Down Expand Up @@ -265,6 +267,12 @@ func defaultFeaturesConfig(ddaSpec *v2alpha1.DatadogAgentSpec) {
}
apiutils.DefaultBooleanIfUnset(&ddaSpec.Features.ServiceDiscovery.Enabled, defaultServiceDiscoveryEnabled)

// GPU monitoring feature
if ddaSpec.Features.GPUMonitoring == nil {
ddaSpec.Features.GPUMonitoring = &v2alpha1.GPUMonitoringFeatureConfig{}
}
apiutils.DefaultBooleanIfUnset(&ddaSpec.Features.GPUMonitoring.Enabled, defaultGPUMonitoringEnabled)

// APM Feature
// APM is enabled by default
if ddaSpec.Features.APM == nil {
Expand Down
Loading
Loading