scheduler: feasibility check that memory_max fits in total

The `resource.memory_max` field is intended to allow memory oversubscription, so
we don't check it in the `AllocsFit` method where we're totalling up all the
request memory for all allocs on a node. But we never check that the value can
even fit in the maximum amount of memory on the node, which can result in
nonsensical placements.

When iterating nodes in the feasibility check phase, check that the `memory_max`
field doesn't exceed the total amount of memory on the node. Note that this
deliberately ignores over "reserved memory", as the feature is intended to allow
oversubscription.

Fixes: #26360

Author: tgross

scheduler/feasible/rank.go

@@ -383,6 +383,13 @@ NEXTNODE:
 			if iter.memoryOversubscription {
 				taskResources.Memory.MemoryMaxMB = safemath.Add(
 					int64(task.Resources.MemoryMaxMB), int64(task.Resources.SecretsMB))
+
+				if taskResources.Memory.MemoryMaxMB > option.Node.NodeResources.Memory.MemoryMB {
+					iter.ctx.Metrics().FilterNode(option.Node,
+						"task memory_max exceeds maximum available memory")
+					netIdx.Release()
+					continue NEXTNODE
+				}
 			}
 
 			// Check if we need a network resource

scheduler/feasible/rank_test.go

@@ -110,6 +110,18 @@ func TestBinPackIterator_NoExistingAlloc(t *testing.T) {
 				},
 			},
 		},
+		{
+			Node: &structs.Node{
+				// Empty but memory_max won't fit
+				NodeResources: &structs.NodeResources{
+					Processors: processorResources4096,
+					Cpu:        legacyCpuResources4096,
+					Memory: structs.NodeMemoryResources{
+						MemoryMB: 1024,
+					},
+				},
+			},
+		},
 	}
 	static := NewStaticRankIterator(ctx, nodes)
 
@@ -119,8 +131,9 @@ func TestBinPackIterator_NoExistingAlloc(t *testing.T) {
 			{
 				Name: "web",
 				Resources: &structs.Resources{
-					CPU:      1024,
-					MemoryMB: 1024,
+					CPU:         1024,
+					MemoryMB:    1024,
+					MemoryMaxMB: 2048,
 				},
 			},
 		},

website/content/docs/upgrade/upgrade-specific.mdx

@@ -18,7 +18,12 @@ used to document those details separately from the standard upgrade flow.
 
 In Nomad 1.11.0, submitting a sysbatch job with a `reschedule` block returns
 an error instead of being silently ignored, as it was in previous versions. The
-same behavior applies to system jobs. 
+same behavior applies to system jobs.
+
+#### Memory oversubscription checked against total memory on node
+
+In Nomad 1.11.0, the scheduler checks that that the `resources.memory_max` of a
+task doesn't exceed the total memory on a given node.
 
 ## Nomad 1.10.2