[BUG][analytics-engine] PPL dedup followed by any pipe stage fails with 500: Project rule encountered unmarked child [LogicalDedup]

[alchemist51]

Describe the bug

Any PPL query using dedup followed by another pipe stage (fields, stats, …) on the analytics-engine (sandbox) route fails with a 500:

{
  "error": {
    "reason": "There was internal problem at backend",
    "details": "Internal error [task_id=…]",
    "type": "RuntimeException"
  },
  "status": 500
}

Server-side stack (identical for every occurrence):

java.lang.IllegalStateException: Project rule encountered unmarked child [LogicalDedup]
    at org.opensearch.analytics.planner.rules.OpenSearchProjectRule.onMatch(OpenSearchProjectRule.java:65)
    at org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:350)
    at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:527)
    ...
    at org.opensearch.analytics.planner.HepPhase.run(HepPhase.java:144)
    at org.opensearch.analytics.planner.PlannerImpl.mark(PlannerImpl.java:406)

Root cause: PPL dedup lowers to a Calcite LogicalDedup rel node, but the marking phase in PlannerImpl.mark() only registers rules for TableScan, Filter, Project, Aggregate, Join, Sort, Union and Values. There is no rule that converts LogicalDedup into an OpenSearchRelNode (and no pre-marking rewrite into a supported shape). When OpenSearchProjectRule fires on the projection above the dedup, its child is still the raw LogicalDedup and the rule throws at OpenSearchProjectRule.java:65.

This is deterministic — it fails 100% of the time for the dedup … | <anything> shape.

This makes 5 sandbox-check CI tests fail on every run (including main)

Test	Failing query
ExtensiveCoveragePplIT.testExtensiveCoveragePplQueries	source=test_data | dedup category | fields category, value (q1.ppl)
KubernetesLogsPplIT.testKubernetesLogsPplQueries	source=kubernetes_logs | dedup \kubernetes.pod_name` | stats count() as unique_pods by `kubernetes.container_image`` (q6.ppl)
MultiSourceJoinsPplIT.testMultiSourceJoinsPplQueries	source=voice_verification | where … | dedup tracking_id | stats … (q5.ppl)
OtelLogsPplIT.testOtelLogsPplQueries	source=otel_logs | dedup serviceName | fields serviceName | sort + serviceName (q38.ppl)
TwoShardShapeIT.testReduceCorrectnessAcrossTwoShards	source=%INDEX% | dedup category | fields category (shape/dedup_category.ppl)

Verified failing on main HEAD (8cac2f50e62) locally and in sandbox-check CI runs of main and multiple unrelated PRs (e.g. run 31135587629 on main, 31150812738 / 31152780093 on PRs). The sandbox-check workflow is continue-on-error: true, so these failures don't block PRs — which is likely why this has gone unnoticed.

Related component

Search:Query Capabilities

To Reproduce

./gradlew -Dsandbox.enabled=true ':sandbox:qa:analytics-engine-rest:integTest' \
  --tests 'org.opensearch.analytics.qa.ExtensiveCoveragePplIT.testExtensiveCoveragePplQueries'

or against any running analytics-engine cluster:

POST /_plugins/_ppl
{"query": "source=<idx> | dedup <field> | fields <field>"}

Expected behavior

dedup queries plan and execute (or fail with a clear UnsupportedFunctionException-style 400 if the command is intentionally unsupported), instead of an opaque 500.

Additional Details

• All 5 failures share seed-independent, deterministic behavior; reproduced with -Dtests.seed=C49DB1133B3B9BEE and without.
• LogicalDedup is defined in the PPL frontend/Calcite integration, not in the sandbox planner — fix is either a marking rule for it or a pre-marking rewrite (e.g. dedup → aggregate/window shape).