ES|QL: NullPointerException planning EVAL

[luigidellaquila]

With csv-spec test dataset

from clientips,* 
| dissect street "%{height_range} %{MNyXV}" 
| rename env AS kxpCK, pod AS etUHW, language_code AS city.country.continent.planet.galaxy
| rename city.country.continent.planet.name as message 
| lookup join message_types_lookup on message
| rename languages.int as language_code 
| lookup join languages_lookup on language_code
| rename language_code as language_code 
| lookup join languages_lookup on language_code 
| sort decade DESC NULLS FIRST, year ASC NULLS FIRST, country.keyword ASC, num ASC NULLS LAST, network.bytes_in ASC, author.keyword ASC, languages.byte DESC NULLS LAST, type ASC NULLS LAST, zip_code NULLS FIRST, height_range, number ASC, language_code ASC NULLS LAST, host_group NULLS LAST, MNyXV ASC NULLS FIRST, kxpCK ASC, language.id ASC, languages.long DESC NULLS FIRST, host.name ASC, city.country.continent.name ASC NULLS FIRST, author DESC NULLS LAST, ip0, event_duration ASC NULLS LAST, scalerank ASC, ip1 DESC, client.ip NULLS FIRST, message NULLS FIRST, city.country.name ASC NULLS FIRST, version ASC NULLS FIRST, airport ASC NULLS FIRST, language.code ASC NULLS LAST, language.name.keyword NULLS FIRST, city.name DESC NULLS LAST, city.country.continent.planet.galaxy NULLS FIRST, publisher ASC NULLS FIRST, abbrev DESC NULLS FIRST, card ASC, salary_change.keyword NULLS FIRST, lk ASC NULLS FIRST	

TODO: simplify the query, it's extreme because it's randomly generated

{
    "error": {
        "root_cause": [
            {
                "type": "null_pointer_exception",
                "reason": "Cannot read field \"nameIds\" because \"set\" is null"
            }
        ],
        "type": "null_pointer_exception",
        "reason": "Cannot read field \"nameIds\" because \"set\" is null"
    },
    "status": 500
}

ESQL request failed with status [INTERNAL_SERVER_ERROR]: java.lang.NullPointerException: Cannot read field "nameIds" because "set" is null
        at org.elasticsearch.xpack.esql.planner.Layout$Builder.build(Layout.java:111)
        at org.elasticsearch.xpack.esql.planner.LocalExecutionPlanner.planEval(LocalExecutionPlanner.java:412)
        at org.elasticsearch.xpack.esql.planner.LocalExecutionPlanner.plan(LocalExecutionPlanner.java:211)
        at org.elasticsearch.xpack.esql.planner.LocalExecutionPlanner.planProject(LocalExecutionPlanner.java:640)
        at org.elasticsearch.xpack.esql.planner.LocalExecutionPlanner.plan(LocalExecutionPlanner.java:217)
        at org.elasticsearch.xpack.esql.planner.LocalExecutionPlanner.planLookupJoin(LocalExecutionPlanner.java:562)
        at org.elasticsearch.xpack.esql.planner.LocalExecutionPlanner.plan(LocalExecutionPlanner.java:243)
        at org.elasticsearch.xpack.esql.planner.LocalExecutionPlanner.planEval(LocalExecutionPlanner.java:406)
        at org.elasticsearch.xpack.esql.planner.LocalExecutionPlanner.plan(LocalExecutionPlanner.java:211)
        at org.elasticsearch.xpack.esql.planner.LocalExecutionPlanner.planProject(LocalExecutionPlanner.java:640)
        at org.elasticsearch.xpack.esql.planner.LocalExecutionPlanner.plan(LocalExecutionPlanner.java:217)
        at org.elasticsearch.xpack.esql.planner.LocalExecutionPlanner.planLookupJoin(LocalExecutionPlanner.java:562)
        at org.elasticsearch.xpack.esql.planner.LocalExecutionPlanner.plan(LocalExecutionPlanner.java:243)

[elasticsearchmachine]

Pinging @elastic/es-analytical-engine (Team:Analytics)

[kanoshiou]

I believe I've found a simpler way to reproduce this issue.

curl --request PUT --url http://localhost:9200/index1
curl --request PUT --url http://localhost:9200/index2
curl --request PUT --url http://localhost:9200/index3
curl --request PUT --url http://localhost:9200/index4
curl --request PUT --url http://localhost:9200/index5
curl --request PUT --url http://localhost:9200/index6
curl --request PUT --url http://localhost:9200/index7
curl --request PUT --url http://localhost:9200/index8
curl --request PUT --url http://localhost:9200/index9
curl --request PUT \
  --url http://localhost:9200/index10 \
  --header 'content-type: application/json' \
  --data '{
  "mappings": {
    "properties": {
      "ip0": {
        "type": "ip"
      }
    }
  }
}'

curl --request PUT \
  --url http://localhost:9200/index_lookup \
  --header 'content-type: application/json' \
  --data '{
  "settings": {
    "index": {
      "mode": "lookup"
    }
  },
  "mappings": {
    "properties": {
      "a": {
        "type": "integer"
      }
    }
  }
}'

curl --request POST \
  --url 'http://localhost:9200/_query?format=txt' \
  --header 'content-type: application/json' \
  --data '{
  "query": "from * | lookup join index_lookup on a | sort a"
}'

[alex-spies]

Thanks a lot @kanoshiou , this is excellent.

This also reproduces for me using your simplified instructions. For whatever reason, the empty indices index1, ..., index9 are relevant. When creating index10 and index_lookup first and running the query, it works fine. Creating the indices index1, ..., index9 suddently makes the query cause an NPE. When I deleted index9, suddenly the query worked again. Creating it again reprodcued the NPE, again.

The fact that this can happen with a simple query like from * | lookup join index_lookup on a | sort a means this is not a low-priority bug and we need to get to the bottom of this.

[kanoshiou]

Thanks @alex-spies , for your detailed additions.

The failed query appears to execute an additional EVAL physical operation compared to the successful query. This discrepancy causes the layout builder to append an existing attribute, leading to a mismatch between the layout map's size and numberOfChannels.

I attempted to resolve this issue, but, as you mentioned, it might not be a low-priority one, so I believe it's best to leave it to you maintainers. However, I am still very willing to assist if needed.

[fang-xing-esql]

I'm able to reproduce the errors with both queries, and a simplified Q1 looks like below, I'll need a sort to make it error out. The two queries stack traces are slightly different, however it is likely that they have the same root case. I'm looking into them further.

+ curl -u elastic:password -H 'Content-Type: application/json' '127.0.0.1:9200/_query?format=txt' -d '{
    "query": "from * 
             | rename city.country.continent.planet.name as message 
             | lookup join message_types_lookup on message
             | sort language_code"
}
'
{"error":{"root_cause":[{"type":"null_pointer_exception","reason":"Cannot read field \"nameIds\" because \"set\" is null"}],"type":"null_pointer_exception","reason":"Cannot read field \"nameIds\" because \"set\" is null","suppressed":[{"type":"null_pointer_exception","reason":"Cannot read field \"nameIds\" because \"set\" is null"}]},"status":500}%