Make Lucene indexing and search work alongside parquet

[expani]

Description

Added a new Index Setting called index.optimized.enabled if true the index is created using the new IndexWriter plugin otherwise the existing Lucene index is created.

SQL PR vinaykpud/sql#21

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

✨ Finishing touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[github-actions]

❌ Gradle check result for 05152ad: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 05dab3a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[expani]

Okay I was able to test Datafusion E2E locally and it's failing due to unmerged SQL plugin PR https://github.com/vinaykpud/sql/pull/21/files which uses the new index setting.

./gradlew :integ-test:integTest --tests "org.opensearch.sql.calcite.clickbench.CalcitePPLClickBenchIT" -Dtests.method="testDataFusion" -Dtests.cluster=localhost:9200 -Dtests.rest.cluster=localhost:9200 -DignorePrometheus=true -Dtests.clustername=opensearch -Dtests.output=true

Response

    PASSING: [1, 2, 3, 5, 6, 7, 8, 9, 13, 15, 16, 17, 18, 19, 21, 22, 23, 28, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 42, 43]
    FAILING WITH 200: [4, 10, 20, 24, 25, 26, 27, 29]
    FAILING: [11, 12, 14]
    [2025-12-09T20:07:30,831][INFO ][o.o.s.c.c.CalcitePPLClickBenchIT] [testDataFusion] after test

CalcitePPLClickBenchIT STANDARD_OUT
    Summary:
...
...
...
...
    Total:
    42
     | Passed:
    31
     | Failed (200):
    8
     | Failed (non-200):
    3

The Lucene indices are getting created in a temp directory with prefix tmp-internal-engine instead of data directory due to changes in InternalEngine #19956 We can pick this up separately later.

cc: @mch2 @bharath-techie

[github-actions]

❌ Gradle check result for bb7b681: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?