Support time modifiers in search command #4224

yuancu · 2025-09-04T09:50:17Z

Description

Support time modifier in search command.

~~Changes are cherry-picked from a branch based on #4152. Waiting for #4152 to be completed.~~

Examples:

source=any earliest='2020-12-10'
source=any earliest='2020-12-10' latest='2025-09-10 13:00:12'
source=any latest=now
source=logs earliest=-7d
source=logs earliest='-1d@d'

Queries with earliest and latest time modifiers will be converted to a query string query with comparison conditions on the implicit timestamp field @timestamp. For example, query source=time_test earliest=-1year latest='-50d@w3' is converted to the following DSL:

{
  "query": {
    "query_string": {
      "query": "(@timestamp:>=now-1y) AND (@timestamp:<=now-50d/w-5d)",
    }
  }
}

Work items

support absolute time like 2012-10-10
support relative time range like now, -30m, -2h@h, @w0
support chained time range offsets like -mon@mon+7d
Unix epoch time like 1 (UTC January 1, 1970 at 12:00:01 AM)

Related Issues

Resolves #4135

Implementation Walk-through

Implementation Details

Time Modifier Processing

The core functionality is implemented in the visitTimeModifierValue and visitTimeModifierExpression methods in AstExpressionBuilder.java:

visitTimeModifierValue: Converts time values to OpenSearch date math expressions
visitTimeModifierExpression: Creates comparison conditions for @timestamp field in query string

Time Format Conversion

The PR extends DateTimeUtils.java with the parseRelativeTime method, which transforms PPL time expressions to OpenSearch date math expressions.

Query String Query Creation

Time modifiers are converted to search comparisons with a query string on the implicit @timestamp field, which is now defined as a constant in OpenSearchConstants.java. For example, search earliest=-10days@month latest=now() is converted to a query string query like the following:

{
  "query": {
    "query_string": {
      "query": "(@timestamp:>=-10d/M) AND (@timestamp:<=now)"
    }
  }
}

PPL Time Modifiers to OpenSearch Date Math Examples

Absolute Times

PPL Time Modifier OpenSearch Date Math Explanation

'2023-01-01' 2023-01-01 Simple date

'2023-01-01 13:45:30' 2023-01-01T13:45:30Z Date with time
Relative Times

PPL Time Modifier OpenSearch Date Math Explanation

now now Current time

-30s now-30s 30 seconds ago

-1h now-1h 1 hour ago

+1d now+1d 1 day in future
Snapping to Time Units (Using @)

PPL Time Modifier OpenSearch Date Math Explanation

-1d@d now-1d/d 1 day ago, rounded to day

@h now/h Round to current hour

@m now/M Round to current month
Week Days

PPL Time Modifier OpenSearch Date Math Explanation

@w0 /w-1d or /w+6d Previous Sunday (depending on current day)

@w1 /w Start of week (Monday)

@w4 /w+3d Thursday of current week
Special Cases

PPL Time Modifier	OpenSearch Date Math	Explanation
@q	/M or /M-1M or /M-2M	Start of current quarter
-2q	-6M	2 quarters ago (6 months)
1234.567	1234567	Unix timestamp (seconds → milliseconds)

Complex Expressions

PPL Time Modifier	OpenSearch Date Math	Explanation
-1d+1y@mon	now-1d+1y/M	1 day ago + 1 year, rounded to month
-3d@d-2h+10m	now-3d/d-2h+10m	3 days ago, day start, minus 2 hours, plus 10 minutes

Check List

New functionality includes testing.
New functionality has been documented.
New functionality has javadoc added.
New functionality has a user manual doc added.
New PPL command checklist all confirmed.
API changes companion pull request created.
Commits are signed per the DCO using --signoff or -s.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

vamsimanohar · 2025-09-05T19:48:55Z

can you review this: #4152
I am changing the grammar for search expression and its functionality.
We might need to reevaluate the implementation.

yuancu · 2025-09-07T03:16:06Z

Hi @vamsimanohar, thank you for reminding! I think the functionality does not overlap, I'll re-implement this based on your grammar.

vamsimanohar · 2025-09-09T22:04:10Z

@yuancu Few things to keep in mind.

We fundamentally want search command to be always push down and translate toquery_stringfunction in DSL which follows Lucene query syntax. Anything we add to search command should get translated to lucene query.

I think it should be possible with your changes.

Signed-off-by: Yuanchun Shen <[email protected]> Unit test search with absolute time range Signed-off-by: Yuanchun Shen <[email protected]> Rephrase timeRange and timeModifier Signed-off-by: Yuanchun Shen <[email protected]> Switch to earliest and latest udf Signed-off-by: Yuanchun Shen <[email protected]> Add a convert util Signed-off-by: Yuanchun Shen <[email protected]> Verify time correctness during coversion Signed-off-by: Yuanchun Shen <[email protected]> Fix quarter parsing bugs Signed-off-by: Yuanchun Shen <[email protected]> Fix week snap parsing Signed-off-by: Yuanchun Shen <[email protected]> Remove old implementation that translates time modifier to time filter Signed-off-by: Yuanchun Shen <[email protected]>

yuancu · 2025-09-16T09:31:08Z

core/src/main/java/org/opensearch/sql/expression/function/udf/condition/EarliestFunction.java

        getRelativeZonedDateTime(
            expression, ZonedDateTime.ofInstant(clock.instant(), clock.getZone()));
-    return earliest.isBefore(candidateDatetime);
+    return !earliest.isAfter(candidateDatetime);


Modified because earliest should be greater or equal.

Signed-off-by: Yuanchun Shen <[email protected]>

yuancu · 2025-09-17T07:25:17Z

ppl/src/main/antlr/OpenSearchPPLParser.g4

+   ;
+
+timeSnap
+  : AT timeModifierUnit;


Asking for help: currently, AT timeModifierUnit can not be matched. Since antlr is parsed bottom-up, plus that a shorter rule matching longer text will be prioritized, AT timeModifierUnit will always be matched as an identifier by the following rule:

// OpenSearchPPLLexer.g4 ID: ID_LITERAL; fragment ID_LITERAL: ([@*A-Z_])+?[*A-Z_\-0-9]*;

// OpenSearchPPLParser.g4 ident : (DOT)? ID

As a result, I can not write something like source=t earliest=-10h@day, but have to write it as source=t earliest='-10h@day' because -10h@day will be matched to the shorter rule ident.

I tried to make the grammar context aware, but had no luck with multiple solutions. Is there any suggestion for a workaround?

@yuancu have you figured out anything on this?

No, I haven't :/

yuancu · 2025-09-17T07:30:01Z

core/build.gradle

    testImplementation group: 'org.hamcrest', name: 'hamcrest-library', version: "${hamcrest_version}"
    testImplementation group: 'org.mockito', name: 'mockito-core', version: "${mockito_version}"
    testImplementation group: 'org.mockito', name: 'mockito-junit-jupiter', version: "${mockito_version}"
+    testImplementation group: 'org.opensearch', name: 'opensearch', version: "${opensearch_version}"


This dependency is added for the access of org.opensearch.common.time.DateMathParser -- I want to make sure that the parsed OpenSearch date math like now-1d/M-2M returns the intended instant for -1d@q

parseRelativeTime() is called by PPL module only, right? if move parseRelativeTime to PPL module, does opensearch depedency still needed?

Yes, it is only called by PPL module. If I still want to assure the correctness of the parsed instant, I'll have to move this dependency to the PPL module.

Is there any concern over this dependency? If it's for the performance, as the dependency is only test time, I assume there would be no harm to the runtime performance.

Signed-off-by: Yuanchun Shen <[email protected]>

vamsimanohar · 2025-09-22T21:51:50Z

@yuancu hey can you update the description with latest implementation where you are translating to Lucene query.

vamsimanohar · 2025-09-22T21:52:48Z

ppl/src/main/antlr/OpenSearchPPLLexer.g4

 HRS:                                'HRS';
 HOURS:                              'HOURS';
 DAYS:                               'DAYS';
+WEEKS:                              'WEEKS';


can you add these new terms to searchKeyWords or keywordCanbeId

Added timeModifierUnit to searchableKeyWord. I wonder how does this change the parsing behavior

vamsimanohar · 2025-09-22T21:53:57Z

ppl/src/main/antlr/OpenSearchPPLParser.g4


 searchExpression
- : LT_PRTHS searchExpression RT_PRTHS                 # groupedExpression
+ : timeModifier                                       # timeModifierExpression


For my understanding: This implies we can write only timeModifier in searchExpression.
Can you update search.rst with these changes.

+1, please update doc.

penghuo · 2025-09-22T22:34:19Z

integ-test/src/test/java/org/opensearch/sql/ppl/SearchCommandIT.java

+    JSONObject result1 =
+        executeQuery(
+            String.format(
+                "search source=%s earliest='2025-08-01 03:47:41' latest=now | fields @timestamp",


What if earliest is a field name? Should add backtick?

penghuo · 2025-09-22T22:35:16Z

ppl/src/main/antlr/OpenSearchPPLParser.g4


 searchExpression
- : LT_PRTHS searchExpression RT_PRTHS                 # groupedExpression
+ : timeModifier                                       # timeModifierExpression


+1, please update doc.

penghuo · 2025-09-22T22:37:36Z

ppl/src/main/antlr/OpenSearchPPLParser.g4

- : LT_PRTHS searchExpression RT_PRTHS                 # groupedExpression
+ : timeModifier                                       # timeModifierExpression
+ | LT_PRTHS searchExpression RT_PRTHS                 # groupedExpression
 | NOT searchExpression                               # notExpression


do timemodifier support NOT? e.g. search source=index NOT latest=now

Yes. Time modifiers are converted conditions with SearchComparison. Any further operation that is applicable to search comparison, such as NOT, OR, AND, are applicable to time modifiers.

penghuo · 2025-09-22T22:42:31Z

core/build.gradle

    testImplementation group: 'org.hamcrest', name: 'hamcrest-library', version: "${hamcrest_version}"
    testImplementation group: 'org.mockito', name: 'mockito-core', version: "${mockito_version}"
    testImplementation group: 'org.mockito', name: 'mockito-junit-jupiter', version: "${mockito_version}"
+    testImplementation group: 'org.opensearch', name: 'opensearch', version: "${opensearch_version}"


parseRelativeTime() is called by PPL module only, right? if move parseRelativeTime to PPL module, does opensearch depedency still needed?

yuancu added the enhancement New feature or request label Sep 4, 2025

yuancu force-pushed the issues/4135 branch from f77ef7a to e8b709e Compare September 12, 2025 04:51

yuancu changed the title ~~Support absolute time range in search command~~ Support time modifiers in search command Sep 12, 2025

yuancu force-pushed the issues/4135 branch from e8b709e to a33f720 Compare September 12, 2025 04:54

yuancu force-pushed the issues/4135 branch from a33f720 to 2161e0e Compare September 16, 2025 08:51

yuancu commented Sep 16, 2025

View reviewed changes

yuancu added 4 commits September 16, 2025 17:54

Fix anomalyzed test & add a todo for an ignored test

8611075

Signed-off-by: Yuanchun Shen <[email protected]>

Support now() in time range

1cdc7e4

Signed-off-by: Yuanchun Shen <[email protected]>

Merge remote-tracking branch 'origin/main' into issues/4135

194840e

Fix time modifier explain ITs

b3a7ee2

Signed-off-by: Yuanchun Shen <[email protected]>

yuancu commented Sep 17, 2025

View reviewed changes

yuancu force-pushed the issues/4135 branch from bf3e1c8 to 9602c9d Compare September 17, 2025 07:40

Support unixtimestamp (second) as a time modifier value

5103b30

Signed-off-by: Yuanchun Shen <[email protected]>

yuancu force-pushed the issues/4135 branch from 9602c9d to 5103b30 Compare September 17, 2025 07:54

yuancu marked this pull request as ready for review September 17, 2025 08:32

yuancu requested review from ps48, kavithacm, derek-ho, joshuali925, dai-chen, YANG-DB, mengweieric, vamsimanohar, Swiddis and penghuo as code owners September 17, 2025 08:32

yuancu requested review from seankao-az, MaxKsyunz, Yury-Fridlyand, anirudha, forestmvey, acarbonetto, GumpacG, ykmr1224, LantaoJin, noCharger and qianheng-aws as code owners September 17, 2025 08:32

yuancu requested a review from RyanL1997 September 19, 2025 05:18

vamsimanohar reviewed Sep 22, 2025

View reviewed changes

penghuo requested changes Sep 22, 2025

View reviewed changes

PPL Time Modifier	OpenSearch Date Math	Explanation
'2023-01-01'	2023-01-01	Simple date
'2023-01-01 13:45:30'	2023-01-01T13:45:30Z	Date with time

PPL Time Modifier	OpenSearch Date Math	Explanation
now	now	Current time
-30s	now-30s	30 seconds ago
-1h	now-1h	1 hour ago
+1d	now+1d	1 day in future

PPL Time Modifier	OpenSearch Date Math	Explanation
-1d@d	now-1d/d	1 day ago, rounded to day
@h	now/h	Round to current hour
@m	now/M	Round to current month

PPL Time Modifier	OpenSearch Date Math	Explanation
@w0	/w-1d or /w+6d	Previous Sunday (depending on current day)
@w1	/w	Start of week (Monday)
@w4	/w+3d	Thursday of current week

Support time modifiers in search command #4224

Are you sure you want to change the base?

Support time modifiers in search command #4224

Conversation

yuancu commented Sep 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Implementation Walk-through

Implementation Details

PPL Time Modifiers to OpenSearch Date Math Examples

Check List

Uh oh!

vamsimanohar commented Sep 5, 2025

Uh oh!

yuancu commented Sep 7, 2025

Uh oh!

vamsimanohar commented Sep 9, 2025

Uh oh!

yuancu Sep 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yuancu Sep 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vamsimanohar commented Sep 22, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

yuancu commented Sep 4, 2025 •

edited

Loading

yuancu Sep 16, 2025 •

edited

Loading

yuancu Sep 17, 2025 •

edited

Loading