Opengrok search and navigation search failing with "a fault occurred in an unsafe memory access operation"

[alighaddar10]

Describe the bug
After upgrading opengrok to version 1.14.15, we faced this issue twice in the last month when search or navigating on the instance with the following error on the webpage:

java.lang.InternalError: a fault occurred in an unsafe memory access operation
	at org.apache.lucene.util.fst.FST.readFirstArcInfo(FST.java:727)
	at org.apache.lucene.util.fst.FST.findTargetArc(FST.java:1101)
	at org.apache.lucene.codecs.lucene90.blocktree.SegmentTermsEnum.seekExact(SegmentTermsEnum.java:482)
	at org.apache.lucene.index.TermStates.loadTermsEnum(TermStates.java:130)
	at org.apache.lucene.index.TermStates.lambda$build$0(TermStates.java:106)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at org.apache.lucene.search.TaskExecutor$TaskGroup$1.run(TaskExecutor.java:120)
	at org.apache.lucene.search.TaskExecutor$TaskGroup.invokeAll(TaskExecutor.java:176)
	at org.apache.lucene.search.TaskExecutor.invokeAll(TaskExecutor.java:84)
	at org.apache.lucene.index.TermStates.build(TermStates.java:116)
	at org.apache.lucene.search.TermQuery.createWeight(TermQuery.java:275)
	at org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:899)
	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:673)
	at org.opengrok.indexer.web.SearchHelper.executeQuery(SearchHelper.java:486)
	at org.apache.jsp.search_jsp._jspService(search_jsp.java:266)
	at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:62)
	at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:658)
	at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:428)
	at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:350)
	at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:301)
	at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:658)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

We are running opengrok in a kubernetes deployment on a physical server running RHEL 9.4. The issue was resolved after scaling down then up the deployment which replaced the failing pod.

To Reproduce
The issue happened after the code was synced and indexed, and the instance was restarted. This is the indexing command we run on a daily basis post sync: bash /scripts/index.sh

#!/bin/bash

LOCKFILE=/var/run/opengrok-indexer
URI="http://localhost:8080"
DEFAULT_BRANCH=/main
# $OPS can be overwritten by environment variable
OPS=${INDEXER_FLAGS:='-P -G'}

if [ -f "$LOCKFILE" ]; then
        date +"%F %T Indexer still locked, skipping indexing"
        exit 1
fi

touch $LOCKFILE

if [ -z $NOMIRROR ]; then
        date +"%F %T Mirroring starting"
        opengrok-mirror --all --uri "$URI"
        date +"%F %T Mirroring finished"
fi

date +"%F %T Indexing starting"
opengrok-indexer -J=-Xmx32g \
    -a /opengrok/lib/opengrok.jar -- \
    -s /opengrok/src \
    -d /opengrok/data \
    -p $DEFAULT_BRANCH \
    -W /opengrok/etc/configuration.xml \
    -U "$URI" \
    --token @/opengrok/etc/webapp_api_token \
    $OPS \
     -i *.tlb -i *.lib -i *.obj -i *.pdb -i *.dll -i *.so.* -i *.a -i *.jar -i *.ncb -i *.zip -i *.exe -i *.idb -i *.ilk -i *.jpg -i *.xdb -i *.psd -i *.dbf -i *.doc -i *.cdu -i *.idl -i *.ppt -i *.gif -i *.dat -i *.idx -i *.nb -i *.pdf -i *.user -i *.exp -i *.lnk -i *.png  -i *.bin -i *.bmp -i *.db -i *.ico -i *.opt -i *.orig -i *.out -i *.p12 -i *.pem -i *.qbk -i *.ser -i *.snk -i *.keep -i *.keep.* -i *.suo -i *.xls -i *.xdc -i *.index -i *.tree -i *.tmp -i *.class -i *.ls \
    $INDEXER_OPT "$@"
date +"%F %T Indexing finished"

rm -f $LOCKFILE

The indexing-out.log (index command output) showed the following error which was missing from the previous runs resulting in a healthy instance after indexing:

SEVERE: Couldn't notify the webapp on http://localhost:8080.
jakarta.ws.rs.NotAuthorizedException: HTTP 401 Unauthorized
	at org.glassfish.jersey.client.JerseyInvocation.convertToException(JerseyInvocation.java:959)
	at org.glassfish.jersey.client.JerseyInvocation.translate(JerseyInvocation.java:770)
	at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$1(JerseyInvocation.java:687)
	at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:709)
	at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:703)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:205)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:391)
	at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:703)
	at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686)
	at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:422)
	at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:318)
	at org.opengrok.indexer.index.IndexerUtil.enableProjects(IndexerUtil.java:96)
	at org.opengrok.indexer.index.Indexer.enableProjectsInWebApp(Indexer.java:463)
	at org.opengrok.indexer.index.Indexer.runMain(Indexer.java:359)
	at org.opengrok.indexer.index.Indexer.main(Indexer.java:170)

This issue as mentioned previously is not consistent, and we had two occurrences over the last month.

Best,
Ali

[alighaddar10]

As an update, we noticed that the 401 unauthorized above is directly correlated, as when we restart a failing pod, this 401 error disappears while the webapp_api_token remains the same.

The pod that was restarted post "401 unauthorized" in indexing logs and "a fault occurred in an unsafe memory access operation":

root@opengrok-2-7d9dc57dc7-44fbn:/# curl -k -X GET -H "Authorization: Bearer $(cat /proc/1/root/opengrok/etc/webapp_api_token)" http://localhost:8080/api/v1/projects
[...] (successful run, redacting data)

The pod that was not restarted:

root@opengrok-d9c779cd9-kjv56:/# curl -k -X GET -H "Authorization: Bearer $(cat /proc/1/root/opengrok/etc/webapp_api_token)" http://localhost:8080/api/v1/projects
<!doctype html><html lang="en"><head><title>HTTP Status 401 – Unauthorized</title><style type="text/css">body {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 401 – Unauthorized</h1><hr class="line" /><p><b>Type</b> Status Report</p><p><b>Message</b> Unauthorized</p><p><b>Description</b> The request has not been applied to the target resource because it lacks valid authentication credentials for that resource.</p><hr class="line" /><h3>Apache Tomcat/10.1.55</h3></body></html>

Upon restarting the latter pod, the error disappeared. I spawned a debug container beside each of the opengrok containers in the aforementioned pods to run curl and such tools, which is why the path to the token was prefixed with /proc/1/root.