[BUG] Warm Index : CompositeDirectory : AccessDenied Exception during indexing doc #19052
Description
Describe the bug
Hi Team,
I created a warm index, on indexing doc getting AccessDeniedException in the logs.
The code base is the main branch, till commit Id d52d40477f94f6f7790ef7e235e7e2ffba3602d3
The stacktrace from the logs is shared below.
From the stacktrace it looks like during execution of CompositeDirectory.deleteFile this issue is coming.
This issue comes for some segment files and for other files the delete is successful. On debugging further looks like when openInput is called for a file then later when the file is being deleted we get the issue.
The remote storage is azure blob storage and repository-azure plugin is being used for that.
The issue does not looks related to setup, but mostly related to code.
Please help to resolve the issue.
java.nio.file.AccessDeniedException: K:\esdata\nodes\0\indices\brpHBehkSxWRLCRYE5BjEw\0\index\_0.kdm
at java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:89)
at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103)
at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108)
at java.base/sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:273)
at java.base/sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:109)
at java.base/java.nio.file.Files.deleteIfExists(Files.java:1191)
at org.opensearch.index.store.remote.filecache.FileCacheFactory.lambda$createDefaultBuilder$0(FileCacheFactory.java:68)
at org.opensearch.ExceptionsHelper.catchAsRuntimeException(ExceptionsHelper.java:360)
at org.opensearch.index.store.remote.filecache.FileCacheFactory.lambda$createDefaultBuilder$1(FileCacheFactory.java:68)
at org.opensearch.index.store.remote.utils.cache.LRUCache.removeNode(LRUCache.java:490)
at org.opensearch.index.store.remote.utils.cache.LRUCache.remove(LRUCache.java:183)
at org.opensearch.index.store.remote.utils.cache.SegmentedCache.remove(SegmentedCache.java:107)
at org.opensearch.index.store.remote.filecache.FileCache.remove(FileCache.java:111)
at org.opensearch.index.store.CompositeDirectory.deleteFile(CompositeDirectory.java:191)
at org.apache.lucene.store.FilterDirectory.deleteFile(FilterDirectory.java:65)
at org.opensearch.index.store.ByteSizeCachingDirectory.deleteFile(ByteSizeCachingDirectory.java:199)
at org.apache.lucene.store.FilterDirectory.deleteFile(FilterDirectory.java:65)
at org.opensearch.index.store.Store$StoreDirectory.deleteFile(Store.java:980)
at org.opensearch.index.store.Store$StoreDirectory.deleteFile(Store.java:985)
at org.apache.lucene.store.LockValidatingDirectoryWrapper.deleteFile(LockValidatingDirectoryWrapper.java:37)
at org.apache.lucene.util.FileDeleter.delete(FileDeleter.java:234)
at org.apache.lucene.util.FileDeleter.delete(FileDeleter.java:227)
at org.apache.lucene.util.FileDeleter.deleteFilesIfNoRef(FileDeleter.java:190)
at org.apache.lucene.index.IndexFileDeleter.deleteNewFiles(IndexFileDeleter.java:644)
at org.apache.lucene.index.IndexWriter.deleteNewFiles(IndexWriter.java:5902)
at org.apache.lucene.index.IndexWriter$1.lambda$deleteUnusedFiles$0(IndexWriter.java:430)
at org.apache.lucene.index.IndexWriter$EventQueue.processEventsInternal(IndexWriter.java:338)
at org.apache.lucene.index.IndexWriter$EventQueue.processEvents(IndexWriter.java:327)
at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:6021)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:586)
at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:382)
at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:356)
at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:346)
at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112)
at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:170)
at org.opensearch.index.engine.OpenSearchReaderManager.refreshIfNeeded(OpenSearchReaderManager.java:72)
at org.opensearch.index.engine.OpenSearchReaderManager.refreshIfNeeded(OpenSearchReaderManager.java:52)
at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167)
at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240)
at org.opensearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:446)
at org.opensearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:426)
at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167)
at org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:213)
at org.opensearch.index.engine.InternalEngine.refresh(InternalEngine.java:1811)
at org.opensearch.index.engine.InternalEngine.maybeRefresh(InternalEngine.java:1790)
at org.opensearch.index.shard.IndexShard.scheduledRefresh(IndexShard.java:4921)
at org.opensearch.index.IndexService.maybeRefreshEngine(IndexService.java:1343)
at org.opensearch.index.IndexService$AsyncRefreshTask.runInternal(IndexService.java:1491)
at org.opensearch.common.util.concurrent.AbstractAsyncTask.run(AbstractAsyncTask.java:177)
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:916)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
Related component
No response
To Reproduce
- Update the Opensearch.yml config for enabling remote storage (segment store, translog store, cluster state store). Enable the warm index experimental feature. Start OpenSearch
- In Create index request send index.warm as true
- After index creation, index a doc, that will be successful. But indexing 2nd doc will fail.
- In the logs the above mentioned stacktrace is visible as warning message.
- The index will be in red state with unassigned shards
- If we check the _cluster/allocation/explain we can see the allocation failure reason being AccessDeniedException
Expected behavior
For warm index we should be able to index and query docs
Additional Details
Plugins
repository-azure
Host/Environment :
- OS: Windows