RocksDB Exception: Incomplete

[amajedi]

A running southpaw java process can occasionally fail with the following stack trace:

Exception in thread "main" java.lang.RuntimeException: org.rocksdb.RocksDBException: Incomplete
	at com.jwplayer.southpaw.state.RocksDBState.delete(RocksDBState.java:317)
	at com.jwplayer.southpaw.index.MultiIndex.remove(MultiIndex.java:252)
	at com.jwplayer.southpaw.Southpaw.scrubParentIndices(Southpaw.java:1069)
	at com.jwplayer.southpaw.Southpaw.createDenormalizedRecords(Southpaw.java:680)
	at com.jwplayer.southpaw.Southpaw.build(Southpaw.java:497)
	at com.jwplayer.southpaw.Southpaw.run(Southpaw.java:1047)
	at com.jwplayer.southpaw.Southpaw.main(Southpaw.java:994)
Caused by: org.rocksdb.RocksDBException: Incomplete
	at org.rocksdb.RocksDB.delete(Native Method)
	at org.rocksdb.RocksDB.delete(RocksDB.java:1186)
	at com.jwplayer.southpaw.state.RocksDBState.delete(RocksDBState.java:315)
	... 6 more

The problem occurs in the underlying RocksDB native code, which makes debugging this difficult. There might be more information in the RocksDB logs in /tmp/RocksDB/southpaw/LOG.

Additionally, the failure does not consistently occur on the same delete operation which points to either memory or concurrency (locking) issues, but this is just a guess.