Books disappear from `booksIDsMap` in `data.json`

[tyler-dot-earth]

Over time, books seem to disappear at random form booksIDsMap in data.json.

Underlying cause is unknown.

Possible suspects:

• Unreliable Readwise plugin code (refactor: improve sync reliability #67 may fix)
• Obsidian Sync performing bad merges (Settings > Sync > Settings version history - View shows the diffs between data.json changes)

MITIGATION: You can fix this locally by doing a Readwise sync in a clean vault and then copying the data.json content from the clean vault into your problematic vault. This is a pretty expansive set of instructions for doing that.

[tyler-dot-earth]

Something that may be helpful is a button/action to re-add books to booksIDsMap that may have been lost by looking at the files in your readwise directory. However, there's one issue: those files do not contain the book IDs. PR #62 adds the ability to save book IDs to frontmatter, which would allow the plugin to restore lost books.

[tyler-dot-earth]

I can confidently say that Obsidian Sync is now the primary culprit of this bug after #67's changes which deleted this plugin's code which removed booksIDsMap entries (as deleted items are tracked another way in that PR).

[tyler-dot-earth]

Got some info from the Obsidian team on how to access Sync's status. It's an internal API, though they note that it's accessible and has never changed.

I'm gonna make a personal branch and do some explroing. Plan is to basically wait for Sync to complete (if the plugin is enabled), then kick off Readwise's sync. I'll also add a fallback in case Obsidian's sync is not present, failing, or non-responsive.

[rvjosh]

I just noticed a lot of books disappearing from booksIDsMap in data.json - do you know if books disappearing from this causes any problems with the highlights in my vault? Put more simply: should I care if books are disappearing from booksIDsMap? Thanks for the contributions you've been making to the plugin!

[tyler-dot-earth]

@rvjosh

I just noticed a lot of books disappearing from booksIDsMap in data.json - do you know if books disappearing from this causes any problems with the highlights in my vault? Put more simply: should I care if books are disappearing from booksIDsMap? Thanks for the contributions you've been making to the plugin!

Shouldn't cause any issues unless you need to resync them from Readwise.

[tyler-dot-earth]

More evidence that this is almost definitely a race condition between Obsidian Sync and Readwise syncs: https://www.reddit.com/r/readwise/comments/1f79gfk/comment/llj533w/

tl;dr:

• user uses Obsidian Sync
• user has several devices
• user uses 1 device significantly less than the others
• ... thus, that device is more "out of sync" (which leads to Obsidian Sync and Readwise syncs competing)
• doing a clean vault install of the plugin synced all of their "missing" Readwise highlights - indicating that Obsidian Sync is clobbering the data.json file in these "race" cases

This is extremely similar to my own setup (multidevice, different usage) with the same problems presenting.

I haven't had time to try exploring like i mentioned in this comment, but i'm hoping to find time soon.

[rvjosh]

For what it's worth, you may be right, but I do not use Obsidian sync but still observed this issue

[tyler-dot-earth]

@rvjosh very interesting! can you speak more to your setup? like:

• are you using another syncing solution?
• are you using the readwise plugin on more than one device or vault?

[rvjosh]

Sure thing.

• I use Git/Github to sync my vault
• Yes, I sync this vault between multiple Mac laptops - I don't use mobile Obsidian at all.

[tyler-dot-earth]

Hmm. After thinking a bit, @rvjosh I would like you to keep an eye out for new occurrences of this issue after v2.1.1 (current version).

Rationale: the changes from #67 made it so the plugin no ever longer deletes entries from booksIDsMap - which was something it previously did under certain logic.

Given that you're using your own syncing solution via git, I think there's a low likelihood that it's caused by some sort of conflict resolution when manually pulling/syncing between your devices (though, hey, it's possible given that Obsidian Sync may use a similar algorithm to determine conflict resolution, albeit with additional sync logic).

So: if the issue occurs for you again, it would mean either (1/less likely) there's some serious oversight in the plugin code that is removing entries, (2/more likely) you're clobbering your plugin data during the git sync/conflict/etc process somehow.

That said, I'm optimistic that you'll have more luck with the removal of the booksIDsMap-entry-deleting code from #67/v2.1.1. Keep this thread posted if that isn't the case :)

[rvjosh]

Will do, thanks!

[mjkerrison]

Another workaround - in case people need it - seems to be just not doing an automatic sync. So it looks like:

• Have plugin settings synced via Sync
• Do not have automatic / on launch / etc. syncs enabled in the Readwise plugin
• Let out-of-date vault catch up with Sync
• Then manually trigger Readwise sync

Basically manually preventing any race problems.