Support for external/mounted sources, or source metadata/grouping?

[sunny-g]

I have a ton of files throughout my Google Drive and Dropbox that I'd love to connect to Open Notebook without having to re-upload them and store them again. Likewise, after uploading directories of files (e.g. gitbooks or other docs), it can be really difficult to manage all the sources I might have created.

Are there any plans for allowing for indexing external sources (i.e., files stored elsewhere) to help make file management easier without having to re-implement a full-blown file/directory management system? If not that, then perhaps a way of grouping and simply organizing sources so that (nested) directories of source documents can be uploaded without flattening everything out?

[lfnovo]

Hey @sunny-g , we've been discussing a couple options for, at least, enabling you to search outside content and query them without pulling them into ON. Can you share more on your use case?

[sunny-g]

Basically, I've got a (locally synced) Google Drive with a TB of books and PDFs, as well as my Obsidian vault on the same machine.

One thing that'd be way better than NotebookLM and be a massive time saver of organizational effort would be to just point to a directory as my set of Sources for a Notebook (and then Open-Notebook could just reference the file paths for each source file and still generate and store embeddings). That way I don't have to duplicate files or manually import them, and can instead just treat my files as a read-only source for deeper analysis.

Does that make sense? Its the difference between bringing my files to Open Notebook and bringing Open Notebook to my files :)

Edit: on second reading of your question, thats pretty much what I'm asking for - search and query outside sources without having to import them (but still compute the embeddings to aid search)

[Sc0rpius91]

I think I have the same thing. I have a (also locally synced) OneDrive knowledge base which exists of hundreds of files with sensitive content. My goal is to be able to have a notebook with that as a source, so I can query that knowledge base when starting a new project.
I tried using the webURL extracted from my sharepoint-onedrive account, but this gives me an error, as the URL landing base doesnt seem to get any further than the login page, even though the link has my credentials embedded. (But maybe I'm doing things wrong :) )

[lfnovo]

Keeping this open as the canonical product Discussion for external or mounted source libraries, folder synchronization, and managing large imported collections.

We are consolidating the related use cases from #355 here so they remain part of the decision:

• using a local directory or Obsidian vault as a source;
• preserving or representing nested folders;
• handling hundreds or thousands of Markdown files;
• avoiding small manual upload batches such as the current 50-file workflow;
• keeping changing external files synchronized without creating duplicates.

There are at least two product layers to resolve: how external content is connected and synchronized, and how large source collections are organized inside Open Notebook. The technical proposal in #988 explores a stable external_id and upsert API that could enable folder watchers and other sync tools.

Please use this Discussion for workflows, constraints, connector expectations, and organization needs. Concrete implementation will be tracked in Issues after the direction is decided.

[lfnovo]

Status refresh: this thread is the evidence base for the connected libraries part of #1315 — Zotero, Obsidian, filesystem searched on demand, cited with a receipt, adopted as Sources only by your decision. Question 2 there is aimed at you: live search on demand, or a local index that syncs? Which matches how your library changes?