Syntactic indexing with heuristics

[varungandhi-src]

Subpart of https://github.com/sourcegraph/sourcegraph/issues/58005

For this quarter:

• Add CLI which can generate SCIP indexes based on heuristics https://github.com/sourcegraph/sourcegraph/pull/57664
• Build and publish CLI The CLI executable can be consumed directly via Bazel, so this isn't needed.
• Syntactic indexing: add an option to CLI for reading data directly from git objects #60048
• Create new worker for batch indexing
  • Scaffold the worker and Bazel build for it: https://github.com/sourcegraph/sourcegraph/pull/59747
  • Add new DB table to hold an indexing queue https://github.com/sourcegraph/sourcegraph/issues/59794
  • Create scheduling worker (similar to autoindexing) to periodically schedule jobs for matching repositories https://github.com/sourcegraph/sourcegraph/issues/59795
  • MVP (1/2): add indexing functionality to the worker - with eager repo
    cloning as part of MVP, moving to a better strategy later https://github.com/sourcegraph/sourcegraph/issues/59802
  • MVP (2/2): add upload functionality to the worker https://github.com/sourcegraph/sourcegraph/issues/59801
  • Implement monitoring and Grafana dashboards for the worker https://github.com/sourcegraph/sourcegraph/issues/59796
  • Implement persistent Git caching - we can consider searcher component for inspiration https://github.com/sourcegraph/sourcegraph/issues/59800
  • Add gRPC (or just HTTP?) endpoint to the worker service to queue a reindex (internal API) (for GRPC see https://sourcegraph.com/docs/dev/background-information/grpc_tutorial)
  • Add a GraphQL mutation to schedule reindexing (public API)
  • Add worker Kubernetes configuration: https://github.com/sourcegraph/deploy-sourcegraph/tree/master/base
  • Add worker docker-compose configuration: https://github.com/sourcegraph/deploy-sourcegraph-docker/
  • Add worker to single-container cmd/server configuration: https://github.com/sourcegraph/sourcegraph/tree/main/cmd/server
  • Update architecture diagram: https://docs.sourcegraph.com/dev/background-information/architecture
  • Update documentation as per the new service checklist: https://docs.sourcegraph.com/dev/background-information/architecture/introducing_a_new_service
• Support ingesting & storing syntactic SCIP indexes
  • https://github.com/sourcegraph/sourcegraph/issues/61422
  • Update "precise-code-intel-worker" (name is now wrong?)

Stretch goals (these will most likely be pushed to the next quarter)

• Update worker with policies & scheduling for batch indexing jobs
• New GraphQL resolver or fields to request batch indexing
• Database schema changes (+ associated migration) for tracking heuristics-based indexes
• Changes to frontend code to prefer heuristics-based results before falling back to text based search
• Observability
• Analytics
• User-facing documentation

Once the feature works end-to-end (which should hopefully be sometime next quarter), we can start rolling out support for the most popular languages first: Java, TypeScript, JavaScript, Python, Go.

[varungandhi-src]

Mar 11 2024 project update:

• Status: We're making incremental progress. However, at the moment, there is still a lot of work that needs to be done (deployments, new GraphQL API design), we continue getting support requests, and the team's capacity will be reduced in April (parental leave). So the project is at risk for not being completed this quarter.
• Velocity: The complexity of the existing backend code, some of it related to scaling concerns, plus unfamiliar coding patterns with lots of indirection, has made on-boarding for us harder than what I had initially estimated.
• Big change in implementation direction for one sub-part: We've decided to change the implementation strategy for symbol matching/lookup, to filtering results from searcher/Zoekt with syntactic data, because about 2 weeks back, we learnt that the existing code for storing SCIP symbols is not well-suited to the kind of fuzzy matching that we want to do, and that storing all the symbols directly would require too much storage to be practical.

Some other challenges that were either unexpected or took longer than expected:

• There was an unexpected (minor) security issue we discovered when looking at some existing code; fixing that took some time.
• The Linear trial has taken up more of Varun's time than planned due to bugs in the issue importing functionality they have.