Allow to execute Y.js and Insight together #11477

JaroslavTulach · 2024-11-04T10:29:35Z

Since the implementation of reload insight script on file change we have a nice and comfortable way to execute Enso Insight scripts like Detect Assign Statements in the IDE and observe how engine executes the code.

The next step is to connect Y.js and Insight together. Let's do it like this:

modify ContextInsightSetup to emulate necessary JS environment
the emulation has already been written and is part of ydoc-server module.
make sure the emulation is also initialized in ContextInsightSetup via the epb language:

    var epbSrc =
        Source.newBuilder("epb", path.toFile())
            .content("js:0#" + codeToExecute)
            .build();

It is essential to use epb language as it sets a proper inner context for JavaScript rather than trying to execute JavaScript directly (as it is not ready for multi threaded access).

all of the above has been solved by Fill in ydoc-polyfill when running backend with the IDE #11822
we just need a demo:

At the end please demonstrate how a simple script showing co-operation between Enso Insight and Y.js structures can be created and used in the IDE via -Denso.dev.insight property.

The text was updated successfully, but these errors were encountered:

JaroslavTulach · 2024-12-10T06:13:25Z

@Frizi, @4e6 I guess it is time to specify what:

demonstrate how a simple script showing co-operation

shall mean? I've just tried 2024.5.1-nightly.2024.12.10 version:

if I perform sbt ydoc-server/test
a new file ./lib/java/ydoc-server/target/classes/org/enso/ydoc/server/ydoc.cjs gets created
I assume that's the script we want to execute to demonstrate Y-doc server can run together with Insight
thus we shall follow the tutorial and just set ENSO_JVM_OPTS=-Denso.dev.insight=$ENSO_ROOT/lib/java/ydoc-server/target/classes/org/enso/ydoc/server/ydoc.cjs

Is that correct assumption guys? Then the whole script would be:

enso$ sbt ydoc-server/test
enso$ export ENSO_JVM_OPTS=-Denso.dev.insight=`pwd`/lib/java/ydoc-server/target/classes/org/enso/ydoc/server/ydoc.cjs
enso$ sbt runProjectManagerDistribution

and launching IDE in another terminal as corepack pnpm run dev:gui.

JaroslavTulach · 2024-12-10T07:11:25Z

With 2024.5.1-nightly.2024.12.10 the above steps fail with:

[ERROR] [2024-12-10T08:07:51.962] [org.enso.languageserver.protocol.json.JsonConnectionController] Failed to initialize resources.
java.util.concurrent.CompletionException: org.graalvm.polyglot.PolyglotException: ReferenceError: crypto is not defined
        at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)
        at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320)
        at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1807)
        at org.enso.akka.wrapper/akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:49)
        at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1423)
        at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387)
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1312)
        at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1843)
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1808)
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:188)
Caused by: org.graalvm.polyglot.PolyglotException: ReferenceError: crypto is not defined
        at <js>.poly_enso_eval(ydoc.cjs:3858)
        at <epb>.(Unknown)
        at org.graalvm.polyglot/org.graalvm.polyglot.Context.initialize(Context.java:579)
        at org.enso.language.server/org.enso.languageserver.boot.resource.TruffleContextInitialization.initComponent(TruffleContextInitialization.java:47)
        at org.enso.language.server/org.enso.languageserver.boot.resource.LockedInitialization.lambda$init$0(LockedInitialization.java:37)
        at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)

We need to modify ContextInsightSetup to emulate necessary JS environment as provided by ydoc-polyfill.

Fill in ydoc-polyfill when running backend with the IDE #11822

Created a #11822 PR to turn on the ydoc-polyfill.

JaroslavTulach · 2024-12-17T14:11:36Z

Once I asked Where is the (type script?) code representing an AST element? Is there information about offset/length? What code needs to be added lib/java/ydoc-server/target/classes/org/enso/ydoc/server/ydoc.cjs to print all AST that shall be updated with expressionUpdates and their offset/length? Having a diff doing this printing would be a good entry point for me.

@Frizi wrote:

There is no offset/length in AST now. It is a mutable tree structure first. Here are the primary parse methods:

enso/app/ydoc-shared/src/ast/parse.ts

Lines 448 to 457 in 9cf7833

/** Parse the input as a complete module. */

export function parseModule(code: string, module?: MutableModule): Owned<MutableBodyBlock> {

return parseModuleWithSpans(code, module).root

}

/** Parse the input as a body block, not the top level of a module. */

export function parseBlock(code: string, module?: MutableModule): Owned<MutableBodyBlock> {

const tree = rawParseBlock(code)

return abstract(module ?? MutableModule.Transient(), tree, code).root

}

This is where we use the parser inside the ydoc server currently:

enso/app/ydoc-server/src/languageServerSession.ts

Line 605 in 9cf7833

const { root, spans } = Ast.parseModuleWithSpans(snapshotCode, syncModule)

4e6 · 2024-12-20T15:06:07Z

I was not able to get enough information from the insight script to prepare update messages that could be exchanged with GUI. For example,

insight.on("return", function(ctx, frame) {
  print(`[return] ctx='${JSON.stringify(ctx)}'`)
  print(`[return] frame='${JSON.stringify(frame)}'`)
}, {
  expressions: true,
  statements: true,
  roots: true,
  rootNameFilter: ".*main"
});

`ctx`

{
   "name":"local.NewProject1.Main::local.NewProject1.Main::main",
   "source":{
      "characters":"from Standard.Base import all\nfrom Standard.Table import all\nfrom Standard.Database import all\nfrom Standard.AWS import all\nfrom Standard.Google_Api import all\nfrom Standard.Snowflake import all\nimport Standard.Examples\nimport Standard.Visualization\n\n## ICON group\nfirst_column file1 =\n    vector1 = file1.column_names\n    any1 = vector1.first\n    any1\n\nmain =\n    file1 = Data.read Examples.welcome\n    any1 = Main.first_column file1\n    any3 = file1.at 'Column 1'\n\n\n\n#### METADATA ####\n[[{\"index\":{\"value\":278},\"size\":{\"value\":5}},\"b9b7d881-fed9-49cc-a3c0-feb247635f89\"],[{\"index\":{\"value\":300},\"size\":{\"value\":18}},\"2bbf7b4a-6866-4783-a793-22b72c1d5161\"],[{\"index\":{\"value\":330},\"size\":{\"value\":13}},\"c3dedc01-bbbc-4660-9dad-7a0f5f50d59a\"],[{\"index\":{\"value\":348},\"size\":{\"value\":4}},\"17961ddd-0306-4c20-a7f5-82eace2f7e5e\"],[{\"index\":{\"value\":373},\"size\":{\"value\":26}},\"fcf199e6-d2b1-44b7-be92-a47545d6a593\"],[{\"index\":{\"value\":411},\"size\":{\"value\":23}},\"3d29652c-e20c-4436-addb-1060cc55013a\"],[{\"index\":{\"value\":446},\"size\":{\"value\":8}},\"fce3a9bd-d7c7-40d9-887c-eceb0ee25240\"],[{\"index\":{\"value\":446},\"size\":{\"value\":19}},\"dfc7341d-aba3-49c6-be18-f5f9e5562d83\"]]\n{\"ide\":{\"node\":{\"b9b7d881-fed9-49cc-a3c0-feb247635f89\":{\"position\":{\"vector\":[-388,282]}},\"2bbf7b4a-6866-4783-a793-22b72c1d5161\":{\"position\":{\"vector\":[-388,210]}},\"c3dedc01-bbbc-4660-9dad-7a0f5f50d59a\":{\"position\":{\"vector\":[-388,138]}},\"17961ddd-0306-4c20-a7f5-82eace2f7e5e\":{\"position\":{\"vector\":[-388,66]}},\"fcf199e6-d2b1-44b7-be92-a47545d6a593\":{\"position\":{\"vector\":[-388,282]},\"visualization\":{\"show\":false,\"width\":1027.671875,\"height\":193}},\"3d29652c-e20c-4436-addb-1060cc55013a\":{\"position\":{\"vector\":[-443,177]}},\"dfc7341d-aba3-49c6-be18-f5f9e5562d83\":{\"position\":{\"vector\":[-145,177]}},\"fce3a9bd-d7c7-40d9-887c-eceb0ee25240\":{\"position\":{\"vector\":[-145,177]}}},\"widget\":{},\"import\":{},\"snapshot\":\"eJx9kD0LwjAQhvf8igMHt4A4O/iFOKhDRcdytokEL7mSplb99ba2FotopiTPk/deoj1biAK6FH0qZ5grMDZjHwCJhO7RPZ7oD15gwNPfgOkx+g1XzGdS8TQzv53IcakJL70Z7baTlje0Gan8CxxMXiCZBwbDTojBANbz3RbOnotMaOPzECdMhXWgDakRTARU66qSwL46NbeyUWKHthpRC+juNW09+QrqgBAWjWuj2liof0p6hSm8u8pSUcJWfQZuqofyu9ZbGXeNMMBw3hijoXgCQg+d7w==\"}}",
      "name":"local.NewProject1.Main",
      "language":"enso",
      "mimeType":null,
      "uri":"truffle:51f7bf11a1e5144c224e96c1c245f7b479c22bbc0fc4fb757bfe5e72ae5c58b4/local.NewProject1.Main"
   },
   "characters":"main =\n    file1 = Data.read Examples.welcome\n    any1 = Main.first_column file1\n    any3 = file1.at 'Column 1'",
   "line":16,
   "startLine":16,
   "endLine":19,
   "column":1,
   "startColumn":1,
   "endColumn":30,
   "charIndex":354,
   "charLength":111,
   "charEndIndex":465
}

`frame`

{
   "file1":{},
   "any1":"Column 1",
   "any3":{}
}

4e6 · 2024-12-20T15:09:06Z

If the issue is just about running ydoc-server from the insight script, then you can already do it since #11822 by pointing enso.dev.insight property to the ydoc-server-polyglot artifact:

ENSO_JVM_OPTS="-Denso.dev.insight=$(pwd)/app/ydoc-server-polyglot/dist/main.cjs" java -jar project-manager.jar

JaroslavTulach · 2024-12-21T12:23:38Z

The goal I'd like to see is:

modify the app/ydoc-server-polyglot/dist/main.cjs to update the AST somehow
modify the IDE to read/use such updated AST somehow

Btw. Use ctx.returnValue: https://www.graalvm.org/jdk20/tools/graalvm-insight/manual/#intercepting-and-altering-execution

GraalVM
GraalVM is an advanced JDK with ahead-of-time Native Image compilation.

enso-bot · 2024-12-27T18:50:43Z

Dmitry Bushev reports a new STANDUP for yesterday (2024-12-26):

Progress: [11477] Added ability to debug JS in Chrome Dev Tools. Implemented parsing of the span map. Implemented setting of the ast metadata from the insight script. It should be finished by 2025-01-03.

Next Day: Next day I will be working on the #11477 task. Continue working on the task

enso-bot · 2024-12-27T18:52:16Z

Dmitry Bushev reports a new STANDUP for today (2024-12-27):

Progress: [11477] Debugging the ast representation in chrome inspector. Fixed an issue with re-setting the metadata. Looking into Yjs connection to ydoc server. It should be finished by 2025-01-03.

Next Day: Next day I will be working on the #11477 task. Continue working on the task

JaroslavTulach · 2024-12-28T05:53:34Z

Added ability to debug JS in Chrome Dev Tools.

Do you mean something other than enso --inspect which has already been working?

enso-bot · 2024-12-31T21:16:48Z

Dmitry Bushev reports a new STANDUP for the provided date (2024-12-30):

Progress: [10923] Looking into performance regression. Removed the truffle boundaries. Updated the PR. Trying to produce the nodes graph for an alternative fix [11477] Started implementing the logic that attaches the Yjs websocket provider. It should be finished by 2025-01-03.

Next Day: Next day I will be working on the #11477 task. Continue working on the task

enso-bot · 2024-12-31T21:21:43Z

Dmitry Bushev reports a new STANDUP for yesterday (2024-12-31):

Progress: [11477] Implemented the Yjs provider that connects to Ydoc and listens to subdocks to mimic the gui behavior. Debugging issues with the websocket connection. It should be finished by 2025-01-03.

Next Day: Next day I will be working on the #11477 task. Continue working on the task

enso-bot · 2025-01-02T18:29:42Z

Dmitry Bushev reports a new STANDUP for yesterday (2025-01-01):

Progress: [11938] Inspecting the truffle asts to discover differences in AtomConstructor implementation. Re-running benchmarks. Merged the PR [11477] Debugging the Yjs websocket provider. It should be finished by 2025-01-03.

Next Day: Next day I will be working on the #11477 task. Continue working on the task

enso-bot · 2025-01-02T18:42:32Z

Dmitry Bushev reports a new STANDUP for today (2025-01-02):

Progress: [11477] Debugging the websocket implementation. Fixed an issue with establishing the connection. Debugging the issue with reading the response. Reduced the test example to a simple websocket connection. Created a draft PR It should be finished by 2025-01-03.

Next Day: Next day I will be working on the #11477 task. Continue working on the task

JaroslavTulach added the -language-server label Nov 4, 2024

github-project-automation bot added this to Issues Board Nov 4, 2024

github-project-automation bot moved this to ❓New in Issues Board Nov 4, 2024

enso-bot bot mentioned this issue Nov 5, 2024

Ensure EnsoMultiValue returns some Meta.type_of #11480

Merged

2 tasks

JaroslavTulach assigned 4e6 Nov 5, 2024

JaroslavTulach mentioned this issue Nov 22, 2024

Ydoc library #11615

Merged

3 tasks

JaroslavTulach mentioned this issue Dec 10, 2024

Fill in ydoc-polyfill when running backend with the IDE #11822

Merged

3 tasks

enso-bot bot mentioned this issue Dec 12, 2024

Allow (multi_value : Table) : Column to succeed #11482

Closed

4e6 moved this from 📤 Backlog to 🔧 Implementation in Issues Board Dec 17, 2024

JaroslavTulach mentioned this issue Jan 9, 2025

Avoid multiple serialize-deserialize cycles when returning results to Language Server clients #5741

Closed

JaroslavTulach linked a pull request Jan 31, 2025 that will close this issue

Execute yjs from insight script #11965

Draft

5 tasks

4e6 moved this from 🔧 Implementation to ⚙️ Design in Issues Board Feb 12, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow to execute Y.js and Insight together #11477

Allow to execute Y.js and Insight together #11477

JaroslavTulach commented Nov 4, 2024 •

edited

Loading

JaroslavTulach commented Dec 10, 2024 •

edited

Loading

JaroslavTulach commented Dec 10, 2024 •

edited

Loading

JaroslavTulach commented Dec 17, 2024

4e6 commented Dec 20, 2024

4e6 commented Dec 20, 2024 •

edited

Loading

JaroslavTulach commented Dec 21, 2024 •

edited by unfurl-links bot

Loading

enso-bot bot commented Dec 27, 2024

enso-bot bot commented Dec 27, 2024

JaroslavTulach commented Dec 28, 2024

enso-bot bot commented Dec 31, 2024

enso-bot bot commented Dec 31, 2024

enso-bot bot commented Jan 2, 2025

enso-bot bot commented Jan 2, 2025

Allow to execute Y.js and Insight together #11477

Allow to execute Y.js and Insight together #11477

Comments

JaroslavTulach commented Nov 4, 2024 • edited Loading

JaroslavTulach commented Dec 10, 2024 • edited Loading

JaroslavTulach commented Dec 10, 2024 • edited Loading

JaroslavTulach commented Dec 17, 2024

4e6 commented Dec 20, 2024

ctx

frame

4e6 commented Dec 20, 2024 • edited Loading

JaroslavTulach commented Dec 21, 2024 • edited by unfurl-links bot Loading

enso-bot bot commented Dec 27, 2024

enso-bot bot commented Dec 27, 2024

JaroslavTulach commented Dec 28, 2024

enso-bot bot commented Dec 31, 2024

enso-bot bot commented Dec 31, 2024

enso-bot bot commented Jan 2, 2025

enso-bot bot commented Jan 2, 2025

JaroslavTulach commented Nov 4, 2024 •

edited

Loading

JaroslavTulach commented Dec 10, 2024 •

edited

Loading

JaroslavTulach commented Dec 10, 2024 •

edited

Loading

`ctx`

`frame`

4e6 commented Dec 20, 2024 •

edited

Loading

JaroslavTulach commented Dec 21, 2024 •

edited by unfurl-links bot

Loading