Using Node.js on mobile

[tdurand]

This started as a casual question from @b-g .. "are you gonna run node.js" on android.. which I answered.. "no no, will not, don't think it is possible..."

And yet here I am 😂.. Exploring the https://github.com/JaneaSystems/nodejs-mobile framework: https://code.janeasystems.com/nodejs-mobile

Turns out you can run Node.js on mobile .. I'm evaluating it on the node-mobile branch : https://github.com/opendatacam/opendatacam-mobile/tree/node-mobile

After battling a lot to integrate it along the NCNN framework.. I think I spent 6 hours on obscure and frustrating CMake builds errors yesterday ( it is also a C++ dependency).. I got a first version running. (not OpenDataCam yet.. but nodejs)

The idea behind this is to:

• Be able to reuse code from the main github.com/opendatacam/opendatacam .. To really evaluate if this will be a win for future maintenance.. The other option I am considering for now is to have all the code from node.js being ported to the client side and having next.js export a static app.. which can be done also in a way we can maintain a common part from what I see... As anyways if we run node.js on the mobile we also need to diverge from the main opendatacam code / build adapters for Mobile ( not launching darknet.. not using mongodb, etc etc )

• Use the Mobile device with the opendatacam app as a "server" like we do with Jetson... This is big feature I didn't think about at first.. but if we can run the node.js server on Android.. then we can connect to OpenDataCam from another device that is on the same network and operate OpenDataCam remotely like we do now. This enables more use cases..

• Be faster to deliver the app ... as the first version deadline is a bit time sensitive. This is not really a clear upside.. I think it is the same amount of work to port the node.js code to the client only as to adapt the node.js code to work on mobile...

Potential/ Confirmed Drawbacks (for now)

• Adds 50 MB to the app size .. Not that problematic I think as NCNN and yolo weights are at least 50 MB already.. so the app won't be lightweight anyways..

• At app install and for each subsequent update.. we need to copy the app assets so node mobile can access them: .. and from first tests it seems it is slow as we are copying all the node_module folder.. like 30s at least right now.. but I'm sure this can be optimized ..

• Next.js is not really made for a node-mobile deployment.. so right now need to hardcode the path where the files are copied to start :

  opendatacam-mobile/android/app/src/main/assets/nodejs-project/server.js

  Line 37 in acba05f

  const app = next({ dir: "/data/data/com.opendatacam/files/nodejs-project" })

  , that was hard to figure out and I'm not sure if this is super portable.. would need more work..

• More CPUs consumption.. node.js running means less power for other things.. not sure if this affects the YOLO performances..

• If a Web version (client side) of OpenDataCam with neural network running via WebAssembly is something we want to do at some point.. already working on a client side only version of OpenDataCam would be a big step forward this

• More complexity.. more problems.. less dependency is always best ;-)

[tdurand]

Update 🔥🙌🎉 !!

First POC working !!! yay !!! Was able to run OpenDataCam on my android !!

I kept exploring the "mobile node.js" road the past days and here are some more updated thoughts:

Upsides

• Be able to reuse code from the main github.com/opendatacam/opendatacam .. To really evaluate if this will be a win for future maintenance.. The other option I am considering for now is to have all the code from node.js being ported to the client side and having next.js export a static app.. which can be done also in a way we can maintain a common part from what I see... As anyways if we run node.js on the mobile we also need to diverge from the main opendatacam code / build adapters for Mobile ( not launching darknet.. not using mongodb, etc etc )

After putting more thoughs (& action) on this.. with going the node.js way on the Android app, we should be able to have just one repository 😁 . And have build task that creates the "darknet" version, the "android" version.. which is not only only appealing to ease the future maintenance but making this step for "android" will be a big step towards the vision of having OpenDataCam beeing able to hook to anything (hardware and software) that is able to give us a video feed, run a neural network framework and run node.js : Raspberry + DepthAI, Jetsons + Deepstream, iOS + NCNN, Android + NCNN, Darknet + Jetsons, NCNN + Jetsons, Tensorflow + any hardware, Pytorch + any hardware.... etc etc..

• Be faster to deliver the app ... as the first version deadline is a bit time sensitive. This is not really a clear upside.. I think it is the same amount of work to port the node.js code to the client only as to adapt the node.js code to work on mobile...

turns out it was "fast" (or not, we will never know for sure until I don't do the client side OpenDataCam version :-) ..)

• Persistence layer: won't run mongodb on android (or may I ? 🤷‍♂️😁), need to do something else that can be done on node.js mobile

• the coordinates of the box are still a bit off.. need to work this out.. some magic video resolution and aspect ratio debugging to do here

• the canvas drawing for the counting lines have some issue.. need to debug

• For remote operating.. for now we don't "forward" the camera view into node.js via a MJPEG stream.. we just use the trick of rendering the camera preview behind the html webview .. not prioritary for first release

• Use the Mobile device with the opendatacam app as a "server" like we do with Jetson... This is big feature I didn't think about at first.. but if we can run the node.js server on Android.. then we can connect to OpenDataCam from another device that is on the same network and operate OpenDataCam remotely like we do now. This enables more use cases..

Keep the ability to operate OpenDataCam remotely with the HTTP APIs (using android phone "headless" like a jetson) is a huge feature.. didn't fully valide this yet.. We might need to open a firewall port on Android to make this work.. but it should + the mjpeg stream needs to be figured out.

Updated Drawback

More CPUs consumption.. node.js running means less power for other things.. not sure if this affects the YOLO performances..

this does not seem to be the case

• Adds 50 MB to the app size

still true ! but not problematic IMO

• At app install and for each subsequent update.. we need to copy the app assets so node mobile can access them: .. and from first tests it seems it is slow as we are copying all the node_module folder.. like 30s at least right now.. but I'm sure this can be optimized ..

• Next.js is not really made for a node-mobile deployment.. so right now need to hardcode the path where the files are copied to start :

  opendatacam-mobile/android/app/src/main/assets/nodejs-project/server.js

  Line 37 in acba05f

  const app = next({ dir: "/data/data/com.opendatacam/files/nodejs-project" })

  , that was hard to figure out and I'm not sure if this is super portable.. would need more work..

Right now it takes 1 minute 40 seconds to start the app after an update or on first install.. (otherwise, just 5 second after subsequent opening).. I didn't work on this but I'm almost sure this copy can be optimized a lot by reducing to the absolute minimum the node_modules dependency we ship and maybe pack them together.. And even if it's not possible.. we need to be creative on the loading screen, that's all !

More complexity.. more problems.. less dependency is always best ;-)

Yeah.. sorry 🤷‍♂️ ! in this case the "less is more" isn't doing it.. But it is very true that we need to test is long term reliability is there (like running for several hours).. a "critical" piece of software we rely on is "nodejs-mobile" https://github.com/JaneaSystems/nodejs-mobile , which is tagged 0.3.2 . This very much fits into our "OpenDataCam is very alpha and we do not provide any guarantee that this will work for your use case, but we conceived it as a starting point from where you can build-on & improve." motto

If a Web version (client side) of OpenDataCam with neural network running via WebAssembly is something we want to do at some point.. already working on a client side only version of OpenDataCam would be a big step forward this

💔 ! Will do it anyways one day I know 😂

Developer experience

need to investigate better, but it is obviously harder.. we need to precompile the node.js app before copying it on android.. and we can't set debug points when running in android.. (or I don't know how to do it yet, we might be able to run the node.js server on the android in development mode and get a nice experience).. If it was just a Webview we could debug "as usual" using chrome remote devtool . + right now every time I touch the code I have to wait 2 min the app starts

[b-g]

Many thanks for the wow update! This sounds extremely promising. Astonishing that the CPU performance with the "node.js layer" is not suffering that much ... and the rest is equally astonishing, full computer magic!

In terms of Developer experience ... if we had the MJPEG stream you might be able to develop the ODC browser part in a webview in an Android simulator running on your mac locally? I developed a long time ago an iOS app in Titanium (js iOS apps framework) and the waiting time after every build drove me nuts :) Sorry about the the 2 mins "delay"!

[tdurand]

In terms of Developer experience ... if we had the MJPEG stream you might be able to develop the ODC browser part in a webview in an Android simulator running on your mac locally? I developed a long time ago an iOS app in Titanium (js iOS apps framework) and the waiting time after every build drove me nuts :) Sorry about the the 2 mins "delay"!

Yep, even without this, all the UI part we can develop it without the mobile / jetson... there is a "simulation mode" that simulates YOLO + camera frames .. But the things I need to develop now are bound to be done on the phone.. Persistence layer + in app purchase etc etc ;-) .. I'm pretty sure I can improve things so I don't worry much.

Many thanks for the wow update! This sounds extremely promising. Astonishing that the CPU performance with the "node.js layer" is not suffering that much ... and the rest is equally astonishing, full computer magic!

🙌 , yeah ! goal this week is to clean up things and launch beta test group!

[tdurand]

First alpha out: #3 (comment)

quick update on the workflow, in regards to the goal of having only one codebase and don't duplicate stuff that will be very hard to merge after

What I started to do now is to have a "mobile" branch on the main OpenDataCam repo where I push all the changes I need in order for the Android version to work..

Github can visualize the diff: opendatacam/opendatacam@development...mobile

It is very light for now.. the next step will be to build config parameters / adapters to be able to choose on the config file if we want the node.js to be build for Android or for Darknet.. But this step isn't prioritary now..

Also started to document the current workflow in this README: https://github.com/opendatacam/opendatacam-mobile