Improve assets copy to android filesystem on first install & on update #8
Comments
tdurand
changed the title
|
Some trial and error progress notes: Found out this project while looking into the "webpack" of node.js : https://github.com/vercel/ncc , it is from the same people doing next.js It is supposed to be able to build your node.js app into a single This seems very much what I'm looking to do.. and I was able to make it work properly on a very simple example project that uses express: https://github.com/vercel/next.js/tree/canary/examples/custom-server . Final app necessary files:
What you need to do is: # create the .next folder containing the client side code
npm i
npm run build
# pack the node.js server side code in the dist/ folder
ncc build server.js
# this creates a dist/ folder with an index.js in it (18 MB with express.js)
# then you need to copy the client side code to the dist/ folder (so the index.js app can use it)
cp -r .next/ dist/
# then launching the app should work
cd dist/
node index.jsAlso very important you need to disable the auto routing of next.js by hooking to next to node.js this way in your server.js file const app = next({ useFileSystemPublicRoutes: false })But while trying on OpenDataCam I run into a couple of issues:
Couldn't figure out how to fix this yet... related to ncc packing several copy of React I think in the big index.js ... Also investigating I've seen someone trying the same thing but failing also (not for the same reason: vercel/next.js#8574 ) I will pause investigation for now as the priority is to get a beta out.. but this ncc tool is promising.... Also I think it might be a good time to get rid of express.js if we move forward with this.. as there are lightweights alternatives now.. |
|
Some update on this.. was able to make very good improvements on this, down to reducing the first load to 5 seconds, same as the subsequent opening of the app 🙌 , and reducing also the size of the app by 20-30 MB.. This improvement was made by adding a build step to statically export the whole front-end app and only ship this to the device instead of having the whole next.js dependencies and have the node.js making the front-end app render dynamically on the device... ( reminder: the slowness didn't come from rendering the app but from copying thousands of small @babel, react etc.. files ) This comes at the cost of increasing a bit the diff between the "opendatacam mobile" branch and the normal branch: opendatacam/opendatacam@development...mobile, documenting here some more changes for my future self
I guess in the future is it would be better to make this "webpack" for node : ncc solution work.. but in the meantime this is a good workaround that makes the experience way better. |
|
Dropping this here as I though about this old issue, maybe will be useful at some point 😅 , next.js now has a standalone output mode that exports the necessary dependencies without needing to cherry-pick node_modules folder: https://nextjs.org/blog/next-12-1?utm_source=next-site&utm_medium=banner&utm_campaign=next-website#self-hosted-nextjs-improvements |
Problem
In order to use nodejs-mobile , you need to first copy the node.js app files to the phone filesystem, you can't start node.js directly with the APK file. (todo when the App in first installed and anytime that the node.js app is changed (each app update) )
Right now we were using the "recommended" method from nodejs-mobile, which is to ship the app + node_module folder in the assets directory and recursively copy this to the phone file system https://code.janeasystems.com/nodejs-mobile/android/node-project-folder#copy-the-nodejs-project-at-runtime-and-start-from-there .
Problem with this is that with a big
node_modulefolder with quite a lot of dependency like OpenDataCam , this means copy pasting24,723 Files, 4,709 Foldersand142 MB.. And on top of adding 140 MB to the app size, copying lots of small files is slow."Out of the box" , the first app install takes 2min30s (and each update).
Two things we are looking to improve here:
copy files faster
Turns out that there was a low hanging fruit here to improve this, instead of copy pasting all the files recursively which is slow, we create a zip at build time of the node.js app and then unzip it on the phone. ( idea from stackoverflow: https://stackoverflow.com/a/42415755 )
I've tried that : c1fcca4 and we get from 2min30s to around 20 seconds .. So this is already a huge win 🙌
Only downside is that we need to integrate that zipping on the build process. for now I'm doing it manually with this command:
zip -0 -r ../nodejs-project.zip .Have less files (and less MBs) to copy
Thinking of having
24,723 Files, 4,709 Foldersfor142Mbof files in the node_modules folder is a bit crazy.. when the client side part of the app (actually loaded in the browser) is max 5 MB .. the server side part has obviously node.js has a dependency but not even included in those 142 MB ( it is 142 MB + the 30-50 MB of node.js) ... My guess is that it might also be only 5 - 20 MB in the end...Normally we never care about this because we do the
npm installon the server and the size of the node_modules folder doesn't matter.. what matters is how much you send to the client ( see explanation here: vercel/next.js#14339 ) . Here we want to also reduce this to avoid.One tool to our service for this is
npm prune --production, which you can run afternpm run buildand delete all thenode_modulesdependencies that are listed as "devDependencies" in the package.json: https://github.com/opendatacam/opendatacam/blob/master/package.json#L43Out of the box doing this already reduce a bit, down to
115 MB (121,226,417 bytes) and 22,298 Files, 4,231 FoldersWhat I need to do next is to improve this, by:
npm prune(which is I think just copy/do not copy) , here is an analyse of the dependencies tree .. for example ournode-moving-things-trackeris 10 MB because it ships MOT17 benchmark files... and the node.js is really using 3 javascript files, maybe 100 kB max 😉The text was updated successfully, but these errors were encountered: