Node script to run Python code using Batavia (beeware#671)

* Added basic command to run some python code using batavia and node

* The compile-and-encode script now outputs the filename and bytecode as JSON
Cleaned up the run-in-batavia script and updated its description

* Replace Python run-in-batavia script with Node-based run_in_batavia script

* Output null if module is not found
Directly compile Python file given a path
Add current directory to Python path so imports work

* Improved error handling.

* Added documentation for the run_in_batavia.js command.

* Added run_in_batavia.js and compile_module.py in the manifest file

* Remove old version of loader.

* Removed special-case handling of Python paths ending with .py
Correction run_with_batavia -> run_in_batavia

* Added npm script to allow running Python code as ``npm run python file.py``

* The module compile is now never passed a file name, just the module name

* Updated tutorial and readme file for running Python code on command line using Batavia

* Allow running ``npm run python`` from subdirectory of project

Author: xitij2000

MANIFEST.in

@@ -4,6 +4,8 @@ include AUTHORS
 include LICENSE
 include Makefile
 include requirements*.txt
+include run_in_batavia.js
+include compile_module.py
 recursive-include docs *.bat
 recursive-include docs *.py
 recursive-include docs *.rst
@@ -12,4 +14,4 @@ recursive-include tests *.py
 recursive-include batavia *.js
 recursive-include testserver *.html
 recursive-include testserver *.py
-recursive-include testserver *.txt
+recursive-include testserver *.txt

README.rst

@@ -127,6 +127,19 @@ For more detailed instructions, see the `Python In The Browser
 <http://batavia.readthedocs.io/en/latest/intro/tutorial-1.html>`_ guide.
 
 
+Running Batavia in the terminal
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you want to run some Python code from a file in the terminal, you can also run Batavia on Node: ::
+
+$ npm run python /path/to/python/file.py
+
+This will should run the Python file and show output on the terminal.
+
+For more details see `Running Python code using Batavia from the command line
+<http://batavia.readthedocs.io/en/latest/intro/tutorial-2.html>`_.
+
+
 Documentation
 -------------
 

compile_module.py

@@ -0,0 +1,62 @@
+# coding=utf-8
+"""Compile Python module and returns its file name and base64-encoded bytecode
+as JSON.
+For use by 'run_in_batavia' script."""
+
+import argparse
+import json
+import py_compile
+import sys
+
+import base64
+import importlib
+import os
+import tempfile
+
+sys.path.insert(0, os.getcwd())
+
+
+def get_module_path(module):
+    module_spec = importlib.util.find_spec(module)
+
+    # TODO: handle importing namespace packages
+    if module_spec is None or module_spec.origin == 'namespace':
+        return
+
+    return module_spec.origin
+
+
+def python_module_to_b64_pyc(module):
+    module_file = get_module_path(module)
+
+    if module_file is None:
+        return
+
+    fp = tempfile.NamedTemporaryFile(delete=False)
+    fp.close()
+
+    try:
+        py_compile.compile(module_file, cfile=fp.name)
+        with open(fp.name, 'rb') as fin:
+            pyc = fin.read()
+    finally:
+        os.unlink(fp.name)
+
+    return {
+        'filename': os.path.basename(module_file),
+        'bytecode': base64.b64encode(pyc).decode('utf8'),
+    }
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument('module', help='Python module')
+
+    args = parser.parse_args()
+
+    print(json.dumps(python_module_to_b64_pyc(args.module),
+                     indent=4))
+
+
+if __name__ == '__main__':
+    main()

docs/intro/index.rst

@@ -13,4 +13,5 @@ reference objects and classes defined natively in JavaScript.
 
    tutorial-0
    tutorial-1
+   tutorial-2
    faq

docs/intro/tutorial-0.rst

@@ -81,8 +81,9 @@ You now have a working Batavia environment!
 Next Steps
 ----------
 
-Next, we can :doc:`setup the sandbox <tutorial-1>`, and try out
-running Python in your browser.
+Next, we can :doc:`setup the sandbox <tutorial-1>`, and try out running Python
+in your browser. Or your can try running some Python code :doc:`from the
+command line <tutorial-2>`.
 
 Troubleshooting Tips
 --------------------

docs/intro/tutorial-2.rst

@@ -0,0 +1,25 @@
+Tutorial: Running Python code using Batavia from the command line
+=================================================================
+
+Batavia includes a simple command-line script that you can use to run your
+Python code. To use this tool you need to have followed the instructions from
+:doc:`tutorial-0`.
+
+You can now run Python code from a code from the command line as follows:
+
+.. code-block:: bash
+
+    npm run python /path/to/python_file.py
+
+This runs the ``run_in_batavia.js`` script which in turn runs the Python code.
+This command will only work if you call it within the Batavia project directory
+and provide it the absolute path to the Python file to run.
+
+You can alternatively directly run the ``run_in_batavia.js`` in Node. If
+you are not in the Batavia project directory you can still use this script as
+follows:
+
+.. code-bloc:: bash
+
+    node /path/to/batavia/run_in_batavia.js /path/to/python_file.py
+

package.json

@@ -14,7 +14,8 @@
     "build:noCache": "DISABLE_WEBPACK_CACHE=true node_modules/.bin/webpack --progress",
     "watch": "node_modules/.bin/webpack --progress --watch",
     "serve": "node_modules/.bin/webpack-dev-server --inline --hot --port 3000",
-    "lint": "node_modules/.bin/eslint batavia"
+    "lint": "node_modules/.bin/eslint batavia",
+    "python": "./run_in_batavia.js"
   },
   "repository": {
     "type": "git",

run_in_batavia.js

@@ -0,0 +1,153 @@
+#!/usr/bin/env node
+/**
+ * Run Python file in a new Batavia VM.
+ */
+
+const fs = require('fs')
+const path = require('path')
+const {spawnSync} = require('child_process')
+const batavia = require('./batavia/batavia.js')
+
+// The compile_module script is in the same directory as this script.
+const compileScriptPath = path.join(__dirname, 'compile_module.py')
+
+function displayHelpText() {
+    console.log(`
+usage: [-h] file
+
+Runs Python file in node JS using using the Batavia virtual machine in Node.
+
+positional arguments:
+  file             Python file to run
+  
+optional arguments:
+  -h, --help       show this help message and exit`.trim())
+}
+
+function main() {
+    if (process.argv.length === 3) {
+        const argument = process.argv[2]
+
+        if (argument === '-h' || argument === '--help') {
+            displayHelpText()
+            return
+        }
+
+        let resolveBase
+        // The INIT_CWD environment variable is set in npm v5.4 and above. It
+        // contains the directory in which the npm was run. If we have that
+        // information we can resolve paths relative to it.
+        // If the file path not relative and INIT_CWD is defined, we'll resolve
+        // the provided path relative to it.
+        if (process.env.INIT_CWD && !path.isAbsolute(argument[0])) {
+            resolveBase = process.env.INIT_CWD
+        } else {
+            resolveBase = ''
+        }
+
+        const filePath = path.resolve(resolveBase, argument)
+        const modulePath = path.basename(filePath, '.py')
+        const basePath = path.dirname(filePath)
+
+        fs.access(
+            filePath,
+            fs.constants.F_OK | fs.constants.R_OK,
+            function(err) {
+                if (err) {
+                    console.log(
+                        'File "' + argument + '" does not exist ' +
+                        'or cannot be accessed by current user.')
+                } else {
+                    try {
+                        runInBatavia(basePath, modulePath)
+                    } catch (error) {
+                        if (error instanceof TypeError) {
+                            console.log('Invalid Python file.')
+                        } else throw error
+                    }
+                }
+            })
+    } else {
+        displayHelpText()
+    }
+}
+
+/**
+ * Runs the specified Python module
+ *
+ * @param basePath
+ * @param module
+ */
+function runInBatavia(basePath, module) {
+    const vm = new batavia.VirtualMachine({
+        loader: makeBataviaLoader(basePath),
+        frame: null
+    })
+
+    vm.run(module, [])
+}
+
+/**
+ * Creates a loader function for the Batavia VM that looks for code around the
+ * specified base path.
+ *
+ * @param basePath
+ * @returns {bataviaLoader}
+ */
+function makeBataviaLoader(basePath) {
+    /**
+     * Compiles the specified Python module and returns its bytecode in the
+     * format that Batavia expects. Returns null if module was not found so an
+     * ImportError can be raised.
+     *
+     * @param modulePath
+     * @returns {*}
+     */
+    function bataviaLoader(modulePath) {
+        const compileProcess = spawnSync(
+            'python3',
+            [compileScriptPath, modulePath],
+            {cwd: basePath}
+        )
+
+        checkForErrors(compileProcess)
+
+        const module = JSON.parse(compileProcess.stdout)
+
+        // Module compiler will output null if the specified module was not
+        // found.
+        if (!module) { return null }
+
+        return {
+            '__python__': true,
+            'bytecode': module.bytecode.trim(),
+            'filename': module.filename.trim()
+        }
+    }
+
+    return bataviaLoader
+}
+
+/**
+ * Checks the Python compile process result for errors. In case of error it
+ * alerts the user quits the program.
+ *
+ * @param processResult
+ */
+function checkForErrors(processResult) {
+    if (processResult.error) {
+        console.log(
+            'There was an error running the Python compile process.\n' +
+            'Ensure that Python 3 is installed and available as "python3".')
+        process.exit(1)
+    }
+    if (processResult.status !== 0) {
+        console.log('There was an error during import.')
+        console.log(processResult.stderr.toString())
+        process.exit(1)
+    }
+}
+
+if (require.main === module) {
+    main()
+}