[build][Jenkins] more efficient Jenkins CI, builds, improved linter config

This is a significant refactoring of the repo and CI.

1) Jenkins CI: more efficient, quicker builds

I propose that we adopt a "run often, run quickly" approach to our Jenkins CI.
ATM we mostly run Jenkins when it's time to release new Blueprint installers
(when a PR is merged on master). The other case is if "jenkins" is detected
in the PR branch or PR title, then almost a full Jenkins build is done, just
stopping short of publishing new Blueprint installers (full signing and
notarization is done, which can take a couple of hours).

I made it so we run a very light version of the Jenkins CI by default, that
builds the "dev" version of the app and pre-packages it using electron builder,
generating a directory that contains the app but no installers.

We really ought to run tests then, on the unpacked pre-packaged app, but
it will need to be done separately. Until then, we do not require the built-in
extensions (plugins), so I gave openvsx.org a rest and skip fetching them, by
default.

When it's detected that the Jenkins CI is processing the result of a merge on
master or if the new "release dry-run" mode is enabled [1], a production version
of the Blueprint apps will be produced, including built-in plugins, which will
then be packaged for "real", producing genuine OS-specific installers, signed
and notarized as required. This will take a lot longer to run, of course, but
may be required only a minority of the time.

Conclusion: with this PR, running Jenkins CI now takes about 12 minutes, and
since this is relatively quick, we can afford to run this when any PR branch
is created or updated.

Running the same CI in "release dry-run mode" takes about an hour. A release
will take longer, a bit over an hour, since additional steps are executed,
like copying the various installers to a release folder.

[1] About the "release dry-run" mode, it can be enabled in JenkinsFile,
"pipeline" definition near the top, in "environment":
  BLUEPRINT_JENKINS_RELEASE_DRYRUN = 'true'

I wish I found a better mechanism to enable this "release dry-run" mode. But
this will serve for now. The "release dry-run" mode should be disabled before
merging a PR to master, but can in the meantime be used to test or troubleshoot
the whole pipeline, in particular signing and notarizing.

2) tsconfig/eslint configs:

  Added configs, to improve linter coverage. This made it possible for some
  source files, not previously covered, to get ts/linter feedback, both while
  editing and when running `yarn lint`. This will help keep code in-line with
  our standards. The config is not perfect and I would welcome further
  improvements. But for now I think it's a nice improvement.

3) Build "scripts" (package.json)

Refactored the build commands ("scripts" section in package.json)

- previously, merely running 'yarn" in the repo's root would rebuild
  every application from scratch. This prevents running a quick
  "yarn install", e.g. just to re-install build dependencies
- the new version permits a granular build, with simple defaults
- inspired from a similar change not so long ago in the main repo
- see updated README for some examples

Other misc items:

- renamed extensions / applications
- made the browser application a first class citizen, equal to the
  Electron application
- all applications now share a common 'plugins' folder rather than
  each having their own. Moved the plugin-related entries to root
  package.json
- to gain flexibility about which `yarn workspaces` are invoked for a
  given `lerna` command, using the `--scope=` CLI option.I renamed the
  repo's extensions and applications. This permits easily composing
  commands that target only the extensions or only the applications.
  e.g.:

  ``` json
  "build:extensions": "lerna run --scope=\"blueprint*ext\" build",
  "build:applications": "lerna run --scope=\"blueprint*app\" build --concurrency 1","
  ```

- renamed the extensions folders, made them more straightforward
- For systems with limited RAM, like on a Raspberry Pi 4B board with 4GB of RAM,
  it's now possible to successfully build Blueprint. e.g. use the following cmd:
  `yarn && yarn build:dev`, optionally followed by a packing command like:
  `yarn electron package`
  - (build:dev will build the Blueprint app in dev mode, which can be achieved
    in 4GB RAM)
- [windows][jenkins] stash only dist folder: Currently, and only for Windows,
  we stash the whole git repo, which is very big and takes long to stash and
  un-stash. For the other OS, we only stash the dist folder, that contains the
  platform-specific installer, that we built. Let's try that for Windows too,
  and see if it works.
- [jenkins] Decrease timeout from 5h to 3h" Looking at the build history, all
  recent builds that succeeded, did do under 2h30. OTOH, when a build hangs,
  it needs to wait for the timeout to expire, wasting time at the current value
  of 5h. Let's compromise at 3 hours and see how it goes?
- [jenkins][installer build] exclude browser app for now: We do not yet publish
  the browser app, so let's skip building it to save time/resources. We may
  revisit when we use the new browser app bundling, recently made available
  upstream in the Theia framework.

Signed-off-by: Marc Dumais <[email protected]>

Author: marcdumais-work

.eslintrc.js

@@ -13,6 +13,6 @@ module.exports = {
     ],
     parserOptions: {
         tsconfigRootDir: __dirname,
-        project: 'tsconfig.json'
+        project: ['./configs/tsconfig.eslint.json', './theia-extensions/*/tsconfig.json', 'applications/electron/tsconfig.eslint.json']
     }
 };

.github/workflows/build.yml

@@ -46,7 +46,9 @@ jobs:
         shell: bash
         run: |
           yarn --skip-integrity-check --network-timeout 100000
-          yarn electron package
+          yarn build:dev
+          yarn download:plugins
+          yarn package:applications:preview
         env:
           NODE_OPTIONS: --max_old_space_size=4096
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} # https://github.com/microsoft/vscode-ripgrep/issues/9
@@ -62,7 +64,7 @@ jobs:
         shell: bash
         run: |
           yarn electron test
-
+          
       - name: Test (macOS)
         if: matrix.tests != 'skip' && runner.os == 'macOS'
         shell: bash

Jenkinsfile

@@ -1,39 +1,60 @@
 /**
  * This Jenkinsfile builds Theia across the major OS platforms
  */
+
+/* groovylint-disable NestedBlockDepth */
 import groovy.json.JsonSlurper
 
-distFolder = "applications/electron/dist"
 releaseBranch = "master"
-// Attempt to detect that a PR is Jenkins-related, by looking-for 
-// the word "jenkins" (case insensitive) in PR branch name and/or 
+distFolder = "applications/electron/dist"
+
+toStashDist = "${distFolder}/**"
+toStashDistInstallers = "${distFolder}/*"
+// default folder to stash
+toStash = toStashDistInstallers
+
+// Attempt to detect whether a PR is Jenkins-related, by looking-for
+// the word "jenkins" (case insensitive) in PR branch name and/or
 // the PR title
 jenkinsRelatedRegex = "(?i).*jenkins.*"
 
 pipeline {
     agent none
     options {
-        timeout(time: 5, unit: 'HOURS')
+        timeout(time: 3, unit: 'HOURS')
         disableConcurrentBuilds()
     }
     environment {
         BLUEPRINT_JENKINS_CI = 'true'
+
+        // to save time and resources, we skip some release-related steps
+        // when not in the process of releasing. e.g. signing/notarizing the
+        // installers. It can sometimes be necessary to run these steps, e.g.
+        // when troubleshooting. Set the variable below to 'true' to do so.
+        // We will still stop short of publishing anything.
+        BLUEPRINT_JENKINS_RELEASE_DRYRUN = 'false'
+        // BLUEPRINT_JENKINS_RELEASE_DRYRUN = 'true'
     }
     stages {
         stage('Build') {
-            // only proceed when merging on the release branch or if the 
-            // PR seems Jenkins-related
             when {
                 anyOf {
                     expression {
                         env.JOB_BASE_NAME ==~ /$releaseBranch/
                     }
                     expression { 
-                        env.CHANGE_BRANCH ==~ /$jenkinsRelatedRegex/ 
+                        env.CHANGE_BRANCH ==~ /$jenkinsRelatedRegex/
                     }
                     expression {
                         env.CHANGE_TITLE ==~ /$jenkinsRelatedRegex/
                     }
+                    expression {
+                        // PR branch? 
+                        env.BRANCH_NAME ==~ /PR-(\d)+/
+                    }
+                    expression {
+                        env.BLUEPRINT_JENKINS_RELEASE_DRYRUN == 'true'
+                    }
                 }
             }
             parallel {
@@ -83,11 +104,11 @@ spec:
                         container('theia-dev') {
                             withCredentials([string(credentialsId: "github-bot-token", variable: 'GITHUB_TOKEN')]) {
                                 script {
-                                    buildInstaller(1200, false)
+                                    buildInstaller(120)
                                 }
                             }
                         }
-                        stash includes: "${distFolder}/*", name: 'linux'
+                        stash includes: "${toStash}", name: 'linux'
                     }
                     post {
                         failure {
@@ -101,9 +122,9 @@ spec:
                     }
                     steps {
                         script {
-                            buildInstaller(60, false)
+                            buildInstaller(60)
                         }
-                        stash includes: "${distFolder}/*", name: 'mac'
+                        stash includes: "${toStash}", name: 'mac'
                     }
                     post {
                         failure {
@@ -125,9 +146,9 @@ spec:
                             bat "wmic OS get FreePhysicalMemory"
                             bat "tasklist"
 
-                            buildInstaller(60, true)
+                            buildInstaller(60)
                         }
-                        stash name: 'win'
+                        stash includes: "${toStash}", name: 'win'
                     }
                     post {
                         failure {
@@ -138,19 +159,23 @@ spec:
             }
         }
         stage('Sign and Upload') {
-            // only proceed when merging on the release branch or if the 
-            // PR seems Jenkins-related
+            // only proceed when merging on the release branch or if the
+            // PR seems Jenkins-related. Note: for PRs, we do not by default
+            // run this stage since it will be of little practical value.
             when {
                 anyOf {
                     expression {
                         env.JOB_BASE_NAME ==~ /$releaseBranch/
                     }
                     expression { 
-                        env.CHANGE_BRANCH ==~ /$jenkinsRelatedRegex/ 
+                        env.CHANGE_BRANCH ==~ /$jenkinsRelatedRegex/
                     }
                     expression {
                         env.CHANGE_TITLE ==~ /$jenkinsRelatedRegex/
                     }
+                    expression {
+                        env.BLUEPRINT_JENKINS_RELEASE_DRYRUN == 'true'
+                    }
                 }
             }
             parallel {
@@ -246,37 +271,53 @@ spec:
     }
 }
 
-def buildInstaller(int sleepBetweenRetries, boolean excludeBrowser) {
-    int MAX_RETRY = 3
+def buildInstaller(int sleepBetweenRetries) {
+    int maxRetry = 3
+    String buildPackageCmd
 
     checkout scm
-    if (excludeBrowser) {
-        sh "npm install -g ts-node typescript '@types/node'"
-        sh "ts-node scripts/patch-workspaces.ts"
+    
+    // only build the Electron app for now
+    buildPackageCmd = 'yarn --frozen-lockfile --force && \
+        yarn build:extensions && yarn electron build'
+
+    if (isRelease()) {
+        // when not a release, build dev to save time
+        buildPackageCmd += ":prod"
     }
-    sh "node --version"
-    sh "export NODE_OPTIONS=--max_old_space_size=4096"
-    sh "printenv && yarn cache dir"
-    sh "yarn cache clean"
+
+    sh 'node --version'
+    sh 'export NODE_OPTIONS=--max_old_space_size=4096'
+    sh 'printenv && yarn cache dir'
     try {
-        sh(script: 'yarn --frozen-lockfile --force')
-    } catch(error) {
-        retry(MAX_RETRY) {
+        sh(script: buildPackageCmd)
+    } catch (error) {
+        retry(maxRetry) {
             sleep(sleepBetweenRetries)
-            echo "yarn failed - Retrying"
-            sh(script: 'yarn --frozen-lockfile --force')        
+            echo 'yarn failed - Retrying'
+            sh(script: buildPackageCmd)
         }
     }
 
-    sh "rm -rf ./${distFolder}"
     sshagent(['projects-storage.eclipse.org-bot-ssh']) {
-        sh "yarn electron deploy"
+        if (isRelease()) {
+            sh 'yarn download:plugins && yarn electron package:prod'
+        } else {
+            // ATM the plugins are not useful for non-releases, so
+            // let's skip ketching them
+            sh 'yarn electron package:preview'
+        }
     }
 }
 
 def signInstaller(String ext, String os) {
+    if (!isRelease()) {
+        echo "This is not a release, so skipping installer signing for branch ${env.BRANCH_NAME}"
+        return
+    }
+
     List installers = findFiles(glob: "${distFolder}/*.${ext}")
-    
+
     // https://wiki.eclipse.org/IT_Infrastructure_Doc#Web_service
     if (os == 'mac') {
         url = 'https://cbi.eclipse.org/macos/codesign/sign'
@@ -296,6 +337,11 @@ def signInstaller(String ext, String os) {
 }
 
 def notarizeInstaller(String ext) {
+    if (!isRelease()) {
+        echo "This is not a release, so skipping installer notarizing for branch ${env.BRANCH_NAME}"
+        return
+    }
+
     String service = 'https://cbi.eclipse.org/macos/xcrun'
     List installers = findFiles(glob: "${distFolder}/*.${ext}")
 
@@ -325,15 +371,20 @@ def notarizeInstaller(String ext) {
 }
 
 def updateMetadata(String executable, String yaml, String platform, int sleepBetweenRetries) {
-    int MAX_RETRY = 4
+    if (!isRelease()) {
+        echo "This is not a release, so skipping updating metadata for branch ${env.BRANCH_NAME}"
+        return
+    }
+
+    int maxRetry = 4
     try {
         sh "export NODE_OPTIONS=--max_old_space_size=4096"
-        sh "rm -rf node_modules"
+        // make sure the npm dependencies are available to the update scripts
         sh "yarn install --force"
         sh "yarn electron update:blockmap -e ${executable}"
         sh "yarn electron update:checksum -e ${executable} -y ${yaml} -p ${platform}"
-    } catch(error) {
-        retry(MAX_RETRY) {
+    } catch (error) {
+        retry(maxRetry) {
             sleep(sleepBetweenRetries)
             echo "yarn failed - Retrying"
             sh "yarn install --force"
@@ -344,7 +395,7 @@ def updateMetadata(String executable, String yaml, String platform, int sleepBet
 }
 
 def uploadInstaller(String platform) {
-    if (env.BRANCH_NAME == releaseBranch) {
+    if (isReleaseBranch()) {
         def packageJSON = readJSON file: "package.json"
         String version = "${packageJSON.version}"
         sshagent(['projects-storage.eclipse.org-bot-ssh']) {
@@ -366,7 +417,7 @@ def uploadInstaller(String platform) {
  * Due to a bug in the nsis-updater the downloaded exe for an update needs to have a different name than initially however.
  */
 def copyInstallerAndUpdateLatestYml(String platform, String installer, String extension, String yaml, String UPDATABLE_VERSIONS) {
-    if (env.BRANCH_NAME == releaseBranch) {
+    if (isReleaseBranch()) {
         def packageJSON = readJSON file: "package.json"
         String version = "${packageJSON.version}"
         sshagent(['projects-storage.eclipse.org-bot-ssh']) {
@@ -385,8 +436,19 @@ def copyInstallerAndUpdateLatestYml(String platform, String installer, String ex
         } else {
             echo "No updateable versions"
         }
-
     } else {
         echo "Skipped copying installer for branch ${env.BRANCH_NAME}"
     }
 }
+
+def isReleaseBranch() {
+    return (env.BRANCH_NAME == releaseBranch)
+}
+
+def isDryRunRelease() {
+    return env.BLUEPRINT_JENKINS_RELEASE_DRYRUN == 'true'
+}
+
+def isRelease() {
+    return isDryRunRelease() || isReleaseBranch()
+}

README.md

@@ -1,7 +1,7 @@
 <br/>
 <div id="theia-logo" align="center">
     <br />
-    <img src="https://raw.githubusercontent.com/eclipse-theia/theia-blueprint/master/theia-extensions/theia-blueprint-product/src/browser/icons/TheiaBlueprintLogo-blue.png" alt="Theia Logo" width="300"/>
+    <img src="https://raw.githubusercontent.com/eclipse-theia/theia-blueprint/master/theia-extensions/product/src/browser/icons/TheiaBlueprintLogo-blue.png" alt="Theia Logo" width="300"/>
     <h3>Eclipse Theia Blueprint</h3>
 </div>
 
@@ -53,24 +53,40 @@ Documentation on how to package Theia as a Desktop Product may be found [here](h
   - `browser` contains a browser based version of Eclipse Theia Blueprint that may be packaged as a Docker image
   - `electron` contains the electron app to package, packaging configuration, and E2E tests for the electron target.
 - `theia-extensions` groups the various custom theia extensions for Blueprint
-  - `theia-blueprint-product` contains a Theia extension contributing the product branding (about dialogue and welcome page).
-  - `theia-blueprint-updater` contains a Theia extension contributing the update mechanism and corresponding UI elements (based on the electron updater).
+  - `product` contains a Theia extension contributing the product branding (about dialogue and welcome page).
+  - `updater` contains a Theia extension contributing the update mechanism and corresponding UI elements (based on the electron updater).
+  - `launcher` contains a Theia extension contributing, for AppImage applications, the option to create a script that allows to start blueprint from the command line by calling the 'theia' command.
 
 ### Build
 
+For development and casual testing of Blueprint, one can build it in "dev" mode. This permits building Blueprint on systems with less resources, like a Raspberry Pi 4B with 4GB of RAM.
+
 ```sh
-yarn
+# Build "dev" version of the Blueprint app. Its quicker, uses less resources, 
+# but the front end app is not "minified"
+yarn && yarn build:dev && yarn download:plugins
 ```
 
-### Package the Electron Application
+Production Blueprint applications:
 
 ```sh
+# Build production version of the Blueprint app
+yarn && yarn build && yarn download:plugins
+```
+
+### Package the Applications
+
+ATM we only produce packages for the Electron application.
+
+```sh
+yarn package:applications
+# or
 yarn electron package
 ```
 
 The packaged application is located in `applications/electron/dist`.
 
-### Create a Preview Electron Application (without packaging it)
+### Create a Preview Electron Electron Application (without packaging it)
 
 ```sh
 yarn electron package:preview
@@ -93,10 +109,6 @@ yarn electron test
 The browser app may be started with
 
 ```sh
-# Download Plugins for browser app
-yarn browser download:plugins
-
-# Start browser app
 yarn browser start
 ```