Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement sass --embedded in pure JS mode #2413

Open
wants to merge 23 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 0 additions & 9 deletions .github/workflows/build-linux.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,6 @@ jobs:
- image: docker.io/library/dart
platform: linux/amd64
target: linux-x64
- image: docker.io/library/dart
platform: linux/amd64
target: linux-ia32
- image: docker.io/library/dart
platform: linux/arm64
target: linux-arm64
Expand All @@ -36,9 +33,6 @@ jobs:
- image: ghcr.io/dart-musl/dart
platform: linux/amd64
target: linux-x64-musl
- image: ghcr.io/dart-musl/dart
platform: linux/amd64
target: linux-ia32-musl
- image: ghcr.io/dart-musl/dart
platform: linux/arm64
target: linux-arm64-musl
Expand All @@ -51,9 +45,6 @@ jobs:
- image: ghcr.io/dart-android/dart
platform: linux/amd64
target: android-x64
- image: ghcr.io/dart-android/dart
platform: linux/amd64
target: android-ia32
- image: ghcr.io/dart-android/dart
platform: linux/arm64
target: android-arm64
Expand Down
2 changes: 0 additions & 2 deletions .github/workflows/build-windows.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,6 @@ jobs:
include:
- arch: x64
runner: windows-latest
- arch: ia32
runner: windows-latest
- arch: arm64
runner: windows-arm64

Expand Down
36 changes: 23 additions & 13 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -128,24 +128,40 @@ jobs:
working-directory: sass-spec

sass_spec_js_embedded:
name: 'JS API Tests | Embedded | Node ${{ matrix.node-version }} | ${{ matrix.os }}'
name: "JS API Tests | Embedded ${{ matrix.js && 'Pure JS' || 'Dart' }} | Node ${{ matrix.node-version }} | ${{ matrix.os }}"
runs-on: ${{ matrix.os }}
if: "github.event_name != 'pull_request' || !contains(github.event.pull_request.body, 'skip sass-embedded')"

strategy:
fail-fast: false
matrix:
js: [true, false]
os: [ubuntu-latest, windows-latest, macos-latest]
node-version: ['lts/*']
include:
# Test older LTS versions
- os: ubuntu-latest
- js: true
os: ubuntu-latest
dart_channel: stable
node-version: lts/-1
- os: ubuntu-latest
- js: true
os: ubuntu-latest
dart_channel: stable
node-version: lts/-2
- os: ubuntu-latest
- js: true
os: ubuntu-latest
dart_channel: stable
node-version: lts/-3
- js: false
os: ubuntu-latest
dart_channel: stable
node-version: lts/-1
- js: false
os: ubuntu-latest
dart_channel: stable
node-version: lts/-2
- js: false
os: ubuntu-latest
dart_channel: stable
node-version: lts/-3

Expand All @@ -168,19 +184,13 @@ jobs:
- name: Initialize embedded host
run: |
npm install
npm run init -- --compiler-path=.. --language-path=../build/language
npm run init -- --compiler-path=.. --language-path=../build/language ${{ matrix.js && '--compiler-js' || '' }}
npm run compile
mv {`pwd`/,dist/}lib/src/vendor/dart-sass
working-directory: embedded-host-node

- name: Version info
run: |
path=embedded-host-node/dist/lib/src/vendor/dart-sass/sass
if [[ -f "$path.cmd" ]]; then "./$path.cmd" --version
elif [[ -f "$path.bat" ]]; then "./$path.bat" --version
elif [[ -f "$path.exe" ]]; then "./$path.exe" --version
else "./$path" --version
fi
run: node dist/bin/sass.js --version
working-directory: embedded-host-node

- name: Run tests
run: npm run js-api-spec -- --sassPackage ../embedded-host-node --sassSassRepo ../build/language
Expand Down
4 changes: 1 addition & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -434,9 +434,7 @@ an API for users to invoke Sass and define custom functions and importers.
* `sass --embedded --version` prints `versionResponse` with `id = 0` in JSON and
exits.

The `--embedded` command-line flag is not available when you install Dart Sass
as an [npm package]. No other command-line flags are supported with
`--embedded`.
No other command-line flags are supported with `--embedded`.

[npm package]: #from-npm

Expand Down
5 changes: 1 addition & 4 deletions bin/sass.dart
Original file line number Diff line number Diff line change
Expand Up @@ -17,10 +17,7 @@ import 'package:sass/src/importer/filesystem.dart';
import 'package:sass/src/io.dart';
import 'package:sass/src/stylesheet_graph.dart';
import 'package:sass/src/utils.dart';
import 'package:sass/src/embedded/executable.dart'
// Never load the embedded protocol when compiling to JS.
if (dart.library.js) 'package:sass/src/embedded/unavailable.dart'
as embedded;
import 'package:sass/src/embedded/executable.dart' as embedded;

Future<void> main(List<String> args) async {
if (args case ['--embedded', ...var rest]) {
Expand Down
69 changes: 61 additions & 8 deletions lib/src/embedded/README.md
Original file line number Diff line number Diff line change
@@ -1,28 +1,81 @@
# Embedded Sass Compiler

This directory contains the Dart Sass embedded compiler. This is a special mode
of the Dart Sass command-line executable, only supported on the Dart VM, in
which it uses stdin and stdout to communicate with another endpoint, the
"embedded host", using a protocol buffer-based protocol. See [the embedded
protocol specification] for details.
of the Dart Sass command-line executable, in which it uses stdin and stdout to
communicate with another endpoint, the "embedded host", using a protocol
buffer-based protocol. See [the embedded protocol specification] for details.

[the embedded protocol specification]: https://github.com/sass/sass/blob/main/spec/embedded-protocol.md

The embedded compiler has two different levels of dispatchers for handling
incoming messages from the embedded host:

1. The [`IsolateDispatcher`] is the first recipient of each packet. It decodes
1. The [`WorkerDispatcher`] is the first recipient of each packet. It decodes
the packets _just enough_ to determine which compilation they belong to, and
forwards them to the appropriate compilation dispatcher. It also parses and
handles messages that aren't compilation specific, such as `VersionRequest`.

[`IsolateDispatcher`]: isolate_dispatcher.dart
[`WorkerDispatcher`]: worker_dispatcher.dart

2. The [`CompilationDispatcher`] fully parses and handles messages for a single
compilation. Each `CompilationDispatcher` runs in a separate isolate so that
compilation. Each `CompilationDispatcher` runs in a separate worker so that
the embedded compiler can run multiple compilations in parallel.

[`CompilationDispatcher`]: compilation_dispatcher.dart

Otherwise, most of the code in this directory just wraps Dart APIs to
Otherwise, most of the code in this directory just wraps Dart APIs or JS APIs to
communicate with their protocol buffer equivalents.

## Worker Communication and Management

The way the Dart VM launches lightweight isolates is very different from how
Node.js launches worker threads. In the Dart VM, the lightweight isolates share
program structures like loaded libraries, classes, functions, and so on, even
including JIT optimized code. This allows main isolate to spawn child isolate
with a reference to the entry point function.

```
┌─────────────────┐ ┌─────────────────┐
│ Main Isolate │ Isolate.spawn(workerEntryPoint, mailbox, sendPort) │ Worker Isolate │
│ ├───────────────────────────────────────────────────►│ │
│ │ │ │
│ ┌─────────────┐ │ Synchronous Messaging │ ┌─────────────┐ │
│ │ Mailbox ├─┼────────────────────────────────────────────────────┼►│ Mailbox │ │
│ └─────────────┘ │ │ └─────────────┘ │
│ │ │ │
│ ┌─────────────┐ │ Asynchronous Messaging │ ┌─────────────┐ │
│ │ ReceivePort │◄┼────────────────────────────────────────────────────┼─┤ SendPort │ │
│ └─────────────┘ │ │ └─────────────┘ │
│ │ │ │
└─────────────────┘ └─────────────────┘
```

In Node.JS, the worker threads do not share program structures. In order to
launch a worker thread, it needs an entry point file, with the entry point
function effectly hard-coded in that file. While it's possible to have a
separate entry point file for the worker threads, it would require complex
packaging changes within `cli_pkg`, so instead the main thread and the worker
threads share [the same entry point file](js/executable.dart), which decides
what to run based on `worker_threads.isMainThread`.

```
if (worker_threads.isMainThread) { if (worker_threads.isMainThread) {
mainEntryPoint(); mainEntryPoint();
} else { } else {
workerEntryPoint(); new Worker(process.argv[1], { workerEntryPoint();
} argv: process.argv.slice(2), }
workerData: channel.port2,
┌────────────────────────────────────┐ transferList: [channel.port2] ┌────────────────────────────────────┐
│ Main Thread │ }) │ Worker Thread │
│ ├───────────────────────────────────────────────►│ │
│ │ │ │
│ ┌────────────────────────────────┐ │ Synchronous Messaging │ ┌────────────────────────────────┐ │
│ │ SyncMessagePort(channel.port1) ├─┼────────────────────────────────────────────────┼►│ SyncMessagePort(channel.port2) │ │
│ └────────────────────────────────┘ │ │ └────────────────────────────────┘ │
│ │ │ │
│ ┌────────────────────────────────┐ │ Asynchronous Messaging │ ┌────────────────────────────────┐ │
│ │ channel.port1 │◄┼────────────────────────────────────────────────┼─┤ channel.port2 │ │
│ └────────────────────────────────┘ │ │ └────────────────────────────────┘ │
│ │ │ │
└────────────────────────────────────┘ └────────────────────────────────────┘
```
43 changes: 23 additions & 20 deletions lib/src/embedded/compilation_dispatcher.dart
Original file line number Diff line number Diff line change
Expand Up @@ -3,17 +3,15 @@
// https://opensource.org/licenses/MIT.

import 'dart:convert';
import 'dart:io';
import 'dart:isolate';
import 'dart:typed_data';

import 'package:native_synchronization/mailbox.dart';
import 'package:path/path.dart' as p;
import 'package:protobuf/protobuf.dart';
import 'package:pub_semver/pub_semver.dart';
import 'package:sass/sass.dart' as sass;
import 'package:sass/src/importer/node_package.dart' as npi;

import '../io.dart';
import '../logger.dart';
import '../value/function.dart';
import '../value/mixin.dart';
Expand All @@ -23,6 +21,7 @@ import 'host_callable.dart';
import 'importer/file.dart';
import 'importer/host.dart';
import 'logger.dart';
import 'sync_receive_port.dart';
import 'util/proto_extensions.dart';
import 'utils.dart';

Expand All @@ -35,8 +34,8 @@ final _outboundRequestId = 0;
/// A class that dispatches messages to and from the host for a single
/// compilation.
final class CompilationDispatcher {
/// The mailbox for receiving messages from the host.
final Mailbox _mailbox;
/// The synchronous receive port for receiving messages from the host.
final SyncReceivePort _receivePort;

/// The send port for sending messages to the host.
final SendPort _sendPort;
Expand All @@ -52,8 +51,8 @@ final class CompilationDispatcher {
late Uint8List _compilationIdVarint;

/// Creates a [CompilationDispatcher] that receives encoded protocol buffers
/// through [_mailbox] and sends them through [_sendPort].
CompilationDispatcher(this._mailbox, this._sendPort);
/// through [_receivePort] and sends them through [_sendPort].
CompilationDispatcher(this._receivePort, this._sendPort);

/// Listens for incoming `CompileRequests` and runs their compilations.
void listen() {
Expand Down Expand Up @@ -305,7 +304,7 @@ final class CompilationDispatcher {
///
/// This is used during compilation by other classes like host callable.
Never sendError(ProtocolError error) {
Isolate.exit(_sendPort, _serializePacket(OutboundMessage()..error = error));
exitWorker(_sendPort, _serializePacket(OutboundMessage()..error = error));
}

InboundMessage_CanonicalizeResponse sendCanonicalizeRequest(
Expand Down Expand Up @@ -407,31 +406,35 @@ final class CompilationDispatcher {
var protobufWriter = CodedBufferWriter();
message.writeToCodedBufferWriter(protobufWriter);

// Add one additional byte to the beginning to indicate whether or not the
// compilation has finished (1) or encountered a fatal error (2), so the
// [IsolateDispatcher] knows whether to treat this isolate as inactive or
// close out entirely.
// Add two bytes to the beginning.
//
// The first byte indicates whether or not the compilation has finished (1)
// or encountered a fatal error (2), so the [WorkerDispatcher] knows
// whether to treat this isolate as inactive or close out entirely.
//
// The second byte is the exitCode when a fatal error occurs.
var packet = Uint8List(
1 + _compilationIdVarint.length + protobufWriter.lengthInBytes,
2 + _compilationIdVarint.length + protobufWriter.lengthInBytes,
);
packet[0] = switch (message.whichMessage()) {
OutboundMessage_Message.compileResponse => 1,
OutboundMessage_Message.error => 2,
_ => 0,
OutboundMessage_Message.compileResponse => 1,
_ => 0
};
packet.setAll(1, _compilationIdVarint);
protobufWriter.writeTo(packet, 1 + _compilationIdVarint.length);
packet[1] = exitCode;
packet.setAll(2, _compilationIdVarint);
protobufWriter.writeTo(packet, 2 + _compilationIdVarint.length);
return packet;
}

/// Receive a packet from the host.
Uint8List _receive() {
try {
return _mailbox.take();
return _receivePort.receive();
} on StateError catch (_) {
// The [_mailbox] has been closed, exit the current isolate immediately
// The [SyncReceivePort] has been closed, exit the current isolate immediately
// to avoid bubble the error up as [SassException] during [_sendRequest].
Isolate.exit();
exitWorker();
}
}
}
41 changes: 2 additions & 39 deletions lib/src/embedded/executable.dart
Original file line number Diff line number Diff line change
@@ -1,42 +1,5 @@
// Copyright 2019 Google Inc. Use of this source code is governed by an
// Copyright 2025 Google Inc. Use of this source code is governed by an
// MIT-style license that can be found in the LICENSE file or at
// https://opensource.org/licenses/MIT.

import 'dart:io';
import 'dart:convert';

import 'package:stream_channel/stream_channel.dart';

import 'isolate_dispatcher.dart';
import 'util/length_delimited_transformer.dart';

void main(List<String> args) {
switch (args) {
case ["--version", ...]:
var response = IsolateDispatcher.versionResponse();
response.id = 0;
stdout.writeln(
JsonEncoder.withIndent(" ").convert(response.toProto3Json()),
);
return;

case [_, ...]:
stderr.writeln(
"sass --embedded is not intended to be executed with additional "
"arguments.\n"
"See https://github.com/sass/dart-sass#embedded-dart-sass for "
"details.",
);
// USAGE error from https://bit.ly/2poTt90
exitCode = 64;
return;
}

IsolateDispatcher(
StreamChannel.withGuarantees(
stdin,
stdout,
allowSinkErrors: false,
).transform(lengthDelimited),
).listen();
}
export 'vm/executable.dart' if (dart.library.js) 'js/executable.dart';
10 changes: 10 additions & 0 deletions lib/src/embedded/js/concurrency.dart
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
// Copyright 2025 Google Inc. Use of this source code is governed by an
// MIT-style license that can be found in the LICENSE file or at
// https://opensource.org/licenses/MIT.

import 'dart:js_interop';

@JS('os.cpus')
external JSArray _cpus();

int get concurrencyLimit => _cpus().length;
Loading