Skip to content

Commit ec064e3

Browse files
committed
JavaScript: make bytes literals use Uint8Array (not number[])
Until now, byte array literals translated for JavaScript were of type `number[]`, as TypeScript would call this type (in pure JS terminology, it's an `Array` object with numbers). For example, a value instance defined as `instances/v/value: '[65, 126]'` would be translated as `this._m_v = [65, 126];` in the generated JS code. However, byte arrays created in any other way would use the [`Uint8Array`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Uint8Array) type, which is the standard type for byte arrays in JavaScript. Namely any byte array parsed from stream would be a `Uint8Array` object and also a *non-literal* byte array like `[123, 246 + 0].as<bytes>` would be generated as `new Uint8Array([123, 246 + 0])`. I believe that using a different type for byte array literals than for byte arrays created in any other way was never intentional. It seems we simply forgot to override `doByteArrayLiteral` in `JavaScriptTranslator` (but `doByteArrayNonLiteral` is overridden correctly, which is weird), so we ended up using the default `BaseTranslator.doByteArrayLiteral` implementation, which produces the `[65, 126]` syntax. JavaScript is dynamically typed, so you can treat `number[]` the same as `Uint8Array` in most cases (e.g. indexing will work the same), but this discrepancy now causes problems when porting our JavaScript runtime library to TypeScript (see kaitai-io/kaitai_struct_javascript_runtime#25), because you would have to annotate a lot of parameter types as `number[] | Uint8Array` (which doesn't really make sense). Moreover, I discovered that generating byte array literals as `number[]` creates a bug in this case: ```ksy meta: id: bytes_to_s instances: literal: value: '[65, 126].to_s("UTF-8")' ``` Before this commit, `[65, 126].to_s("UTF-8")` would be translated as `KaitaiStream.bytesToStr([65, 126], "UTF-8")` for JavaScript. But our `bytesToStr` implementation won't actually accept `number[]` for non-ASCII encodings. See [`KaitaiStream.js:650-657`](https://github.com/kaitai-io/kaitai_struct_javascript_runtime/blob/2acb0d8b09fde9c95fc77ee3019f6d6b1c73f040/KaitaiStream.js#L650-L657): ```js KaitaiStream.bytesToStr = function(arr, encoding) { if (encoding == null || encoding.toLowerCase() === "ascii") { return KaitaiStream.createStringFromArray(arr); } else { if (typeof TextDecoder === 'function') { // we're in the browser that supports TextDecoder return (new TextDecoder(encoding)).decode(arr); } else { ``` If `TextDecoder` is available (which is the case in a browser or since Node.js 11), the error when trying to access the `literal` instance above will look like this in the browser (Google Chrome 129): ``` TypeError: Failed to execute 'decode' on 'TextDecoder': parameter 1 is not of type 'ArrayBuffer'. at KaitaiStream.bytesToStr (https://ide.kaitai.io/devel/lib/_npm/kaitai-struct/KaitaiStream.js:656:42) at BytesToS.get (eval at initCode (https://ide.kaitai.io/devel/js/v1/kaitaiWorker.js:156:9), <anonymous>:27:38) at fetchInstance (https://ide.kaitai.io/devel/js/v1/kaitaiWorker.js:123:20) ... ``` ... and like this in Node.js 12: ``` TypeError [ERR_INVALID_ARG_TYPE]: The "input" argument must be an instance of ArrayBuffer or ArrayBufferView. Received an instance of Array at TextDecoder.decode (internal/encoding.js:412:15) at Function.KaitaiStream.bytesToStr (/runtime/javascript/KaitaiStream.js:656:42) at BytesToS.get (compiled/javascript/BytesToS.js:26:38) at /tests/spec/javascript/test_bytes_to_s.js:8:24 ... ``` So using `Uint8Array` for all byte arrays will not only be more consistent, but will also fix this bug.
1 parent 8d913de commit ec064e3

File tree

2 files changed

+6
-4
lines changed

2 files changed

+6
-4
lines changed

jvm/src/test/scala/io/kaitai/struct/translators/TranslatorSpec.scala

+4-4
Original file line numberDiff line numberDiff line change
@@ -406,7 +406,7 @@ class TranslatorSpec extends AnyFunSpec {
406406
CSharpCompiler -> "new byte[] { 34, 0, 10, 64, 65, 66, 92 }",
407407
GoCompiler -> "[]uint8{34, 0, 10, 64, 65, 66, 92}",
408408
JavaCompiler -> "new byte[] { 34, 0, 10, 64, 65, 66, 92 }",
409-
JavaScriptCompiler -> "[34, 0, 10, 64, 65, 66, 92]",
409+
JavaScriptCompiler -> "new Uint8Array([34, 0, 10, 64, 65, 66, 92])",
410410
LuaCompiler -> "\"\\034\\000\\010\\064\\065\\066\\092\"",
411411
PerlCompiler -> "pack('C*', (34, 0, 10, 64, 65, 66, 92))",
412412
PHPCompiler -> "\"\\x22\\x00\\x0A\\x40\\x41\\x42\\x5C\"",
@@ -419,7 +419,7 @@ class TranslatorSpec extends AnyFunSpec {
419419
CSharpCompiler -> "new byte[] { 255, 0, 255 }",
420420
GoCompiler -> "[]uint8{255, 0, 255}",
421421
JavaCompiler -> "new byte[] { -1, 0, -1 }",
422-
JavaScriptCompiler -> "[255, 0, 255]",
422+
JavaScriptCompiler -> "new Uint8Array([255, 0, 255])",
423423
LuaCompiler -> "\"\\255\\000\\255\"",
424424
PerlCompiler -> "pack('C*', (255, 0, 255))",
425425
PHPCompiler -> "\"\\xFF\\x00\\xFF\"",
@@ -434,7 +434,7 @@ class TranslatorSpec extends AnyFunSpec {
434434
CSharpCompiler -> "new byte[] { 0, 1, 2 }.Length",
435435
GoCompiler -> "len([]uint8{0, 1, 2})",
436436
JavaCompiler -> "new byte[] { 0, 1, 2 }.length",
437-
JavaScriptCompiler -> "[0, 1, 2].length",
437+
JavaScriptCompiler -> "new Uint8Array([0, 1, 2]).length",
438438
LuaCompiler -> "#\"\\000\\001\\002\"",
439439
PerlCompiler -> "length(pack('C*', (0, 1, 2)))",
440440
PHPCompiler -> "strlen(\"\\x00\\x01\\x02\")",
@@ -816,7 +816,7 @@ class TranslatorSpec extends AnyFunSpec {
816816
CSharpCompiler -> "new byte[] { }",
817817
GoCompiler -> "[]uint8{}",
818818
JavaCompiler -> "new byte[] { }",
819-
JavaScriptCompiler -> "[]",
819+
JavaScriptCompiler -> "new Uint8Array([])",
820820
LuaCompiler -> "\"\"",
821821
PerlCompiler -> "pack('C*', ())",
822822
PHPCompiler -> "\"\"",

shared/src/main/scala/io/kaitai/struct/translators/JavaScriptTranslator.scala

+2
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,8 @@ import io.kaitai.struct.format.{EnumSpec, Identifier}
88
import io.kaitai.struct.languages.JavaScriptCompiler
99

1010
class JavaScriptTranslator(provider: TypeProvider, importList: ImportList) extends BaseTranslator(provider) {
11+
override def doByteArrayLiteral(arr: Seq[Byte]): String =
12+
s"new Uint8Array([${arr.map(_ & 0xff).mkString(", ")}])"
1113
override def doByteArrayNonLiteral(elts: Seq[Ast.expr]): String =
1214
s"new Uint8Array([${elts.map(translate).mkString(", ")}])"
1315

0 commit comments

Comments
 (0)