Sourcemap improvements (#23741)

Provide source map settings to emcc (resolves #23686, resolves #22189) :
- for embedding the sources content in the source map :
`-gsource-map=inline`.
- for applying path prefix substitution :
`-sSOURCE_MAP_PREFIXES=["<old>=<new>"]`.

Update documentation accordingly. 

Fix source file resolver : 
- Always fall back to the given filepath if prefix not provided or
doesn't match.
- Fix source content loading when no `--load-prefix` is given/needed.
- Fix relative path in "sources" field when`--prefix` is given but
doesn't match the given file.
- Cache filepaths with no/unmatched prefix as well.
- Resolve deterministic prefix when loading source content (related:
#20779).
- Don't emit relative paths for sources with a deterministic prefix.

Improve existing test for wasm-sourcemap.py :
- Parameterize `test_wasm_sourcemap()` and do proper checks according to
the combinations of options given.
- Fix regex for checking the "mappings" field (was checking only the
first character).

Add test for emcc covering the basic use cases where :
- no option is given (users do path substitution as needed via their
client or server configuration).
- different prefixes are provided for source files and emscripten
dependencies.
- source content is embedded in the sourcemap.

Author: lvlte

ChangeLog.md

@@ -20,6 +20,10 @@ See docs/process.md for more on how version tagging works.
 
 4.0.6 (in development)
 ----------------------
+- Added support for applying path prefix substitution to the sources of the
+  source map : use `-sSOURCE_MAP_PREFIXES=["<old>=<new>"]` with `-gsource-map`.
+  Alternatively, you can now embed the sources content into the source map file
+  using `-gsource-map=inline`. (#23741)
 
 4.0.5 - 03/12/25
 ----------------

docs/emcc.txt

@@ -181,15 +181,23 @@ Options that are modified or new in *emcc* are listed below:
    alongside the wasm object files. This option must be used together
    with "-c".
 
-"-gsource-map"
+"-gsource-map[=inline]"
    [link] Generate a source map using LLVM debug information (which
    must be present in object files, i.e., they should have been
-   compiled with "-g"). When this option is provided, the **.wasm**
-   file is updated to have a "sourceMappingURL" section. The resulting
-   URL will have format: "<base-url>" + "<wasm-file-name>" + ".map".
-   "<base-url>" defaults to being empty (which means the source map is
-   served from the same directory as the Wasm file). It can be changed
-   using --source-map-base.
+   compiled with "-g").
+
+   When this option is provided, the **.wasm** file is updated to have
+   a "sourceMappingURL" section. The resulting URL will have format:
+   "<base-url>" + "<wasm-file-name>" + ".map". "<base-url>" defaults
+   to being empty (which means the source map is served from the same
+   directory as the Wasm file). It can be changed using --source-map-
+   base.
+
+   Path substitution can be applied to the referenced sources using
+   the "-sSOURCE_MAP_PREFIXES" (link). If "inline" is specified, the
+   sources content is embedded in the source map (in this case you
+   don't need path substitution, but it comes with the cost of having
+   a large source map file).
 
 "-g<level>"
    [compile+link] Controls the level of debuggability. Each level

emcc.py

@@ -1196,8 +1196,8 @@ def consume_arg_file():
           else:
             settings.SEPARATE_DWARF = True
           settings.GENERATE_DWARF = 1
-        elif requested_level == 'source-map':
-          settings.GENERATE_SOURCE_MAP = 1
+        elif requested_level in ['source-map', 'source-map=inline']:
+          settings.GENERATE_SOURCE_MAP = 1 if requested_level == 'source-map' else 2
           settings.EMIT_NAME_SECTION = 1
           newargs[i] = '-g'
         else:

site/source/docs/tools_reference/emcc.rst

@@ -172,16 +172,23 @@ Options that are modified or new in *emcc* are listed below:
 
 .. _emcc-gsource-map:
 
-``-gsource-map``
+``-gsource-map[=inline]``
   [link]
   Generate a source map using LLVM debug information (which must
   be present in object files, i.e., they should have been compiled with ``-g``).
+
   When this option is provided, the **.wasm** file is updated to have a
   ``sourceMappingURL`` section. The resulting URL will have format:
   ``<base-url>`` + ``<wasm-file-name>`` + ``.map``. ``<base-url>`` defaults
   to being empty (which means the source map is served from the same directory
   as the Wasm file). It can be changed using :ref:`--source-map-base <emcc-source-map-base>`.
 
+  Path substitution can be applied to the referenced sources using the
+  ``-sSOURCE_MAP_PREFIXES`` (:ref:`link <source_map_prefixes>`).
+  If ``inline`` is specified, the sources content is embedded in the source map
+  (in this case you don't need path substitution, but it comes with the cost of
+  having a large source map file).
+
 .. _emcc-gN:
 
 ``-g<level>``

site/source/docs/tools_reference/settings_reference.rst

@@ -3131,6 +3131,21 @@ This is enabled automatically when using -gsource-map with sanitizers.
 
 Default value: false
 
+.. _source_map_prefixes:
+
+SOURCE_MAP_PREFIXES
+===================
+
+List of path substitutions to apply in the "sources" field of the source map.
+Corresponds to the ``--prefix`` option used in ``tools/wasm-sourcemap.py``.
+Must be used with ``-gsource-map``.
+
+This setting allows to map path prefixes to the proper ones so that the final
+(possibly relative) URLs point to the correct locations :
+``-sSOURCE_MAP_PREFIXES=/old/path=/new/path``
+
+Default value: []
+
 .. _default_to_cxx:
 
 DEFAULT_TO_CXX

src/settings.js

@@ -2046,6 +2046,17 @@ var USE_OFFSET_CONVERTER = false;
 // This is enabled automatically when using -gsource-map with sanitizers.
 var LOAD_SOURCE_MAP = false;
 
+// List of path substitutions to apply in the "sources" field of the source map.
+// Corresponds to the ``--prefix`` option used in ``tools/wasm-sourcemap.py``.
+// Must be used with ``-gsource-map``.
+//
+// This setting allows to map path prefixes to the proper ones so that the final
+// (possibly relative) URLs point to the correct locations :
+// ``-sSOURCE_MAP_PREFIXES=/old/path=/new/path``
+//
+// [link]
+var SOURCE_MAP_PREFIXES = [];
+
 // Default to c++ mode even when run as ``emcc`` rather then ``emc++``.
 // When this is disabled ``em++`` is required linking C++ programs. Disabling
 // this will match the behaviour of gcc/g++ and clang/clang++.

src/settings_internal.js

@@ -204,7 +204,10 @@ var USE_READY_PROMISE = true;
 // If true, building against Emscripten's wasm heap memory profiler.
 var MEMORYPROFILER = false;
 
-var GENERATE_SOURCE_MAP = false;
+// Set automatically to :
+// - 1 when using `-gsource-map`
+// - 2 when using `gsource-map=inline` (embed sources content in souce map)
+var GENERATE_SOURCE_MAP = 0;
 
 var GENERATE_DWARF = false;
 

test/test_other.py

@@ -50,6 +50,7 @@
 import line_endings
 from tools import webassembly
 from tools.settings import settings
+from tools.system_libs import DETERMINISTIC_PREFIX
 
 scons_path = shutil.which('scons')
 emmake = shared.bat_suffix(path_from_root('emmake'))
@@ -10432,25 +10433,84 @@ def test_check_sourcemapurl_default(self, *args):
     source_mapping_url_content = webassembly.to_leb(len('sourceMappingURL')) + b'sourceMappingURL' + webassembly.to_leb(len('a.wasm.map')) + b'a.wasm.map'
     self.assertIn(source_mapping_url_content, output)
 
-  def test_wasm_sourcemap(self):
-    # The no_main.c will be read (from relative location) due to speficied "-s"
+  @parameterized({
+    '': ([], [], []),
+    'prefix_wildcard': ([], ['--prefix', '=wasm-src://'], []),
+    'prefix_partial': ([], ['--prefix', '/emscripten/=wasm-src:///emscripten/'], []),
+    'sources': (['--sources'], [], ['--load-prefix', '/emscripten/test/other/wasm_sourcemap=.'])
+  })
+  @parameterized({
+    '': ('/',),
+    'basepath': ('/emscripten/test',)
+  })
+  def test_wasm_sourcemap(self, sources, prefix, load_prefix, basepath):
+    # The no_main.c will be read from relative location if necessary (depends
+    # on --sources and --load-prefix options).
     shutil.copy(test_file('other/wasm_sourcemap/no_main.c'), '.')
+    DW_AT_decl_file = '/emscripten/test/other/wasm_sourcemap/no_main.c'
     wasm_map_cmd = [PYTHON, path_from_root('tools/wasm-sourcemap.py'),
-                    '--sources', '--prefix', '=wasm-src://',
-                    '--load-prefix', '/emscripten/test/other/wasm_sourcemap=.',
+                    *sources, *prefix, *load_prefix,
                     '--dwarfdump-output',
                     test_file('other/wasm_sourcemap/foo.wasm.dump'),
                     '-o', 'a.out.wasm.map',
                     test_file('other/wasm_sourcemap/foo.wasm'),
-                    '--basepath=' + os.getcwd()]
+                    '--basepath=' + basepath]
     self.run_process(wasm_map_cmd)
     output = read_file('a.out.wasm.map')
-    # has "sources" entry with file (includes also `--prefix =wasm-src:///` replacement)
-    self.assertIn('wasm-src:///emscripten/test/other/wasm_sourcemap/no_main.c', output)
-    # has "sourcesContent" entry with source code (included with `-s` option)
-    self.assertIn('int foo()', output)
-    # has some entries
-    self.assertRegex(output, r'"mappings":\s*"[A-Za-z0-9+/]')
+    # "sourcesContent" contains source code iff --sources is specified.
+    self.assertIn('int foo()' if sources else '"sourcesContent":[]', output)
+    if prefix: # "sources" contains URL with prefix path substition if provided
+      sources_url = 'wasm-src:///emscripten/test/other/wasm_sourcemap/no_main.c'
+    else: # otherwise a path relative to the given basepath.
+      sources_url = utils.normalize_path(os.path.relpath(DW_AT_decl_file, basepath))
+    self.assertIn(sources_url, output)
+    # "mappings" contains valid Base64 VLQ segments.
+    self.assertRegex(output, r'"mappings":\s*"(?:[A-Za-z0-9+\/]+[,;]?)+"')
+
+  @parameterized({
+    '': ([], 0),
+    'prefix': ([
+      '<cwd>=file:///path/to/src',
+      DETERMINISTIC_PREFIX + '=file:///path/to/emscripten',
+    ], 0),
+    'sources': ([], 1)
+  })
+  def test_emcc_sourcemap_options(self, prefixes, sources):
+    wasm_sourcemap = importlib.import_module('tools.wasm-sourcemap')
+    cwd = os.getcwd()
+    src_file = shutil.copy(test_file('hello_123.c'), cwd)
+    lib_file = DETERMINISTIC_PREFIX + '/system/lib/libc/musl/src/stdio/fflush.c'
+    if prefixes:
+      prefixes = [p.replace('<cwd>', cwd) for p in prefixes]
+    self.set_setting('SOURCE_MAP_PREFIXES', prefixes)
+    args = ['-gsource-map=inline' if sources else '-gsource-map']
+    self.emcc(src_file, args=args, output_filename='test.js')
+    output = read_file('test.wasm.map')
+    # Check source file resolution
+    p = wasm_sourcemap.Prefixes(prefixes, base_path=cwd)
+    self.assertEqual(len(p.prefixes), len(prefixes))
+    src_file_url = p.resolve(utils.normalize_path(src_file))
+    lib_file_url = p.resolve(utils.normalize_path(lib_file))
+    if prefixes:
+      self.assertEqual(src_file_url, 'file:///path/to/src/hello_123.c')
+      self.assertEqual(lib_file_url, 'file:///path/to/emscripten/system/lib/libc/musl/src/stdio/fflush.c')
+    else:
+      self.assertEqual(src_file_url, 'hello_123.c')
+      self.assertEqual(lib_file_url, '/emsdk/emscripten/system/lib/libc/musl/src/stdio/fflush.c')
+    # "sources" contains resolved filepath.
+    self.assertIn(f'"{src_file_url}"', output)
+    self.assertIn(f'"{lib_file_url}"', output)
+    # "sourcesContent" contains source code iff -gsource-map=inline is specified.
+    if sources:
+      p = wasm_sourcemap.Prefixes(prefixes, preserve_deterministic_prefix=False)
+      for filepath in [src_file, lib_file]:
+        resolved_path = p.resolve(utils.normalize_path(filepath))
+        sources_content = json.dumps(read_file(resolved_path))
+        self.assertIn(sources_content, output)
+    else:
+      self.assertIn('"sourcesContent":[]', output)
+    # "mappings" contains valid Base64 VLQ segments.
+    self.assertRegex(output, r'"mappings":\s*"(?:[A-Za-z0-9+\/]+[,;]?)+"')
 
   def test_wasm_sourcemap_dead(self):
     wasm_map_cmd = [PYTHON, path_from_root('tools/wasm-sourcemap.py'),

tools/building.py

@@ -1137,6 +1137,13 @@ def emit_wasm_source_map(wasm_file, map_file, final_wasm):
                    '--dwarfdump=' + LLVM_DWARFDUMP,
                    '-o',  map_file,
                    '--basepath=' + base_path]
+
+  if settings.SOURCE_MAP_PREFIXES:
+    sourcemap_cmd += ['--prefix', *settings.SOURCE_MAP_PREFIXES]
+
+  if settings.GENERATE_SOURCE_MAP == 2:
+    sourcemap_cmd += ['--sources']
+
   check_call(sourcemap_cmd)
 
 

tools/wasm-sourcemap.py

@@ -25,6 +25,10 @@
 sys.path.insert(0, __rootdir__)
 
 from tools import utils
+from tools.system_libs import DETERMINISTIC_PREFIX
+from tools.shared import path_from_root
+
+EMSCRIPTEN_PREFIX = utils.normalize_path(path_from_root())
 
 logger = logging.getLogger('wasm-sourcemap')
 
@@ -46,42 +50,57 @@ def parse_args():
 
 
 class Prefixes:
-  def __init__(self, args):
+  def __init__(self, args, base_path=None, preserve_deterministic_prefix=True):
     prefixes = []
     for p in args:
       if '=' in p:
         prefix, replacement = p.split('=')
         prefixes.append({'prefix': prefix, 'replacement': replacement})
       else:
-        prefixes.append({'prefix': p, 'replacement': None})
+        prefixes.append({'prefix': p, 'replacement': ''})
+    self.base_path = base_path
+    self.preserve_deterministic_prefix = preserve_deterministic_prefix
     self.prefixes = prefixes
     self.cache = {}
 
   def resolve(self, name):
     if name in self.cache:
       return self.cache[name]
 
+    source = name
+    if not self.preserve_deterministic_prefix and name.startswith(DETERMINISTIC_PREFIX):
+      source = EMSCRIPTEN_PREFIX + utils.removeprefix(name, DETERMINISTIC_PREFIX)
+
+    provided = False
     for p in self.prefixes:
-      if name.startswith(p['prefix']):
-        if p['replacement'] is None:
-          result = utils.removeprefix(name, p['prefix'])
-        else:
-          result = p['replacement'] + utils.removeprefix(name, p['prefix'])
+      if source.startswith(p['prefix']):
+        source = p['replacement'] + utils.removeprefix(source, p['prefix'])
+        provided = True
         break
-    self.cache[name] = result
-    return result
+
+    # If prefixes were provided, we use that; otherwise if base_path is set, we
+    # emit a relative path. For files with deterministic prefix, we never use
+    # a relative path, precisely to preserve determinism, and because it would
+    # still point to the wrong location, so we leave the filepath untouched to
+    # let users map it to the proper location using prefix options.
+    if not (source.startswith(DETERMINISTIC_PREFIX) or provided or self.base_path is None):
+      try:
+        source = os.path.relpath(source, self.base_path)
+      except ValueError:
+        source = os.path.abspath(source)
+      source = utils.normalize_path(source)
+
+    self.cache[name] = source
+    return source
 
 
 # SourceMapPrefixes contains resolver for file names that are:
 #  - "sources" is for names that output to source maps JSON
 #  - "load" is for paths that used to load source text
 class SourceMapPrefixes:
-  def __init__(self, sources, load):
-    self.sources = sources
-    self.load = load
-
-  def provided(self):
-    return bool(self.sources.prefixes or self.load.prefixes)
+  def __init__(self, sources, load, base_path):
+    self.sources = Prefixes(sources, base_path=base_path)
+    self.load = Prefixes(load, preserve_deterministic_prefix=False)
 
 
 def encode_vlq(n):
@@ -259,15 +278,20 @@ def read_dwarf_entries(wasm, options):
   return sorted(entries, key=lambda entry: entry['address'])
 
 
-def build_sourcemap(entries, code_section_offset, prefixes, collect_sources, base_path):
+def build_sourcemap(entries, code_section_offset, options):
+  base_path = options.basepath
+  collect_sources = options.sources
+  prefixes = SourceMapPrefixes(options.prefix, options.load_prefix, base_path)
+
   sources = []
-  sources_content = [] if collect_sources else None
+  sources_content = []
   mappings = []
   sources_map = {}
   last_address = 0
   last_source_id = 0
   last_line = 1
   last_column = 1
+
   for entry in entries:
     line = entry['line']
     column = entry['column']
@@ -277,20 +301,11 @@ def build_sourcemap(entries, code_section_offset, prefixes, collect_sources, bas
     # start at least at column 1
     if column == 0:
       column = 1
+
     address = entry['address'] + code_section_offset
-    file_name = entry['file']
-    file_name = utils.normalize_path(file_name)
-    # if prefixes were provided, we use that; otherwise, we emit a relative
-    # path
-    if prefixes.provided():
-      source_name = prefixes.sources.resolve(file_name)
-    else:
-      try:
-        file_name = os.path.relpath(file_name, base_path)
-      except ValueError:
-        file_name = os.path.abspath(file_name)
-      file_name = utils.normalize_path(file_name)
-      source_name = file_name
+    file_name = utils.normalize_path(entry['file'])
+    source_name = prefixes.sources.resolve(file_name)
+
     if source_name not in sources_map:
       source_id = len(sources)
       sources_map[source_name] = source_id
@@ -316,6 +331,7 @@ def build_sourcemap(entries, code_section_offset, prefixes, collect_sources, bas
     last_source_id = source_id
     last_line = line
     last_column = column
+
   return {'version': 3,
           'sources': sources,
           'sourcesContent': sources_content,
@@ -334,10 +350,8 @@ def main():
 
   code_section_offset = get_code_section_offset(wasm)
 
-  prefixes = SourceMapPrefixes(sources=Prefixes(options.prefix), load=Prefixes(options.load_prefix))
-
   logger.debug('Saving to %s' % options.output)
-  map = build_sourcemap(entries, code_section_offset, prefixes, options.sources, options.basepath)
+  map = build_sourcemap(entries, code_section_offset, options)
   with open(options.output, 'w') as outfile:
     json.dump(map, outfile, separators=(',', ':'))