Create content loaders

CHANGELOG.md

@@ -2,6 +2,7 @@
 
 ## Unreleased
 
+* add content loaders to the configuration file to process more than Lua or Luau files with darklua ([#354](https://github.com/seaofvoices/darklua/pull/354))
 * improve file watching to handle rename events and remove empty folders after processing if they weren't present before the initial run ([#351](https://github.com/seaofvoices/darklua/pull/351))
 * add a new parameter for the `rename_variables` rule so that globals can be detected automatically and then avoided in the renaming pass ([#348](https://github.com/seaofvoices/darklua/pull/348))
 * add support for `const` declaration of variables and functions (e.g. `const var = true` or `const function test() end`) and add rule (`make_assignment_local`) to convert those assignments to `local` assignments ([#346](https://github.com/seaofvoices/darklua/pull/346))

Cargo.toml

@@ -27,6 +27,7 @@ tracing = ["dep:tracing"]
 
 [dependencies]
 anstyle = "1.0.14"
+base64 = "0.22.1"
 bstr = "1.12.1"
 clap = { version = "4.6.1", features = ["derive"] }
 durationfmt = "0.1.1"

site/content/docs/config/index.md

@@ -24,6 +24,24 @@ From the directory where you run `darklua process`, darklua will attempt to read
 
 To provide a different configuration file, this subcommand also accept a specific path to a configuration file with `--config <path>`.
 
+## Content Loaders
+
+darklua chooses how to process each file based on its extension: Lua and Luau files are parsed, recognized data files (such as JSON, TOML, and YAML) are converted to Lua modules, and other files are left alone. With content loaders, you can override these defaults or tell darklua what to do with other files: copy them to the output, turn their content into a Lua module, or skip them.
+
+Loaders are configured with the `loaders` field, which maps glob patterns to loader names. For example, to copy Rojo model files, ignore `.yml` files and turn Markdown files into string modules:
+
+```json5
+{
+  loaders: {
+    "**/*.model.json": "copy",
+    "**/*.yml": "skip",
+    "**/*.md": "string",
+  },
+}
+```
+
+See the [content loaders](/docs/content-loaders) page for the full list of loaders and how each one works.
+
 ## Filtering
 
 It is possible to limit which files are processed using `apply_to_files` and `skip_files`. Both accept glob patterns [from this implementation](https://github.com/olson-sean-k/wax/blob/master/README.md#patterns) (same syntax as `bundle.excludes`). Each field can be a single pattern string or an array of patterns.
@@ -47,6 +65,15 @@ Any missing field will be replaced with its default value.
   // Exclude applying rules to some files
   skip_files: ["**/*.test.lua"],
 
+  // Tell darklua how to process files
+  loaders: {
+    "**/*.model.json": "copy",
+    "**/*.md": "string",
+    "**/*.png": "buffer/base64",
+  },
+  // Some content loaders create Lua modules which will use this extension
+  lua_extension: "lua", // or "luau"
+
   bundle: {
     // Identifier used by darklua to store the bundled modules
     modules_identifier: "__DARKLUA_BUNDLE_MODULES",

site/content/docs/content-loaders/index.md

@@ -0,0 +1,133 @@
+---
+title: Content Loaders
+description: How darklua processes files
+group: Configuration
+order: 7
+---
+
+darklua picks a loader for each file based on its extension: Lua and Luau files are parsed, recognized data files are converted to Lua modules, and everything else is left alone. The `loaders` field lets you override these defaults or assign a loader to any extension darklua does not recognize. It maps [glob patterns](https://github.com/olson-sean-k/wax/blob/master/README.md#patterns) to loader names. Patterns are matched in order, and the first one that matches a file wins. Files that match no pattern fall back to the extension-based defaults (see [Reference](#reference)), and files with no recognized loader are skipped.
+
+```json5
+{
+  loaders: {
+    "**/*.model.json": "copy",
+    "**/*.md": "string",
+    "**/*.png": "buffer/base64",
+  },
+}
+```
+
+## Copying files
+
+The `copy` loader writes files to the output directory unchanged. This is useful for assets that need to end up alongside your processed code, such as Rojo model files:
+
+```json5
+{
+  loaders: {
+    "**/*.model.json": "copy",
+  },
+}
+```
+
+Previously, the usual workaround was to copy these files manually (for example with `cp`) and then run darklua in-place. The `copy` loader removes that extra step.
+
+## Processing data files
+
+darklua can convert structured data into a Lua module that returns the data as a table. JSON, TOML, and YAML are supported:
+
+```json5
+{
+  loaders: {
+    "**/*.json": "json",
+    "**/*.toml": "toml",
+    "**/*.yaml": "yaml",
+  },
+}
+```
+
+A file like `config.json` becomes `config.lua` (or `config.luau`) that returns the converted table. This already worked when bundling; loaders make it available in the regular `process` flow too.
+
+The `json` loader accepts JSON5 syntax, so files may include comments, trailing commas, and unquoted keys. The `yaml` loader also accepts `yml` as an alias, so `"**/*.yml": "yaml"` and `"**/*.yml": "yml"` are equivalent.
+
+## Embedding file content
+
+These loaders read a file and create a Lua module that returns its content. They work with any file, including binary files.
+
+- `string`: returns the content as a string.
+- `buffer`: returns the content as a buffer (`buffer.fromstring(...)`).
+- `bytes`: returns the content as an array of byte values.
+
+```json5
+{
+  loaders: {
+    "**/*.md": "string",
+    "**/*.bin": "buffer",
+  },
+}
+```
+
+Each of these also has a `/base64` variant that encodes the content as base64 before embedding it: `string/base64`, `buffer/base64`, and `bytes/base64`. This keeps binary data safe inside the generated Lua source. Note that base64 content is **not** decoded at runtime, so your code receives the encoded value.
+
+Unlike the data loaders (`json`, `toml`, `yaml`) and `luau`, which require valid UTF-8 input, the embedding loaders accept any file content and work correctly with binary files.
+
+## Skipping files
+
+The `skip` loader ignores matching files. Use it to exclude files that would otherwise be picked up by a default loader:
+
+```json5
+{
+  loaders: {
+    "**/*.test.json": "skip",
+  },
+}
+```
+
+## Choosing the Lua extension
+
+Loaders that produce a Lua module write a file with the same name but a Lua extension. By default this is `.lua`. Set `lua_extension` to control which extension is used:
+
+```json5
+{
+  loaders: {
+    "**/*.json": "json",
+  },
+  lua_extension: "luau", // or "lua"
+}
+```
+
+Files that already end in `.lua` or `.luau` keep their extension.
+
+## Requiring loaded files
+
+Loaders that produce a Lua module work with `require` during bundling. You can `require` a JSON config, an embedded Markdown template, or any other file whose loader outputs Lua, and darklua will inline the generated module. The path in the `require` call is rewritten to use the `.lua` (or `.luau`) extension automatically, so `require("./data.json")` becomes a valid require after processing.
+
+Files matched by `skip` or `copy` cannot be `require`d. Requiring a skipped file (or a file with no recognized loader) produces the error "configure content loaders to load this file". Requiring a copied file produces "configure content loaders to copy this file". In both cases the error message points to the relevant loader to use instead.
+
+## Reference
+
+Loaders available for use in the `loaders` field:
+
+| Loader          | What it does                                               |
+| --------------- | ---------------------------------------------------------- |
+| `luau`          | Parses and processes the file as Lua/Luau code.            |
+| `copy`          | Copies the file to the output unchanged.                   |
+| `skip`          | Ignores the file.                                          |
+| `string`        | Returns the content as a string.                           |
+| `string/base64` | Returns the content as a base64-encoded string.            |
+| `buffer`        | Returns the content as a buffer.                           |
+| `buffer/base64` | Returns the content as a base64-encoded buffer.            |
+| `bytes`         | Returns the content as an array of bytes.                  |
+| `bytes/base64`  | Returns the content, base64-encoded, as an array of bytes. |
+| `json`          | Converts JSON data to a Lua module.                        |
+| `toml`          | Converts TOML data to a Lua module.                        |
+| `yaml`          | Converts YAML data to a Lua module.                        |
+
+Default loaders assigned by file extension when no pattern matches. Anything not listed here is skipped unless you assign a loader to it.
+
+| Extension         | Loader   |
+| ----------------- | -------- |
+| `.lua`, `.luau`   | `luau`   |
+| `.json`, `.json5` | `json`   |
+| `.toml`           | `toml`   |
+| `.yaml`, `.yml`   | `yaml`   |
+| `.txt`            | `string` |