WIP: Write a Preset Authoring Guide

Author: kblaschke

content/1.docs/3.preset-authoring/1.index.md

@@ -8,14 +8,14 @@ description: A comprehensive guide to Milkdrop presets.
 This section contains all information required to create new and edit existing Milkdrop presets.
 
 Starting with a general overview on the file format and general rendering process, the guide also covers each built-in
-effect, syntax and usage of the Milkdrop-specific expressions and th use of HLSL shaders to further improve the
+effect, syntax and usage of the Milkdrop-specific expressions and making use of HLSL shaders to further improve the
 resulting visuals.
 
 ### Available Editors
 
-The original Milkdrop plug-in for WinAmp<span>&trade;</span> has a built-in editor which allows editing all parameters
+The original Milkdrop plug-in for WinAmp<span>&reg;</span> has a built-in editor which allows editing all parameters
 and code blocks. While not being overly user-friendly, it is the editor which was used to create basically all presets
 available in the wild.
 
-Recently, more community projects have started implementing their own editor, including projectM. As of now, none of
-these editors are released publicly. Once they become available, we'll add links here. 
+Recently, more community projects have started implementing their own, improved editors, including projectM. As of now,
+none of these editors are released publicly. Once they become available, we'll add links here. 

content/1.docs/3.preset-authoring/6.expression-reference/1.index.md

@@ -2,3 +2,53 @@
 title: Using Expressions
 description: Writing math expressions to control preset effect rendering.
 ---
+
+While some effects already produce a certain amount of movement and reactivity, the real magic in Milkdrop presets
+happens in the expression code.
+
+**Important:** Expressions are not shaders! They use a different language and solely run on the CPU.
+
+## Features
+
+These expressions are written in a simple scripting language which is similar to what scientific programmable
+calculators accept as inputs. It is not a fully-featured scripting language, but powerful enough for basically any use
+case. It supports the following language features:
+
+- A single data type: floating-point numbers.
+- User-defined variables with names up to 16 characters.
+- The usual set of operators for math, logic and assignment.
+- A good number of built-in math functions.
+- Several control-structure functions for implementing conditionals and simple loops.
+
+The expressions do _not_ support custom functions, writing reusable code blocks and defining data structures.
+
+## Performance
+
+Expressions generally perform very well, at near-native speeds of compiled code.
+
+The original Milkdrop and most derivatives use
+the original ["ns-eel2"](https://github.com/projectM-visualizer/milkdrop2/tree/master/src/ns-eel2) (short for "NullSoft
+Expression Evaluation Library 2") implementation. It uses handwritten win32 assembly code - while there are ports of the
+assembly to x86_64 and PPC architectures, it generally only compiles and runs on the Windows 32-bit platform.
+
+projectM, being a cross-platform, cross-architecture implementation, uses a newly written parser
+called ["projectm-eval"](https://github.com/projectM-visualizer/projectm-eval), which is written in C and thus compiles
+on basically any platform supporting floating-point math and the C99 standard library.
+
+### Milkdrop / ns-eel2
+
+In the original Milkdrop, code is parsed and the resulting structure is then built from blocks of pre-compiled Intel 486
+assembly instructions.
+
+This makes the code basically as performant as natively- or JIT-compiled code. The downside is that the assembly
+instructions are quite old and there are probably more performant variants of doing things in modern CPUs, which can't
+be used since the compiler cannot optimize the assembly code.
+
+### projectM / projectm-eval
+
+projectM's custom approach builds a simple tree structure from the code, then recursively calls the functions
+implementing the language features to execute the code.
+
+Since compilers can optimize the implementation to run faster on modern CPUs and the code has been designed with the
+lowest possible runtime overhead (e.g. not using return values, omitting stack frame pointers) in mind, the average
+performance should be nearly identical to ns-eel2.

content/1.docs/3.preset-authoring/6.expression-reference/2.syntax.md

@@ -8,14 +8,13 @@ description: General syntax of the expression code.
 The expression language only knows a few syntactic features:
 
 - Numbers, with or without decimals. Can also contain an _integer_ base-10 exponent using the `e[+-]n` notation.
-- Variables to hold a single number, both user-defined and predefined. Variable names may contain characters a-z, 0-9
-  and the underscore (`_`), while 0-9 must not be the first letter. The maximum length of a variable name defaults to 16
-  characters and thus should not exceed this count.
-- Special constants starting with `$` (see [Undocumented Features](undocumented-features)).
+- [Variables](variables) to hold a single number, both user-defined and predefined.
+- [Special constants](undocumented-features#constants) starting with `$` (
+  see [Undocumented Features](undocumented-features)).
 - A single expression, separated from other expressions in the same list via semicolons (`;`), with the last semicolon
   being optional.
-- Calls to built-in functions in the form `func(arg1, arg2, ...)`.
-- Mathematical, assignment and other operators.
+- Calls to [built-in functions](functions) in the form `func(arg1, arg2, ...)`.
+- Mathematical, assignment and other [operators](operators).
 - Subscript access `[]` for [megabuf/gmegabuf access](megabuf).
 - Parentheses (`()`) to group statements and override precedence.
 - In-line comments starting with `//` until the end of the current line
@@ -67,19 +66,7 @@ Functions always need to be followed by parentheses (`sin(expr)`), while variabl
 Milkdrop imposes a 16-character limit on variable names. projectm-eval does not have a hard limit, but it's advisable to
 not use more than 16 characters for Milkdrop compatibility.
 
-Milkdrop also has a limit of 64 user-defined variables, which also isn't applied by projectm-eval.
-
-### Global "Register" Variables
-
-In addition to gmegabuf, there are an additional 100 variables which can be used to store global values. These variables
-are named `reg00`, `reg01` etc. up to `reg99`. Values stored in those variables are available in al execution contexts,
-in the same way as gmegabuf.
-
-Note the index must always be written with two digits. `reg3` is _not_ considered a global variable and will only have a
-local scope as any other variable. The same is true for more digits like `reg123`.
-
-Same as with gmegabuf, global variables are not necessarily `0` when a preset is initialized, and they can change at any
-time when two presets using the same global variables are blended during a transition.
+Milkdrop also has a limit of 64 user-defined variables, which also isn't applied by projectM.
 
 ## Numbers
 

content/1.docs/3.preset-authoring/6.expression-reference/3.variables.md

@@ -0,0 +1,54 @@
+---
+title: Variables
+description: User-defined and special reg/q/t variables.
+---
+
+In expression code, variables can be used to store and retrieve a single number. There are three logical types of
+variables:
+
+- Predefined variables depending on the execution context which act as inputs and outputs for a specific effect.
+- Up to 64 (in the original Milkdrop and derivatives, projectM does not impose a limit) user-defined variables, which
+  can simply be used without declaration.
+- The special `reg`, `q` and `t` variables, which are used to carry over values from context to context in a certain
+  way.
+
+The `q` and `t` variables predate the `reg` variables and the megabuf feature and were the only way to pass values along
+between preset effects in earlier (pre-2.0) Milkdrop versions.
+
+## `reg` Variables
+
+The global `reg` variables are 100 additional, pre-defined variables which can be used to store global values. These
+variables are named `reg00`, `reg01` etc. up to `reg99`. Values stored in those variables are available in all execution
+contexts, in the same way as gmegabuf.
+
+Register variables are not reassigned/coped between effects, so the values stored within will be passed form one effect
+to the next [in the order they are drawn](../preset-rendering-process).
+
+**Important:** The index must always be written with two digits. `reg3` or `reg100` are _not_ considered global
+variables and will behave like any other user-defined variable.
+
+Same as with gmegabuf, in Milkdrop and derivatives, global variables are not necessarily set to `0` when a preset is
+initialized, and they can change at any time when two presets using the same global variables are blended during a
+transition. In projectM, `reg` variables are always preset-specific and set to `0` when the preset starts.
+
+## `q` Variables
+
+In a preset, 32 variables named `q1` to `q32` can be used to pass values around within a preset, including the Warp and
+Composite shaders. These variables are passed (copied) in a tree-like manner, which is best visualized as a diagram:
+
+![Flow of q Variables](/content/preset-guide/q-var-diagram.png)
+
+The data flow from preset init code to custom wave/shape init code was not shown in the `q` var diagram Ryan
+Geiss' original guide, only documented as being present but mostly useless. In reality, the `q` vars actually _can_ be
+useful to pass the values of some predefined per-frame variables to the wave/shape init code which are not available in
+those code blocks, such as the aspect ratio.
+
+## `t` Variables
+
+Similar to the `q` variables, an additional 8 variables named `t1` to `t8` are available in the code of Custom Wave and
+Custom Shape effects.
+
+For each Custom Shape or Wave, an individual copy of those variables is used, meaning `t1` in shape 1 is different from
+`t1` in shape 2.
+
+![Flow of t Variables](/content/preset-guide/t-var-diagram.png)

content/1.docs/3.preset-authoring/6.expression-reference/3.operators.md renamed to content/1.docs/3.preset-authoring/6.expression-reference/4.operators.md

@@ -0,0 +1,166 @@
+---
+title: Memory Buffers
+description: Large memory buffers to store values.
+---
+
+Since Milkdrop 2, a new feature to store values called "megabuf" was added. It adds several memory buffers which can be
+used in all expressions, with each buffer being able to store up to 8,388,608 numbers.
+
+Each execution context in Milkdrop has its own memory buffer, plus an additional one called "gmegabuf" that is globally
+available across all code blocks. Execution contexts are generally tied to a specific effect. This is a list of all
+10 contexts and which ones share a megabuf, with one context per line:
+
+- `per_frame_init_` and `per_frame_`
+- `per_pixel_`
+- `wave_1_init`, `wave_1_per_frame` and `wave_1_per_point`
+- `wave_2_init`, `wave_2_per_frame` and `wave_2_per_point`
+- `wave_3_init`, `wave_3_per_frame` and `wave_3_per_point`
+- `wave_4_init`, `wave_4_per_frame` and `wave_4_per_point`
+- `shape_1_init` and `shape_1_per_frame`
+- `shape_2_init` and `shape_2_per_frame`
+- `shape_3_init` and `shape_3_per_frame`
+- `shape_4_init` and `shape_4_per_frame`
+
+This means there are 11 separate buffers per preset, allowing to store 92,274,688 different values. Using all slots
+would require 704 MB of RAM (numbers are stored as 64-bit/8-byte double-precision floats internally).
+
+The global memory buffer can be used to circumvent the way of how values are passed using `qNN` and `tN` variables,
+similar to how `regNN` variables work, but having a larger capacity. Since this buffer is also not touched between
+frames, it can be used to carry values from an effect late in a frame to another effect drawn in the next frame, like
+custom waveform/spectrum values to be used in custom shapes. While this will have a single frame of lag, it opens up a
+few additional possibilities such as drawing spectrum data using custom shapes or the warp grid.
+
+Warp and composite shaders do not have access to these buffers, as they run on the GPU.
+
+## Buffer Allocation
+
+To reduce the number of memory allocations at runtime, values are not individually allocated in system memory. Rather,
+Milkdrop and projectM allocate up to 128 blocks of 65,536 values per buffer.
+
+Each time a value index is accessed, it is checked whether the block this index is stored is already allocated. If it
+is, the existing block will be accessed. If not, a new memory block is allocated and filled with zeroes.
+
+### Memory Usage Warning
+
+This in turn means if a preset accesses the first index of each block once, _all_ memory block would be allocated. In
+theory, the original ns-eel2 interpreter in Milkdrop supports setting a memory limit, but this feature is not used and
+thus there is no memory cap except the size of all buffers combined.
+
+When writing presets, it's highly recommended to not spread memory indices too far, e.g. trying to keep them within one
+or two memory blocks if possible. In practice, most presets will probably only make use of the global buffer, ignoring
+the context-specific ones altogether. On modern PCs, a few MB of RAM usage won't have a big impact anyway.
+
+## Accessing Memory Buffers
+
+The global and context-specific memory buffers can be accessed with different methods.
+
+### megabuf()
+
+The `megabuf()` functions returns a reference to a single value in the memory buffer of the current execution context.
+The return value is a reference, and thus can be both read and assigned to:
+
+```
+x = megabuf(1000);
+megabuf(1001) = x + 1;
+```
+
+### gmegabuf()
+
+The `gmegabuf()` functions returns a reference to a single value in the preset-global memory buffer. The return value is
+a reference, and thus can be both read and assigned to:
+
+```
+x = gmegabuf(1000);
+gmegabuf(1001) = x + 1;
+```
+
+### Subscript Operator
+
+The subscript operator or "array access operator" has a slightly different meaning than in other programming languages.
+The Milkdrop expression syntax only knows a single data type, individual numbers, so there are no arrays.
+
+If any index value is not an integer, it is _rounded_ to the nearest integer after calculating the final index (e.g.
+adding index and offset first, then rounding).
+
+The subscript operator is instead used to address memory locations in megabuf and gmegabuf. There are three possible
+syntax variants:
+
+#### Global Memory Access
+
+Accessing the gmegabuf is possible by using the special keyword `gmem`, followed by an index in the subscript brackets.
+The following statement sets gmegabuf location 10.000 to the current value of `x`:
+
+```
+gmem[10000] = x;
+```
+
+Any memory index from 0 to 8.388.607 (= 128 * 65536) can be addressed.
+
+#### Local Memory Access
+
+Accessing the current context memory buffer (megabuf) can be done by writing the index before a set of empty brackets.
+The following example is analogous to the above, but it sets index 10.000 of the local memory instead:
+
+```
+10000[] = x;
+```
+
+#### Local Memory Access with Offset
+
+An optional offset can be provided in the brackets, which is simply added to the index on the outside. The following
+example will set memory index 10.123 to the value of `x`:
+
+```
+10000[123] = x;
+```
+
+Both index and offset values can of course be calculated with expressions. So the following expression is valid:
+
+```
+if(x > 5,5000,1000)[(sin(y) + 1 * .5) * 1000] = z;
+```
+
+## Initializing a Range
+
+**Note:** This works only on context-specific memory buffers, _not_ gmegabuf!
+
+If a large portion of a context-specific memory buffer needs to be initialized with a specific, single value, the
+`memset` function can be used instead of `loop` and iterating over all indices. The following lines are equivalent, with
+the second one being more performant - both set 1000 values to 1.123, starting at index 4000:
+
+```
+idx=0; loop(1000, 4000[idx] = 1.123; idx += 1); 
+memset(4000, 1.123, 1000);
+```
+
+## Copying Ranges
+
+**Note:** This works only on context-specific memory buffers, _not_ gmegabuf! There is no way of either copying ranges
+within gmegabuf or between gmegabuf and a context megabuf.
+
+In the case a large, continuous portion of values need to be copied, the `loop` function in expression code is way
+slower than using [the `memcpy`function](functions#memcpydest-src-count). This function takes a destination index, a
+source index and a count as arguments and then copies `count` values starting at the source index to the destination
+index.
+
+To copy 1000 values from index 2000 to 4000, both of the following lines would do this, with the second one being
+way more performant:
+
+```
+idx=0; loop(1000, 4000[idx] = 2000[idx]; idx += 1); 
+memcpy(4000, 2000, 1000);
+```
+
+## Freeing Memory
+
+While the expression language defines a function [`freembuf`](functions#freembufindex), it doesn't actually do anything
+in Milkdrop or projectM. Once memory has been allocated, it'll stay that way until either the preset is unloaded
+(context-specific buffers and gmegabuf in projectM) or Milkdrop is closed (gmegabuf in Milkdrop).
+
+## Cross-Preset Sharing
+
+In the original Milkdrop and most derivatives, the global memory buffer is shared across presets being blended. While
+rare, this can occasionally break presets making heavy use of gmegabuf, as the newly loaded preset may use the same
+buffer indices as the previous one.
+
+projectM does use one gmegabuf instance per preset, so each preset is fully independent regarding gmegabuf use.