Revise xml parser (settings.xml)

doc/settings.md

@@ -5,21 +5,21 @@ graph TB
     subgraph Settings loading order
     direction LR
         A1["1"] --- B1("Init hardcoded <include=.../> list")
-        B1 --> C1("Take <include=.../> list from the $VTM_CONFIG value or file it referencing")
-        C1 --> G1("Take <include=.../> list from the received DirectVT packet")
-        G1 --> H1("Take <include=.../> list from the --config' CLI option value or file it referencing")
+        B1 --> C1("Take <include=.../> file list from the $VTM_CONFIG value or from the file it directly references")
+        C1 --> G1("Take <include=.../> file list from the received DirectVT packet")
+        G1 --> H1("Take <include=.../> file list from the --config' CLI option value or from the file it directly references")
     direction LR
-        A2["2"] --- B2("Overlay <config/> subsections from the resultant <include=.../> list")
-        B2 --> C2("Overlay <config/> subsection from the $VTM_CONFIG value or file it referencing")
-        C2 --> G2("Overlay <config/> subsection from the received DirectVT packet")
-        G2 --> H2("Overlay <config/> subsection from the --config' CLI option value or file it referencing")
+        A2["2"] --- B2("Overlay settings from the resultant <include=.../> file list")
+        B2 --> C2("Overlay settings from the $VTM_CONFIG value or from the file it directly references")
+        C2 --> G2("Overlay settings from the received DirectVT packet")
+        G2 --> H2("Overlay settings from the --config' CLI option value or from the file it directly references")
     end
 ```
 
 ## TL;DR
 
-The settings are stored in a Pure 'XML' which is an XML-like format, storing a hierarchical list of key=value pairs.  
-See [`/src/vtm.xml`](../src/vtm.xml) for reference.
+The settings are stored in the "Pure XML" file format, which looks like classical XML but with dynamic element refrencing and templating.  
+See [`/src/vtm.xml`](../src/vtm.xml) for example.
 
 We call the text data in the settings file "plain XML data" even though our file format is not technically XML, but only visually resembles it.
 
@@ -30,11 +30,11 @@ There are two predefined settings source locations and this can be changed as ne
 ```
 
 The process of loading settings consists of the following steps:
-- Build an ordered list of the setting source files by looking for the root `<include=.../>` subsections.
-- Overlay the XML data from the source files in the specified order.
-- Overlay the XML data from the value of the `$VTM_CONFIG` environment variable or from a settings file it references.
-- Overlay the XML data from the DirectVT config payload received from the parent process.
-- Overlay the XML data from the specified `--config <...>` CLI option value or from a settings file it referencing.
+- Build an ordered list of the setting source files by looking for the root `<include=.../>` elements.
+- Overlay settings from the source files in the specified order.
+- Overlay settings from the value of the `$VTM_CONFIG` environment variable or from a settings file it directly references.
+- Overlay settings from the DirectVT config payload received from the parent process.
+- Overlay settings from the specified `--config <...>` CLI option value or from a settings file it directly references.
 
 The file list is built in the following order from the following sources:
 - The settings file list from the hardcoded configuration containing a list of two files:
@@ -44,24 +44,26 @@ The file list is built in the following order from the following sources:
   <include="~/.config/vtm/settings.xml"/>   <!-- Default user-wise settings source. -->
   ...
   ```
-- The settings file list from the `$VTM_CONFIG` environment variable value or from a settings file it referencing.
+- The settings file list from the `$VTM_CONFIG` environment variable value or from a settings file it directly references.
   - A case with a plain XML-data:
     - `$VTM_CONFIG=<include*/><include='/path/to/override_defaults.xml'/>...` - Clear the current file list and begin a new file list containing a single file '/path/to/override_defaults.xml'.
     - `$VTM_CONFIG=<include='/path/to/first.xml'/><include='/path/to/second.xml'/>...` - Append the current file list with the files '/path/to/first.xml' and '/path/to/second.xml'.
   - A case with a file reference:
     - `$VTM_CONFIG='/path/to/override_defaults.xml'` - Take the file list from the '/path/to/override_defaults.xml'.
 - The settings file list from the DirectVT config received from the parent process.
-- The settings file list from the specified `--config <...>` CLI option value or from a settings file it referencing.
+- The settings file list from the specified `--config <...>` CLI option value or from a settings file it directly references.
   - A case with a plain XML-data:
     - `./vtm --config "<include*/><include='/path/to/override_defaults.xml'/>..."` - Clear the current file list and begin a new file list containing a single file '/path/to/override_defaults.xml/'.
   - A case with a file reference:
     - `./vtm --config "/path/to/override_defaults.xml"` - Take the file list from the '/path/to/override_defaults.xml'.
 
 ## Pure XML
 
-### Key differences from classical XML
+### Differences from classical XML
 
- - Document encoding is UTF-8.
+Pure XML is based on the XML 1.1 standard, with the following exceptions:
+
+ - Document encoding is UTF-8 only.
  - Any Unicode characters are allowed, including the U+0000 (null) character.
  - There is no support for named XML character entities.
  - The stored data forms a hierarchical list of `name=value` pairs.
@@ -71,21 +73,24 @@ The file list is built in the following order from the following sources:
      - `<... name="value" />`, `<...> <name> "value" </name> </...>`, and `<...> <name="value" /> </...>` have the same meaning.
    - The XML-attribute `param` in `<name param="value"/>` and the XML-element `param` in `<name> <param="value"/> </name>` are semantically identical sub-elements of the `name` element.
  - No spaces are allowed between the opening angle bracket and the element name:
-   - `... < name ...`, `... <= ...`, `... << ...` are treated as parts of the element's value content.
+   - `... < name ...` should not be treated as an opening tag.
  - Every element has its own text value.
    - For example, `<name="names_value" param="params_value"/>` - the `name` element has the text value `names_value`, and its `param` sub-element has the text value `params_value`.
  - All stored values are strings (the data requester decides on its side how to interpret it):
    - `name=2000` and `name="2000"` have the same meaning.
  - All value strings, except those that begin with a decimal digit character (ASCII `0` - `9`), must be quoted with either double or single quotes (`"` U+0022 or `'` U+0027).
  - The value string can be fragmented. Fragments can be located after the equal sign following the element name, as well as between the opening and closing tags.
- - The fragments located between the opening and closing tags can be either quoted or in raw form. The quoted form sets strict boundaries for the string value. The raw form pulls all characters between the opening and closing tags, including line breaks.
- - The following compact syntax for elements is allowed:
+ - The fragments located between the opening and closing tags can be either quoted or in raw form. The quoted form sets strict boundaries for the string value. The raw form pulls all characters between the opening and closing tags, excluding trailing whitespaces (whitespaces immediately before a nested opening tag or an element's closing tag).
+ - The following compact syntax for element declaration is allowed:
    - `<node0/node1/thing name="value"/>` and `<node0><node1><thing name="value"/></node1></node0>` have the same meaning.
- - Elements can reference any element using relative and absolute references, in the form of an unquoted name or an XML path to the referenced element.
+ - Elements can reference any other elements using relative and absolute references, in the form of an unquoted name or an XML path to the referenced element.
    - `thing2` refers to the value `/node1/thing1` in `<node1 thing1="value1"/><node2 thing2=/node1/thing1 />`.
    - `thing2` refers to the value `thing1` within the scope of `<node1 thing1="value1"><node2 thing2=thing1 /></node1>`.
+   - Each element forms its own namespace.
+   - The value of an element containing relative references is obtained by traversing the element's namespace and all its surrounding namespaces until the first hit.
+   - A recursive reference is a reference encountered during the resolving of another reference.
+   - All recursive references are resolved starting from the element's namespace, regardless of where the recursive references are encountered.
    - Circular references are silently ignored.
-   - //todo describe the reference resolution order.
  - The element reference includes all of the element's contents, including the element's value and all nested elements.
  - The element's content may include any number of substrings, as well as references to other elements, combined in the required order using the vertical bar character ASCII 0x7C `|`.
    - `<thing1="1"/><thing2="2"/><thing21=thing2 | thing1/>` and `<thing1="1"/><thing2="2"/><thing21="21"/>` have the same meaning.

src/netxs/desktopio/application.hpp

@@ -22,7 +22,7 @@ namespace netxs::app
 
 namespace netxs::app::shared
 {
-    static const auto version = "v2025.05.28";
+    static const auto version = "v2025.06.03";
     static const auto repository = "https://github.com/directvt/vtm";
     static const auto usr_config = "~/.config/vtm/settings.xml"s;
     static const auto sys_config = "/etc/vtm/settings.xml"s;
@@ -699,35 +699,33 @@ namespace netxs::app::shared
             }
             return faux;
         }
-        auto attach_file_list(xml::document& defcfg, xml::document& cfg)
+        auto attach_file_list(txts& file_list, xml::document& src_cfg)
         {
-            if (cfg)
+            auto file_ptr_list = src_cfg.take_ptr_list<true>("/include");
+            if (file_ptr_list.size())
             {
-                auto file_list = cfg.take_ptr_list<true>("/include");
-                if (file_list.size())
+                log("%%Update settings source files from %src%", prompt::apps, src_cfg.page.file);
+                for (auto& file_ptr : file_ptr_list)
                 {
-                    log("%%Update settings source files from %src%", prompt::apps, cfg.page.file);
-                    for (auto& file : file_list)
+                    if (file_ptr->base)
                     {
-                        if (file && !file->base)
-                        {
-                            log("%%%file%", prompt::pads, file->_concat_values());
-                        }
+                        file_list.clear();
+                    }
+                    auto file_path = file_ptr->_concat_values();
+                    if (file_path.size())
+                    {
+                        log("%%%file%", prompt::pads, file_path);
+                        file_list.emplace_back(std::move(file_path));
                     }
-                    defcfg.attach("/", file_list);
                 }
             }
         }
-        auto overlay_config(xml::document& defcfg, xml::document& cfg)
+        auto overlay_config(xml::document& def_cfg, xml::document& src_cfg)
         {
-            if (cfg)
+            if (src_cfg)
             {
-                auto config_data = cfg.take_ptr_list("/");
-                if (config_data.size())
-                {
-                    log(prompt::pads, "Merging settings from ", cfg.page.file);
-                    defcfg.overlay(config_data.front(), "");
-                }
+                log(prompt::pads, "Merging settings from ", src_cfg.page.file);
+                def_cfg.combine_item(src_cfg.root_ptr);
             }
         }
         auto settings(qiew cliopt, bool print = faux)
@@ -740,6 +738,7 @@ namespace netxs::app::shared
             auto envcfg = xml::document{};
             auto dvtcfg = xml::document{};
             auto clicfg = xml::document{};
+            auto file_list = txts{};
 
             auto show_cfg = [&](auto& cfg){ if (print && cfg) log("%source%:\n%config%", cfg.page.file, cfg.page.show()); };
 
@@ -781,16 +780,15 @@ namespace netxs::app::shared
                 show_cfg(clicfg);
             }
 
-            attach_file_list(defcfg, envcfg);
-            attach_file_list(defcfg, dvtcfg);
-            attach_file_list(defcfg, clicfg);
+            attach_file_list(file_list, defcfg);
+            attach_file_list(file_list, envcfg);
+            attach_file_list(file_list, dvtcfg);
+            attach_file_list(file_list, clicfg);
 
-            auto config_sources = defcfg.take_ptr_list("/include");
-            for (auto& file_rec : config_sources) if (file_rec && !file_rec->base) // Overlay configs from the specified sources if it is.
+            for (auto& file_path : file_list) // Overlay configs from the specified sources if it is.
             {
-                auto src_file = file_rec->_concat_values();
                 auto src_conf = xml::document{};
-                load_from_file(src_conf, src_file);
+                load_from_file(src_conf, file_path);
                 show_cfg(src_conf);
                 overlay_config(defcfg, src_conf);
             }
@@ -799,7 +797,7 @@ namespace netxs::app::shared
             overlay_config(defcfg, dvtcfg);
             overlay_config(defcfg, clicfg);
 
-            auto resultant = xml::settings{ defcfg };
+            auto resultant = xml::settings{ std::move(defcfg) };
             return resultant;
         }
     }

src/netxs/desktopio/utf.hpp

@@ -18,7 +18,7 @@ namespace netxs
     using txts = std::vector<text>;
     using namespace std::literals;
 
-    static constexpr auto whitespaces = " \n\r\t"sv;
+    static constexpr auto whitespaces = " \t\r\n\v\f"sv;
     static constexpr auto onlydigits  = "0123456789"sv;
     static constexpr auto alphabetic  = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ_"sv;
     static constexpr auto base64code  = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
@@ -1164,6 +1164,21 @@ namespace netxs::utf
         }
         return from.substr(0, s_size);
     }
+    void replace_all(view utf8, view what, view to, text& dest)
+    {
+        auto last = 0_sz;
+        if (!what.empty() && utf8.length() >= what.length())
+        {
+            auto spot = 0_sz;
+            while ((spot = utf8.find(what, last)) != text::npos)
+            {
+                dest += utf8.substr(last, spot - last);
+                dest += to;
+                last = spot + what.size();
+            }
+        }
+        dest += utf8.substr(last);
+    }
     void replace_all(text& utf8, auto const& from, auto const& to)
     {
         auto frag = view{ from };
@@ -1366,6 +1381,19 @@ namespace netxs::utf
             return crop;
         }
     }
+    auto split2(view utf8, char delimiter, auto proc)
+    {
+        auto cur = 0_sz;
+        auto pos = 0_sz;
+        while ((pos = utf8.find(delimiter, cur)) != text::npos)
+        {
+            auto frag = view{ utf8.data() + cur, pos - cur };
+            if (!proc(frag, faux)) return faux;
+            cur = pos + 1;
+        }
+        auto end = view{ utf8.data() + cur, utf8.size() - cur };
+        return proc(end, true);
+    }
     template<bool SkipEmpty = faux, feed Direction = feed::fwd, class P, bool Plain = std::is_same_v<void, std::invoke_result_t<P, view>>>
     auto split(view utf8, char delimiter, P proc)
     {
@@ -1852,18 +1880,28 @@ namespace netxs::utf
     {
         utf::trim_back_if(utf8, [&](char c){ return delims.find(c) == text::npos; });
     }
-    // utf: Trim the utf8 front and return trims.
-    auto pop_front_chars(view& utf8, view delims)
+    // utf: Trim the utf8 while any of delims front and return trims.
+    auto pop_front_chars(view& utf8, view while_any_of)
     {
         auto temp = utf8;
-        utf::trim_front(utf8, delims);
+        utf::trim_front(utf8, while_any_of);
         return temp.substr(0, temp.size() - utf8.size());
     }
-    // utf: Trim the utf8 back and return trims.
-    auto pop_back_chars(view& utf8, view delims)
+    // utf: Trim the utf8 front until any of delims is found.
+    template<bool Lazy = true>
+    void pop_front_until(view& utf8, auto until_any_of)
+    {
+        auto head = utf8.begin();
+        auto tail = utf8.end();
+        auto stop = utf::find_char(head, tail, until_any_of);
+        auto prefix_len = stop - head;
+        utf8.remove_prefix(prefix_len);
+    }
+    // utf: Trim the utf8 back while any of delims and return trims.
+    auto pop_back_chars(view& utf8, view while_any_of)
     {
         auto temp = utf8;
-        utf::trim_back(utf8, delims);
+        utf::trim_back(utf8, while_any_of);
         return temp.substr(utf8.size());
     }
 
@@ -2063,29 +2101,28 @@ namespace netxs::utf
             return std::pair{ qiew{}, utf8 };
         }
     }
+    // utf: Trim utf8 until any of delims is found, and return trims.
     template<bool Lazy = true>
     auto take_front(view& utf8, view delims)
     {
-        auto head = utf8.begin();
-        auto tail = utf8.end();
-        auto stop = find_char(head, tail, delims);
-        if (stop == tail)
+        auto temp = qiew{ utf8 };
+        utf::pop_front_until(utf8, delims);
+        if (utf8.empty()) // If not found.
         {
             if constexpr (Lazy)
             {
-                utf8 = {};
-                return qiew{ utf8 };
+                return qiew{};
             }
             else
             {
-                auto crop = qiew{ utf8 };
-                utf8 = {};
-                return crop;
+                return temp;
             }
         }
-        auto str = qiew{ head, stop };
-        utf8.remove_prefix(str.size());
-        return str;
+        else
+        {
+            auto crop = temp.substr(0, temp.size() - utf8.size());
+            return crop;
+        }
     }
     template<bool Lazy = true, class ...ViewList>
     auto take_front(view& utf8, std::tuple<ViewList...> const& delims)
@@ -2168,6 +2205,7 @@ namespace netxs::utf
         }
         return args;
     }
+    // utf: Remove utf8 tail until any of delims (including delim).
     auto eat_tail(view& utf8, view delims)
     {
         auto head = utf8.begin();