DATAMIMIC CE v4.0.0

[ake2l]

The largest DSL surface expansion since 1.0: weighted state machines, composite references, scriptable memstore, native DB sequences, fixed-width/XLSX/DbUnit formats, control flow — plus an agent-ready authoring toolchain over MCP. Seeded runs are now machine-independent; existing seeded projects re-baseline once (see Migration Notes).

✨ Agent-Ready DSL Authoring (MCP)

The MCP server exposes generate, datamimic_check, datamimic_run, and datamimic_reference — a coding agent can look up the DSL contract, lint a descriptor, dry-run it, and generate data without leaving the editor. That shortens the loop from "describe the data you need" to "validated descriptor in the repo."

• datamimic_check lints descriptors with DM-rules and concrete fix hints.
• Dry-run execution caps counts and neutralizes side-effecting targets, so an agent can test a descriptor safely against live client configs.
• datamimic_reference serves element schemas, generators, targets, distributions, entities, and recipes — derived from the engine's registries, so the reference cannot drift from shipped behavior.

---

✨ Weighted State Machines

<state-machine> and <transition> generate reusable weighted state progressions — order lifecycles, claim flows, session journeys — as a named generator.

<setup rngSeed="1">
    <state-machine id="order_flow" start="created">
        <transition from="created" to="paid" weight="0.9"/>
        <transition from="created" to="cancelled" weight="0.1"/>
        <transition from="paid" to="shipped" weight="1"/>
    </state-machine>
    <generate name="orders" count="500" target="CSV">
        <key name="status" generator="order_flow"/>
    </generate>
</setup>

✨ Composite References

Composite <reference> blocks map multiple target fields from the same source row — a multi-column foreign key in one declaration — including DB and MongoDB sources.

<generate name="order_items" count="5000" target="db">
    <reference name="order_ref" source="db" sourceType="orders" distribution="random">
        <field target="order_id" sourceKey="id"/>
        <field target="order_country" sourceKey="country"/>
    </reference>
    <key name="qty" type="int" min="1" max="9"/>
</generate>

✨ Scriptable Memstore and Client IDs

Declared memstore, database, and mongodb ids are available in script= and <execute> scopes: aggregate staged data, filter it against a live table, and report — all inside the descriptor. Memstore aggregation is lenient for non-numeric cells, and a whole entity bound into a scalar DB/Mongo field is stringified deterministically.

<setup defaultSeparator=",">
    <memstore id="mem"/>
    <database id="db" system="postgresql"/>
    <iterate name="orders" source="orders.csv" target="mem"/>
    <execute>
        total = mem.sumEntityColumn('orders', 'amount')
        mem.removeNotExistingIds('orders', 'customer_id', 'customers', db)
    </execute>
    <generate name="summary" count="1" target="JSON">
        <key name="order_count" script="mem.entityCount('orders')"/>
        <key name="total" script="total"/>
    </generate>
</setup>

✨ Native DB Sequences with Explicit Names

SequenceTableGenerator(sequence='schema.seq_name') binds to a named Postgres sequence — ids come from the database's own counter, so generated rows interleave safely with application writes. Omit sequence= to keep the {type}_{name}_seq convention. The enclosing <generate>/<iterate> must carry an explicit count= — the generator partitions its id range up front.

<generate name="orders" type="orders" count="1000" target="db">
    <key name="id" database="db" generator="SequenceTableGenerator(sequence='shop.order_id_seq')"/>
    <key name="status" values="'open','shipped','done'" weights="0.2,0.3,0.5"/>
</generate>

Known carve-out: An explicit sequence= name that does not exist is auto-created starting at 1 (schema-qualified names included). A typo in the sequence name creates a fresh sequence instead of failing. Verify sequence names in dry-run before executing.

✨ Value Picks, Counts, Decimals, and Numeric Distributions

Native attributes now cover weighted values, unique value picks, minCount/maxCount row ranges, exact decimals, bell-shaped numeric draws, and deterministic numeric sequences — no generator expression needed.

<generate name="txns" minCount="100" maxCount="200" target="CSV">
    <key name="channel" values="'web','pos','app'" weights="0.5,0.3,0.2"/>
    <key name="amount" type="decimal" min="0.01" max="999.99" granularity="0.01"/>
    <key name="score" type="int" min="1" max="100" distribution="cumulated"/>
    <key name="sequence_id" type="int" min="1" max="1000" distribution="step"/>
</generate>

unique="True" guarantees each value is picked at most once, and fails loudly when the row count exceeds the pool — size the count accordingly:

<generate name="vouchers" count="4" target="CSV">
    <key name="code" values="'A','B','C','D'" unique="True"/>
</generate>

Numeric range key distributions: uniform (default per-row random draw), cumulated (bell-shaped, mean at midpoint), step / increment (ascending sequence), shuffle (deterministic strided walk), wedge (min, max, min+d, max-d, converging), bitreverse (bit-reversed counter order), fibonacci / padovan (recurrence values clipped to range), randomWalk (bounded walk from min, saturating at max).

Finite positional sequences end instead of wrapping — generation stops when the sequence is exhausted — and are rejected under multiprocessing. Use single-process execution or a per-row distribution for those descriptors.

✨ Control Flow and Assertions

<execute>, <while>, and <assert> are supported for descriptor-local setup, looping, and hard quality gates: an <assert> that fails stops the run instead of shipping bad data downstream.

<setup>
    <execute>threshold = 42</execute>
    <generate name="rows" count="10" target="JSON">
        <key name="v" type="int" min="0" max="100"/>
        <assert condition="v >= 0"/>
    </generate>
</setup>

✨ Read/Enrich Workflows: <iterate>, <id>, <comment>, and offset

<iterate> is an alias of <generate> for read/enrich workflows, <id> an identifier-oriented alias of <key>, and <comment> inline documentation with no runtime effect. offset="N" skips the first N source rows; count defaults, page windows, cyclic wrap-around, and shuffled pools all operate on the post-offset region.

<iterate name="products" source="products.import.fcw" offset="2" count="1000" distribution="ordered" target="db">
    <comment>legacy import - id from a DBA-named native sequence</comment>
    <id name="id" database="db" generator="SequenceTableGenerator(sequence='zsv.t_product_id_seq')"/>
</iterate>

offset= applies to file sources only. DB and memstore sources reject it; use selector-side skipping for those sources.

<setup defaultSeparator=",">
    <iterate name="tail" source="rows.csv" offset="2" distribution="ordered" target="JSON"/>
    <iterate name="wrapped" source="rows.csv" offset="2" cyclic="True" count="7" distribution="ordered" target="JSON"/>
</setup>

(The engine's default column separator is |; set defaultSeparator="," on <setup> — or separator="," per element — when reading comma CSVs.)

✨ Formats and Exporters

XLSX, DbUnit, and fixed-width are now first-class formats, alongside dot-notation SQL write targets (db.upsert) and a native binary type.

Fixed-width .fcw files are self-describing — the first line carries the column specification; DATAMIMIC reads and writes both:

# ean_code[13],name[30],price[8r0]
8000353006386Limoncello Liqueur            00009.85

<setup>
    <generate name="export" count="100" target="FixedWidth(columns='id[8r0],name[30]')" exportUri="out">
        <key name="id" generator="IncrementGenerator"/>
        <key name="name" pattern="[A-Z][a-z]{5,12}"/>
    </generate>
    <iterate name="reimport" source="products.import.fcw" target="ConsoleExporter"/>
</setup>

<setup>
    <database id="db" system="postgresql"/>
    <iterate name="from_excel" source="input.xlsx" target="XLSX" exportUri="out"/>
    <iterate name="seeded" source="fixture.dbunit.xml" sourceEntity="customers" target="DbUnit"/>
    <generate name="upserts" count="100" target="db.upsert">
        <key name="id" generator="IncrementGenerator"/>
    </generate>
    <generate name="files" count="10" target="JSON">
        <key name="thumbnail" type="binary" minLength="64" maxLength="256" mimeType="image/png"/>
    </generate>
</setup>

✨ Region-Group Datasets

Dataset groups — europe, western_europe, central_europe, southern_europe, eastern_europe, north_america, oceania, french, and iberia — select an eligible country per row, so one descriptor produces regionally coherent multi-country data.

<generate name="addresses" count="1000" target="CSV">
    <variable name="addr" entity="Address" dataset="western_europe"/>
    <key name="country" script="addr.country_code"/>
    <key name="city" script="addr.city"/>
</generate>

---

Improvements

Seeded runs are machine-independent. Literal generators (pattern, values, typed keys), domain generators (names, addresses), DateTimeGenerator, and cross-page unique picks now draw from host-independent RNG paths bound to <setup rngSeed>. Existing seeded projects gain cross-machine reproducibility with no DSL change; seeded output changes once — see Migration Notes.

<setup rngSeed="42">
  <generate name="orders" count="20" target="JSON">
    <key name="order_id"  type="int" />
    <key name="reference" pattern="REF-[0-9]{6}-[A-Z]{3}" />
    <key name="status"    values="('active','inactive','pending')" />
    <key name="comment"   type="string" nullQuota="0.3" />
    <nestedKey name="items" type="list" minCount="1" maxCount="5">
      <key name="sku"      pattern="SKU-[0-9]{4}" />
      <key name="quantity" type="int" />
    </nestedKey>
  </generate>
</setup>

Tolerant entity field access. Entity fields resolve case and underscore differences — person.givenName and person.given_name address the same field, so migrated descriptors keep their spelling.

<generate name="people" count="100" target="CSV">
    <variable name="p" entity="Person" dataset="DE" locale="de"/>
    <key name="first" script="p.givenName"/>
    <key name="last" script="p.family_name"/>
</generate>

CSV compatibility. Padded or aligned CSV headers are trimmed on read, so this.name resolves even when the file header carries surrounding whitespace. .wgt.csv weighted files accept an optional header row.

Script and converter ergonomics.

• this, parent, and root script-scope aliases.
• Computed count="{expr}".
• Dynamic <include uri="{...}">.
• Substring(start[, end]) converter with Python slice semantics (string input).
• Nested <generate> exports after its parent, preserving FK-safe write order.
• Data-source length cache keyed by (statement, source).
• Mongo Decimal128 round-trips and CRUD collection routing fixes.
• Luhn-valid credit card numbers.
• Empty CSV files are empty sources, not crashes.

<generate name="parents" count="10" target="JSON">
    <key name="id" generator="IncrementGenerator"/>
    <generate name="children" count="3" target="JSON">
        <key name="parent_id" script="parent.id"/>
        <key name="code" script="str(parent.id)" converter="Substring(-2)"/>
    </generate>
</generate>

---

Migrating from Benerator

The 4.0.0 DSL surface — <iterate>/<id>/<comment>, explicit DB sequence names, .fcw files, the scriptable memstore, weighted values, minCount/maxCount, and tolerant field access — closes the descriptor vocabulary gap for long-lived Benerator projects. Descriptors can be brought across with engine-level support instead of manual rewrites; JVM-specific custom script imports remain manual ports.

---

Migration Notes from 3.x to 4.0.0

Seeded output changes. A 3.x seeded run and a 4.0.0 seeded run with the same seed produce different data. Every 4.0.0 run with the same model and seed is then stable and reproducible across machines. Re-baseline golden files.

Seeded generation now runs single-process. File/stream identity requires stable ordering, which single-process execution provides; worker count can no longer affect seeded output. This is a throughput trade-off — drop rngSeed when parallel throughput matters more than reproducibility, and benchmark on your own infrastructure.

Headerless .wgt.csv files are rejected as sources. A weighted file without column names is valid for per-field weighted picks, not row iteration. For row iteration, include a header line such as value|weight.

Nested <generate> export order changed. Children now export after their parent. This preserves FK-safe write order for nested SQL inserts.

count= above the source length warns and caps when cyclic is off. Prevents silent truncation.

DSL backward compatibility. New attributes and elements (offset, minCount/maxCount on <generate>, weights/unique on <key values>, distribution on numeric keys, sourceEntity/targetEntity, exportUri, <iterate>, <id>, <comment>, <state-machine>, <assert>, <while>) are additive. Existing 3.x XML models parse and run unchanged (except seeded output changes).

---

Known Limitations

• SequenceTableGenerator is Postgres-only, and requires an explicit count= on the enclosing <generate>/<iterate> (it partitions its id range up front).
• An explicit sequence= name that does not exist is auto-created starting at 1. A typo can create a fresh sequence instead of failing. Verify in dry-run.
• offset= applies to file sources only. DB and memstore sources reject it; use selector-side skipping for those sources.
• The default column separator is |; comma CSVs need defaultSeparator="," on <setup> or separator="," on the element.
• JVM-specific custom script imports in migrated projects remain manual ports.

---

This discussion was created from the release DATAMIMIC CE v4.0.0.