From 31f5ea6b57fed2c0a8df883a9c3237315c30e844 Mon Sep 17 00:00:00 2001 From: Alice Cecile Date: Thu, 20 Jan 2022 11:36:45 -0500 Subject: [PATCH 1/8] Basic explanation --- rfcs/46-atomic-groups.md | 117 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 117 insertions(+) create mode 100644 rfcs/46-atomic-groups.md diff --git a/rfcs/46-atomic-groups.md b/rfcs/46-atomic-groups.md new file mode 100644 index 00000000..e6e0f550 --- /dev/null +++ b/rfcs/46-atomic-groups.md @@ -0,0 +1,117 @@ +# Feature Name: (fill me in with a unique ident, `my_awesome_feature`) + +## Summary + +One paragraph explanation of the feature. + +## Motivation + +Why are we doing this? What use cases does it support? + +## User-facing explanation + +**Atomic groups** are collections of systems (and run criteria) that must be executed "together": once the group has been started, no systems can mutate data that they rely on. +Like the atom, they are indivisible: nothing can get between them! +Run criteria are always part of the same atomic group as the systems they control. + +**Atomic ordering constraints** are a form of ordering constraints that ensure that two systems are run directly after each other. +At least, as far as "directly after" is a meaningful concept in a parallel, nonlinear ordering. + +In addition to their use in run criteria, atomic groups are a useful (if quite advanced) feature for forcing tight interlocking behavior between systems, preserving plugin privacy and avoiding leaking sensitive timing information. +They allow us to ensure that the state read from and written to the earlier system cannot be invalidated. +This functionality is extremely useful when updating indexes (used to look up components by value): +allowing you to ensure that the index is still valid when the system using the index uses it. + +The **atomicity** of each label can be controlled using `.configure_label(MySystemLabel::Variant.set_atomicity(AtomicityLevel::Coherent))`: + +```rust +enum AtomicityLevel { + // Read and write-locks are released when a system completes + None, + // Read and write-locks are held until they are no longer needed by a member of this atomic group + Coherent, + // Read and write-locks are only released when this atomic group is complete + Isolated, +} +``` + +Labels are non-atomic by default, and atomic ordering constraints create coherent atomic groups. +Isolated atomic groups are relatively niche, used only when you want to be certain that no information about internal progress can leak out. + +Calling `a.atomically_before(b)` will create a strict ordering constraint between `a` and `b`, and add them to the same atomic group (with a uniquely-generated system label). + +Of course, atomic groups (and thus atomic ordering constraints) should be used thoughtfully. +As the strictest of the three types of system ordering dependency, they can easily result in unsatisfiable schedules if applied to large groups of systems at once. +For example, consider the simple case, where we have an atomic ordering constraint from system A to system B. +If for any reason commands must be flushed between systems A and system B, we cannot satisfy the schedule. +The reason for this is quite simple: the lock on data from system A will not be released when it completes, but we cannot run exclusive systems unless all locks on the `World's` data are clear. +As we add more of these atomic constraints (particularly if they share a single ancestor), the risk of these problems grows. +In effect, atomically-connected systems look "almost like" a single system from the scheduler's perspective. + +On the other hand, atomic ordering constraints can be helpful when attempting to split large complex systems into multiple parts. +By guaranteeing that the initial state of your complex system cannot be altered before the later parts of the system are complete, +you can safely parallelize the rest of the work into separate systems, improving both performance and maintainability. + +## Implementation strategy + +For all systems that are part of the same atomic system group: + +- When a system in an atomic group completes, its write-locks and read-locks are downgraded to **virtual locks**. +- Virtual locks are treated as if they were real locks for all systems outside of the atomic group. +- Virtual locks are ignored by systems in the same atomic group. +- If the atomic group is **coherent**, virtual locks are removed once all systems in the atomic group have finished with it. +- If the atomic group is **isolated**, virtual locks are held until all systems in the atomic group complete. + +This can be modelled as: + +```rust +enum DataLock{ + Read, + Write, + VirtualRead(Box), + VirtualWrite(Box), +} +``` + +Multiple virtual locks on the same data can exist at once, and membership in an atomic group is *not* transitive. +Suppose we have two atomic groups, `GroupX = {A, B}` and `GroupY = {B, C}`. + +Systems `A` and `C` do not share an atomic group, and data that is virtually locked by both `GroupX` and `GroupY` can only be accessed by system `B`. + +This reduces the risk of an unsatisfiable schedule by allowing locks to be released in a maximally permissive fashion. + +## Drawbacks + +Why should we *not* do this? + +## Rationale and alternatives + +- Why is this design the best in the space of possible designs? +- What other designs have been considered and what is the rationale for not choosing them? +- What objections immediately spring to mind? How have you addressed them? +- What is the impact of not doing this? +- Why is this important to implement as a feature of Bevy itself, rather than an ecosystem crate? + +## \[Optional\] Prior art + +Discuss prior art, both the good and the bad, in relation to this proposal. +This can include: + +- Does this feature exist in other libraries and what experiences have their community had? +- Papers: Are there any published papers or great posts that discuss this? + +This section is intended to encourage you as an author to think about the lessons from other tools and provide readers of your RFC with a fuller picture. + +Note that while precedent set by other engines is some motivation, it does not on its own motivate an RFC. + +## Unresolved questions + +- Should we allow overlapping atomic groups that don't have a subset-superset relationship? + - An atomic group is supposed to make multiple systems look like one system to outside systems. + - What do overlapping subsets mean conceptually? + +## \[Optional\] Future possibilities + +- Indexes! +- Relations! +- Automatic flush inference! From 94e95f712cff33e286be579089a8624184e997f1 Mon Sep 17 00:00:00 2001 From: Alice Cecile Date: Thu, 20 Jan 2022 12:03:44 -0500 Subject: [PATCH 2/8] Flesh out supporting sections --- rfcs/46-atomic-groups.md | 40 ++++++++++++++++++++-------------------- 1 file changed, 20 insertions(+), 20 deletions(-) diff --git a/rfcs/46-atomic-groups.md b/rfcs/46-atomic-groups.md index e6e0f550..2d2dd6a7 100644 --- a/rfcs/46-atomic-groups.md +++ b/rfcs/46-atomic-groups.md @@ -1,12 +1,21 @@ -# Feature Name: (fill me in with a unique ident, `my_awesome_feature`) +# Feature Name: `atomic-groups` ## Summary -One paragraph explanation of the feature. +Atomic groups ensure that systems are evaluated "together", ensuring that nothing can break their internal state during execution. +While this would be feasible by running systems in strict sequence, atomicity ensures safe "as-if" parallel execution of other systems while a group is being evaluated. ## Motivation -Why are we doing this? What use cases does it support? +Atomic groups are a basic building block of scheduling, and are essential to support more complex user-facing tools that rely on setting up and then using state. +Critically, indexes (used to efficiently store and then look up components by value) require some form of atomicity to ensure that users are not reading from bad state. + +In addition, atomicity can be a powerful tool for plugin authors, allowing them to impose strong guarantees of privacy without disabling configuration. + +While system piping creates a simple form of atomicity, atomic groups are substantially more expressive. They: + +- allows for systems with conflicting data accesses to co-exist +- allows for nonlinear atomicity ## User-facing explanation @@ -82,7 +91,8 @@ This reduces the risk of an unsatisfiable schedule by allowing locks to be relea ## Drawbacks -Why should we *not* do this? +- This is a complex feature! +- It could slow down evaluation of the schedule, including in cases where this feature is not used. ## Rationale and alternatives @@ -92,26 +102,16 @@ Why should we *not* do this? - What is the impact of not doing this? - Why is this important to implement as a feature of Bevy itself, rather than an ecosystem crate? -## \[Optional\] Prior art - -Discuss prior art, both the good and the bad, in relation to this proposal. -This can include: - -- Does this feature exist in other libraries and what experiences have their community had? -- Papers: Are there any published papers or great posts that discuss this? - -This section is intended to encourage you as an author to think about the lessons from other tools and provide readers of your RFC with a fuller picture. - -Note that while precedent set by other engines is some motivation, it does not on its own motivate an RFC. - ## Unresolved questions - Should we allow overlapping atomic groups that don't have a subset-superset relationship? - An atomic group is supposed to make multiple systems look like one system to outside systems. - What do overlapping subsets mean conceptually? -## \[Optional\] Future possibilities +## Future possibilities -- Indexes! -- Relations! -- Automatic flush inference! +- Built-in index primitives + - Reduces boilerplate + - Allows the community to reuse optimized, featureful code +- Automatic inference of index rebuilds + - Increases usability by deduplicating state refreshes without risking stale state From abfcf1ab7f25d035c717ab2fd6b251451ea57d49 Mon Sep 17 00:00:00 2001 From: Alice Cecile Date: Thu, 20 Jan 2022 12:11:27 -0500 Subject: [PATCH 3/8] Rationale --- rfcs/46-atomic-groups.md | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/rfcs/46-atomic-groups.md b/rfcs/46-atomic-groups.md index 2d2dd6a7..1b92a31a 100644 --- a/rfcs/46-atomic-groups.md +++ b/rfcs/46-atomic-groups.md @@ -96,11 +96,17 @@ This reduces the risk of an unsatisfiable schedule by allowing locks to be relea ## Rationale and alternatives -- Why is this design the best in the space of possible designs? -- What other designs have been considered and what is the rationale for not choosing them? -- What objections immediately spring to mind? How have you addressed them? -- What is the impact of not doing this? -- Why is this important to implement as a feature of Bevy itself, rather than an ecosystem crate? +### Why do we want to distinguish between coherent and isolated groups? + +In most cases, coherent groups will be desired. +They ensure correctness while releasing locks sooner. +However, isolated groups have a few advantages: + +- They are conceptually simpler: isolated atomic groups are run "as-if" they are a single system. +- They preserve privacy more strictly and do not leak timing information about their internal components. +- They work correctly in the face of interior mutability. +- Isolated atomic groups can efficiently be run as part of a single task, without communication upstream. + - They release their locks all at once, which can have lower overhead for simple systems. ## Unresolved questions From a228b643fd36594e3b751ad934b9fc32e1ea0e98 Mon Sep 17 00:00:00 2001 From: Alice Cecile Date: Thu, 20 Jan 2022 12:18:02 -0500 Subject: [PATCH 4/8] Editing pass --- rfcs/46-atomic-groups.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/rfcs/46-atomic-groups.md b/rfcs/46-atomic-groups.md index 1b92a31a..84e063d6 100644 --- a/rfcs/46-atomic-groups.md +++ b/rfcs/46-atomic-groups.md @@ -68,8 +68,8 @@ For all systems that are part of the same atomic system group: - When a system in an atomic group completes, its write-locks and read-locks are downgraded to **virtual locks**. - Virtual locks are treated as if they were real locks for all systems outside of the atomic group. - Virtual locks are ignored by systems in the same atomic group. -- If the atomic group is **coherent**, virtual locks are removed once all systems in the atomic group have finished with it. -- If the atomic group is **isolated**, virtual locks are held until all systems in the atomic group complete. +- If the atomic group is coherent, virtual locks are removed once all systems in the atomic group have finished with it. +- If the atomic group is isolated, virtual locks are held until all systems in the atomic group complete. This can be modelled as: @@ -91,8 +91,8 @@ This reduces the risk of an unsatisfiable schedule by allowing locks to be relea ## Drawbacks -- This is a complex feature! -- It could slow down evaluation of the schedule, including in cases where this feature is not used. +- This is a complex feature! It exposes the details of scheduling and locking to the end user. +- Depending on the exact detail of implmentation, it could slow down scheduling, including in cases where this feature is not used. ## Rationale and alternatives @@ -121,3 +121,6 @@ However, isolated groups have a few advantages: - Allows the community to reuse optimized, featureful code - Automatic inference of index rebuilds - Increases usability by deduplicating state refreshes without risking stale state +- Automatic creation of isolated groups for schedule optimization + - Isolated groups effectively allow us to "batch" systems, evaluating them in a single task with lower overhead + - This could be (optionally) inferred, as a schedule optimization From 24e88ae27e9abe8f58fef2f917f38a0abfe221d2 Mon Sep 17 00:00:00 2001 From: Alice Cecile Date: Thu, 20 Jan 2022 12:48:11 -0500 Subject: [PATCH 5/8] Complete index example --- rfcs/46-atomic-groups.md | 99 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 99 insertions(+) diff --git a/rfcs/46-atomic-groups.md b/rfcs/46-atomic-groups.md index 84e063d6..185a3221 100644 --- a/rfcs/46-atomic-groups.md +++ b/rfcs/46-atomic-groups.md @@ -61,6 +61,105 @@ On the other hand, atomic ordering constraints can be helpful when attempting to By guaranteeing that the initial state of your complex system cannot be altered before the later parts of the system are complete, you can safely parallelize the rest of the work into separate systems, improving both performance and maintainability. +### Using atomicity to ensure fresh indexes + +**Indexes** are a common and powerful tool for accelerating working with queries. +By caching the values of components in an optimized resource, we can find components with a particular value (or range of values) without needing to look through every single component. + +Here's a rather hand-waved example of how that might look: + +```rust +#[derive(Component, Hash)] +struct Name(String); + +impl Name { + fn new(str: &str) -> Self { + Name(string.to_string()) + } +} + +// This is our index resource +struct Names { + // This is some HashMap like data structure, that allows for fast lookup in both directions + map: BidirectionalMap +} + +impl Names { + /// Fetches the entity with the given name from our index + fn get(&self, name: &Name) -> Option { + // This can be done in O(1) time, due to the internal hashing + } + + /// Updates the Names index based on the current list of names from the `World` + fn update(&mut self, name_query: Query){ + // This can often be done faster using careful Changed filters + // plus RemovedComponents + } +} + +/// Refreshes the current cache of the index +fn update_names_index(name_query: Query, names: ResMut){ + names.update(name_query) +} + +/// Uses the index to efficiently look up a specific entity +fn eliminate_bavy(names: Res, mut commands: Commands){ + if let Some(bavy_entity) = names.get(Name("Bavy")) { + commands.despawn(bavy_entity); + } +} + +fn main(){ + App::new() + // This ordering is required, in order to ensure that the data in our index is fresh + .add_system(update_names_index.before("eliminate_bavy").label("update_names_index")) + .add_system(eliminate_bavy.label("eliminate_bavy")) + .run(); +} +``` + +However, this strategy is not robust! +Because systems that run "before" other systems are not necessarily run "immediately before", our index could be distorted before it is read from. + +```rust +// This system deliberately breaks our index! +// Curse you Bavy! +fn cunning_ruse(mut name_query: Query<&mut Name>, names: Res){ + // Of *course* Bavy just unwraps the returned option... + let bavy_entity = names.get(Name("Bavy")); + let mut bavy_name = name_query.get_mut(bavy_entity); + bavy_name = "Bevy"; + + let bevy_entity = names.get(Name("Bevy")); + let mut bevy_name = name_query.get_mut(bevy_entity); + bevy_name = "Bavy"; +} + +fn main(){ + App::new() + .add_system(update_names_index.before("eliminate_bavy").label("update_names_index")) + .add_system(eliminate_bavy.label("eliminate_bavy")) + // Oh no! We've despawned Bevy instead!! + .add_system(cunning_ruse.after("update_names_index").before("eliminate_bavy")) + .run(); +} +``` + +However, we can use the power of atomic groups to ensure that our index is fresh when it is read from. + +```rust +fn main(){ + App::new() + // Upgrading the constraint ensure the index is refreshed "just before" it is read + .add_system(update_names_index.atomically_before("eliminate_bavy").label("update_names_index")) + .add_system(eliminate_bavy.label("eliminate_bavy")) + // This causes the compilation to fail + // Drat! Foiled again! + .add_system(cunning_ruse.after("update_names_index").before("eliminate_bavy")) + .run(); +} +``` + ## Implementation strategy For all systems that are part of the same atomic system group: From 8ff65df07619013e3a7c47f86d8f285ae12a56e9 Mon Sep 17 00:00:00 2001 From: Alice Cecile Date: Thu, 20 Jan 2022 12:48:38 -0500 Subject: [PATCH 6/8] Quick note on unsatisfiability --- rfcs/46-atomic-groups.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/rfcs/46-atomic-groups.md b/rfcs/46-atomic-groups.md index 185a3221..1de86f30 100644 --- a/rfcs/46-atomic-groups.md +++ b/rfcs/46-atomic-groups.md @@ -188,6 +188,10 @@ Systems `A` and `C` do not share an atomic group, and data that is virtually loc This reduces the risk of an unsatisfiable schedule by allowing locks to be released in a maximally permissive fashion. +### Unsatisfiability + +This RFC adds an additional satisfiability constraint: if a data access is required during the execution of an atomic group, the requesting system's data access must be compatible with the data accesses of that group at the appropriate time. + ## Drawbacks - This is a complex feature! It exposes the details of scheduling and locking to the end user. From 6ef899b3c4a058b4d9cd93da5738da8ccef2dbe4 Mon Sep 17 00:00:00 2001 From: Alice Cecile Date: Thu, 20 Jan 2022 12:56:27 -0500 Subject: [PATCH 7/8] Use a multimap... --- rfcs/46-atomic-groups.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/rfcs/46-atomic-groups.md b/rfcs/46-atomic-groups.md index 1de86f30..cec75333 100644 --- a/rfcs/46-atomic-groups.md +++ b/rfcs/46-atomic-groups.md @@ -80,8 +80,8 @@ impl Name { // This is our index resource struct Names { - // This is some HashMap like data structure, that allows for fast lookup in both directions - map: BidirectionalMap + // Multiple entities can share the same name + map: MultiMap } impl Names { @@ -104,8 +104,10 @@ fn update_names_index(name_query: Query, names: ResMut){ /// Uses the index to efficiently look up a specific entity fn eliminate_bavy(names: Res, mut commands: Commands){ - if let Some(bavy_entity) = names.get(Name("Bavy")) { - commands.despawn(bavy_entity); + if let Some(bavy_entities) = names.get_vec(Name("Bavy")) { + for bavy in bavy_entities { + commands.despawn(bavy); + } } } From 182dad3eef02d6db048e82ebda09e493fdcdd70e Mon Sep 17 00:00:00 2001 From: Alice Cecile Date: Sun, 6 Feb 2022 23:20:21 -0500 Subject: [PATCH 8/8] More thoughts --- rfcs/46-atomic-groups.md | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/rfcs/46-atomic-groups.md b/rfcs/46-atomic-groups.md index cec75333..9aeed6c8 100644 --- a/rfcs/46-atomic-groups.md +++ b/rfcs/46-atomic-groups.md @@ -190,10 +190,42 @@ Systems `A` and `C` do not share an atomic group, and data that is virtually loc This reduces the risk of an unsatisfiable schedule by allowing locks to be released in a maximally permissive fashion. +### Is atomicity transitive? + +In other words, can we "chain" atomicity between groups in meaningful ways? + +**Yes, both isolated and coherent atomicity are transitive. Mixing the two causes the larger group to become coherent.** + +Suppose that we have two atomic groups `X = {A, B}` and `Y = {B, C}`, where `X` runs before `Y`. +From an outside perspective, does the collection of `X` and `Y` look as though it's atomic? + +Let's walk through what happens. + +1. `X` begins, applying a virtual lock on the data for `A` and `B`. +2. `Y` checks if it can run, checking the locks for `B` and `C`. + 1. If `B` is compatible with itself (as it only reads data), we can immediately begin `Y`. + 2. If `B` is not compatible with itself: + 1. If `X` is coherent, the locks are released as `X` completes. + +### Are atomic groups strictly hierachical? + +**For isolated groups, the answer is "effectively yes". For coherent groups however "surprisingly, no!".** + +This is conceptually important, as it shapes the API users should use to interact with atomic groups. + +Suppose that we have a minimal pair of two atomic groups `X = {A, B}` and `Y = {B, C}`, where `A`, `B` and `C` are distinct systems. +Atomic groups must be strictly hierarchical if-and-only if the schedule is unsatisfiable. + +TODO: Complete. + +### Scheduling + ### Unsatisfiability This RFC adds an additional satisfiability constraint: if a data access is required during the execution of an atomic group, the requesting system's data access must be compatible with the data accesses of that group at the appropriate time. +TODO: Complete. + ## Drawbacks - This is a complex feature! It exposes the details of scheduling and locking to the end user.