Sync dependencies to Redis

[yhabteab]

As the title of the PR already implies, this is mainly about serialising the in-memory representation of all checkable dependencies according to the specifications given by Icinga/icingadb#347 (comment) and dumping them into Redis so that they can be processed further and written to the database by Icinga/icingadb#889. Furthermore, it should be noted that, until this point, redundancy_groups were merely an additional flag on the actual Dependency objects, but for Icinga 2 itself they didn't represent any kind of object or state and there was no grouping involved at all as one might expect from its name. However, in order to facilitate dependency management and serialisations, redundancy groups now represent a concrete Icinga 2 object. The inline comment in the code describes how and what it exactly represents, but I'll just copy and paste it here with a bit of formatting adjustment.

A DependencyGroup now represents a set of dependencies that are somehow related to each other. Specifically, a DependencyGroup is a container for Dependency objects of different Checkables that share the same child -> parent relationship config, thus forming a group of dependencies. All dependencies of a Checkable that have the same redundancy_group attribute value set are guaranteed to be part of the same DependencyGroup object, and another Checkable will join that group if and only if it has identical set of dependencies, that is, the same parent(s), same redundancy group name and all other dependency attributes required to form a composite key. The composite key, consisting of the Dependency parent checkable, the time period (if configured), the states filter and the ignore_soft_states attributes, uniquely identifies the DependencyGroup object to which it belongs.

More specifically, let's say we have a dependency graph like this:

graph BT;
pp1((PP1));
pp2((PP2));
p1((P1));
p2((P2));
c1((C1));
c2((C2));
rg1(RG1);
p1-->pp1;
p2-->pp2;
rg1-->p1;
rg1-->p2;
c1-->rg1;
c2-->rg1;

Loading

The arrows represent a dependency relationship from bottom to top, i.e. both C1 and C2 depend on their RG1 redundancy group that depends from P1 and P2, and P1 and P2 depend each on their respective parents (PP1, PP2 - no group). Now, as one can see, both C1 and C2 have identical dependencies, that is, they both depend on the same redundancy group RG1 (they might have been constructed via some Apply Rules). So, instead of having to maintain two separate copies of that dependency graph as it's the case with the master branch, we can bring that imaginary redundancy group into reality by putting both P1 and P2 into the new DependencyGroup object. However, we don't really put P1 and P2 objects into that group, but rather the actual Dependency objects of both child Checkables. Therefore, the group wouldn't just contain 2 dependencies, but 4 in total, i.e. 2 for each child Checkable (C1 -> {P1, P2} and C2 -> {P1, P2}). This way, both child Checkables can just refer to that very same DependencyGroup object.

However, since not all dependencies are part of a redundancy group, we also have to consider the case where a Checkable has dependencies that are not part of any redundancy group, like P1 -> PP1. In such situations, each of the child Checkables (e.g. P1, P2) will have their own (sharable) DependencyGroup object just like for RGs. This allows us to keep the implementation simple and treat redundant and non-redundant dependencies in the same way, without having to introduce any special cases everywhere. So, in the end, we'd have 3 dependency groups in total, i.e. one for the redundancy group RG1 (shared by C1 and C2), and two distinct groups for P1 and P2.

With this structure in hand, Icinga DB can now easily serialise them for each checkable if it needs to. As previously stated, a single DependencyGroup object may be shared by multiple checkables, and Icinga DB must only serialise such group once. To achieve this, Icinga DB now caches all the already processed DependencyGroup objects in the new m_DumpedGlobals#DependencyGroup for each initial config dump and resets it when it is complete.

The process of serialisation on its own works as follows, and each part of the results is dumped into Redis using the appropriate Redis keys. There're now 5 new Redis keys introduced mainly for this purposes.

• icinga:dependency:node contains dependency node data representing each host, service, and redundancy group in any given dependency graph.
• icinga:dependency:edge contains information representing all connections between the dependency nodes.
• icinga:redundancygroup contains the all the redundancy group data (redundancy_group database table) of any dependency graph.
• icinga:redundancygroup:state represents state information for all redundancy groups.
• icinga:dependency:edge:state likewise contains state information for (every) dependency edge in a dependency graph. It is noteworthy that multiple edges may share the same state, a decision that is made by the serialisation/grouping process.

Please note, as per Icinga/icingadb#347 (comment) only Checkables that are part of some dependency graph should be serialised. That's they either depend on other Checkables or others depend on it, checkables without any child or parent dependencies and Service objects with only implicit dependencies to their hosts are ignored. So, firstly, it produces dependency node entries according to the new Icinga DB schema for a given checkable. In cases where the checkable is also part of a redundancy group (note redundancy group not just dependency group), each of its redundancy groups including their state information will also be serialised according to the Icinga DB (Go) schema spec. Subsequently, all the relevant edges are extracted from that checkable and also dumped into Redis (for more details, please refer to the inline comments). It is important to note that there are some special cases to be aware of, as of today Icinga 2 allows you to have n Dependency objects pointing from C1 -> P1 as long as they have different names. However, in order to render these in Icinga DB Web in a meaningful way, we needed to somehow reduce all the duplicate edges connecting C1 and P1 to just a single one in the database. Consequently, during the serialisation process, Icinga DB will be using all their short names separated by comma as dependency_edge.display_name, and querying the state of each of them and selecting only the severed one. This approach did not require us to introduce a breaking change in Icinga 2 itself, prohibiting the use of more than one dependency objects referencing the same parent, but solved it in a more convenient way.

Last but not least, as part of the restructured dependency management, a bunch of existing issues have been solved automatically, which are listed below.

Well, that's it! This pretty much describes in general what this PR is all about, but you'll find more implementation-specific details as inline comments in the respective code blocks.

fixes #10158
fixes #10190
fixes #10227
fixes #10014
fixes #10143

[julianbrost]

I'm somewhat confused by the DependencyGroup class as it doesn't really map to the mental model I had from our discussions on that topic.

So in my understanding, a DependencyGroup would represent a set a set of checkables that are used as parents in dependency config objects combined with the attributes from that dependency object that affect how the availability of that dependency is determined, i.e. ignore_soft_states, period, and states. For dependencies without a redundancy group, that information is all that's needed to determine if all dependency objects that share the parent and these attribute mark all their children as unreachable. With a redundancy group, you have to look at all the parents from the redundancy group with the three aforementioned additional attributes. So that would be how to determine what creates a DependencyGroup object for redundancy groups.

For dependencies without a redundancy group, this grouping provides no value in itself, the dependency objects can be considered individually. There are two reasons why we might instantiate such trivial groups explicitly nonetheless: for one, it may allow simpler code by being able to treat both cases consistently, but more importantly, there was a request from Johannes that if two children depend on the same parent in such a way that the state of these dependencies is always the same (i.e. the three aforementioned attributes are identical), then the different graph edges should refer to the same shared state. These groups may be used for this deduplication as well.

Consider the following example (P = parent checkable, RG = redundancy group as represented in the generated graph, C = child checkable):

graph BT;
p1((P1));
p2((P2));
p3((P3));
c1((C1));
c2((C2));
c3((C3));
c4((C4));
c5((C5));
rg1(RG1);
c1-->rg1;
c2-->rg1;
c3-->rg1;
rg1-->p1;
rg1-->p2;
c4-->p3;
c5-->p3;

Loading

Here I'd expect the creation of the following two DependencyGroups (... refers to the three magic attributes attached to the parent in the corresponding dependency objects):

• {(P1, ...), (P2, ...)}: This basically represents RG1
• {(P3, ...)}: This is a if there was an imaginary second redundancy with only one parent, P3.

[julianbrost]

I've written a few inline comments regarding the term "member" but stopped at some point. Can you please go over all newly introduced things named that way and consider making the naming a bit more clear by changing that towards something like children/dependencies/parents?

[julianbrost]

Oh damn it, I've completely missed something so far: for Icinga DB, each row must be completely "described" by their ID (that is, if a value in any column changes, that yields a new ID and deletes the old row) or needs a checksum (see properties_checksum) columns in existing tables in the schema.

[julianbrost]

Code-wise fine for me now, however, the PR description is quite outdated now and also references commits no longer part of the PR at all.

Apart for that, for merging, this has to be coordinated with Icinga/icingadb#889, best way would probably be the following:

• Merge this
• Rerun the tests in Sync dependencies icingadb#889 after the new Docker images were pushed
• Merge Sync dependencies icingadb#889

Keep in mind that Icinga/icingadb#889 shouldn't be merged right now (see Icinga/icingadb#889 (review)).

[yhabteab]

the PR description is quite outdated now and also references commits no longer part of the PR at all.

Done ✔️!

[julianbrost]

Keep the merge order requirements from #10290 (comment) in mind (in particular: only start merging after Icinga DB v1.3.0 was released).