feature request: Migrate NSX security tags together with VM's

[martijnbruins]

When migrating a virtual machine that has an NSX (-V or- T) connection, also migrate the NSX security tags with each VM
And set these as user.xyz=abc instance metadata.

As this metadata will be used for migrating NSX fw rules to incus + OVN

nxs-v security tags are single values, so these values could be combined as user.nsxv=prod|||swift|||icto-12345 ( example ||| separator)

nsx-t security tags are value pairs, so these valuepairs could be combined as user.nsxt=env^^^prod|||app^^^swift|||appcode^^^icto-12345 (example ( ^^^= and ||| separator, eg. env=swift , app=swift nsx-t security tags)

This will enable any future NSX microsegmentation migrations to Inucs

[stgraber]

@masnax I thought we had generic logic to carry all the custom attributes and tags from VMs and apply those to the migrated instance?

@martijnbruins where are those set exactly within VMware? Are they regular tags and custom attributes on the VMs or are they hiding elsewhere?

[masnax]

Looking at what we do now, we record all tags from VMware under tag.{category}={tag1},{tag2},{tag3} and all attributes as attribute.{key}={value}

They're currently used primarily for filtering in MM at the moment, but we can easily add them as user. keys to Incus.

The NSX API documentation (https://developer.broadcom.com/xapis/nsx-t-data-center-rest-api/latest/types_VirtualMachine.html) idicates NSX itself has some tags set on its internal VM definition. We do already record these properties when an NSX type network is imported from a vmware source, and its matching NSX source is registered in MM as an nsx source.

We currently don't do anything with this information, but can record these keys on Incus migrated VMs as user.network.{segment}.tag.{scope}={tag1},{tag2},{tag3} or something similar.

[martijnbruins]

in vmware you have:

1. vcenter tags: not often used for automation
2. NSX security tags, practically ALWAYS used for defining "intent based light" microsegmentation in NSX

It is the NSX -tags i am referring to, as safeguarding these during migration per VM will offer the possibilities of migrating a NSX microsegmentation to incus/OVN,

There is a difference between NSX-V (old) and NSX-T (current) , in the way that NSX-V are single values and NSX-T are value pairs

This is also why I propose this very particular mapping to user.metadata per instance

nxs-v security tags are single values, so these values could be combined as user.nsxv=prod|||swift|||icto-12345 ( example ||| separator)

nsx-t security tags are value pairs, so these valuepairs could be combined as user.nsxt=env^^^prod|||app^^^swift|||appcode^^^icto-12345 (example ( ^^^= and ||| separator, eg. env=swift , app=swift nsx-t security tags)

IMO the network segment in the name as in " user.network.{segment}.tag.{scope}={tag1},{tag2},{tag3}" will not be relevant after migration anymore, as its ip address is leading in all. You can migrate the segment IDs, etc., but I would put them in a separate user metadata value for later reference.
Listing the NSX segment id's here in the naming convention will only make any automation more difficult for migrating an existing NSX micro-segmentation
An overal uniform user.nsxt (or user.nsxt-mgration=, whatever chosen) and then the format i suggest will keep an uniformous possibility to reinstate the microsegmentation of NSX customer in a similar fashion via automation.
This is also why I proposed two formats for NSX-V and NSX-T, one being single values and the other one being value pairs.

[stgraber]

For single values, we should just comma separate in the same key (user.nsxv=foo,bar,baz).
For key/value, we should just put the key in the key name (user.nsxt.foo=bar).

That's in line to how we normally handle tags (single value) and custom properties (key/value pairs).

[stgraber]

Actually, thinking about this a bit more. The NSX connection happens at the NIC level. So a VMware VM could be attached to 3 different segments, each with their own tags, right?

Would then probably make more sense to apply those user.nsxt and user.nsxv to the NIC device in question rather than globally to the instance, correctly allowing it to have different tags per segment/network it's attached to.

[masnax]

Yeah, we won't need to expose the segment ID (or alternatively the source network name) if we add the tags directly to the NIC device. I had forgotten that devices have user keys too, that's convenient :)

One slight annoyance (this is actually true for regular vCenter attribute scopes as well) is that the NSX-T tag key (or scope) seems to be functionally optional (also mentioned in the API spec). Which means we would need to be opinionated about a default scope name (something like default or global), which has the risk of clashing with an actual defined scope.

I see two alternatives, one is we use 3 key formats (user.nsxv=tags, user.nsxt=tags, user.nsxt.scope=tags), or that we add some default scope override per network. So far we've been opinionated about this (vCenter VM attribute scopes default to global if unset, and we use comma delimiters for values).

Another one is if we really want to directly expose nsxt and nsxv to Incus, given that the actual network is going to be OVN, and not NSX. Similarly, should we explicitly differentiate between nsxt with an empty scope, versus nsxv?

[stgraber]

Yeah, I think just putting the scope-less nsxt tags as user.nsxt is fine, that avoids the potential clash.

And I agree with your last point too. We don't carry all the VMware tags and custom properties as user.vmware.XYZ because the whole point of migration manager is to migrate things, though at the same time, not losing configuration data that could be useful.

Given that we'd be putting the tags as NIC config and not global, it could be as simple as user.tags on the NIC device with it being mapped as user.tags=foo,bar,baz for either NSX-V or for the scope-less NSX-V tags and user.tags.FOO=BAR for the scoped NSX-T tags.

[martijnbruins]

NSX-T security tags can be applied at two different scopes,
Scope - What it attaches to- Granularity
VM tag (the common case) - The virtual machine object in the NSX inventory - The whole VM — all of its vNICs
Segment Port tag - A single logical/segment port = one vNIC connection - A single vNIC

The default / 99%-of-the-time path is VM-level at customer NSX-T DFW microsegmentation implementations.
This is why i proposed this format:

nxs-v security tags are single values, so these values could be combined as user.nsxv=prod|||swift|||icto-12345 ( example ||| separator) (instance metadata)

nsx-t security tags are value pairs, so these valuepairs could be combined as user.nsxt=env^^^prod|||app^^^swift|||appcode^^^icto-12345 (example ( ^^^= and ||| separator, eg. env=swift , app=swift nsx-t security tags) (instance metadata)

As this will then be very easy to consume for any automation to reproduce same microsegmentation results as the migrated NSX.

[stgraber]

Okay, so for NSX-T, the segment port tags should become user.tags on the NIC as described above.
The VM tags should go to user.network.tags instead with the format suggested above.

When dealing with key/value pairs, we use user.tags.FOO=BAR (NIC) or user.network.tags.FOO=BAR (VM).
When dealing with single values, we use user.tags=foo,bar (NIC) or user.network.tags=foo,bar (VM).

That should correctly handle both the port-specific and VM-wide tags with both formats (single value or key/value pairs) and keep the naming generic.

Standard Incus separators are , or , so that's what we'd use here. When dealing with key-value, we use sub-keys as shown above. ^ and | are separators we've never used within Incus, on purpose, as they both as special meanings in shell scripting making it likely that a user would miss escaping them and get some very unfortunate behavior.

[martijnbruins]

As per Stephane:
"For single values, we should just comma separate in the same key (user.nsxv=foo,bar,baz).
For key/value, we should just put the key in the key name (user.nsxt.foo=bar).

That's in line to how we normally handle tags (single value) and custom properties (key/value pairs)."

Yes that makes perfect sense in a logical world, but your "opponents" here are 2012-2022 corporate vmware administrators ;-)
(E.g. in Dell landscape i saw plenty usage of comma's and "=" in NSX security tag values :-( (I know) )
And that is exactly why i came up with the ||| and ^^^ separator values in my proposed format, to prevent killing the logic with such shenanigans.

So if we could do it like this:
nxs-v security tags are single values, so these values could be combined as user.nsxv=prod|||swift|||icto-12345 ( example ||| separator)

nsx-t security tags are value pairs, so these valuepairs could be combined as user.nsxt=env^^^prod|||app^^^swift|||appcode^^^icto-12345 (example ( ^^^= and ||| separator, eg. env=swift , app=swift nsx-t security tags)

Then that will prevent separator syntax hell and also clearly document if it's an NSX-V or an NSX-T migration in a very identifiable and unique metadata value (nsxv or nsxt).

And that value is then is purely only for a (potential) NSX Dynamic security tags microsegmentation migration. In one single instance metadata value that can be easily decommissioned if/when not needed anymore.

After the migration to the destination Incus environment customer can do it properly with proper tag syntax

[stgraber]

I don't think that makes sense. Migration Manager is a migration tool, its whole point is to convert from one ecosystem to another.

Incus has been rolling out more and more restrictions on special values within its configuration and those kind of separators are exactly what we've had to disable in various places due to security issues, I would strongly recommend against their use in Incus configuration.

Migration Manager has a variety of webhooks and scriptlets which can be used to access additional source data and mangle things if needed. Here we're talking about the default clean solution which should feel native to the target platform.

Tags including "=" are not a problem with what I've suggested above as that doesn't have any special meaning. The only problem here is with tags including commas but that's only relevant for single-value tags, so only for NSX-V.

One easy way to cleanly avoid that particular problem would be to turn them into key/value instead.

NSX-T with tags ["foo,bar,baz", "foo", "bar"] would turn into:

• user.network.tags.0 = foo,bar,baz
• user.network.tags.1 = foo
• user.network.tags.2 = bar

That effectively eliminates the suggested comma separate storage of NSX-T tags and moves both NSX-T and NSX-V to either user.network.tags.KEY or user.network.tags.N in the VM-wide case, or user.tags.KEY and user.tags.N in the segment-specific case.

[martijnbruins]

Do what you guys think is best for Incus of course.
Any post-migration logic that consumes these migrated NSX security tags to replicate teh microsegemtation to OVN can be amended.