[skip ci] Investigation on NSX integration by amandafeng001 · Pull Request #4071 · vmware/vic

amandafeng001 · 2017-02-28T18:28:01Z

hickeng · 2017-03-06T19:32:47Z

doc/design/networking/Investigation-on-NSX-integration.md

+### Create VCH 
+specifying 'bridge-network' as the port group 'vxw-dvs-55-virtualwire-1-sid-5000-logical-switch-1':
+  * I use this VCH to create containers, they are automatically added to this port group (i.e. logical switch)
+  * containerVMs get their IPs in the internal network


These IPs are currently assigned by the VCH - is there any IPAM mechanism embedded in NSX that would allow us to configure automatic DHCP response for a portgroup for example?

This would depends on the NSX configuration and topology. NSX edge can act like a DHCP server, but can only serve interfaces directly connected to it. or NSX Distrubted Logical Router(DLR) can have external DHCP servers via DHCP relay configuration. However, we cannnot configure a DHCP on an ESG and then use the DHCP Relay function from the DLR to get an IP address from the ESG.

hickeng · 2017-03-06T19:33:33Z

doc/design/networking/Investigation-on-NSX-integration.md

+### docker network create network-a
+Our goal:
+  * VMs can communicate with VMs in the same security group.
+  * VMs cannot talk to VMs in a differrent security group.


Are security groups coupled to IP ranges or completely independent?

I would say they are not completely independent.
we can tag a VM to make it belong to one security group. but NSX uses this VM's IP to identify and apply security policies.

hickeng · 2017-03-06T19:37:45Z

doc/design/networking/Investigation-on-NSX-integration.md

+Our goal:
+  * VMs can communicate with VMs in the same security group.
+  * VMs cannot talk to VMs in a differrent security group.
+  * one VM to be added to more than one security group.


Is it possible to associate a NIC with a security group, or just VMs? I'm asking for the case where a containerVM is attached to a container-network to publish a service. In that case the container-network NIC may well need to take traffic from any source, but the other NIC on the container that's attached to an internal network for communication with other containers would be Policy to Policy (if I'm reading this right).

Yes. We can associate a NIC with a security group by configuring dynamic membership as entity belong-to <the vNIC>.
I just experimented with VMs with multiple vNICs. The policy on one vNIC does not affect another vNIC.
If I understand correctly, you mean a containerVM may be attached to some external container-network and also have an identity in VIC created internal network? I will need to verify if it works in this case.

hickeng · 2017-03-06T20:39:47Z

doc/design/networking/Investigation-on-NSX-integration.md

+     * POST /2.0/services/ipset/{scopeMoref}
+  * or create Security Tag
+     * POST /2.0/services/securitytags/tag
+  * create security group withe IP Sets or Security Tag as members


withe --> with

thanks. will update!

hickeng · 2017-03-06T20:41:51Z

doc/design/networking/Investigation-on-NSX-integration.md

+  * list all the security groups
+    * GET /2.0/services/securitygroup/internal/scope/{scopeId}
+  * filter security group names by `<vch name>-SecurityGroup` prefix
+  * or if we keep KV pairs of each created network, then we just need to use that to list all the networks


The infrastructure is supposed to be the source of truth as this allows a degree of dual-mode management. So, pulling from the infrastructure would be preferred.

Thanks. Get it. I will update the document.

hickeng · 2017-03-06T20:43:25Z

doc/design/networking/Investigation-on-NSX-integration.md

+
+## State Storage
+  * we can map a docker network name to a security group name as `<vch name>-SecurityGroup-<network name>`
+  * we can store KV pairs for each created network and then everything else will be in the NSX control plane


what state needs to be stored outside of NSX? One obvious possibility is the IPAM configuration, but worth being explicit.

sorry I was not clear here. I meant storing some key/value pair like:
[key]: docker network name, [value]: information of the related objects created on NSX side.
can help fetching network info faster.
but as your previous comment mentioned, pulling from infrastructure would be preferred. Then no states need to be stored outside of NSX.

hickeng · 2017-03-06T20:49:00Z

doc/design/networking/Investigation-on-NSX-integration.md

+## Unclear / need discussion:
+### Security Tag v.s IP Sets
+  * Security Tag we only need to add the tag to a VM then it is added to a security group, which makes adding VMs to multiple security groups convinient. We do not need to update security groups everytime we create a new containerVM in a network.
+  * With IP Sets, we need to add the VM's IP to an IP Set. The downside is that 1. it requires the VM's IP to be static. 2. if we have more than one vch in the environment, containerVMs' IPs overlap, we cannot allow one VM in multiple security groups. 3. it may limit the number of VMs added to an IP Set (how many IPs can be added one IP set?). 4. everytime adding a new containerVM in a network, we need to add its IP to the IP set.


Please reflow the numbered points so they render as a list.

Consideration of security tag:

Seems to apply to a VM instead of an interface on a VM - does this allow for different policies per vNIC?

Other possible considerations of IPs:
5. currently we only assign an IP to a VM when it is powered on so VMs wouldn't have a security group membership when powered off (meaning possible issues with out-of-band power on)
6. no way to prevent a container changing it's IP address

Thanks Geroge. This is very important input.
Actually at the point when I sent this PR, security tag did not work on VIC. that's why I was considering IP Sets at that time.
After Doug's change #4057, security tag works on VIC.
But associating security group with security tag/VM may not work for the case you mentioned, where a container VM may have an external and internal identities.
I will investigate more on associating security group with vNIC and update the document.

hickeng · 2017-03-06T20:50:11Z

doc/design/networking/Investigation-on-NSX-integration.md

+  * all the user cases mentioned in the networking design document should still be valid or improved
+
+### To support both cases: with or without NSX?
+  * need to find out how to verify if a bridge network specified is a logic switch?


bridge network is currently specified as a portgroup

yes. I meant, if we want to support both cases:

user has NSX setup.

user does not have NSX setup.
then we need a way to figure out if the port group here is an NSX created logical switch or just a regular port group.
I think we can search for the sid- inside the name of the port group (for example, a logical switch name may be like this vxw-dvs-55-virtualwire-12-sid-5010-logical-switch-23)
but this approach may give false positives if someone creates a regular port group name as vxw-dvs-55-virtualwire-12-sid-1

hickeng · 2017-03-06T20:53:47Z

doc/design/networking/Investigation-on-NSX-integration.md

+
+### User management
+  * NSX allows RBAC, we can add Vcenter users and assign roles for them
+  * to my understanding, in terms of using NSX for 'docker network create', we will not need RBAC controls? Unless if we want to restrict users from creating networks, otherwise we may not need RBAC at this point?


It depends what else can be done via the NSX management connection.
We started using admin users for VC and as a result have some awkward usages and have had to experimentally determine what privileges we need for what operations.

It's much easier to start with zero privileges while developing and document the addition as they are needed.

The other related question is can NSX use vCenter users for authZ or do we need to add a capability to add/specify an NSX user.
If the latter then we may need two users, one that can create the bridge network for vic-machine, the other to create security groups on it for endpointVM operations.

amandafeng001 · 2017-03-07T04:32:38Z

@hickeng , Thanks for the comments.
I'll investigate on associating vNIC with security group for VIC, and update the document soon.

anchal-agrawal

Suggestions for typos.

anchal-agrawal · 2017-03-08T20:52:53Z

doc/design/networking/Investigation-on-NSX-integration.md

+
+
+## A Working Installation of NSX-v
+For VMWare engineers using Nimbus, here is a document about how to install and configure NSX with Vsphere: How to set up VM-to-VM communication on NSX logical switch with Nimbus


s/VMWare/VMware

s/Vsphere/vSphere

anchal-agrawal · 2017-03-08T20:53:12Z

doc/design/networking/Investigation-on-NSX-integration.md

+Let us use this example to explain the workflow.
+
+### Prerequisite
+We assume VSphere admin already has NSX installed and configured on VCenter. That means there is an NSX logical switch created by the admin, which automatically creates a port group on distributed switch.


s/VSphere/vSphere

s/VCenter/vCenter

anchal-agrawal · 2017-03-08T20:54:39Z

doc/design/networking/Investigation-on-NSX-integration.md

+      * vCenter Objects: VMs, Distributed Switches, Clusters, etc.
+      * VM Properties: vNICs, VM names, VM operating Systems, etc.
+      * NSX Objects: Logical Switches, Security Tags, Logical Routers, etc. 
+    In this document, we propose to use vNICs as the grouping creteria. We will dicuss more on this later.


s/creteria/criteria

s/dicuss/discuss

anchal-agrawal · 2017-03-08T20:56:36Z

doc/design/networking/Investigation-on-NSX-integration.md

+
+![NSX-v multi-tenancy] (pics/nsx-multi-tenants.png)
+
+It worths to mention that we would expect only one VCH is created for one tenant. Otherwise there IP overlapping would issue. 


We can replace this with something like:

"It is worth to mention that we would expect only one VCH to be created for one tenant. Otherwise IP overlapping would occur."

anchal-agrawal · 2017-03-08T20:57:15Z

doc/design/networking/Investigation-on-NSX-integration.md

+  * need to find out how to verify if a bridge network specified is a logic switch?
+
+### User Management
+  * NSX allows RBAC, we can add Vcenter users and assign roles for them


s/Vcenter/vCenter

Thanks Anchal. I will update!

anchal-agrawal · 2017-03-08T20:57:28Z

doc/design/networking/Investigation-on-NSX-integration.md

+    * Pros:
+      * It serves our goals.
+    * Cons:
+      * Everytime a containerVM is created, we need to update related security groups' by adding one more membership criterias. Not sure the maximum number of membership criterias NSX-v supports. 


s/criterias/criteria

fdawg4l · 2017-03-20T17:48:18Z

doc/design/networking/Investigation-on-NSX-integration.md

+Let us use this example to explain the workflow.
+
+### Prerequisite
+We assume vSphere admin already has NSX installed and configured on vCenter. That means there is an NSX logical switch created by the admin, which automatically creates a port group on distributed switch.


I'm just curious; can we create the port group for the user in the installer?

Yes. We can do that. As long as NSX has been installed, in the installer we can make API calls to NSX to create logical switch, which will create corresponding port group :)

fdawg4l · 2017-03-20T17:51:32Z

doc/design/networking/Investigation-on-NSX-integration.md

+
+### docker run --name=containerVM1 <image name>
+  * VIC create containerVM1
+  * NSX manager creates security group `<vch name>-SG-bridge` with dynamic membership: NIC-1 (of containerVM1)


Is the intent that we create a bridge security group for every container?

Sorry for the confusion here. I mean we create a security group for each container network. And then everytime we add a container (say with NIC-1) to that network, we update the security group by adding a criteria of 'entitiy equals to NIC-1'. This is how the container's NIC-1 associated to the security group.

fdawg4l · 2017-03-20T17:54:08Z

doc/design/networking/Investigation-on-NSX-integration.md

+https://{nsxmanager's hostname or IP}/api 
+
+### API calls
+(details in http://pubs.vmware.com/nsx-63/topic/com.vmware.ICbase/PDF/nsx_63_api.pdf , note that many NSX API methods reference vCenter object IDs in URI parameters, query parameters, request bodies, and response bodies, also it takes XML as request body (sad) )


LOL @ "sad". Is it SOAP by chance? If so, you could probably reuse some of the client code in govmomi to do the wsdl stuff.

it is a REST API... I think the plan is to add code in govmomi. Then make calls to govmomi from port layer. I'm not familiar with wsdl..what is it? and do you think at least I can use it to handle the xml part?

fdawg4l · 2017-03-20T17:56:04Z

doc/design/networking/Investigation-on-NSX-integration.md

+### Adding a ContainerVM to Multiple Networks
+This is not supported in current VIC implementation. 
+  * Is adding another NIC to a containerVM the right way to do it?
+  * Does having multiple interfaces matter ? How would user use multiple NICs?


I feel like this overlaps with the "scopes" work @hmahmood did. @hmahmood do we add a new scope for a new set of containers that doesn't exactly fit the smaller set or does the scope get extended?

Basically my question is whether or not the semantic is different than we currently have with scopes.

If we go with the Security Group with vNIC association approach, we can build it on top of what we already have. I think it is loosely like this: scope == docker network versus scope + security group == docker network.
The scopes will still be there. We add security groups on top. The difference is when we add a container to another network, we need to create another vNIC to give it an identify in the new network. Would love to hear @hmahmood 's input on this.

Currently it is possible to add a container to multiple networks. A new vNIC is added if there is no NIC that is currently connected to the port group in question. It sounds like scope == security group == docker network to me.

fdawg4l · 2017-03-20T17:58:07Z

doc/design/networking/Investigation-on-NSX-integration.md

+      * We only need to tag a VM then it is added to a security group. 
+      * we do not need to update security groups everytime a new containerVM is created. 
+    * Cons:
+      * It seems at the backend of NSX, IP address is used to identify the membership of a security group. So it does not work in this scenario, for example, when there are two port groups on the same VDS and each of them have a VIC (say VIC-a and VIC-b) created. Then the security group and policy on VIC-a aslo affects VIC-b. (One NSX-v tenant is not allowed to have overlapping IPs. But VIC users might be the same tenant (if our understanding of a NSX tenant is correct).)


RE: IP mapping to SG. I'm really surprised by this. You can't tag by something more unique and immutable like mac or MoRef?

I actually tried to use mac to associate security group. Turns out it eventually still uses IP to identify membership. I found only with vNIC, security group membership can be correctly identified even with overlapping IP.
I asked a NSX-v expert, he said "As dataplane is an IP classification engine, all the members of the security group will be converted to the IP addresses before sending them down to the ESXi host."

amandafeng001 · 2017-05-08T20:46:36Z

This work is paused.

vmwclabot added the cla-not-required label Feb 28, 2017

hickeng suggested changes Mar 6, 2017

View reviewed changes

amandafeng001 force-pushed the NSX-integration branch from 06ec3e2 to b94895a Compare March 8, 2017 07:14

anchal-agrawal reviewed Mar 8, 2017

View reviewed changes

amandafeng001 added 10 commits March 9, 2017 16:18

Investigation on NSX integration

477dfde

small typo

23e6f02

update: use IP Sets instead of security tag

ef29706

typo

9d31fdf

update

93004c5

update

bfb8751

update - using vNIC as membership criteria

783d5dd

typo

55d75da

update with approach comparison

c0dd69d

fix typo and reformat chart

650f399

amandafeng001 force-pushed the NSX-integration branch from 40231de to 650f399 Compare March 10, 2017 00:19

amandafeng001 mentioned this pull request Mar 14, 2017

Investigate the compatibility of containerVM with NSX #4097

Closed

fix graph display

75add66

fdawg4l reviewed Mar 20, 2017

View reviewed changes

amandafeng001 closed this May 8, 2017



		## A Working Installation of NSX-v
		For VMWare engineers using Nimbus, here is a document about how to install and configure NSX with Vsphere: How to set up VM-to-VM communication on NSX logical switch with Nimbus


		![NSX-v multi-tenancy] (pics/nsx-multi-tenants.png)

		It worths to mention that we would expect only one VCH is created for one tenant. Otherwise there IP overlapping would issue.

Conversation

amandafeng001 commented Feb 28, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amandafeng001 Mar 7, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amandafeng001 Mar 7, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amandafeng001 Mar 7, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amandafeng001 commented Mar 7, 2017

Uh oh!

anchal-agrawal left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amandafeng001 Mar 20, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

amandafeng001 commented Feb 28, 2017 •

edited

Loading

amandafeng001 Mar 7, 2017 •

edited

Loading

amandafeng001 Mar 7, 2017 •

edited

Loading

amandafeng001 Mar 7, 2017 •

edited

Loading

amandafeng001 Mar 20, 2017 •

edited

Loading

fdawg4l Mar 20, 2017 •

edited

Loading

amandafeng001 Mar 20, 2017 •

edited

Loading