Skip to content
This repository was archived by the owner on Dec 17, 2022. It is now read-only.

Commit a75b971

Browse files
mpuncelmpuncel
authored
Add v3 xDS support (envoyproxy#140)
* Add v3 xDS support This commit adds support for the v3 xDS transport as well as support for storing and serving v3 resources out of the resource cache. Note that resource versions and xDS transport version are decoupled to provide a migration path to v3. End users can decide to generate v3 resources and serve them over v2 transport during migration, or continue to generate v2 resources and serve them over v3 transport. For implementation simplicity, it is not possible to generate a combination of v2 and v3 resources in the control plane, which would require either up or down conversion. Signed-off-by: Michael Puncel <[email protected]> * PR comments: move type URLs to their own static class. Fix broken javadoc link. Use method overloading instead of different names for v2 vs v3 node hash Signed-off-by: Michael Puncel <[email protected]> * PR comments: use package names to differentiate v2 and v3 cache types Signed-off-by: Michael Puncel <[email protected]> * put tests in v2 and v3 packages Signed-off-by: Michael Puncel <[email protected]> * rename v2 onStreamRequest callback, remove default implementation Signed-off-by: Michael Puncel <[email protected]> * rename onStreamRequest on DiscoveryServerCallbacks Signed-off-by: Michael Puncel <[email protected]> * DRY most of the DiscoveryServerCallbacks implementations in integration tests Signed-off-by: Michael Puncel <[email protected]> * make v3 transport integration tests also use v3 resource version Note that there are not tests that cover a different transport version from resource version in the interest of not doubling the number of integration tests. Signed-off-by: Michael Puncel <[email protected]> * update README and add v2 -> v3 guide Signed-off-by: Michael Puncel <[email protected]>
1 parent df5fac6 commit a75b971

File tree

60 files changed

+5371
-782
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

60 files changed

+5371
-782
lines changed

‎README.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,12 @@ in [data-plane-api](https://github.com/envoyproxy/data-plane-api). It started li
77
[go-control-plane](https://github.com/envoyproxy/go-control-plane), but building an idiomatic Java implementation is
88
prioritized over exact interface parity with the Go implementation.
99

10+
Both v2 and v3 resources as well as transport versions are supported. Migrating
11+
to v3 is recommended as Envoy will drop v2 support at EOY 2020 (see
12+
[API_VERSIONING.md](https://github.com/envoyproxy/envoy/blob/4c6206865061591155d18b55972b4d626e1703dd/api/API_VERSIONING.md))
13+
14+
See the (v2-to-v3 migration guide)[V2_TO_V3_GUIDE.md] for an exmplanation of migration paths.
15+
1016
### Requirements
1117

1218
1. Java 8+

‎V2_TO_V3_GUIDE.md‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
# Migrating from xDS v2 to v3
2+
3+
To faciliate migrating from the v2 xDS APIs to v3, this repo supports both the
4+
v2 and v3 gRPC transports, with each transport supporting type URL rewriting of
5+
DiscoveryResponse to whatever version the client requests with
6+
api_resource_version.
7+
8+
The migration requires care - for example, using v3-only fields too soon or trying to use
9+
deprecated v2 fields too late can cause Envoy to reject or improperly apply config.
10+
11+
### Recommended Sequence
12+
13+
This section assumes you have sufficient control over Envoy sidecar versions that you do
14+
not need to run v2 and v3 simultaneously for a long migration period.
15+
16+
1. Make sure your oldest Envoy client supports final v2 message versions.
17+
2.
18+
1. Ensure your control plane is not using any deprecated v2 fields.
19+
Deprecated v2 fields will cause errors when they are translated to v3
20+
(because deprecated v2 fields are dropped in v3).
21+
2. Configure a V3DiscoveryServer alongside the V2DiscoveryServer in your
22+
control plane. You can (and should) use the same (v2) Cache implementation
23+
in both servers.
24+
3. Deploy all Envoy clients to switch to both the v3 transport_api_version and
25+
resource_api_version in each respective xDS configs. As this happens, the V3DiscoveryServer
26+
will be translating your v2 resources to v3 automatically, and the V2DiscoveryServer will
27+
stop being used.
28+
4.
29+
1. Rewrite your control plane code to use v3 resources, which means using
30+
V3SimpleCache (if you use SimpleCache). You may now start using v3-only
31+
message fields if you choose.
32+
2. Drop the V2DiscoveryServer.
33+
34+
### Alternative
35+
36+
Another possible path to the one above is to switch to generating v3 in the
37+
control plane first (e.g. by using V3SimpleCache) and then deploying Envoy
38+
clients to use v3 transport and resource versions.
39+
40+
This approach requires care to not use new V3-only fields until the client side
41+
upgrade is complete (or at least understand the consequences of doing so).

‎cache/src/main/java/io/envoyproxy/controlplane/cache/Cache.java‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,5 @@
11
package io.envoyproxy.controlplane.cache;
22

3-
import io.envoyproxy.envoy.api.v2.core.Node;
43
import java.util.Collection;
54
import javax.annotation.concurrent.ThreadSafe;
65

@@ -11,13 +10,13 @@
1110
public interface Cache<T> extends ConfigWatcher {
1211

1312
/**
14-
* Returns all known {@link Node} groups.
13+
* Returns all known groups.
1514
*
1615
*/
1716
Collection<T> groups();
1817

1918
/**
20-
* Returns the current {@link StatusInfo} for the given {@link Node} group.
19+
* Returns the current {@link StatusInfo} for the given group.
2120
*
2221
* @param group the node group whose status is being fetched
2322
*/

‎cache/src/main/java/io/envoyproxy/controlplane/cache/ConfigWatcher.java‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,5 @@
11
package io.envoyproxy.controlplane.cache;
22

3-
import io.envoyproxy.envoy.api.v2.DiscoveryRequest;
43
import java.util.Set;
54
import java.util.function.Consumer;
65
import javax.annotation.concurrent.ThreadSafe;
@@ -25,7 +24,7 @@ public interface ConfigWatcher {
2524
*/
2625
Watch createWatch(
2726
boolean ads,
28-
DiscoveryRequest request,
27+
XdsRequest request,
2928
Set<String> knownResourceNames,
3029
Consumer<Response> responseConsumer,
3130
boolean hasClusterChanged);

‎cache/src/main/java/io/envoyproxy/controlplane/cache/NodeGroup.java‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,4 +15,11 @@ public interface NodeGroup<T> {
1515
* @param node identifier for the envoy instance that is requesting config
1616
*/
1717
T hash(Node node);
18+
19+
/**
20+
* Returns a consistent identifier of the given {@link io.envoyproxy.envoy.config.core.v3.Node}.
21+
*
22+
* @param node identifier for the envoy instance that is requesting config
23+
*/
24+
T hash(io.envoyproxy.envoy.config.core.v3.Node node);
1825
}

‎cache/src/main/java/io/envoyproxy/controlplane/cache/Resources.java‎

Lines changed: 197 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,13 @@
11
package io.envoyproxy.controlplane.cache;
22

33
import static com.google.common.base.Strings.isNullOrEmpty;
4+
import static io.envoyproxy.controlplane.cache.Resources.ApiVersion.V2;
5+
import static io.envoyproxy.controlplane.cache.Resources.ApiVersion.V3;
6+
import static io.envoyproxy.controlplane.cache.Resources.ResourceType.CLUSTER;
7+
import static io.envoyproxy.controlplane.cache.Resources.ResourceType.ENDPOINT;
8+
import static io.envoyproxy.controlplane.cache.Resources.ResourceType.LISTENER;
9+
import static io.envoyproxy.controlplane.cache.Resources.ResourceType.ROUTE;
10+
import static io.envoyproxy.controlplane.cache.Resources.ResourceType.SECRET;
411
import static io.envoyproxy.envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager.RouteSpecifierCase.RDS;
512

613
import com.google.common.base.Preconditions;
@@ -30,33 +37,113 @@
3037

3138
public class Resources {
3239

40+
/**
41+
* Version-agnostic representation of a resource. This is useful when the version qualifier
42+
* isn't needed.
43+
*/
44+
public enum ResourceType {
45+
CLUSTER,
46+
ENDPOINT,
47+
LISTENER,
48+
ROUTE,
49+
SECRET
50+
}
51+
52+
public enum ApiVersion {
53+
V2,
54+
V3
55+
}
56+
3357
private static final Logger LOGGER = LoggerFactory.getLogger(Resources.class);
3458

3559
static final String FILTER_ENVOY_ROUTER = "envoy.router";
3660
static final String FILTER_HTTP_CONNECTION_MANAGER = "envoy.http_connection_manager";
3761

38-
private static final String TYPE_URL_PREFIX = "type.googleapis.com/envoy.api.v2.";
39-
40-
public static final String CLUSTER_TYPE_URL = TYPE_URL_PREFIX + "Cluster";
41-
public static final String ENDPOINT_TYPE_URL = TYPE_URL_PREFIX + "ClusterLoadAssignment";
42-
public static final String LISTENER_TYPE_URL = TYPE_URL_PREFIX + "Listener";
43-
public static final String ROUTE_TYPE_URL = TYPE_URL_PREFIX + "RouteConfiguration";
44-
public static final String SECRET_TYPE_URL = TYPE_URL_PREFIX + "auth.Secret";
45-
46-
public static final List<String> TYPE_URLS = ImmutableList.of(
47-
CLUSTER_TYPE_URL,
48-
ENDPOINT_TYPE_URL,
49-
LISTENER_TYPE_URL,
50-
ROUTE_TYPE_URL,
51-
SECRET_TYPE_URL);
52-
53-
public static final Map<String, Class<? extends Message>> RESOURCE_TYPE_BY_URL = ImmutableMap.of(
54-
CLUSTER_TYPE_URL, Cluster.class,
55-
ENDPOINT_TYPE_URL, ClusterLoadAssignment.class,
56-
LISTENER_TYPE_URL, Listener.class,
57-
ROUTE_TYPE_URL, RouteConfiguration.class,
58-
SECRET_TYPE_URL, Secret.class
59-
);
62+
public static class V2 {
63+
private static final String TYPE_URL_PREFIX = "type.googleapis.com/envoy.api.v2.";
64+
public static final String SECRET_TYPE_URL = TYPE_URL_PREFIX + "auth.Secret";
65+
public static final String ROUTE_TYPE_URL = TYPE_URL_PREFIX + "RouteConfiguration";
66+
public static final String LISTENER_TYPE_URL = TYPE_URL_PREFIX + "Listener";
67+
public static final String ENDPOINT_TYPE_URL = TYPE_URL_PREFIX + "ClusterLoadAssignment";
68+
public static final String CLUSTER_TYPE_URL = TYPE_URL_PREFIX + "Cluster";
69+
70+
public static final List<String> TYPE_URLS = ImmutableList.of(
71+
CLUSTER_TYPE_URL,
72+
ENDPOINT_TYPE_URL,
73+
LISTENER_TYPE_URL,
74+
ROUTE_TYPE_URL,
75+
SECRET_TYPE_URL);
76+
}
77+
78+
public static class V3 {
79+
80+
public static final String CLUSTER_TYPE_URL = "type.googleapis.com/envoy.config.cluster.v3"
81+
+ ".Cluster";
82+
public static final String ENDPOINT_TYPE_URL = "type.googleapis.com/envoy.config.endpoint.v3"
83+
+ ".ClusterLoadAssignment";
84+
public static final String LISTENER_TYPE_URL = "type.googleapis.com/envoy.config.listener.v3"
85+
+ ".Listener";
86+
public static final String ROUTE_TYPE_URL = "type.googleapis.com/envoy.config.route.v3"
87+
+ ".RouteConfiguration";
88+
public static final String SECRET_TYPE_URL = "type.googleapis.com/envoy.extensions"
89+
+ ".transport_sockets.tls.v3.Secret";
90+
91+
public static final List<String> TYPE_URLS = ImmutableList.of(
92+
CLUSTER_TYPE_URL,
93+
ENDPOINT_TYPE_URL,
94+
LISTENER_TYPE_URL,
95+
ROUTE_TYPE_URL,
96+
SECRET_TYPE_URL);
97+
}
98+
99+
public static final List<ResourceType> RESOURCE_TYPES_IN_ORDER = ImmutableList.of(
100+
CLUSTER,
101+
ENDPOINT,
102+
LISTENER,
103+
ROUTE,
104+
SECRET);
105+
106+
public static final Map<String, String> V3_TYPE_URLS_TO_V2 = ImmutableMap.of(
107+
Resources.V3.CLUSTER_TYPE_URL, Resources.V2.CLUSTER_TYPE_URL,
108+
Resources.V3.ENDPOINT_TYPE_URL, Resources.V2.ENDPOINT_TYPE_URL,
109+
Resources.V3.LISTENER_TYPE_URL, Resources.V2.LISTENER_TYPE_URL,
110+
Resources.V3.ROUTE_TYPE_URL, Resources.V2.ROUTE_TYPE_URL,
111+
Resources.V3.SECRET_TYPE_URL, Resources.V2.SECRET_TYPE_URL);
112+
113+
public static final Map<String, String> V2_TYPE_URLS_TO_V3 = ImmutableMap.of(
114+
Resources.V2.CLUSTER_TYPE_URL, Resources.V3.CLUSTER_TYPE_URL,
115+
Resources.V2.ENDPOINT_TYPE_URL, Resources.V3.ENDPOINT_TYPE_URL,
116+
Resources.V2.LISTENER_TYPE_URL, Resources.V3.LISTENER_TYPE_URL,
117+
Resources.V2.ROUTE_TYPE_URL, Resources.V3.ROUTE_TYPE_URL,
118+
Resources.V2.SECRET_TYPE_URL, Resources.V3.SECRET_TYPE_URL);
119+
120+
public static final Map<String, ResourceType> TYPE_URLS_TO_RESOURCE_TYPE =
121+
new ImmutableMap.Builder<String, ResourceType>()
122+
.put(Resources.V3.CLUSTER_TYPE_URL, CLUSTER)
123+
.put(Resources.V2.CLUSTER_TYPE_URL, CLUSTER)
124+
.put(Resources.V3.ENDPOINT_TYPE_URL, ENDPOINT)
125+
.put(Resources.V2.ENDPOINT_TYPE_URL, ENDPOINT)
126+
.put(Resources.V3.LISTENER_TYPE_URL, LISTENER)
127+
.put(Resources.V2.LISTENER_TYPE_URL, LISTENER)
128+
.put(Resources.V3.ROUTE_TYPE_URL, ROUTE)
129+
.put(Resources.V2.ROUTE_TYPE_URL, ROUTE)
130+
.put(Resources.V3.SECRET_TYPE_URL, SECRET)
131+
.put(Resources.V2.SECRET_TYPE_URL, SECRET)
132+
.build();
133+
134+
public static final Map<String, Class<? extends Message>> RESOURCE_TYPE_BY_URL =
135+
new ImmutableMap.Builder<String, Class<? extends Message>>()
136+
.put(Resources.V2.CLUSTER_TYPE_URL, Cluster.class)
137+
.put(Resources.V2.ENDPOINT_TYPE_URL, ClusterLoadAssignment.class)
138+
.put(Resources.V2.LISTENER_TYPE_URL, Listener.class)
139+
.put(Resources.V2.ROUTE_TYPE_URL, RouteConfiguration.class)
140+
.put(Resources.V2.SECRET_TYPE_URL, Secret.class)
141+
.put(Resources.V3.CLUSTER_TYPE_URL, io.envoyproxy.envoy.config.cluster.v3.Cluster.class)
142+
.put(Resources.V3.ENDPOINT_TYPE_URL, io.envoyproxy.envoy.config.endpoint.v3.ClusterLoadAssignment.class)
143+
.put(Resources.V3.LISTENER_TYPE_URL, io.envoyproxy.envoy.config.listener.v3.Listener.class)
144+
.put(Resources.V3.ROUTE_TYPE_URL, io.envoyproxy.envoy.config.route.v3.RouteConfiguration.class)
145+
.put(Resources.V3.SECRET_TYPE_URL, io.envoyproxy.envoy.extensions.transport_sockets.tls.v3.Secret.class)
146+
.build();
60147

61148
/**
62149
* Returns the name of the given resource message.
@@ -84,6 +171,26 @@ public static String getResourceName(Message resource) {
84171
return ((Secret) resource).getName();
85172
}
86173

174+
if (resource instanceof io.envoyproxy.envoy.config.cluster.v3.Cluster) {
175+
return ((io.envoyproxy.envoy.config.cluster.v3.Cluster) resource).getName();
176+
}
177+
178+
if (resource instanceof io.envoyproxy.envoy.config.endpoint.v3.ClusterLoadAssignment) {
179+
return ((io.envoyproxy.envoy.config.endpoint.v3.ClusterLoadAssignment) resource).getClusterName();
180+
}
181+
182+
if (resource instanceof io.envoyproxy.envoy.config.listener.v3.Listener) {
183+
return ((io.envoyproxy.envoy.config.listener.v3.Listener) resource).getName();
184+
}
185+
186+
if (resource instanceof io.envoyproxy.envoy.config.route.v3.RouteConfiguration) {
187+
return ((io.envoyproxy.envoy.config.route.v3.RouteConfiguration) resource).getName();
188+
}
189+
190+
if (resource instanceof io.envoyproxy.envoy.extensions.transport_sockets.tls.v3.Secret) {
191+
return ((io.envoyproxy.envoy.extensions.transport_sockets.tls.v3.Secret) resource).getName();
192+
}
193+
87194
return "";
88195
}
89196

@@ -133,6 +240,17 @@ public static Set<String> getResourceReferences(Collection<? extends Message> re
133240
refs.add(c.getName());
134241
}
135242
}
243+
} else if (r instanceof io.envoyproxy.envoy.config.cluster.v3.Cluster) {
244+
io.envoyproxy.envoy.config.cluster.v3.Cluster c = (io.envoyproxy.envoy.config.cluster.v3.Cluster) r;
245+
246+
// For EDS clusters, use the cluster name or the service name override.
247+
if (c.getType() == io.envoyproxy.envoy.config.cluster.v3.Cluster.DiscoveryType.EDS) {
248+
if (!isNullOrEmpty(c.getEdsClusterConfig().getServiceName())) {
249+
refs.add(c.getEdsClusterConfig().getServiceName());
250+
} else {
251+
refs.add(c.getName());
252+
}
253+
}
136254
} else if (r instanceof Listener) {
137255
Listener l = (Listener) r;
138256

@@ -144,17 +262,56 @@ public static Set<String> getResourceReferences(Collection<? extends Message> re
144262
}
145263

146264
try {
147-
HttpConnectionManager.Builder config = HttpConnectionManager.newBuilder();
148-
149-
// TODO: Filter#getConfig() is deprecated, migrate to use Filter#getTypedConfig().
150-
structAsMessage(filter.getConfig(), config);
265+
HttpConnectionManager config;
266+
267+
if (filter.hasTypedConfig()) {
268+
config = filter.getTypedConfig().unpack(HttpConnectionManager.class);
269+
} else {
270+
HttpConnectionManager.Builder builder = HttpConnectionManager.newBuilder();
271+
structAsMessage(filter.getConfig(), builder);
272+
config = builder.build();
273+
}
151274

152275
if (config.getRouteSpecifierCase() == RDS && !isNullOrEmpty(config.getRds().getRouteConfigName())) {
153276
refs.add(config.getRds().getRouteConfigName());
154277
}
155278
} catch (InvalidProtocolBufferException e) {
156279
LOGGER.error(
157-
"Failed to convert HTTP connection manager config struct into protobuf message for listener {}",
280+
"Failed to convert v2 HTTP connection manager config struct into protobuf "
281+
+ "message for listener {}",
282+
getResourceName(l),
283+
e);
284+
}
285+
}
286+
}
287+
} else if (r instanceof io.envoyproxy.envoy.config.listener.v3.Listener) {
288+
289+
io.envoyproxy.envoy.config.listener.v3.Listener l =
290+
(io.envoyproxy.envoy.config.listener.v3.Listener) r;
291+
292+
// Extract the route configuration names from the HTTP connection manager.
293+
for (io.envoyproxy.envoy.config.listener.v3.FilterChain chain : l.getFilterChainsList()) {
294+
for (io.envoyproxy.envoy.config.listener.v3.Filter filter : chain.getFiltersList()) {
295+
if (!filter.getName().equals(FILTER_HTTP_CONNECTION_MANAGER)) {
296+
continue;
297+
}
298+
299+
try {
300+
io.envoyproxy.envoy.extensions.filters.network
301+
.http_connection_manager.v3.HttpConnectionManager config = filter
302+
.getTypedConfig().unpack(
303+
io.envoyproxy.envoy.extensions.filters.network
304+
.http_connection_manager.v3.HttpConnectionManager.class);
305+
306+
if (config.getRouteSpecifierCase() == io.envoyproxy.envoy.extensions.filters.network
307+
.http_connection_manager.v3.HttpConnectionManager.RouteSpecifierCase.RDS
308+
&& !isNullOrEmpty(config.getRds().getRouteConfigName())) {
309+
refs.add(config.getRds().getRouteConfigName());
310+
}
311+
} catch (InvalidProtocolBufferException e) {
312+
LOGGER.error(
313+
"Failed to convert v3 HTTP connection manager config struct into protobuf "
314+
+ "message for listener {}",
158315
getResourceName(l),
159316
e);
160317
}
@@ -166,6 +323,19 @@ public static Set<String> getResourceReferences(Collection<? extends Message> re
166323
return refs.build();
167324
}
168325

326+
/**
327+
* Returns the API version (v2 or v3) for a given type URL.
328+
*/
329+
public static ApiVersion getResourceApiVersion(String typeUrl) {
330+
if (Resources.V2.TYPE_URLS.contains(typeUrl)) {
331+
return V2;
332+
} else if (Resources.V3.TYPE_URLS.contains(typeUrl)) {
333+
return V3;
334+
}
335+
336+
throw new RuntimeException(String.format("Unsupported API version for type URL %s", typeUrl));
337+
}
338+
169339
private static void structAsMessage(Struct struct, Message.Builder messageBuilder)
170340
throws InvalidProtocolBufferException {
171341

0 commit comments

Comments
 (0)