-
Notifications
You must be signed in to change notification settings - Fork 26
/
Copy pathoperator-information.html.md.erb
409 lines (268 loc) · 24.5 KB
/
operator-information.html.md.erb
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
---
title: Operator Information
owner: Spring Cloud Services
---
<strong><%= modified_date %></strong>
See below for information about Spring Cloud Services' deployment model and other information which may be useful in operating its services or client applications.
## <a id="service-orchestration-and-instance-management"></a>Service Orchestration and Instance Management
Spring Cloud Services provides a series of [Managed Services](https://docs.cloudfoundry.org/services/) on [Pivotal Cloud Foundry®](https://network.pivotal.io/products/pivotal-cf) (PCF). It uses Cloud Foundry's [Service Broker API](https://docs.cloudfoundry.org/services/api.html) to manage the three services---Config Server, Service Registry, and Circuit Breaker Dashboard---that it makes available. See below for information about Spring Cloud Services's broker implementation and details about how it manages service instances.
<%= image_tag("/spring-cloud-services/images/operator-information/ops1.png") %>
### <a id="the-service-broker"></a>The Service Broker
The service broker's functionality is divided between the following two Spring Boot applications, which are deployed in the "system" org to the "p-spring-cloud-services" space.
* spring-cloud-broker ("the SB"): Implements the Service Broker API.
* spring-cloud-broker-worker ("the Worker"): Acts on provision, deprovision, bind, and unbind requests.
The broker relies on two other Pivotal Cloud Foundry® products, [MySQL for Pivotal Cloud Foundry®](https://network.pivotal.io/products/p-mysql) and [RabbitMQ for Pivotal Cloud Foundry®](https://network.pivotal.io/products/pivotal-rabbitmq-service), for the following service instances.
* **spring-cloud-broker-db**: A MySQL database used by the SB.
* **spring-cloud-broker-rmq**: A RabbitMQ queue used for communication between the SB and the Worker.
You can obtain the broker username and password from the Spring Cloud Services tile in Pivotal Cloud Foundry® Operations Manager. Click the Spring Cloud Services tile, and in the **Credentials** tab, copy the Broker Credentials.
<%= image_tag("/spring-cloud-services/images/operator-information/broker-credentials.png") %>
#### Broker Upgrades
The Spring Cloud Services product upgrade process checks before redeploying the service broker to see whether the broker applications' version has changed. If the version has not changed, the upgrade process will continue without redeploying either the SB or the Worker.
The SB application and the Worker application are deployed using a [blue-green deployment strategy](http://martinfowler.com/bliki/BlueGreenDeployment.html). During an upgrade of the service broker, the broker will continue processing requests to provision, deprovision, bind, and unbind service instances, without downtime.
#### <a id="access-broker-applications-in-apps-manager"></a>Access Broker Applications in Apps Manager
To view the broker applications in Pivotal Cloud Foundry® Apps Manager, log into Apps Manager as an admin user and select the "system" org.
<%= image_tag("/spring-cloud-services/images/operator-information/broker-apps-apps-manager-org.png") %>
The applications are deployed in the "p-spring-cloud-services" space.
<%= image_tag("/spring-cloud-services/images/operator-information/broker-apps-apps-manager-space.png") %>
### <a id="service-instances"></a>Service Instances
A service instance is backed by one or more Spring Boot applications deployed by the Worker in the "p-spring-cloud-services" org to the "instances" space.
A service instance is assigned a GUID at provision time. Backing applications for services include the GUID in their names:
**Config Server**
* config-[GUID]: A [Spring Cloud Config](http://cloud.spring.io/spring-cloud-config/spring-cloud-config.html) server application.
**Service Registry**
* eureka-[GUID]: A [Spring Cloud Netflix](http://cloud.spring.io/spring-cloud-netflix/spring-cloud-netflix.html) Eureka server application.
**Circuit Breaker Dashboard**
* hystrix-[GUID]: A [Spring Cloud Netflix](http://cloud.spring.io/spring-cloud-netflix/spring-cloud-netflix.html) Hystrix server application.
* turbine-[GUID]: A [Spring Cloud Netflix](http://cloud.spring.io/spring-cloud-netflix/spring-cloud-netflix.html) Turbine application.
* **rabbitmq-[GUID]**: A [RabbitMQ for PCF](http://docs.pivotal.io/rabbitmq-cf/index.html) service instance.
#### Access Service Instance Backing Applications in Apps Manager
To view a backing application for a service instance in Pivotal Cloud Foundry® Apps Manager, get the service instance's GUID and look for the corresponding application in the "instances" space.
Target the org and space of the service instance.
<pre class="terminal">
$ cf target -o myorg -s development
API endpoint: https://api.cf.wise.com (API version: 2.43.0)
User: admin
Org: myorg
Space: development
$ cf services
Getting services in org myorg / space development as admin...
OK
name service plan bound apps last operation
config-server p-config-server standard cook update succeeded
</pre>
Run `cf service` with the `CF_TRACE` environment variable set to `true`. Copy the value of `resources.metadata.guid`, which is the service instance GUID.
<pre class="terminal">
$ CF_TRACE=true cf service config-server
REQUEST: [2016-06-27T20:45:24-05:00]
[...]
RESPONSE: [2016-06-27T20:45:25-05:00]
[...]
{
"total_results": 1,
"total_pages": 1,
"prev_url": null,
"next_url": null,
"resources": [
{
"metadata": {
"guid": "51711835-4626-4823-b5a1-e5d91012f3f2",
</pre>
Log into Apps Manager as an admin user and select the "p-spring-cloud-services" org.
<%= image_tag("/spring-cloud-services/images/operator-information/instance-apps-apps-manager-org.png") %>
The applications are deployed in the "instances" space. Look for the backing application named as described above, with the prefix specific to the service type and the service instance GUID.
<%= image_tag("/spring-cloud-services/images/operator-information/instance-apps-apps-manager-space.png") %>
### <a id="set-buildpack-for-service-instances"></a>Set Buildpack for Service Instances
By default, the SB stages service instance backing applications using the buildpack chosen by the platform's buildpack detection; normally, this will be the highest-priority Java buildpack. To cause the SB to use a particular Java buildpack regardless of priority, you can set the name of the buildpack to use in the Spring Cloud Services tile settings.
In the Installation Dashboard of Ops Manager, click the Spring Cloud Services tile. Navigate to the **Settings** tab.
<%= image_tag("/spring-cloud-services/images/operator-information/set-buildpack.png") %>
Click the **Spring Cloud Services** tab and enter the name of the desired Java buildpack in the **Buildpack** field. Click **Save**, return to the Installation Dashboard, and apply your changes. The SB will now use the selected buildpack to stage service instance backing applications.
## <a id="uaa-identity-zones-and-clients"></a>UAA Identity Zones and Clients
Spring Cloud Services uses the multi-tenancy capabilities of the Cloud Foundry User Account and Authentication server (UAA). It creates a new Identity Zone ("the Spring Cloud Services Identity Zone") for all authorization from Config Server and Service Registry service instances to client applications which have bindings to these service instances.
Within the platform Identity Zone ("the UAA Identity Zone"), Spring Cloud Services creates a `p-spring-cloud-services` UAA client for the SB and a `p-spring-cloud-services-worker` UAA client for the Worker. In the Spring Cloud Services Identity Zone, it creates a `p-spring-cloud-services-worker` UAA client for the Worker.
In the UAA Identity Zone, it also creates the following clients, where `[GUID]` is the service instance's GUID:
* A `eureka-[GUID]` UAA client per Service Registry service instance.
* A `hystrix-[GUID]` UAA client per Circuit Breaker Dashboard service instance.
In the Spring Cloud Services Identity Zone, it also creates the following clients, where `[GUID]` is the service instance's GUID:
* A `config-server-[GUID]` UAA client per Config Server service instance.
* A `eureka-[GUID]` UAA client per Service Registry service instance.
### <a id="get-access-token-for-direct-requests-to-a-service-instance"></a>Get Access Token for Direct Requests to a Service Instance
To make requests directly against a Config Server service instance's Spring Cloud Config Server backing application or a Service Registry service instance's Eureka backing application, you can get an access token using the binding credentials of an application that is bound to the service instance. To do this, you must have [uaac](https://github.com/cloudfoundry/cf-uaac) installed.
1. Run `cf env`, giving the name of an application that is bound to the service instance:
<pre class="terminal">
$ cf services
Getting services in org myorg / space development as admin...
OK
name service plan bound apps last operation
config-server p-config-server standard cook create succeeded
$ cf env cook
Getting env variables for app cook in org myorg / space development as admin...
OK
System-Provided:
{
"VCAP_SERVICES": {
"p-config-server": [
{
"credentials": {
"access_token_uri": "https://p-spring-cloud-services.uaa.cf.wise.com/oauth/token",
"client_id": "p-config-server-876cd13b-1564-4a9a-9d44-c7c8a6257b73",
"client_secret": "rU7dMUw6bQjR",
"uri": "https://config-86b38ce0-eed8-4c01-adb4-1a651a6178e2.apps.wise.com"
},
[...]
</pre>
2. From the service instance's entry in the application's `VCAP_SERVICES` environment variable, copy the URL in `credentials.access_token_uri`. Remove the `/oauth/token` path from that URL and target it using `uaac target`:
<pre class="terminal">
$ uaac target https://p-spring-cloud-services.uaa.cf.wise.com
Target: https://p-spring-cloud-services.uaa.cf.wise.com
Context: p-config-server-876cd13b-1564-4a9a-9d44-c7c8a6257b73, from client p-config-server-876cd13b-1564-4a9a-9d44-c7c8a6257b73
</pre>
3. From the service instance's entry in the application's `VCAP_SERVICES` environment variable, copy the values of `credentials.client_id` and `credentials.client_secret`. Run `uaac token client get`, providing those values:
<pre class="terminal">
$ uaac token client get p-config-server-876cd13b-1564-4a9a-9d44-c7c8a6257b73 -s rU7dMUw6bQjR
Successfully fetched token via client credentials grant.
Target: https://p-spring-cloud-services.uaa.cf.wise.com
Context: p-config-server-876cd13b-1564-4a9a-9d44-c7c8a6257b73, from client p-config-server-876cd13b-1564-4a9a-9d44-c7c8a6257b73
</pre>
4. Run `uaac context` to display the current context:
<pre class="terminal">
$ uaac context
[2]*[https://p-spring-cloud-services.uaa.cf.wise.com]
[8]*[p-config-server-876cd13b-1564-4a9a-9d44-c7c8a6257b73]
client_id: p-config-server-876cd13b-1564-4a9a-9d44-c7c8a6257b73
access_token: TOKEN_STRING
token_type: bearer
expires_in: 43199
scope: p-config-server.86b38ce0-eed8-4c01-adb4-1a651a6178e2.read
jti: 8c122f33-0d63-446d-9549-6e349ce447eb
</pre>
Copy the value of `access_token`.
With the access token, you can now make requests directly against the service instance, e.g. using curl. Use the URL in `credentials.uri` from the service instance's entry in the application's `VCAP_SERVICES` environment variable.
For a Config Server service instance:
<pre class="terminal">
curl -H 'Authorization: bearer TOKEN_STRING' https://config-86b38ce0-eed8-4c01-adb4-1a651a6178e2.apps.wise.com/cook/default
{"name":"cook","profiles":["default"],"label":null,"version":"7b5da0d68d9237f2852f7ac6c5e7474fd433c3f3","propertySources":[{"name":"https://github.com/spring-cloud-samples/cook-config/cook.properties","source":{"cook.special":"Pickled Cactus"}}]}
</pre>
For a Service Registry service instance:
<%# Encoded output below because otherwise the browser can attempt to render the XML %>
<pre class="terminal">
$ curl -H 'Authorization: bearer TOKEN_STRING' https://eureka-493b6b17-512a-4961-8f85-6178251fe2fa.apps.wise.com/eureka/apps
<applications>
<versions__delta>1</versions__delta>
<apps__hashcode>UP_2_</apps__hashcode>
<application>
<name>GREETER</name>
<instance>
<instanceId>greeter.apps.wise.com:88e152ff-b476-44e1-4078-470b0f80eb25</instanceId>
<hostName>greeter.apps.wise.com</hostName>
<app>GREETER</app>
<ipAddr>10.68.224.43</ipAddr>
<status>UP</status>
[...]
</pre>
## <a id="service-instances-dashboard"></a>The Service Instances Dashboard
To view the status of individual service instances, visit the Service Instances dashboard. You can access it at the following URL, where `SCS_BROKER_URL` is the URL of the SB (spring-cloud-broker):
```
SCS_BROKER_URL/admin/serviceInstances
```
Locate the SB as described in the [Access Broker Applications in Apps Manager](#access-broker-applications-in-apps-manager) section. The Service Instances dashboard is also linked to on the main page of the SB.
The dashboard shows the version and status of each service instance, as well as other information including the number of applications that are bound to the instance. It also provides an **Upgrade** button for each service instance and an **Upgrade Service Instances** button; these buttons may be enabled if any service instances shown in the dashboard are not at the version included with the currently-installed Spring Cloud Services product. For more information about upgrading service instances, see the [Service Instance Upgrade Steps](/spring-cloud-services/upgrades.html#service-instance-upgrade-steps) section of the [Upgrades](/spring-cloud-services/upgrades.html) topic.
<%= image_tag("/spring-cloud-services/images/operator-information/si-dashboard.png") %>
You can click the **Instance Name** of a listed service instance to view the instance's dashboard. Click the **+** icon beside the count of an instance's **Bound Apps** or **Service Keys** to view the names of the applications or service keys that are bound to that instance.
## <a id="application-health-and-status"></a>Application Health and Status
For more visibility into how Spring Cloud Services service instances and the broker applications are behaving, or for troubleshooting purposes, you may wish to access those applications directly. See below for information about accessing their output.
### <a id="broker-application-logs"></a>Read Broker Application Logs
To access logs for the SB and Worker applications, target the "system" org and its "p-spring-cloud-services" space:
<pre class="terminal">
$ cf target -o system -s p-spring-cloud-services
API endpoint: https://api.cf.wise.com (API version: 2.43.0)
User: admin
Org: system
Space: p-spring-cloud-services
Ben-Kleins-MacBook-Pro:Work bklein$ cf apps
Getting apps in org system / space p-spring-cloud-services as admin...
OK
name requested state instances memory disk urls
spring-cloud-broker started 1/1 1G 1G spring-cloud-broker.apps.wise.com
spring-cloud-broker-worker started 1/1 1G 1G spring-cloud-broker-worker.apps.wise.com
</pre>
Then you can use ```cf logs``` to tail logs for either the SB:
<pre class="terminal">
$ cf logs spring-cloud-broker
Connected, tailing logs for app spring-cloud-broker in org system / space p-spring-cloud-services as admin...
</pre>
or the Worker:
<pre class="terminal">
$ cf logs spring-cloud-broker-worker
Connected, tailing logs for app spring-cloud-broker-worker in org system / space p-spring-cloud-services as admin...
</pre>
### <a id="backing-application-logs"></a>Read Backing Application Logs
To access logs for service instance backing applications, target the "p-spring-cloud-services" org and its "instances" space:
<pre class="terminal">
$ cf target -o p-spring-cloud-services -s instances
API endpoint: https://api.cf.wise.com (API version: 2.43.0)
User: admin
Org: p-spring-cloud-services
Space: instances
$ cf apps
Getting apps in org p-spring-cloud-services / space instances as admin...
OK
name requested state instances memory disk urls
config-86b38ce0-eed8-4c01-adb4-1a651a6178e2 started 1/1 1G 1G config-86b38ce0-eed8-4c01-adb4-1a651a6178e2.apps.wise.com
eureka-493b6b17-512a-4961-8f85-6178251fe2fa started 1/1 1G 1G eureka-493b6b17-512a-4961-8f85-6178251fe2fa.apps.wise.com
hystrix-ff835bde-5b3a-4623-a57f-8d88028b6376 started 1/1 1G 1G hystrix-ff835bde-5b3a-4623-a57f-8d88028b6376.apps.wise.com
turbine-ff835bde-5b3a-4623-a57f-8d88028b6376 started 1/1 1G 1G turbine-ff835bde-5b3a-4623-a57f-8d88028b6376.apps.wise.com
</pre>
Then you can use ```cf logs``` to tail logs for a backing application.
<pre class="terminal">
$ cf logs config-86b38ce0-eed8-4c01-adb4-1a651a6178e2
Connected, tailing logs for app config-86b38ce0-eed8-4c01-adb4-1a651a6178e2 in org p-spring-cloud-services / space instances as admin...
</pre>
### <a id="access-actuator-endpoints"></a>Access Actuator Endpoints
The SB and Worker applications, as well as backing applications for service instances, use [Spring Boot Actuator](http://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#production-ready). Actuator adds a number of endpoints to these applications; some of the added endpoints are summarized below.
| ID | Function |
|-----------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------|
| <code>env</code> | Displays profiles, properties, and property sources from the application environment |
| <code>health</code> | Displays information about the health and status of the application |
| <code>metrics</code> | Displays metrics information from the application |
| <code>mappings</code> | Displays list of <code>@RequestMapping</code> paths in the application |
| <code>shutdown</code> | Allows for graceful shutdown of the application. Disabled by default; not enabled in Spring Cloud Services broker or instance-backing applications |
See the <a href="http://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#production-ready-endpoints">Endpoints</a> section of the Actuator documentation for the full list of endpoints.
#### Endpoints on SB, Worker, Service Registry, or Circuit Breaker Dashboard
The SB application, the Worker application, the Service Registry's Eureka backing application, and the Circuit Breaker Dashboard's Hystrix backing application use [Dashboard SSO](http://docs.cloudfoundry.org/services/dashboard-sso.html). To access Actuator endpoints on one of these applications, you must be authenticated as a Space Developer in the application's space and have a current session with the application.
1. Log in to Apps Manager as an admin user and navigate to the relevant application: for the SB or Worker, access the SB application or Worker application as described in the [The Service Broker](#the-service-broker) section; for a Service Registry or Circuit Breaker Dashboard service instance, access the backing application as described in the [Service Instances](#service-instances) section.
<%= image_tag("/spring-cloud-services/images/operator-information/hystrix.png") %>
2. Visit the application's URL, appending the endpoint ID to that URL; e.g., for the `health` endpoint on a Circuit Breaker Dashboard service instance’s Hystrix backing application, this would be something like `https://hystrix-ff835bde-5b3a-4623-a57f-8d88028b6376.apps.wise.com/health`.
#### Endpoints on Config Server
The Config Server's Spring Cloud Config Server backing application uses HTTP Basic authentication. To access Actuator endpoints on this application, you must authenticate with the credentials stored in the application's environment variables.
1. Log in to Apps Manager as an admin user and navigate to the relevant application, as described in the [Service Instances](#service-instances) section.
<%= image_tag("/spring-cloud-services/images/operator-information/config-server.png") %>
2. From the **Env Variables** tab, copy the values of the `SECURITY_USER_NAME` and `SECURITY_USER_PASSWORD` environment variables.
<%= image_tag("/spring-cloud-services/images/operator-information/config-server-env-variables.png") %>
3. Visit the application's URL, appending the endpoint ID to that URL; e.g., for the `health` endpoint, this would be something like `config-86b38ce0-eed8-4c01-adb4-1a651a6178e2.apps.wise.com/health`. When challenged, provide the values of `SECURITY_USER_NAME` and `SECURITY_USER_PASSWORD` from Step 2.
### <a id="invoke-config-server-endpoints"></a>Invoke Config Server Endpoints
To view the configuration properties that a Config Server service instance is returning for an application, you can access the configuration endpoints on the service instance's Spring Cloud Config Server backing application directly.
1. Obtain an access token for interacting with the service instance, as described in the [Get Access Token for Direct Requests to a Service Instance](#get-access-token-for-direct-requests-to-a-service-instance) section. In Step 1 of that section, copy the URL in `credentials.uri` from the service instance's entry in the `VCAP_SERVICES` environment variable.
2. Make a request of the Config Server service instance backing application in the format `http://SERVER/APPLICATION_NAME/PROFILE`, where `SERVER` is the URL from `credentials.uri` as mentioned in Step 1, `APPLICATION_NAME` is the application name as set in the `spring.application.name` property, and `PROFILE` is the name of a profile that has a configuration file in the repository.
<br />
An example request URL might look like this:
```
https://config-86b38ce0-eed8-4c01-adb4-1a651a6178e2.apps.wise.com/cook/production
```
Or, for the `default` profile:
```
https://config-86b38ce0-eed8-4c01-adb4-1a651a6178e2.apps.wise.com/cook/default
```
You can make the request using curl, providing the token---`TOKEN` in the below example---in a header with the `-H` or `--header` option:
<pre class="terminal">
$ curl -H 'Authorization: bearer TOKEN_STRING' https://config-86b38ce0-eed8-4c01-adb4-1a651a6178e2.apps.wise.com/cook/default
{"name":"cook","profiles":["default"],"label":null,"version":"7b5da0d68d9237f2852f7ac6c5e7474fd433c3f3","propertySources":[{"name":"https://github.com/spring-cloud-samples/cook-config/cook.properties","source":{"cook.special":"Pickled Cactus"}}]}
</pre>
For a list of path formats that the Config Server accepts, see the [Request Paths](/spring-cloud-services/config-server/server.html#request-paths) section of the [The Config Server](/spring-cloud-services/config-server/server.html) topic in the [Config Server documentation](/spring-cloud-services/config-server/).
## <a id="capacity-requirements"></a>Capacity Requirements
Below are usage requirements of the applications backing a single service instance (SI) of each Spring Cloud Services service type.
| Service Type | Memory Limit / SI | Disk Limit / SI |
|---------------------------|---------------------|------------------|
| Config Server | 1 GB | 1 GB |
| Service Registry | 1 GB | 1 GB |
| Circuit Breaker Dashboard | 2 GB | 2 GB |
Spring Cloud Services service types may also be backed by non-SCS service instances. For example, a Circuit Breaker Dashboard service instance uses a [RabbitMQ for Pivotal Cloud Foundry®](http://docs.pivotal.io/rabbitmq-cf/index.html) service instance for communication between its two backing applications.