forked from cloudfoundry/docs-loggregator
-
Notifications
You must be signed in to change notification settings - Fork 0
/
nozzle-tutorial.html.md.erb
351 lines (270 loc) · 14.1 KB
/
nozzle-tutorial.html.md.erb
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
---
title: Deploying a Nozzle to your Cloud Foundry Loggregator Firehose
owner: Logging and Metrics
---
You can deploy a nozzle app to the <%= vars.app_runtime_first %> Loggregator Firehose.
For more information about nozzles and the Loggregator Firehose, see
[Loggregator Architecture](./architecture.html). The <%= vars.app_runtime_abbr %> Loggregator team created an example
nozzle app for use with this tutorial.
The following procedure deploys an example nozzle to the Firehose of a <%= vars.app_runtime_abbr %>
installation deployed locally with BOSH Lite v2. For more information about BOSH Lite v2, see
[VirtualBox BOSH Lite v2](https://bosh.io/docs/bosh-lite/) in the BOSH documentation.
To reduce the load on custom nozzles that you develop, you can request Firehose subscriptions that emit only
metrics on an allow list. For examples, see `rlpreader` and `rlptypereader` in the
[Loggregator Tools](https://github.com/cloudfoundry-incubator/loggregator-tools) repository and
[V2 Subscriptions](https://github.com/cloudfoundry/loggregator-release/blob/main/docs/v2-subscriptions.md) in the
Loggregator Release repository in GitHub.
<p> You can deactivate the Firehose. In place of the Firehose, you can configure an
aggregate log and metric drain for your foundation. <%= vars.configure_system_logging_link %></p>
## <a id='prerequisites'></a> Prerequisites
Before you deploy a nozzle to the Loggregator Firehose, you must have:
* BOSH CLI installed locally. For more information, see [BOSH CLI](https://bosh.io/docs/bosh-cli.html) in the BOSH
documentation.
* Spiff installed locally and added to the load path of your shell. For more information, see the
[Spiff](https://github.com/cloudfoundry-incubator/spiff) repository on GitHub.
* BOSH Lite v2 deployed locally using VirtualBox. For more information about BOSH Lite v2, see
[VirtualBox BOSH Lite v2](https://bosh.io/docs/bosh-lite/) in the BOSH documentation.
* A working <%= vars.app_runtime_abbr %> deployment, including Loggregator, deployed with your local BOSH Lite v2. This
deployment serves as the source of data. Use the `provision_cf` script included in the BOSH Lite v2 release.
<p class="note important">
<span class="note__title">Important</span> Deploying <%= vars.app_runtime_abbr %> can take several hours depending on your
internet bandwith, even when using the automated <code>provision_cf</code> script.</p>
## <a id='download-manifest'></a> Step 1: Downloading Cloud Foundry BOSH manifest
To download the BOSH manifest:
<%= partial 'download_bosh_manifest' %>
## <a id='add-client'></a> Step 2: Adding UAA client
You must authorize the example nozzle as a UAA client for your <%= vars.app_runtime_abbr %> deployment. To do this, add
an entry for the example nozzle as a `client` for `uaa` under the `properties` key in your <%= vars.app_runtime_abbr %>
deployment manifest YAML file. You must enter the example nozzle object in the correct location in the manifest, and
with the correct indentation.
To add the nozzle as a UAA client for your deployment:
1. Open the deployment manifest in a text editor.
1. Locate the left-aligned `properties` key.
1. Under the `properties` key, locate `uaa` at the next level of indentation.
1. Under the `uaa` key, locate the `clients` key at the next level of indentation.
1. Enter properties for the `example-nozzle` at the next level of indentation, exactly as shown. The `...` in the
text indicates other properties that might populate the manifest at each level in the hierarchy.
```
properties:
...
uaa:
...
clients:
...
example-nozzle:
access-token-validity: 1209600
authorized-grant-types: client_credentials
override: true
secret: example-nozzle
authorities: oauth.login,doppler.firehose
```
1. Save the deployment manifest file.
## <a id='redeploy'></a> Step 3: Redeploying <%= vars.app_runtime_abbr %>
To redeploy <%= vars.app_runtime_abbr %> with BOSH, run:
```
bosh -e BOSH-ENVIRONMENT deploy
```
Where `BOSH-ENVIRONMENT` is the alias that you set for your BOSH Director.
For example:
<pre class="terminal">
$ bosh -e dev deploy
Acting as user 'admin' on deployment 'cf-warden' on 'Bosh Lite Director'
Getting deployment properties from director...
Detecting deployment changes
----------------------------
Releases
No changes
Compilation
No changes
Update
No changes
Resource pools
No changes
Disk pools
No changes
Networks
No changes
Jobs
No changes
Properties
uaa
clients
example-nozzle
+ access-token-validity: 1209600
+ authorized-grant-types: authorization_code,client_credentials,refresh_token
+ override: true
+ secret: example-nozzle
+ scope: openid,oauth.approvals,doppler.firehose
+ authorities: oauth.login,doppler.firehose
Meta
No changes
Please review all changes carefully
Deploying
---------
Are you sure you want to deploy? (type 'yes' to continue):yes
</pre>
## <a id='clone-example-release'></a> Step 4: Cloning an example release
The <%= vars.app_runtime_abbr %> Loggregator team created an example nozzle app for use with this tutorial.
To clone the example nozzle release:
1. Clone the main release repository from the example-nozzle-release repository on GitHub by running:
```
git clone https://github.com/cloudfoundry-incubator/example-nozzle-release.git
```
1. Go to the `example-nozzle-release` directory by running:
```
cd example-nozzle-release
```
1. Update all of the included submodules by running:
```
git submodule update --init --recursive
```
For example:
<pre class="terminal">
$ git submodule update --init --recursive
Submodule 'src/github.com/cloudfoundry-incubator/example-nozzle' (<span>git@</span>github.com:cloudfoundry-incubator/example-nozzle.git) registered for path 'src/github.com/cloudfoundry-incubator/example-nozzle'
Submodule 'src/github.com/cloudfoundry-incubator/uaago' (<span>git@</span>github.com:cloudfoundry-incubator/uaago.git) registered for path 'src/github.com/cloudfoundry-incubator/uaago'
...
Cloning into 'src/github.com/cloudfoundry-incubator/example-nozzle'...
...
</pre>
## <a id='prepare-nozzle-manifest'></a> Step 5: Preparing nozzle manifest
To prepare the nozzle deployment manifest:
1. In the `example-nozzle-release` directory, navigate to the `templates` directory by running:
```
cd templates
```
Within this directory, examine the two YAML files. `bosh-lite-stub.yml` contains the values used to populate the
missing information in `template.yml`. By combining these two files, you can create a deployment manifest for the
nozzle.
1. Create a `tmp` directory for the compiled manifest.
1. Use Spiff to compile a deployment manifest from the template and stub by running:
```
spiff merge templates/template.yml templates/bosh-lite-stub.yml > tmp/manifest_bosh_lite.yml
```
Save this manifest.
1. To identify the names of all deployments in the environment that you specify, run:
```
bosh -e BOSH-ENVIRONMENT deployments
```
Where `BOSH-ENVIRONMENT` is the alias that you set for your BOSH Director.
1. To obtain your BOSH Director UUID, run:
```
bosh -e BOSH-ENVIRONMENT env --column=uuid
```
Where `BOSH-ENVIRONMENT` is the alias that you set for your BOSH Director. For example:
<pre class="terminal">
$ bosh -e dev env --column=uuid
</pre>
1. In the compiled nozzle deployment manifest, locate the `director_uuid` property. Replace `PLACEHOLDER-DIRECTOR-UUID`
with your BOSH Director UUID.
```
compilation:
cloud_properties:
name: default
network: example-nozzle-net
reuse_compilation_vms: true
workers: 1
director_uuid: PLACEHOLDER-DIRECTOR-UUID
```
<p> If you do not want to see the complete deployment procedure, run
<code>scripts/make_manifest_spiff_bosh_lite</code> to prepare the manifest.</p>
## <a id='create-nozzle-bosh-release'></a> Step 6: Creating nozzle BOSH release
To create a nozzle BOSH release, run:
```
bosh -e BOSH-ENVIRONMENT create-release --name RELEASE-NAME
```
Where:
* `BOSH-ENVIRONMENT` is the alias that you set for your BOSH Director.
* `RELEASE-NAME` is `example-nozzle` to match the UAA client that you created in the <%= vars.app_runtime_abbr %>
deployment manifest.
For example:
<pre class="terminal">
$ bosh -e dev create-release --name example-nozzle
Syncing blobs...
...
</pre>
## <a id='upload-nozzle-bosh-release'></a> Step 7: Uploading nozzle BOSH release
Upload the nozzle BOSH release that you created in [Step 6: Create nozzle BOSH release](#create-nozzle-bosh-release).
To upload the BOSH release, run:
```
bosh -e BOSH-ENVIRONMENT upload-release
```
Where `BOSH-ENVIRONMENT` is the alias that you set for your BOSH Director.
For example:
<pre class="terminal">
$ bosh -e dev upload-release
Acting as user 'admin' on 'Bosh Lite Director'<br>
Copying packages
\----------------
example-nozzle
golang1.7<br>
Copying jobs
\------------
example-nozzle<br>
Generated /var/folders/4n/qs1rjbmd1c5gfb78m3_06j6r0000gn/T/d20151009-71219-17a5m49/d20151009-71219-rts928/release.tgz
Release size: 59.2M<br>
Verifying release...
...
Release info
\------------
Name: nozzle-test
Version: 0+dev.2<br>
Packages
- example-nozzle (b0944f95eb5a332e9be2adfb4db1bc88f9755894)
- golang1.7 (b68dc9557ef296cb21e577c31ba97e2584a5154b)<br>
Jobs
- example-nozzle (112e01c6ee91e8b268a42239e58e8e18e0360f58)<br>
License
- none<br>
Uploading release
</pre>
## <a id='deploy-nozzle'></a> Step 8: Deploying nozzle
To deploy the nozzle, run:
```console
bosh -e BOSH-ENVIRONMENT deploy
```
Where `BOSH-ENVIRONMENT` is the alias that you set for your BOSH Director.
For example:
<pre class="terminal">
$ bosh -e dev deploy
Acting as user 'admin' on deployment 'example-nozzle-lite' on 'Bosh Lite Director'
Getting deployment properties from director...
Unable to get properties list from director, trying without it...
Cannot get current deployment information from director, possibly a new deployment
Please review all changes carefully<br>
Deploying
\---------
Are you sure you want to deploy? (type 'yes' to continue):yes
</pre>
## <a id='view-nozzle-output'></a> Step 9: Viewing nozzle output
The example nozzle outputs all of the data originating coming from the Firehose to its log files. To view this data,
SSH into the `example-nozzle` VM and examine the logs.
To view nozzle output:
1. Access the nozzle VM at the IP address configured in the nozzle manifest template stub,
`./templates/bosh-lite-stub.yml`, by running:
```yml
bosh -e BOSH-ENVIRONMENT ssh
```
Where `BOSH-ENVIRONMENT` is the alias that you set for your BOSH Director.
For example:
<pre class="terminal">
$ bosh -e dev ssh example-nozzle
Welcome to Ubuntu 14.04.1 LTS (GNU/Linux 3.19.0-25-generic x86_64)
Documentation: <span>https<span>://help.ubuntu.com/
Last login: Wed Sep 23 21:29:50 2015 from 192.0.2.1
</pre>
1. Use the `cat` command to output the `stdout` log file.
<pre class="terminal">
$ cat /var/vcap/sys/log/example-nozzle/example-nozzle.stdout.log
===== Streaming Firehose (will only succeed if you have admin credentials)
origin:"bosh-system-metrics-forwarder" eventType:ValueMetric timestamp:1541091851000000000 deployment:"cf-c42ae2c4dfb6f67b6c27" job:"loggregator_trafficcontroller" index:"d1dffe15-5894-44de-b7f0-ad43161a0a7b" ip:"" tags:>key:"id" value:"d1dffe15-5894-44de-b7f0-ad43161a0a7b" > tags:>key:"source_id" value:"bosh-system-metrics-forwarder" > valueMetric:>"system.swap.percent" value:0 unit:"Percent" >
origin:"bosh-system-metrics-forwarder" eventType:ValueMetric timestamp:1541091851000000000 deployment:"cf-c42ae2c4dfb6f67b6c27" job:"loggregator_trafficcontroller" index:"d1dffe15-5894-44de-b7f0-ad43161a0a7b" ip:"" tags:>key:"id" value:"d1dffe15-5894-44de-b7f0-ad43161a0a7b" > tags:>key:"source_id" value:"bosh-system-metrics-forwarder" > valueMetric:>"system.swap.kb" value:0 unit:"Kb" >
origin:"bosh-system-metrics-forwarder" eventType:ValueMetric timestamp:1541091851000000000 deployment:"cf-c42ae2c4dfb6f67b6c27" job:"loggregator_trafficcontroller" index:"d1dffe15-5894-44de-b7f0-ad43161a0a7b" ip:"" tags:>key:"id" value:"d1dffe15-5894-44de-b7f0-ad43161a0a7b" > tags:>key:"source_id" value:"bosh-system-metrics-forwarder" > valueMetric:>"system.disk.ephemeral.percent" value:3 unit:"Percent" >
origin:"bosh-system-metrics-forwarder" eventType:ValueMetric timestamp:1541091851000000000 deployment:"cf-c42ae2c4dfb6f67b6c27" job:"loggregator_trafficcontroller" index:"d1dffe15-5894-44de-b7f0-ad43161a0a7b" ip:"" tags:>key:"id" value:"d1dffe15-5894-44de-b7f0-ad43161a0a7b" > tags:>key:"source_id" value:"bosh-system-metrics-forwarder" > valueMetric:>"system.healthy" value:1 unit:"b" >
origin:"gorouter" eventType:ValueMetric timestamp:1541091851218590916 deployment:"cf-c56ab7c4dfb6f67b6c28" job:"router" index:"d5d1b5a4-2497-4679-8d3b-66ffc978d829" ip:"10.0.4.13" tags:>key:"source_id" value:"gorouter" > valueMetric:>"uptime" value:3.273478e+06 unit:"seconds" >
origin:"netmon" eventType:ValueMetric timestamp:1541091851234217334 deployment:"cf-c56ab7c4dfb6f67b6c28" job:"diego_cell" index:"8007afda-3bff-4856-857f-a47a43cbf994" ip:"10.0.4.18" tags:>key:"source_id" value:"netmon" > valueMetric:>name:"numGoRoutines" value:13 unit:"count" >
origin:"netmon" eventType:ValueMetric timestamp:1541091851234129669 deployment:"cf-c56ab7c4dfb6f67b6c28" job:"diego_cell" index:"8007afda-3bff-4856-857f-a47a43cbf994" ip:"10.0.4.18" tags:>key:"source_id" value:"netmon" > valueMetric:>"numCPUS" value:2 unit:"count" >
origin:"netmon" eventType:ValueMetric timestamp:1541091851234292367 deployment:"cf-c56ab7c4dfb6f67b6c28" job:"diego_cell" index:"8007afda-3bff-4856-857f-a47a43cbf994" ip:"10.0.4.18" tags:>key:"source_id" value:"netmon" > valueMetric:>"memoryStats.numBytesAllocated" value:542328 unit:"count" >
origin:"netmon" eventType:ValueMetric timestamp:1541091851234279470 deployment:"cf-c56ab7c4dfb6f67b6c28" job:"diego_cell" index:"8007afda-3bff-4856-857f-a47a43cbf994" ip:"10.0.4.18" tags:>key:"source_id" value:"netmon" > valueMetric:>"memoryStats.numBytesAllocatedStack" value:655360 unit:"count" >
...
</pre>