You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: buffer/file.md
+4-46Lines changed: 4 additions & 46 deletions
Original file line number
Diff line number
Diff line change
@@ -9,49 +9,11 @@ The `file` buffer plugin provides a persistent buffer implementation. It uses fi
9
9
10
10
### `path`
11
11
12
-
| type | required | default | version |
13
-
| :--- | :---: | :--- | :--- |
14
-
| string | ✔ || 0.9.0 |
15
-
16
-
The path where buffer chunks are stored. The '\*' is replaced with random characters. This parameter is required.
17
-
18
-
This parameter must be unique to avoid the race condition problem. For example, you cannot use a fixed path parameter in `fluent-plugin-forest`. `${tag}` or similar placeholder is needed. Of course, this parameter must also be unique between fluentd instances.
19
-
20
-
In addition, `path` should not be another `path` prefix. For example, the following configuration does not work well. `/var/log/fluent/foo` resumes `/var/log/fluent/foo.bar`'s buffer files during start phase and it causes `No such file or directory` in `/var/log/fluent/foo.bar` side.
21
-
22
-
```text
23
-
<match pattern1>
24
-
<buffer>
25
-
@type file
26
-
path /var/log/fluent/foo
27
-
</buffer>
28
-
</match>
29
-
30
-
<match pattern2>
31
-
<buffer>
32
-
@type file
33
-
path /var/log/fluent/foo.bar
34
-
</buffer>
35
-
</match>
36
-
```
37
-
38
-
Here is the correct version to avoid the prefix problem:
12
+
| type | default | version |
13
+
| :--- | :--- | :--- |
14
+
| string | nil | TBD |
39
15
40
-
```text
41
-
<match pattern1>
42
-
<buffer>
43
-
@type file
44
-
path /var/log/fluent/foo.baz
45
-
</buffer>
46
-
</match>
47
-
48
-
<match pattern2>
49
-
<buffer>
50
-
@type file
51
-
path /var/log/fluent/foo.bar
52
-
</buffer>
53
-
</match>
54
-
```
16
+
The path where buffer chunks are stored.
55
17
56
18
Please make sure that you have **enough space in the path directory**. Running out of disk space is a problem frequently reported by users.
57
19
@@ -88,10 +50,6 @@ This parameter is useful when `.log` is not fit for your environment. See also [
88
50
89
51
## Tips
90
52
91
-
### Multi-Process Environment
92
-
93
-
If you use this plugin under the multi-process environment, you need to use `@id`/`root_dir` parameters instead of fixed `path` parameter. See [Multi Process Workers](../deployment/multi-process-workers.md#root_dir-id-parameter) article.
94
-
95
53
## Limitation
96
54
97
55
Caution: `file` buffer implementation depends on the characteristics of the local file system. Don't use `file` buffer on remote file systems e.g. NFS, GlusterFS, HDFS, etc. We observed major data loss by using the remote file system.
Copy file name to clipboardExpand all lines: deployment/multi-process-workers.md
-39Lines changed: 0 additions & 39 deletions
Original file line number
Diff line number
Diff line change
@@ -131,45 +131,6 @@ As of Fluentd v1.4.0, `<worker N-M>` syntax has been introduced:
131
131
132
132
With this directive, you can specify multiple workers per worker directive.
133
133
134
-
### `root_dir/@id` parameter
135
-
136
-
These parameters must be specified when you use the file buffer.
137
-
138
-
With multi-process workers, you cannot use the fixed `path` configuration for file buffer because it conflicts buffer file path between processes.
139
-
140
-
```text
141
-
<system>
142
-
workers 2
143
-
</system>
144
-
145
-
<match pattern>
146
-
@type forward
147
-
<buffer>
148
-
@type file
149
-
path /var/log/fluentd/forward # This is not allowed
150
-
</buffer>
151
-
</match>
152
-
```
153
-
154
-
Instead of a fixed configuration, fluentd provides the dynamic buffer path based on `root_dir` and `@id` parameters. The stored path is `${root_dir}/worker${worker index}/${plugin @id}/buffer` directory.
155
-
156
-
```text
157
-
<system>
158
-
workers 2
159
-
root_dir /var/log/fluentd
160
-
</system>
161
-
162
-
<match pattern>
163
-
@type forward
164
-
@id out_fwd
165
-
<buffer>
166
-
@type file
167
-
</buffer>
168
-
</match>
169
-
```
170
-
171
-
With this configuration, `forward` output buffer files are stored into `/var/log/fluentd/worker0/out_fwd/buffer` and `/var/log/fluentd/worker1/out_fwd/buffer` directories.
172
-
173
134
## Operation
174
135
175
136
Each worker consumes memory and disk space separately. Take care while configuring buffer spaces and monitoring memory/disk consumption.
0 commit comments