File tree Expand file tree Collapse file tree 1 file changed +12
-1
lines changed Expand file tree Collapse file tree 1 file changed +12
-1
lines changed Original file line number Diff line number Diff line change @@ -14,6 +14,7 @@ Read files on Hdfs.
14
14
- ** config** overwrites configuration parameters (hash, default: ` {} ` )
15
15
- ** input_path** file path on Hdfs. you can use glob and Date format like ` %Y%m%d/%s ` .
16
16
- ** rewind_seconds** When you use Date format in input_path property, the format is executed by using the time which is Now minus this property.
17
+ - ** partition** when this is true, partition input files and increase task count. (default: ` true ` )
17
18
18
19
## Example
19
20
24
25
- /opt/analytics/etc/hadoop/conf/core-site.xml
25
26
- /opt/analytics/etc/hadoop/conf/hdfs-site.xml
26
27
config :
27
- fs.defaultFS : ' hdfs://hdp -nn1:8020'
28
+ fs.defaultFS : ' hdfs://hadoop -nn1:8020'
28
29
dfs.replication : 1
29
30
fs.hdfs.impl : ' org.apache.hadoop.hdfs.DistributedFileSystem'
30
31
fs.file.impl : ' org.apache.hadoop.fs.LocalFileSystem'
31
32
input_path : /user/embulk/test/%Y-%m-%d/*
32
33
rewind_seconds : 86400
34
+ partition : true
33
35
decoders :
34
36
- {type: gzip}
35
37
parser :
50
52
- {name: c3, type: long}
51
53
` ` `
52
54
55
+ ## Note
56
+ - the feature of the partition supports only 3 line terminators.
57
+ - ` \n`
58
+ - ` \r `
59
+ - ` \r\n `
60
+
61
+ # # The Reference Implementation
62
+ - [hito4t/embulk-input-filesplit](https://github.com/hito4t/embulk-input-filesplit)
63
+
53
64
# # Build
54
65
55
66
```
You can’t perform that action at this time.
0 commit comments