Skip to content

parser_csv: skip empty or unparseable lines#5355

Merged
kenhys merged 1 commit into
fluent:masterfrom
Watson1978:fix-parser_csv
May 11, 2026
Merged

parser_csv: skip empty or unparseable lines#5355
kenhys merged 1 commit into
fluent:masterfrom
Watson1978:fix-parser_csv

Conversation

@Watson1978

@Watson1978 Watson1978 commented May 9, 2026

Copy link
Copy Markdown
Contributor

Which issue(s) this PR fixes:
Fixes #

What this PR does / why we need it:
This PR fixes an issue in parser_csv where processing an empty line causes a TypeError.

When parser_csv receives an empty line, CSV.parse_line returns nil.
Currently, passing this nil value to Array#zip throws a TypeError: wrong argument type NilClass (must respond to :each).

Reproduce

config:

<source>
  @type tail
  path "#{File.expand_path '~/tmp/fluentd/test.csv'}"
  tag test.csv
  read_from_head true
  <parse>
    @type csv
    keys k1,k2,k3
  </parse>
</source>

<match test.csv>
  @type stdout
</match>

Prepare input data:

$ echo "A,B,C"  > ~/tmp/fluentd/test.csv
$ echo ""      >> ~/tmp/fluentd/test.csv
$ echo "D,E,F" >> ~/tmp/fluentd/test.csv

Result of before changing:

2026-05-09 17:23:14 +0900 [info]: #0 following tail of /home/watson/tmp/fluentd/test.csv
2026-05-09 17:23:14 +0900 [warn]: #0 invalid line found 
2026-05-09 17:23:14 +0900 [warn]: #0 this parameter is highly recommended to save the position to resume tailing.
2026-05-09 17:23:14 +0900 [info]: #0 starting fluentd worker pid=41671 ppid=41646 worker=0
2026-05-09 17:23:14 +0900 [info]: #0 following tail of /home/watson/tmp/fluentd/test.csv
2026-05-09 17:23:14 +0900 [warn]: #0 invalid line found file="/home/watson/tmp/fluentd/test.csv" line="" error="wrong argument type NilClass (must respond to :each)"
2026-05-09 17:23:14.692710744 +0900 test.csv: {"k1":"A","k2":"B","k3":"C"}
2026-05-09 17:23:14.692801343 +0900 test.csv: {"k1":"D","k2":"E","k3":"F"}
2026-05-09 17:23:14 +0900 [info]: #0 fluentd worker is now running worker=0

Docs Changes:
N/A

Release Note:

  • parser_csv: skip empty or unparseable lines

@Watson1978 Watson1978 requested a review from kenhys May 9, 2026 08:25
@Watson1978 Watson1978 added the backport to v1.19 We will backport this fix to the LTS branch label May 9, 2026
@Watson1978 Watson1978 added this to the v1.20.0 milestone May 9, 2026
@Watson1978

Copy link
Copy Markdown
Contributor Author

Simple reproduction:

irb(main):001> require 'csv'
=> true
irb(main):002> @keys = ('k1'..'k3').to_a
=> ["k1", "k2", "k3"]
irb(main):003> values = CSV.parse_line('')
=> nil
irb(main):004> Hash[@keys.zip(values)]
(irb):4:in 'Array#zip': wrong argument type NilClass (must respond to :each) (TypeError)
        from (irb):4:in '<main>'
        from /home/watson/.rbenv/versions/4.0.3/lib/ruby/gems/4.0.0/gems/irb-1.18.0/exe/irb:9:in '<top (required)>'
        from /home/watson/.rbenv/versions/4.0.3/lib/ruby/site_ruby/4.0.0/rubygems.rb:305:in 'Kernel#load'
        from /home/watson/.rbenv/versions/4.0.3/lib/ruby/site_ruby/4.0.0/rubygems.rb:305:in 'Gem.activate_and_load_bin_path'
        from /home/watson/.rbenv/versions/4.0.3/bin/irb:25:in '<main>'

Signed-off-by: Shizuo Fujita <fujita@clear-code.com>

@kenhys kenhys left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@kenhys kenhys merged commit 0547ff9 into fluent:master May 11, 2026
21 checks passed
@Watson1978 Watson1978 deleted the fix-parser_csv branch May 11, 2026 01:30
Watson1978 added a commit that referenced this pull request May 17, 2026
…5359)

**Which issue(s) this PR fixes**: 
Backport #5355
Fixes #

**What this PR does / why we need it**: 
This PR fixes an issue in `parser_csv` where processing an empty line
causes a `TypeError`.

When `parser_csv` receives an empty line, `CSV.parse_line` returns
`nil`.
Currently, passing this `nil` value to `Array#zip` throws a `TypeError:
wrong argument type NilClass (must respond to :each)`.

### Reproduce
config:
```
<source>
  @type tail
  path "#{File.expand_path '~/tmp/fluentd/test.csv'}"
  tag test.csv
  read_from_head true
  <parse>
    @type csv
    keys k1,k2,k3
  </parse>
</source>

<match test.csv>
  @type stdout
</match>
```

Prepare input data:
```
$ echo "A,B,C"  > ~/tmp/fluentd/test.csv
$ echo ""      >> ~/tmp/fluentd/test.csv
$ echo "D,E,F" >> ~/tmp/fluentd/test.csv
```

Result of before changing:
```
2026-05-09 17:23:14 +0900 [info]: #0 following tail of /home/watson/tmp/fluentd/test.csv
2026-05-09 17:23:14 +0900 [warn]: #0 invalid line found 
2026-05-09 17:23:14 +0900 [warn]: #0 this parameter is highly recommended to save the position to resume tailing.
2026-05-09 17:23:14 +0900 [info]: #0 starting fluentd worker pid=41671 ppid=41646 worker=0
2026-05-09 17:23:14 +0900 [info]: #0 following tail of /home/watson/tmp/fluentd/test.csv
2026-05-09 17:23:14 +0900 [warn]: #0 invalid line found file="/home/watson/tmp/fluentd/test.csv" line="" error="wrong argument type NilClass (must respond to :each)"
2026-05-09 17:23:14.692710744 +0900 test.csv: {"k1":"A","k2":"B","k3":"C"}
2026-05-09 17:23:14.692801343 +0900 test.csv: {"k1":"D","k2":"E","k3":"F"}
2026-05-09 17:23:14 +0900 [info]: #0 fluentd worker is now running worker=0
```

**Docs Changes**:
N/A

**Release Note**: 
* parser_csv: skip empty or unparseable lines

Signed-off-by: Shizuo Fujita <fujita@clear-code.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Shizuo Fujita <fujita@clear-code.com>
@Watson1978 Watson1978 added the backported "backport to LTS" is done label May 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport to v1.19 We will backport this fix to the LTS branch backported "backport to LTS" is done

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants