Breaks in streaming mode

[mcqj]

When used in streaming mode with a reasonable size file, it seems to choke at regular points - I'm guessing its at chunk boundaries. Each time it stops part way through a record and inserts a closing and opening bracket ][ and then continues from the next record.

The csv file I'm using has about 5000 lines and the breaks occur at roughly every 560 lines.

Line lengths are variable but less than 150 characters.

There are 12 columns in the data.

[mcqj]

Actually, looking at the code, it seems that streaming mode is not going to work since it calls the functions in app.js for every chunk and those functions output a complete array.

[charanguru]

I am also experiencing this issue. @pradeep-mishra please remove support for streaming data since it's clear that this is not supported at this time.

[ezcolombo]

bumped into this as well with not so big csv files ... (8914 lines / 760K) - think I solved it by piping through the following Transform before toObject. Anyway will be great to solve this correctly ...

const { Transform } = require('stream')

function friendlyChunks() {
    const queue = []

    return new Transform({
      transform(chunk, encoding, callback) {
        if (queue.length) {
          chunk = Buffer.concat([queue.pop(), chunk])
        }

        let offset = chunk.length

        for (let i = offset, found = false;
          (!found && i);
          i -= 1) {
          if (chunk[i] === 10
              || chunk[i] === 13) {
            found = true
            offset = i
          }
        }

        queue.push(Buffer.from(chunk.slice(offset + 1, chunk.length)))

        this.push(chunk.slice(0, offset))
        callback()
      },
    })
  }

read.pipe(FriendlyChunks()).pipe(toObject).pipe(stringify).pipe(write)