Do mlr or mlr cat have an option for specifying the name of a file that contains a list of CSV file arguments?

[derekmahar]

Do mlr or mlr cat have an option for specifying the name of a file that contains a list of CSV file arguments? I'm thinking of a command like find . -type f -name "*.csv" | mlr --csv --infile-list=- cat that could replace the longer and more cumbersome command find . -type f -name "*.csv" | (read first; mlr --csv cat $first; xargs mlr --csv --headerless-csv-output).

[aborruso]

I don't think there is. Like you, I would use bash

find /sample/path-type f -name "*.csv" | while IFS= read -r file; do
  mlr --csv whateverYouWant "$file"
done

[aborruso]

This won't work either for the same reason as in your previous solution.

Okay, but it is not at all the same reason as my previous solution 🙂

As you said, it won't work in the case where the total length of the command and input file names exceeds the maximum command line length.

[aborruso]

@derekmahar is something that can be done in append? If yes (using something without header) you could use a while loop.

If you could paste the sample command you need to run, I could try to help you better

[derekmahar]

Absent an option that accepts a file that contains the file names of all of the CSV file arguments, there are only two general solutions that will work.

The first solution is the one that I gave in my first example which prints the first CSV file with the header followed by the remaining files without their headers.

The second solution is to recursively concatenate the CSV files in groups until there is only one CSV file left. One way to implement this solution would be to invoke xargs mlr --csv cat on groups of the CSV files, capture the result of each invocation in a temporary intermediate CSV file, and then repeat the command on groups of the intermediate CSV files until there is just one CSV file left.

This second method isn't very efficient, though, because it copies the same rows more than once.

[derekmahar]

If you could paste the sample command you need to run, I could try to help you better

The sample command is the one that I gave in my first message where I mlr --csv cat the first file with the header followed by the remaining files without their headers.

[aborruso]

The sample command is the one that I gave in my first message where I mlr --csv cat the first file with the header followed by the remaining files without their headers.

Many times, for cases like these I write in json lines format, which allows me to go into append and at the same time keep the header.

Using some while loop or some subshell like yours.

[johnkerl]

Tracked in #1422

[johnkerl]

For posterity: this is now mlr --mfrom *.csv -- ...