Cfrgtkky

I have a bunch of .csv files with N columns and different number of rows (lines). I would like to add as many empty lines ;...; (N semicolons) to make them the same length. I can get the length of the longest file manually but it would also be good to get this done automatically.

For example:

I have,

file1.csv

128; pep; 93; 22:22:10; 3; 11

127; qep; 93; 12:52:10; 3; 15

171; pep; 73; 22:26:10; 3; 72

file2.csv

128; pep; 93; 22:22:10; 3; 11

127; qep; 93; 12:52:10; 3; 15

121; fng; 96; 09:42:10; 3; 52

141; gep; 53; 21:22:10; 3; 62

171; pep; 73; 22:26:10; 3; 72

221; ahp; 93; 23:52:10; 3; 892

file3.csv

121; fng; 96; 09:42:10; 3; 52

171; pep; 73; 22:26:10; 3; 72

221; ahp; 93; 23:52:10; 3; 892

141; gep; 53; 21:22:10; 3; 62

I need,

file1.csv

128; pep; 93; 22:22:10; 3; 11

127; qep; 93; 12:52:10; 3; 15

171; pep; 73; 22:26:10; 3; 72

;;;;;

;;;;;

;;;;;

file2.csv

128; pep; 93; 22:22:10; 3; 11

127; qep; 93; 12:52:10; 3; 15

121; fng; 96; 09:42:10; 3; 52

141; gep; 53; 21:22:10; 3; 62

171; pep; 73; 22:26:10; 3; 72

221; ahp; 93; 23:52:10; 3; 892

file3.csv

121; fng; 96; 09:42:10; 3; 52

171; pep; 73; 22:26:10; 3; 72

221; ahp; 93; 23:52:10; 3; 892

141; gep; 53; 21:22:10; 3; 62

;;;;;

;;;;;

Thanks @Sparhawk for the suggestions in the comments, I update based on those,

#!/bin/bash



emptyLine=;;;;;;;

rr=($(wc -l files*pattern.txt |  awk '{print $1}' | sed '$ d'))

max=$(echo "${rr[*]}" | sort -nr | head -n1)

for name in files*pattern.txt;do

    lineNumber=$(wc -l < $name)

    let missing=max-lineNumber

    for((i=0;i<$missing;i++));do

        echo $emptyLine >> $name

    done

done

Well, not elegand nor efficient. Actually, it takes a couple of seconds which sounds an eternity given the small size of the data. Nevertheless it works,

#!/bin/bash



emptyLine=;;;;;;;

rr=($(wc -l files*pattern.txt |  awk '{print $1}' | sed '$ d'))

max=$(echo "${rr[*]}" | sort -nr | head -n1)

for name in $(ls files*pattern.txt);do

    lineNumber=$(cat $name | wc -l )

    let missing=max-lineNumber

    for((i=0;i<$missing;i++));do

        echo $emptyLine >> $name

    done

done

I just put this file together in the directory where I have the files provided that there is a pattern I can use to list them with files*pattern.txt

An improvement of @myradio's answer.

The part inside the loop written in awk which should be much faster.

max=$(wc -l file*.csv | sed '$ d' | sort -n | tail -n1 | awk '{print $1}' )

for f in file*.csv; do

    awk -F';' -v max=$max 

      'END{

         s=sprintf("%*s",FS,"");

         gsub(/ /,"-",s);

         for(i=NR;i<max;i++)

           print s;

       }' "$f" >> "$f"

done

With -F you set the correct field separator of your files (here -F';').

The s=sprintf();gsub(); part dynamically sets the right amount of the FS (= field separator) (via).

You could simply replace that with print ";;;;;" or other static content if you like.

In order to count the lines in each file only once:

wc -l *csv |sort -nr| sed 1d | {

    read max file

    pad=$(sed q "$file"|tr -cd ";")  # extract separators from first record

    while read lines file ; do

        while [ $((lines+=1)) -le $max ] ; do

                echo "$pad" >> "$file"

        done

    done

}

Note that any newlines in the filenames will cause problems for both sort and the while read loop, but they can handle filenames containing normal spaces.