Linux shell逐行读取文件的方法

首先我们用脚本 (脚本见附件)生成一个 70000行的文件,文件位置在/scripts/bigfile 。然后通过下面的脚本来测试各个方法的执行效率,脚本很简单,不再解释。

read_line.sh
#!/bin/bash
FILENAME="$1"
TIMEFILE="/tmp/loopfile.out" > $TIMEFILE
SCRIPT=$(basename $0)
 
function usage(){
echo -e "\nUSAGE: $SCRIPT file \n"
 
exit 1
}
 
function while_read_bottm(){
 
while read LINE
do
echo $LINE
 
done < $FILENAME
 
}
 
function while_read_line(){
 
cat $FILENAME | while read LINE
do
echo $LINE
done
 
}
 
 function while_read_line_fd (){
 
exec 3 <&0
exec 0 < $FILENAME
while read LINE
do
 echo $LINE
done
 exec 0<&3
}
 
function for_in_file(){
for i in   `cat $FILENAME`
do
echo $i
done
}
 
if [ $# -lt 1 ] ; then
usage
fi
 echo -e " \n starting file processing of each method\n"
 echo -e "method 1:"
 echo -e "function while_read_bottm"
 time while_read_bottm >> $TIMEFILE
 
echo -e "\n"
 
echo -e "method 2:"
echo -e "function while_read_line "
time while_read_line >> $TIMEFILE
 
echo -e "\n"
echo -e "method 3:"
echo "function while_read_line_fd"
time while_read_line_fd >>$TIMEFILE
 
echo -e "\n"
echo -e "method 4:"
echo -e "function  for_in_file"
time  for_in_file >> $TIMEFILE
./make.sh 300000
./while.sh /scripts/bigfile
method 1:
function while_read_bottm
real    0m13.913s
user    0m12.947s
sys     0m0.961s

method 2:
function while_read_line
real    0m18.102s
user    0m13.067s
sys     0m5.025s

method 3:
function while_read_line_fd
real    0m13.456s
user    0m12.360s
sys     0m1.089s

method 4:
function   for_in_file
real    0m13.777s
user    0m12.717s
sys     0m1.055s
ll -h /scripts/bigfile
-rw-r--r-- 1 root root 24M Jun 26 10: 56 /scripts /bigfile

由此可见在各个方法中, for语句效率最高

while read LINE
do
echo $LINE
done < $FILENAME
  • linux/shell/shell逐行读取文件.txt
  • 最后更改: 2019/04/16 18:31
  • (外部编辑)