[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
improve performance of a script
From: |
xeon Mailinglist |
Subject: |
improve performance of a script |
Date: |
Tue, 25 Mar 2014 07:12:16 -0700 (PDT) |
User-agent: |
G2/1.0 |
For each file inside the directory $output, I do a cat to the file and generate
a sha256 hash. This script takes 9 minutes to read 105 files, with the total
data of 556MB and generate the digests. Is there a way to make this script
faster? Maybe generate digests in parallel?
for path in $output
do
# sha256sum
digests[$count]=$( $HADOOP_HOME/bin/hdfs dfs -cat "$path" | sha256sum | awk
'{ print $1 }')
(( count ++ ))
done
Thanks,
- improve performance of a script,
xeon Mailinglist <=