[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Error for large input files >2GB
From: |
Greg Wooledge |
Subject: |
Re: Error for large input files >2GB |
Date: |
Wed, 8 Feb 2012 08:40:38 -0500 |
User-agent: |
Mutt/1.4.2.3i |
I don't quite understand what your error report is saying, but when I
was trying to read your script, I couldn't help noticing a few things:
On Tue, Feb 07, 2012 at 06:24:11PM +0100, Hardy Flor wrote:
> #!/bin/bash
>
> inputfile="<large list of files>"
Actually it's just one file name.
> leer_max=" "
> for i in $(seq -f "%02g" 3 ${#leer_max}); do
> eval "leer_$i=\"${leer_max:0:10#$i}\""
> done
Huh?
> line=0
> while read inode size dat1 dat2 filepath; do
> let "line++"
> echo -en "${leer_08:${#line}}$line\r"
If you're trying to do a continuously overwritten counter, you're making
it much more complicated than it needs to be. Consider:
for ((i=7; i<=13; i++)); do
printf "\r%-3d" $i
sleep 1
done
echo
You definitely don't need a whole fleet of leer_nn variables. Even if
you actually did need that many of them, you'd be better served by making
an array instead of a bunch of separate strings.
> if [ -r "$filepath" ]; then
> set -- $(stat -c "%i %Y" "$filepath")
An alternative here would be:
read real_inode real_mtime < <(stat -c "%i %Y" "$filepath")
That's not a dramatic improvement, though.
> test $inode -ne $1 && echo "Inode from \"$filepath\" is wrong"
> let "ut_diff = $(date -d "$dat1 $dat2" +%s) - $2"
> test $ut_diff -ne 0 -a $ut_diff -ne -3600 -a $ut_diff -ne 3600 && echo
> "Date from \"$filepath\" faulty ($ut_diff)"
Chaining with -a in the "test" command is not supported. Since you're
already using bash syntax (let, ${string:start:len}, etc.), you might as
well just use [[ which can actually do that:
[[ $ut_diff != 0 && $ut_diff != -3600 && $ut_diff != 3600 ]] && echo ...
If you must use test or [ in future scripts (for POSIX compatibility),
then you must use more than one of them:
test $ut_diff != 0 && test $ut_diff != -3600 && ...
> else
> echo "File \"$filepath\" is missing"
> fi
> done <$inputfile
I don't see anything here which would cause the script to malfunction
after reading multiple gigabytes of input, unless your operating system
has errors when reading from a file of that size.