Re: Loading a large and unusually formatted dataset into an Octave matrix

[Przemek Klosowski]

On 06/14/2013 02:04 PM, Elliot Gorokhovsky wrote:
> > > Hello! I am a new octave user and I am trying to predict the price of
> > > bitcoins 15 minutes in advance via neural networks for use on the
> > > website btcoracle.com <http://btcoracle.com> <http://btcoracle.com>.
> > I have about a gigabyte of
> > > data that looks like this:

{"data":1279408157,"price":"0.04951","amount":"20","price_int_":"4951","amount_int":"2000000000","tid":"1","price_currency":"USD","item":"BTC","trade_type":""},
{"data":1279424586,"price":"0.05941","amount":"50.01","price_int_":"5941","amount_int":"5001000000","tid":"2","price_currency":"USD","item":"BTC","trade_type":""},
{"data":1279475336,"price":"0.08080","amount":"5","price_int_":"8080","amount_int":"500000000","tid":"3","price_currency":"USD","item":"BTC","trade_type":""}

I responded, trying   a=fscanf(FD,"%d %d")

Eh, make it a=fscanf(FD,"%f %f") of course:

command="perl -F'\"' -lane 'print \"$F[5] $F[9]\"' /tmp/bitcoin";
a=reshape(fscanf(popen(command,'r'),"%f"),2,[]);

-lane is a very useful Perl idiom that splits every line into words into 
array F ($F[0], $F[1], etc). -F" changes the word break character to the 
double quote. After that, some judicious quoting of special characters 
et voila.

As usual, I got bitten by *scanf code failing silently unless the format 
string is perfect. Does anyone have good tips on debugging issues like 
that? a way to figure out how far into the input did the format string 
match?