DCG over a large file...?

[emacstheviking]

I am getting stuck into something and I was wondering if there is way
(I am still thinking about this myself) to be able to use "DCG" over a
file ?

My gut reaction is that no you can't because the current push-back
limit is only eight characters according to gprolog and backtracking
over large strings could easily thwart that in no time.

A complete DCG grammar will only work if the entire source text is
loaded into memory in the initial list as far as I understand, for
very large files this may not be an option and it smacks of
inefficiency too.

I was thinking more along the lines of streaming iterators like you
get in Haskell, attoparsec and such like use them to good effect.

Having run into issues processing large files in the last day or so, I
want to focus on writing some kind of FSA that uses no more than 1
character look-ahead to parse my file BUT I would like to be able to
use DCG notation if possible. The goal (no pun intended) is that the
only real memory consumption is going to be the list of tokens created
as the file is consumed.

Can this be done, I am sure it has been done a million times, how
would one wire up a "list" to file-read descriptor. I guess there is
"C" code but I want to know if native Prolog can cut it. Have I missed
something fundamental perhaps?

Thanks,
Sean.