Re: File search

[Mathieu Othacehe]

Hello Ludo!

> Lately I found myself going several times to
> <https://packages.debian.org> to look for packages providing a given
> file and I thought it’s time to do something about it.

Yeah, I'm also thinking regularly about it but giving up because setting
up this mechanism properly turns out to be much more complex than
initially expected.

> The script below creates an SQLite database for the current set of
> packages, but only for those already in the store:
>
>   guix repl file-database.scm populate
>
> That creates /tmp/db; it took about 25mn on berlin, for 18K packages.
> Then you can run, say:
>
>   guix repl file-database.scm search boot-9.scm

Nice proof of concept :).

> I think accuracy (making sure you get results that correspond precisely
> to, say, your current channel revisions and your current system) is not
> a high priority: some result is better than no result.  Likewise for
> freshness: results for an older version of a given package may still be
> valid now.

Agreed.

> In terms of privacy, I think it’s better if we can avoid making one
> request per file searched for.  Off-line operation would be sweet, and
> it comes with responsiveness; fast off-line search is necessary for
> things like ‘command-not-found’ (where the shell tells you what package
> to install when a command is not found).

Yeah, that's the tricky part. In term of maintenance, it would probably
be easier to have Cuirass index the packages it's building, store the
results in the PostgreSQL database and serve them using the Cuirass web
server. The pros are that we only rely on one database which is very
important in my opinion. It's also relatively easy to setup. The cons
are that you need to be online to access this API.

If we instead decide to build periodically an sqlite database indexing
all the packages in a cronjob or so, it would still be needed for the
users to download it, which would be an expensive operation as you
mentioned. It would also be difficult to index custom Guix channels with
that approach.

Another solution could be to have guix publish index the files from the
NAR in its cache and provide a file searching API. That would still
require to be online, but it would allow to search from multiple publish
servers hence possibly multiple Guix channels. The packages that do not
have substitutes couldn't be searched which is a strong cons. I would
still maybe have a preference for that option.

WDYT?

Thanks,

Mathieu