Re: File search

[Ludovic Courtès]

Hi!

Mathieu Othacehe <othacehe@gnu.org> skribis:

>> I think accuracy (making sure you get results that correspond precisely
>> to, say, your current channel revisions and your current system) is not
>> a high priority: some result is better than no result.  Likewise for
>> freshness: results for an older version of a given package may still be
>> valid now.
>
> Agreed.
>
>> In terms of privacy, I think it’s better if we can avoid making one
>> request per file searched for.  Off-line operation would be sweet, and
>> it comes with responsiveness; fast off-line search is necessary for
>> things like ‘command-not-found’ (where the shell tells you what package
>> to install when a command is not found).
>
> Yeah, that's the tricky part. In term of maintenance, it would probably
> be easier to have Cuirass index the packages it's building, store the
> results in the PostgreSQL database and serve them using the Cuirass web
> server. The pros are that we only rely on one database which is very
> important in my opinion. It's also relatively easy to setup. The cons
> are that you need to be online to access this API.

Like I wrote, I don’t think we should do on-line only; we need users to
be able to download a database at least for ‘command-not-found’.

> If we instead decide to build periodically an sqlite database indexing
> all the packages in a cronjob or so, it would still be needed for the
> users to download it, which would be an expensive operation as you
> mentioned. It would also be difficult to index custom Guix channels with
> that approach.

True!  Though I for this matter I’d be very pragmatic and wouldn’t mind
sacrificing third-party channels until we have a better idea.

> Another solution could be to have guix publish index the files from the
> NAR in its cache and provide a file searching API. That would still
> require to be online, but it would allow to search from multiple publish
> servers hence possibly multiple Guix channels. The packages that do not
> have substitutes couldn't be searched which is a strong cons. I would
> still maybe have a preference for that option.

I also thought about doing it in ‘guix publish’.  One problem is that
it’s not the right level of abstraction: it publishes everything, not
just packages, and it can only guess whether something is a package and
what its name and version are.

Another option would be to have ‘guix publish’ provide digests (file
lists), similar to what I did in:

  https://lists.gnu.org/archive/html/guix-devel/2021-01/msg00080.html
  https://git.savannah.gnu.org/cgit/guix.git/log?h=wip-digests

That way, ‘file-database.scm populate’ could fetch those digests instead
of whole nars (or solely local info).  Users would have to run it
regularly.


The Nix folks have <https://github.com/bennofs/nix-index>, which
apparently creates a database based on substitutes.  Looking at
<https://github.com/bennofs/nix-index/blob/master/src/hydra.rs#L401>, it
seems Hydra provides “file listings”, similar to digests.

Then it’s up to users to regularly run the indexer so they can use
‘nix-locate’, which is typically done via a local cron job (similar to
how one would use ‘updatedb’.)  Populating the database the first time
may be rather costly though.

Maybe that’s a more reasonable approach?


All that said, I think we could very much have, in parallel, a fancier
database, be it in the Data Service or in Cuirass, that one could query
on-line.  Its implementation would actually be less constrained.

Ludo’.