Re: Getting rid of the mandb profile hook?

[Ludovic Courtès]

Hello!

Brice Waegeneire <brice@waegenei.re> skribis:

> On 2021-03-03 15:13, Ludovic Courtès wrote:
>>>> I’m thinking we could get rid of the mandb hook.  However, the

[...]

> What about using mandoc¹, the manpage compiler from OpenBSD, instead of
> man-db? As from it's manual it support specifying the database location:
>
> “makewhatis -d dir [file ...]”²

I recently packaged it, but I’m not impressed; I’m not even sure how
that’s supposed to work:

  $ guix environment --ad-hoc mandoc -- makewhatis -d /tmp/foo $(find -L 
~/.guix-profile/share/man -name \*.[0-9].gz)

exits successfully but does nothing.

At this point my preference would be to build a custom tool (I’m not
aware of any existing tool like that, but if you do, please share) that
would lazily build a database, ideally full-text, and search through it;
attached a super rough example that uses Guile-Xapian and inserts man-db
synopses into a Xapian database.

The tool would index man pages and Info pages.  It would be smart enough
to index only info/man files that have actually changed (it could look
at the inode number to determine in a way that avoids unnecessary cache
invalidation).  I’m not sure how to implement this part though.  It
sounds like a good hack for our Xapian experts—I’m looking at you Arun,
Ricardo, zimoun.  :-)

Thoughts?

I’d really like to have a rough solution so we can remove the
‘manual-database’ hook in time for the release.

Thoughts?

Ludo’.

(use-modules (xapian wrap)
             (xapian xapian)
             (ice-9 match)
             (guix man-db)
             (srfi srfi-1)
             (srfi srfi-26))

;; eval: (put 'call-with-writable-database 'scheme-indent-function 1)

(define (index-mandb-entry db entry)
  (define (mandb-entry-id-term entry)
    (string-append "Q" "man:" (mandb-entry-name entry) "."
                   (number->string
                    (mandb-entry-section entry))))

  (when (mandb-entry-name entry)
    (let* ((idterm (mandb-entry-id-term entry))
           (doc (make-document
                 #:data (object->string
                         `((name . ,(mandb-entry-name entry))
                           (section . ,(number->string
                                        (mandb-entry-section entry)))
                           (file . ,(canonicalize-path
                                     (mandb-entry-file-name entry)))))
                 #:terms `((,idterm . 0))))
           (term-generator
            (make-term-generator #:stem (make-stem "en")
                                 #:document doc)))
      (index-text! term-generator (mandb-entry-name entry) #:prefix "A")
      (index-text! term-generator
                   (number->string (mandb-entry-section entry))
                   #:prefix "B")
      (index-text! term-generator (mandb-entry-synopsis entry))
      (replace-document! db idterm doc))))

(define (index-mandb-entries)
  (call-with-writable-database "/tmp/db"
    (lambda (db)
      (for-each (cut index-mandb-entry db <>)
                ;; (mandb-entries "/run/current-system/profile/share/man")
                (append-map mandb-entries
                            (string-split (getenv "MANPATH") #\:))
                ))))

(define* (parse-query* querystring #:key stemmer stemming-strategy
                       (prefixes '())
                       (boolean-prefixes '()))
  (let ((queryparser (new-QueryParser)))
    (QueryParser-set-stemmer queryparser stemmer)
    (when stemming-strategy
      (QueryParser-set-stemming-strategy queryparser stemming-strategy))
    (for-each (match-lambda
                ((field . prefix)
                 (QueryParser-add-prefix queryparser field prefix)))
              prefixes)
    (for-each (match-lambda
                ((field . prefix)
                 (QueryParser-add-boolean-prefix queryparser field prefix)))
              boolean-prefixes)
    (let ((query (QueryParser-parse-query queryparser querystring)))
      (delete-QueryParser queryparser)
      query)))

(define* (search querystring #:key (pagesize 100))
  (call-with-database "/tmp/db"
    (lambda (db)
      (let* ((query (parse-query querystring
                                  #:stemmer (make-stem "en")
                                  #:prefixes
                                  '(("name"    . "A")
                                    ("section" . "B"))))
             (enq (enquire db query)))
        ;; (Enquire-set-sort-by-value enq 0 #f)
        (reverse (mset-fold (lambda (item acc)
                              (cons (call-with-input-string
                                        (document-data (mset-item-document 
item))
                                      read)
                                    acc))
                            '()
                            (enquire-mset enq
                                          #:maximum-items pagesize)))))))