[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[bug#27876] [PATCH v2 2/3] cuirass: add Hydra compatible HTTP API.
|
From: |
Mathieu Othacehe |
|
Subject: |
[bug#27876] [PATCH v2 2/3] cuirass: add Hydra compatible HTTP API. |
|
Date: |
Tue, 1 Aug 2017 21:51:23 +0200 |
* doc/cuirass.texi (Sections)[Web API]: New section describing the HTTP API.
* src/cuirass/http.scm (spec->json-string): Move it to utils.scm and rename it
object->json-string.
(object->json-scm): Move it utils.scm.
(handle-*-request): New helpers procedures.
(request-parameters): New procedure to parse a request query.
(url-handler): Add new API's.
* src/cuirass/utils.scm (object->json-scm, object->json-string): Exported
procedures moved from http.scm.
---
doc/cuirass.texi | 191 +++++++++++++++++++++++++++++++++++++++++++++++
src/cuirass/database.scm | 29 +++----
src/cuirass/http.scm | 150 ++++++++++++++++++++++++++++++-------
src/cuirass/utils.scm | 22 +++++-
4 files changed, 350 insertions(+), 42 deletions(-)
diff --git a/doc/cuirass.texi b/doc/cuirass.texi
index 443b53c..bac2e54 100644
--- a/doc/cuirass.texi
+++ b/doc/cuirass.texi
@@ -57,6 +57,7 @@ Tutorial sections:
Reference sections:
* Invocation:: How to run Cuirass.
* Database:: About the database schema.
+* Web API:: Description of the Web API.
* Contributing:: Your help needed!
* GNU Free Documentation License:: The license of this manual.
@@ -380,8 +381,198 @@ This text field holds the name of the output.
This text field holds the path of the output.
@end table
+
address@hidden
*********************************************************************
address@hidden Web API
address@hidden Web API
address@hidden web api
+
+Cuirass web API is derived from Hydra one, see
@url{https://github.com/NixOS/hydra/blob/master/doc/manual/api.xml, Hydra API
description}.
+
+For now only a subset of this API is implemented.
+
address@hidden API description
address@hidden description, json
+
address@hidden Build information
+
+It is possible to query Cuirass web server for build informations. The
+dedicated API is "/build/@var{build-id}" where @var{build-id} is the
+unique id associated to the build in database.
+
+For instance, querying a local Cuirass web server can be done with
address@hidden and @code{jq} to format the JSON response :
+
address@hidden
+$ curl -s "http://localhost:8080/build/2"; | jq
+
address@hidden
+ "id": 2,
+ "project": "guix",
+ "jobset": "master",
+ "job": "acpica-20150410-job",
+ "timestamp": 1501347493,
+ "starttime": 1501347493,
+ "stoptime": 1501347493,
+ "buildoutputs": @{
+ "out": @{
+ "path": "/gnu/store/6g3njhfzqpdm335s7qhvmwvs5l7gcbq1-acpica-20150410"
+ @}
+ @},
+ "system": "x86_64-linux",
+ "nixname": "acpica-20150410",
+ "buildstatus": 0,
+ "busy": 0,
+ "priority": 0,
+ "finished": 1,
+ "buildproducts": null,
+ "releasename": null,
+ "buildinputs_builds": null
address@hidden
address@hidden example
+
+If requested @var{build-id} is not known, the HTTP code 404 is
+answered with a JSON error message. For example :
+
address@hidden
+$ curl -s "http://localhost:8080/build/fff";
+
address@hidden"error" : "Build with ID fff doesn't exist."@}
address@hidden example
+
+The nominal output is a JSON object whose fields are described
+hereafter.
+
address@hidden @code
address@hidden id
+The unique build id.
+
address@hidden project
+The associated specification name, as a string.
+
address@hidden jobset
+The associated specification branch, as a string.
+
address@hidden job
+The associated job-name, as a string.
+
address@hidden timestamp
+Timestamp taken at build creation time.
+
address@hidden starttime
+Timestamp taken at build start time.
+
address@hidden stoptime
+Timestamp taken at build stop time.
+
address@hidden buildoutputs
+Build outputs as a JSON object. The keys names are referring to the
+eventual output names. The associated value is another JSON object which
+only key is @code{path}. @code{path} value is the output directory in
+store as a string.
+
address@hidden system
+System name of the build, as a string.
+
address@hidden nixname
+Derivation name, as a string.
+
address@hidden buildstatus
+Build status, as an integer. Possible values are :
+
address@hidden
+0 -> succeeded
+1 -> failed
+2 -> failed dependency
+3 -> failed other
+4 -> cancelled
address@hidden example
+
address@hidden busy
+Whether the build is pending, as an integer (not implemented yet).
+
address@hidden priority
+Build priority, as an integer (not implemented yet).
+
address@hidden finished
+Build finished, as an integer (not implemented yet : always 1).
+
address@hidden buildproducts
+Build products in store as a JSON object (not implemented yet).
+
address@hidden releasename
+Unknown, not implemented yet.
+
address@hidden buildinputs_builds
+Inputs used for the build, as a JSON object (not implemented yet).
+
address@hidden table
+
address@hidden Build raw log output
+
+It is possible to ask Cuirass for the raw build output log with the API
+"/build/@var{build-id}/log/raw" where @var{build-id} is the
+unique id associated to the build in database.
+
+The output is a raw text, for example :
+
address@hidden
+$ curl http://localhost:8080/build/2/log/raw
+
+starting phase `set-SOURCE-DATE-EPOCH'
+phase `set-SOURCE-DATE-EPOCH' succeeded after 0.0 seconds
+starting phase `set-paths'
+...
address@hidden example
+
+If requested @var{build-id} is not known, the HTTP code 404 is
+answered with a JSON error message. For example :
+
address@hidden
+$ curl -s "http://localhost:8080/build/fff/log/raw";
+
address@hidden"error" : "Build with ID fff doesn't exist."@}
address@hidden example
+
address@hidden Latest builds
+
+The list of latest builds can be obtained with the API
+"/api/latestbuilds". The output is a JSON array of
+builds. Builds are represented as in "/build/@var{build-id} API.
+
+This request accepts a mandatory parameter and multiple optional ones.
+
address@hidden @code
address@hidden nr
+Limit query result to nr elements. This parameter is @emph{mandatory}.
+
address@hidden project
+Filter query result to builds with the given @code{project}.
+
address@hidden jobset
+Filter query result to builds with the given @code{jobset}.
+
address@hidden job
+Filter query result to builds with the given @code{job} name.
+
address@hidden system
+Filter query result to builds with the given @code{system}.
+
@end table
+For example, to ask for the ten last builds :
+
address@hidden
+$ curl "http://localhost:8080/api/latestbuilds?nr=10";
address@hidden example
+
+or the five last builds which project is ``guix'' and jobset ``master' :
+
address@hidden
+$ curl "http://localhost:8080/api/latestbuilds?nr=5&project=guix&jobset=master";
address@hidden example
+
+If no builds matching given parameters are found and empty JSON array is
returned.
@c *********************************************************************
@node Contributing
diff --git a/src/cuirass/database.scm b/src/cuirass/database.scm
index 5f60fac..6b84a48 100644
--- a/src/cuirass/database.scm
+++ b/src/cuirass/database.scm
@@ -226,7 +226,7 @@ INSERT INTO Outputs (build, name, path) VALUES ('~A', '~A',
'~A');"
outputs))))))
(define db-build-request "\
-SELECT Builds.id, Builds.timestamp, Builds.starttime, Builds.stoptime,
Builds.log, Builds.status,\
+SELECT Builds.id, Builds.timestamp, Builds.starttime, Builds.stoptime,
Builds.log, Builds.status, Builds.derivation,\
Derivations.job_name, Derivations.system, Derivations.nix_name,\
Specifications.repo_name, Specifications.branch \
FROM Builds \
@@ -236,20 +236,21 @@ INNER JOIN Specifications ON Evaluations.specification =
Specifications.repo_nam
(define (db-format-build db build)
(match build
- (#(id timestamp starttime stoptime log status job-name system
+ (#(id timestamp starttime stoptime log status derivation job-name system
nix-name repo-name branch)
- `((#:id . ,id)
- (#:timestamp . ,timestamp)
- (#:starttime . ,starttime)
- (#:stoptime . ,stoptime)
- (#:log . ,log)
- (#:status . ,status)
- (#:job-name . ,job-name)
- (#:system . ,system)
- (#:nix-name . ,nix-name)
- (#:repo-name . ,repo-name)
- (#:outputs . ,(db-get-outputs db id))
- (#:branch . ,branch)))))
+ `((#:id . ,id)
+ (#:timestamp . ,timestamp)
+ (#:starttime . ,starttime)
+ (#:stoptime . ,stoptime)
+ (#:log . ,log)
+ (#:status . ,status)
+ (#:derivation . ,derivation)
+ (#:job-name . ,job-name)
+ (#:system . ,system)
+ (#:nix-name . ,nix-name)
+ (#:repo-name . ,repo-name)
+ (#:outputs . ,(db-get-outputs db id))
+ (#:branch . ,branch)))))
(define (db-get-build db id)
"Retrieve a build in database DB which corresponds to ID."
diff --git a/src/cuirass/http.scm b/src/cuirass/http.scm
index 33cd37b..23c3ad7 100644
--- a/src/cuirass/http.scm
+++ b/src/cuirass/http.scm
@@ -1,5 +1,6 @@
;;;; http.scm -- HTTP API
;;; Copyright © 2016 Mathieu Lirzin <address@hidden>
+;;; Copyright © 2017 Mathieu Othacehe <address@hidden>
;;;
;;; This file is part of Cuirass.
;;;
@@ -19,52 +20,147 @@
(define-module (cuirass http)
#:use-module (cuirass database)
#:use-module (cuirass utils)
- #:use-module (ice-9 hash-table)
+ #:use-module (guix config)
+ #:use-module (guix build utils)
+ #:use-module (guix utils)
#:use-module (ice-9 match)
+ #:use-module (ice-9 popen)
#:use-module (json)
#:use-module (web request)
#:use-module (web response)
#:use-module (web server)
#:use-module (web uri)
- #:export (spec->json-string
- run-cuirass-server))
+ #:export (run-cuirass-server))
-;;;
-;;; JSON format.
-;;;
+(define (build->hydra-build build)
+ "Convert BUILD to an assoc list matching hydra API format."
+ `((#:id . ,(assq-ref build #:id))
+ (#:project . ,(assq-ref build #:repo-name))
+ (#:jobset . ,(assq-ref build #:branch))
+ (#:job . ,(assq-ref build #:job-name))
+ (#:timestamp . ,(assq-ref build #:timestamp))
+ (#:starttime . ,(assq-ref build #:starttime))
+ (#:stoptime . ,(assq-ref build #:stoptime))
+ (#:buildoutputs . ,(assq-ref build #:outputs))
+ (#:system . ,(assq-ref build #:system))
+ (#:nixname . ,(assq-ref build #:nix-name))
+ (#:buildstatus . ,(assq-ref build #:status))
+
+ ;; TODO: Fill the fields above with correct values.
+ (#:busy . 0)
+ (#:priority . 0)
+ (#:finished . 1)
+ (#:buildproducts . #nil)
+ (#:releasename . #nil)
+ (#:buildinputs_builds . #nil)))
+
+(define (handle-build-request db build-id)
+ "Retrieve build identified by BUILD-ID in DB and convert it to hydra
+ format. Return #f is not build was found."
+ (let ((build (db-get-build db build-id)))
+ (and=> build build->hydra-build)))
-(define (object->json-scm obj)
- "Prepare OBJ for JSON usage."
- (cond ((string? obj) obj)
- ((number? obj) obj)
- ((boolean? obj) obj)
- ((null? obj) obj)
- ((symbol? obj) (symbol->string obj))
- ((keyword? obj) (object->json-scm (keyword->symbol obj)))
- ((alist? obj) (alist->hash-table (map object->json-scm obj)))
- ((pair? obj) (cons (object->json-scm (car obj))
- (object->json-scm (cdr obj))))
- (else (object->string obj))))
-
-(define* (spec->json-string spec #:key pretty)
- "Return SPEC as a JSON object."
- (scm->json-string (object->json-scm spec) #:pretty pretty))
+(define (handle-builds-request db filters)
+ "Retrieve all builds matched by FILTERS in DB and convert them to hydra
+ format."
+ (let ((builds (db-get-builds db filters)))
+ (map build->hydra-build builds)))
+
+(define (handle-log-request db build)
+ "Retrieve the log file of BUILD. Return a lambda which PORT argument is an
+ input port from which the content of the log file can be read or #f if the
+ log file is not readable."
+ (let* ((log (assq-ref build #:log))
+ (access (and (string? log)
+ (access? log R_OK))))
+ (and access
+ (lambda (out-port)
+ (let ((in-pipe-port
+ (open-input-pipe
+ (format #f "~a -dc ~a" %bzip2 log))))
+ (dump-port in-pipe-port out-port)
+ (close-pipe in-pipe-port))))))
+
+(define (request-parameters request)
+ "Parse the REQUEST query parameters and return them under the form
+ '((parameter value) ...)."
+ (let* ((uri (request-uri request))
+ (query (uri-query uri)))
+ (and query
+ (map (lambda (param)
+ (match (string-split param #\=)
+ ((key param)
+ (list (string->symbol key) param))))
+ (string-split query #\&)))))
;;;
;;; Web server.
;;;
+;;; The api is derived from the hydra one. It is partially described here :
+;;;
+;;; https://github.com/NixOS/hydra/blob/master/doc/manual/api.xml
+;;;
(define (request-path-components request)
(split-and-decode-uri-path (uri-path (request-uri request))))
(define (url-handler request body db)
+
(define* (respond response #:key body (db db))
(values response body db))
+
+ (define-syntax-rule (respond-json body ...)
+ (respond '((content-type . (application/json)))
+ #:body body ...))
+
+ (define-syntax-rule (respond-text body ...)
+ (respond '((content-type . (text/plain)))
+ #:body body ...))
+
+ (define-syntax-rule (respond-json-with-error error-code message)
+ (respond
+ (build-response #:headers '((content-type . (application/json)))
+ #:code error-code)
+ #:body
+ (object->json-string
+ `((error . ,message)))))
+
+ (define (respond-build-not-found build-id)
+ (respond-json-with-error
+ 404
+ (format #f "Build with ID ~a doesn't exist." build-id)))
+
+ (define (respond-build-log-not-found build)
+ (let ((drv (assq-ref build #:derivation)))
+ (respond-json-with-error
+ 404
+ (format #f "The build log of derivation ~a is not available." drv))))
+
(match (request-path-components request)
(((or "jobsets" "specifications") . rest)
- (respond '((content-type . (application/json)))
- #:body (spec->json-string (car (db-get-specifications db)))))
+ (respond-json (object->json-string (car (db-get-specifications db)))))
+ (("build" build-id)
+ (let ((hydra-build (handle-build-request db build-id)))
+ (if hydra-build
+ (respond-json (object->json-string hydra-build))
+ (respond-build-not-found build-id))))
+ (("build" build-id "log" "raw")
+ (let ((build (db-get-build db build-id)))
+ (if build
+ (let ((log-response (handle-log-request db build)))
+ (if log-response
+ (respond-text log-response)
+ (respond-build-log-not-found build)))
+ (respond-build-not-found build-id))))
+ (("api" "latestbuilds")
+ (let* ((params (request-parameters request))
+ ;; 'nr parameter is mandatory to limit query size.
+ (valid-params? (assq-ref params 'nr)))
+ (if valid-params?
+ (respond-json (object->json-string
+ (handle-builds-request db params)))
+ (respond-json-with-error 500 "Parameter not defined!"))))
(_
(respond (build-response #:code 404)
#:body (string-append "Resource not found: "
@@ -73,6 +169,6 @@
(define* (run-cuirass-server db #:key (port 8080))
(format (current-error-port) "listening on port ~A~%" port)
(run-server url-handler
- 'http ;server implementation
- `(#:port ,port) ;implementation parameters
- db)) ;state
+ 'http
+ `(#:port ,port)
+ db))
diff --git a/src/cuirass/utils.scm b/src/cuirass/utils.scm
index d966543..a932674 100644
--- a/src/cuirass/utils.scm
+++ b/src/cuirass/utils.scm
@@ -21,9 +21,29 @@
(define-module (cuirass utils)
#:use-module (ice-9 match)
#:use-module (srfi srfi-1)
- #:export (alist?))
+ #:use-module (json)
+ #:export (alist?
+ object->json-scm
+ object->json-string))
(define (alist? obj)
"Return #t if OBJ is an alist."
(and (list? obj)
(every pair? obj)))
+
+(define (object->json-scm obj)
+ "Prepare OBJ for JSON usage."
+ (cond ((string? obj) obj)
+ ((number? obj) obj)
+ ((boolean? obj) obj)
+ ((null? obj) obj)
+ ((symbol? obj) (symbol->string obj))
+ ((keyword? obj) (object->json-scm (keyword->symbol obj)))
+ ((alist? obj) (map object->json-scm obj))
+ ((pair? obj) (cons (object->json-scm (car obj))
+ (object->json-scm (cdr obj))))
+ (else (object->string obj))))
+
+(define* (object->json-string object #:key pretty)
+ "Return OBJECT as a JSON object."
+ (scm->json-string (object->json-scm object) #:pretty pretty))
--
2.13.2