[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
01/02: website: Add toolchain blog post.
|
From: |
Ludovic Courtès |
|
Subject: |
01/02: website: Add toolchain blog post. |
|
Date: |
Sat, 11 Mar 2023 07:09:26 -0500 (EST) |
civodul pushed a commit to branch master
in repository guix-artwork.
commit d0477f40482fba9415e3deeac52ee09c7d339bf8
Author: Mitchell Schmeisser <mitchellschmeisser@librem.one>
AuthorDate: Mon Feb 27 10:20:32 2023 -0500
website: Add toolchain blog post.
* website/drafts/custom-toolchains-with-guix.md: New file.
Signed-off-by: Ludovic Courtès <ludo@gnu.org>
---
website/drafts/custom-toolchains-with-guix.md | 576 ++++++++++++++++++++++++++
1 file changed, 576 insertions(+)
diff --git a/website/drafts/custom-toolchains-with-guix.md
b/website/drafts/custom-toolchains-with-guix.md
new file mode 100644
index 0000000..3bd38be
--- /dev/null
+++ b/website/drafts/custom-toolchains-with-guix.md
@@ -0,0 +1,576 @@
+title: Building Toolchains with Guix
+author: Mitchell Schmeisser <mitchellschmeisser@librem.one>
+date: 2023-02-24 12:00
+tags: Software Development, Embedded, Zephyr, Scheme API
+---
+
+In order to deploy embedded software using Guix we first need to teach Guix
+how to build it. Since Guix bootstraps everything this means we must teach Guix
+how to build our toolchain.
+
+The [Zephyr Project](https://zephyrproject.org) uses its own fork of GCC with
custom configs for
+the architectures supported by the project.
+
+This is implemented as a Guix Channel.
+All code is available at [here](https://github.com/paperclip4465/guix-zephyr).
+
+# About ZephyrRTOS
+
+ZephyrRTOS is a Real Time Operating System from the Linux Foundation.
+It aims to provide a common environment which can target even the most
+resource constrained devices.
+
+Zephyr introduces a module system which allows third parties to share code
+in a uniform way. Zephyr uses CMake to perform _physical component composition_
+of these modules. It searches the filesystem and generates scripts which
+the toolchain will use to successfully combine those components into a
+firmware image.
+
+The fact that Zephyr provides this mechanism is one reason I chose to
+target it in the first place.
+
+This separation of modules in an embedded context is a really great thing.
+It brings many of the advantages that it brings to the Linux world such as
+code re-use, smaller binaries, more efficient cache/RAM usage, etc.
+It also allows us to work as independent groups and compose
+contributions from many teams.
+
+It also brings all of the complexity. Suddenly most of the problems
+that plague traditional deployment now apply to our embedded
+system. The fact that the libraries are statically linked at compile
+time instead of dynamically at runtime is simply an implementation detail.
+I say most because everything is statically linked so there is no runtime
+component discovery that needs to be accounted for.
+
+
+# Anatomy of a Toolchain
+
+Toolchains are responsible for taking high level descriptions of programs
+and lowering them down to a series of equivalent machine instructions.
+This process involves more than just a compiler. The compiler uses the
`binutils`
+to manipulate it's internal representation down to a given architecture.
+It also needs the use of the C standard library as well as a few other
libraries
+needed for some compiler optimizations.
+
+The C library provides the interface to the underlying kernel. System calls
like `write`
+and `read` are provided by Glibc on most Linux distributions.
+
+In embedded systems smaller implementations like Redhat's newlib and
+newlib-nano are used.
+
+# Bootstrapping a Toolchain
+
+In order to compile GCC we need a C library that's been compiled for
+our target architecture. How can we cross compile our C library if we
+need our C library to build a cross compiler? The solution is to build
+a simpler compiler that doesn't require the C library to function.
+It will not be capable of as many optimizations and it will be very slow,
+however it will be able to build the C libraries as well as the complete
version
+of GCC.
+
+In order to build the simpler compiler we need to compile the Binutils to
+work with our target architecture.
+The `binutils` can be bootstrapped with our host GCC and have no target
dependencies.
+
+[For more information read
this.](https://crosstool-ng.github.io/docs/toolchain-construction/)
+
+Doesn't sound so bad right? It isn't... in theory.
+However internet forums since time immemorial have been
+littered with the laments of those who came before.
+From incorrect versions of ISL to the wrong C library being linked
+or the host linker being used, etc.
+The one commonality between all of these issues is the environment.
+Building GCC is difficult because isolating build environments is hard.
+
+In fact as of =v0.14.2= the zephyr SDK repository took down the build
+instructions and posted a sign that read
+"Building this is too complicated, don't worry about it."
+(I'm paraphrasing, but
+[not by
much](https://github.com/zephyrproject-rtos/sdk-ng/tree/v0.14.2#build-process).)
+
+We will neatly side step all of these problems and not
+risk destroying or polluting our host system with garbage
+by using Guix to manage our environments for us.
+
+Our toolchain only requires the first pass compiler because
+newlib(-nano) is statically linked and introduced to the toolchain
+by normal package composition.
+
+
+# Defining the Packages
+
+All of the base packages are defined in `zephyr/packages/zephyr.scm`.
+Zephyr modules (coming soon!) are defined in `zephyr/packages/zephyr-xyz.scm`,
+following the pattern of other module systems implemented by Guix.
+
+## Binutils
+
+First thing we need to build is the `arm-zephyr-eabi` binutils.
+This is very easy in Guix.
+
+```scheme
+(define-public arm-zephyr-eabi-binutils
+ (let ((xbinutils (cross-binutils "arm-zephyr-eabi")))
+ (package (inherit xbinutils)
+ (name "arm-zephyr-eabi-binutils")
+ (version "2.38")
+ (source
+ (origin (method git-fetch)
+ (uri (git-reference
+ (url "https://github.com/zephyrproject-rtos/binutils-gdb";)
+ (commit "6a1be1a6a571957fea8b130e4ca2dcc65e753469")))
+ (file-name (git-file-name name version))
+ (sha256 (base32
"0ylnl48jj5jk3jrmvfx5zf8byvwg7g7my7jwwyqw3a95qcyh0isr"))))
+ (arguments
+ `(#:tests? #f
+ ,@(substitute-keyword-arguments (package-arguments xbinutils)
+ ((#:configure-flags flags)
+ `(cons "--program-prefix=arm-zephyr-eabi-" ,flags)))))
+ (native-inputs
+ (append
+ (list texinfo
+ bison
+ flex
+ gmp
+ dejagnu)
+ (package-native-inputs xbinutils)))
+ (home-page "https://zephyrproject.org";)
+ (synopsis "binutils for zephyr RTOS"))))
+```
+
+The function `cross-binutils` returns a package which has been
+configured for the given gnu triplet. We simply inherit that package
+and replace the source.
+The zephyr build system expects the binutils to be prefixed with
+`arm-zephyr-eabi-` which is accomplished by adding another flag to the
+`#:configure-flags` argument.
+
+We can test our package definition using the =-L= flag with `guix build`
+to add our packages.
+
+```sh
+guix build -L guix-zephyr zephyr-binutils
+
+/gnu/store/...-zephyr-binutils-2.38
+```
+
+This directory contains the results of `make install`.
+
+## GCC sans libc
+
+This one is a bit more involved. Don't be afraid!
+This version of GCC wants ISL version 0.15. It's easy enough
+to make that happen. Inherit the current version of ISL and swap
+out the source and update the version. For most packages the build process
doesn't
+change that much between versions.
+
+```scheme
+(define-public isl-0.15
+ (package
+ (inherit isl)
+ (version "0.15")
+ (source (origin
+ (method url-fetch)
+ (uri (list (string-append "mirror://sourceforge/libisl/isl-"
+ version ".tar.gz")))
+ (sha256
+ (base32
+ "11vrpznpdh7w8jp4wm4i8zqhzq2h7nix71xfdddp8xnzhz26gyq2"))))))
+
+```
+
+Like the binutils, there is a function for creating cross-gcc packages.
+This one accepts keywords specifying which binutils and libc to use.
+If libc isn't given (like here), gcc is configured with many options disabled
+to facilitate being built without libc. Therefore we need to add the extra
options
+we want (I got them from the SDK configuration scripts on the
+[sdk github](https://github.com/zephyrproject-rtos/sdk-ng) as well as the
+commits to use for each of the tools).
+
+
+```scheme
+(define-public gcc-arm-zephyr-eabi-12
+ (let ((xgcc (cross-gcc "arm-zephyr-eabi"
+ #:xbinutils zephyr-binutils)))
+ (package
+ (inherit xgcc)
+ (version "12.1.0")
+ (source (origin (method git-fetch)
+ (uri (git-reference
+ (url "https://github.com/zephyrproject-rtos/gcc";)
+ (commit
"0218469df050c33479a1d5be3e5239ac0eb351bf")))
+ (file-name (git-file-name (package-name xgcc) version))
+ (sha256
+ (base32
"1s409qmidlvzaw1ns6jaanigh3azcxisjplzwn7j2n3s33b76zjk"))
+ (patches
+ (search-patches
"gcc-12-cross-environment-variables.patch"
+ "gcc-cross-gxx-include-dir.patch"))))
+ (native-inputs
+ (modify-inputs (package-native-inputs xgcc)
+ ;; Get rid of stock ISL
+ (delete "isl")
+ ;; Add additional dependencies that xgcc doesn't have
+ ;; including our special ISL
+ (prepend flex
+ perl
+ python-3
+ gmp
+ isl-0.15
+ texinfo
+ python
+ mpc
+ mpfr
+ zlib)))
+ (arguments
+ (substitute-keyword-arguments (package-arguments xgcc)
+ ((#:phases phases)
+ `(modify-phases ,phases
+ (add-after 'unpack 'fix-genmultilib
+ (lambda _
+ (substitute# "gcc/genmultilib"
+ (("#!/bin/sh") (string-append "#!" (which "sh"))))
+ #t))
+
+ (add-after 'set-paths 'augment-CPLUS_INCLUDE_PATH
+ (lambda# (#:key inputs #:allow-other-keys)
+ (let ((gcc (assoc-ref inputs "gcc")))
+ ;; Remove the default compiler from CPLUS_INCLUDE_PATH to
+ ;; prevent header conflict with the GCC from native-inputs.
+ (setenv "CPLUS_INCLUDE_PATH"
+ (string-join
+ (delete (string-append gcc "/include/c++")
+ (string-split (getenv
"CPLUS_INCLUDE_PATH")
+ #\:))
+ ":"))
+ (format #t
+ "environment variable `CPLUS_INCLUDE_PATH' changed
to `a`%"
+ (getenv "CPLUS_INCLUDE_PATH"))
+ #t)))))
+
+ ((#:configure-flags flags)
+ ;; The configure flags are largely identical to the flags used by
the
+ ;; "GCC ARM embedded" project.
+ `(append (list "--enable-multilib"
+ "--with-newlib"
+ "--with-multilib-list=rmprofile"
+ "--with-host-libstdcxx=-static-libgcc
-Wl,-Bstatic,-lstdc++,-Bdynamic -lm"
+ "--enable-plugins"
+ "--disable-decimal-float"
+ "--disable-libffi"
+ "--disable-libgomp"
+ "--disable-libmudflap"
+ "--disable-libquadmath"
+ "--disable-libssp"
+ "--disable-libstdcxx-pch"
+ "--disable-nls"
+ "--disable-shared"
+ "--disable-threads"
+ "--disable-tls"
+ "--with-gnu-ld"
+ "--with-gnu-as"
+ "--enable-initfini-array")
+ (delete "--disable-multilib" ,flags)))))
+ (native-search-paths
+ (list (search-path-specification
+ (variable "CROSS_C_INCLUDE_PATH")
+ (files '("arm-zephyr-eabi/include")))
+ (search-path-specification
+ (variable "CROSS_CPLUS_INCLUDE_PATH")
+ (files '("arm-zephyr-eabi/include"
+ "arm-zephyr-eabi/c++"
+ "arm-zephyr-eabi/c++/arm-zephyr-eabi")))
+ (search-path-specification
+ (variable "CROSS_LIBRARY_PATH")
+ (files '("arm-zephyr-eabi/lib")))))
+ (home-page "https://zephyrproject.org";)
+ (synopsis "GCC for zephyr RTOS"))))
+```
+
+This GCC can be built like so.
+
+```sh
+guix build -L guix-zephyr gcc-cross-sans-libc-arm-zephyr-eabi
+
+/gnu/store/...-gcc-cross-sans-libc-arm-zephyr-eabi-12.1.0-lib
+/gnu/store/...-gcc-cross-sans-libc-arm-zephyr-eabi-12.1.0
+
+```
+Great! We now have our stage-1 compiler.
+
+## Newlib(-nano)
+
+The newlib package package is quite straight forward (relatively).
+It is mostly adding in the relevent configuration flags and patching
+the files the `patch-shebangs` phase missed.
+
+```scheme
+(define-public zephyr-newlib
+ (package
+ (name "zephyr-newlib")
+ (version "3.3")
+ (source (origin
+ (method git-fetch)
+ (uri (git-reference
+ (url "https://github.com/zephyrproject-rtos/newlib-cygwin";)
+ (commit "4e150303bcc1e44f4d90f3489a4417433980d5ff")))
+ (sha256
+ (base32
"08qwjpj5jhpc3p7a5mbl7n6z7rav5yqlydqanm6nny42qpa8kxij"))))
+ (build-system gnu-build-system)
+ (arguments
+ `(#:out-of-source? #t
+ #:configure-flags '("--target=arm-zephyr-eabi"
+ "--enable-newlib-io-long-long"
+ "--enable-newlib-io-float"
+ "--enable-newlib-io-c99-formats"
+ "--enable-newlib-retargetable-locking"
+ "--enable-newlib-lite-exit"
+ "--enable-newlib-multithread"
+ "--enable-newlib-register-fini"
+ "--enable-newlib-extra-sections"
+ "--disable-newlib-wide-orient"
+ "--disable-newlib-fseek-optimization"
+ "--disable-newlib-supplied-syscalls"
+ "--disable-newlib-target-optspace"
+ "--disable-nls")
+ #:phases
+ (modify-phases %standard-phases
+ (add-after 'unpack 'fix-references-to-/bin/sh
+ (lambda _
+ (substitute# '("libgloss/arm/cpu-init/Makefile.in"
+ "libgloss/arm/Makefile.in"
+ "libgloss/libnosys/Makefile.in"
+ "libgloss/Makefile.in")
+ (("/bin/sh") (which "sh")))
+ #t)))))
+ (native-inputs
+ `(("xbinutils" ,zephyr-binutils)
+ ("xgcc" ,gcc-arm-zephyr-eabi-12)
+ ("texinfo" ,texinfo)))
+ (home-page "https://www.sourceware.org/newlib/";)
+ (synopsis "C library for use on embedded systems")
+ (description "Newlib is a C library intended for use on embedded
+systems. It is a conglomeration of several library parts that are easily
+usable on embedded products.")
+ (license (license:non-copyleft
+ "https://www.sourceware.org/newlib/COPYING.NEWLIB";))))
+```
+
+And the build.
+
+```sh :exports both
+$ guix build -L guix-zephyr zephyr-newlib
+
+/gnu/store/...-zephyr-newlib-3.3
+```
+
+## Complete Toolchain
+
+_Mostly_ complete. libstdc++ does not build because
+`arm-zephyr-eabi` is not `arm-none-eabi` so a dynamic link check is
+performed/failed. I cannot figure out how crosstool-ng handles this.
+
+Now that we've got the individual tools it's time to create our complete
toolchain.
+For this we need to do some package transformations.
+Because these transformations are going to have to be done for every
combination of
+binutils/gcc/newlib it is best to create a function which we can reuse for
every version
+of the SDK.
+
+```scheme :exports code
+ (define (arm-zephyr-eabi-toolchain xgcc newlib version)
+ "Produce a cross-compiler zephyr toolchain package with the compiler XGCC
and the C
+ library variant NEWLIB."
+ (let ((newlib-with-xgcc (package (inherit newlib)
+ (native-inputs
+ (alist-replace "xgcc" (list xgcc)
+ (package-native-inputs
newlib))))))
+ (package
+ (name (string-append "arm-zephyr-eabi"
+ (if (string=? (package-name newlib-with-xgcc)
+ "newlib-nano")
+ "-nano" "")
+ "-toolchain"))
+ (version version)
+ (source #f)
+ (build-system trivial-build-system)
+ (arguments
+ '(#:modules ((guix build union)
+ (guix build utils))
+ #:builder
+ (begin
+ (use-modules (ice-9 match)
+ (guix build union)
+ (guix build utils))
+ (let ((out (assoc-ref %outputs "out")))
+ (mkdir-p out)
+ (match %build-inputs
+ (((names . directories) ...)
+ (union-build (string-append out "/arm-zephyr-eabi")
+ directories)
+ #t))))))
+ (inputs
+ `(("binutils" ,zephyr-binutils)
+ ("gcc" ,xgcc)
+ ("newlib" ,newlib-with-xgcc)))
+ (synopsis "Complete GCC tool chain for ARM zephyrRTOS development")
+ (description "This package provides a complete GCC tool chain for ARM
+ bare metal development with zephyr rtos. This includes the GCC
arm-zephyr-eabi cross compiler
+ and newlib (or newlib-nano) as the C library. The supported programming
+ language is C.")
+ (home-page (package-home-page xgcc))
+ (license (package-license xgcc)))))
+```
+
+This function creates a special package which consists of the toolchain
+in a special directory hierarchy, i.e `arm-zephyr-eabi/`.
+Our complete toolchain definition looks like this.
+
+```scheme
+(define-public arm-zephyr-eabi-toolchain-0.15.0
+ (arm-zephyr-eabi-toolchain
+ gcc-arm-zephyr-eabi-12
+ zephyr-newlib
+ "0.15.0"))
+```
+
+To build:
+
+```sh
+guix build -L guix-zephyr arm-zephyr-eabi-toolchain
+/gnu/store/...-arm-zephyr-eabi-toolchain-0.15.0
+```
+
+# Integrating with Zephyr Build System
+
+Zephyr uses CMake as its build system. It contains numerous CMake files in
both the so-called `ZEPHYR_BASE`,
+the zephyr source code repository, as well as a handful in the SDK which help
select the correct toolchain
+for a given board.
+
+There are standard locations the build system will look for the SDK. We are
not using any of them.
+Our SDK lives in the store, immutable forever.
+According to
[[https://docs.zephyrproject.org/latest/develop/west/without-west.html][this]]
the variable `ZEPHYR_SDK_INSTALL_DIR` needs to point to our custom spot.
+
+We also need to grab the cmake files from the
+[repository](https://github.com/zephyrproject-rtos/sdk-ng)
+and create a file, `sdk_version`, which
+contains the version string `ZEPHYR_BASE` uses to find a compatible SDK.
+
+Along with the SDK proper we need to include a number of
+python packages required by the build system.
+
+```scheme
+(define-public zephyr-sdk
+ (package
+ (name "zephyr-sdk")
+ (version "0.15.0")
+ (home-page "https://zephyrproject.org";)
+ (source (origin (method git-fetch)
+ (uri (git-reference
+ (url "https://github.com/zephyrproject-rtos/sdk-ng";)
+ (commit "v0.15.0")))
+ (file-name (git-file-name name version))
+ (sha256 (base32
"04gsvh20y820dkv5lrwppbj7w3wdqvd8hcanm8hl4wi907lwlmwi"))))
+ (build-system trivial-build-system)
+ (arguments
+ `(#:modules ((guix build union)
+ (guix build utils))
+ #:builder
+ (begin
+ (use-modules (guix build union)
+ (ice-9 match)
+ (guix build utils))
+ (let# ((out (assoc-ref %outputs "out"))
+ (cmake-scripts (string-append (assoc-ref %build-inputs "source")
+ "/cmake"))
+ (sdk-out (string-append out "/zephyr-sdk-0.15.0")))
+ (mkdir-p out)
+
+ (match (assoc-remove! %build-inputs "source")
+ (((names . directories) ...)
+ (union-build sdk-out directories)))
+
+ (copy-recursively cmake-scripts
+ (string-append sdk-out "/cmake"))
+
+ (with-directory-excursion sdk-out
+ (call-with-output-file "sdk_version"
+ (lambda (p)
+ (format p "0.15.0")))
+ #t)))))
+ (propagated-inputs
+ (list
+ arm-zephyr-eabi-toolchain-0.15.0
+ zephyr-binutils
+ dtc
+ python-3
+ python-pyelftools
+ python-pykwalify
+ python-pyyaml
+ python-packaging))
+ (native-search-paths
+ (list (search-path-specification
+ (variable "ZEPHYR_SDK_INSTALL_DIR")
+ (files '("")))))
+ (synopsis "SDK for zephyrRTOS")
+ (description "zephyr-sdk contains bundles a complete gcc toolchain as well
+as host tools like dtc, openocd, qemu, and required python packages.")
+ (license license:apsl2)))
+```
+
+# Testing
+
+In order to test we will need an environment with the SDK installed.
+We can take advantage of `guix shell` to avoid installing test packages into
+our home environment. This way if it causes problems we can just exit the shell
+and try again.
+
+```sh
+guix shell -L guix-zephyr zephyr-sdk cmake ninja git
+```
+
+`ZEPHYR_BASE` can be cloned into a temporary workspace to test our toolchain
functionality.
+(For now. Eventually we will need to create a package for `zephyr-base` that
+our guix zephyr-build-system can use.)
+
+```sh
+mkdir /tmp/zephyr-project
+cd /tmp/zephyr-project
+git clone https://github.com/zephyrproject-rtos/zephyr
+export ZEPHYR_BASE=/tmp/zephyr-project/zephyr
+```
+
+In order to build for the test board (k64f in this case) we need to get a hold
of the vendor
+Hardware Abstraction Layers and CMSIS.
+(These will also need to become guix packages to allow the build system to
compose modules).
+
+```sh
+git clone https://github.com/zephyrproject-rtos/hal_nxp &&
+git clone https://github.com/zephyrproject-rtos/cmsis
+```
+
+To inform the build system about this module we pass it in with
`-DZEPHYR_MODULES=` which is
+a semicolon separated list of paths containing a module.yml file.
+
+To build the hello world sample we use the following incantation.
+```sh
+cmake -Bbuild $ZEPHYR_BASE/samples/hello_world \
+ -GNinja \
+ -DBOARD=frdm_k64f \
+ -DBUILD_VERSION=3.1.0 \
+
-DZEPHYR_MODULES="/tmp/zephyr-project/hal_nxp;/tmp/zephyr-project/cmsis" \
+ && ninja -Cbuild
+```
+
+If everything is set up correctly we will end up with a =./build=
+directory with all our build artifacts. The SDK is installed correctly!
+
+# Conclusion
+
+A customized cross toolchain is one of the most difficult pieces of
+software to build. Using Guix, we do not need to be afraid of the
+complexity! We can fiddle with settings, swap out components, and do
+the most brain dead things to our environments without a care in the
+world. Just exit the environment and it's like it never happened at
+all.
+
+It highlights one of my favorite aspects of Guix, every package is a
+working reference design for you to modify and learn from.
\ No newline at end of file