[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[task #15686] Removing original software URLs from Maneage?
From: |
Mohammad Akhlaghi |
Subject: |
[task #15686] Removing original software URLs from Maneage? |
Date: |
Wed, 24 Jun 2020 21:03:41 -0400 (EDT) |
User-agent: |
Mozilla/5.0 (X11; Linux x86_64; rv:77.0) Gecko/20100101 Firefox/77.0 |
Update of task #15686 (project reproduce):
Status: In Progress => Ready For Test
_______________________________________________________
Follow-up Comment #11:
!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!! VERY IMPORTANT !!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!
Implementing this task forced me to go a little low-level, and very soon
started a chain-reaction(!), in one way or another, merging into these tasks:
task #15399, task #15389, task #15272, task #15359 and task #15330!
It effectively took my free time in the whole week, and resulted in an
almost-complete rewrite of the software building phase of Maneage, making it
SO MUCH more robust and easier to read/modify.
You can see the changes in Commit 603ba6d459e
<https://gitlab.com/maneage/project-dev/-/commit/603ba6d459e> on my
development branch. Just for the record, I am also putting the full commit
message at the end of this post.
I already checked the building of all the software (with the new
'--all-highlevel' option) in a 'slim' Debian-stable Docker image (while
testing task #15389!) and it worked nicely (after almost two days of fixing
issues due to the low-level change in each package!). But it would be great if
you could also try it out and look into it in the next two days. If a problem
isn't found or a suggestion isn't made by Saturday. I'll merge it into the
core Maneage branch.
In particular, Raul, it would be great if you could try this branch on your
macOS (don't forget the '--all-highlevel' option ;-)).
IMPORTANT: Software recipes use GNU Make, many other improvements
POSSIBLE EFFECT ON YOUR PROJECT: The changes in this commit may only cause
conflicts to your project if you have changed the software building
Makefiles in your project's branch (e.g., 'basic.mk', 'high-level.mk' and
'python.mk'). If your project has only added analysis, it shouldn't affect
it, but still read through the points for using the newly added
features. In general, things have become much more clear and easier to
read/execute now ;-).
This is a large commit, involving a long series of corrections in a
differnt branch for an initial low-level change. Other changes naturally
came up as the low-level structure was improved.
- Until now, Maneage needed the host to have a 'make' implementation which
was necessary to build Lzip (Lzip is then used to uncompress the source
of GNU Make). But this is not the case in the minimalist/slim versions
of operating systems (for example used to build Docker images). After
consulting Antonio Diaz Diaz (creator of Lzip), he kindly added the
necessary functionality to Lzip and Lzip can now be built without Make.
Hence we don't need this assumption any more. With this commit, Lzip and
GNU Make are built without Make, allowing everything else to be safely
built with our own custom version of GNU Make and not using the host's
'make' at all.
- Until recently (Commit 3d8aa595) GNU Make was built in
'basic.mk'. Therefore 'basic.mk' was written in a way that it can be
used with other 'make' implementations also (important shell commands
starting with '&&' and ending in '\', it also didn't allow any
comments!). Furthermore, to help in style uniformity, the rules in
'high-level.mk' and 'python.mk' also followed a similar structure. But
due to the point above, we can now guarantee that GNU Make is used from
the very first Makefile, so this hard-to-read structure has been removed
in the software build recipes.
- Until now, the default backup servers where at some fixed URLs, on our
own pages or on Gitlab. But recently we uploaded all the necessary
software to Zenodo (https://doi.org/10.5281/zenodo.3883409) which is
more suitable for this task (it promises longevity, has a fixed DOI,
while allowing us to add new content, or new software tarball
versions). With this commit, a small script has been written to extract
the most recent Zenodo upload link from the Zenodo DOI and use it for
downloading the software source codes.
- Until now, we primarily used the webpage of each software as the primary
source for downloading its tarball. But this caused many problems: 1)
Some of them needed Javascript before the download, 2) Some URLs had a
complex dependency on the version number, 3) some servers would be
randomly down for maintenance and etc. So thanks to the point above, we
now use the Zenodo server as the primary download location. However, if
a user wants to use a custom software that is not (yet!) on Zenodo, the
download script gives priority to a custom URL that the users can give
as Make variables. If that variable is defined, then the script will use
that URL before going onto Zenodo. We now have a special place for such
URLs: 'reproduce/software/config/urls.conf'. The old URLs (which are a
good documentation themselvs) are preserved here, but are commented by
default.
- The downloading and checksum verification step has been moved into a
Make function called 'import-source' (defined in the 'build-rules.mk'
and loaded in all software Makefiles). Having taken all the low-level
steps there, I noticed that there is no more need for having the tarball
as a separate target! So with this commit, a single rule is the only
place that needs to be edited/added (greatly simplifying the software
building Makefiles).
- Following task #15272, A new option has been added to the './project'
script called '--all-highlevel'. When this option is given, the contents
of 'TARGETS.conf' are ignored and a script goes through all the software
(by parsing the 'versions.conf' file) and sets all high-level software
as targets to be built. I added and used this new option to confirm the
extensive changes made in all the software building recipes!
- Many of the software hadn't been tested for a long time! So after using
the newly added '--all-highlevel', I found that some need to be
updated. In general, with this commit 'libpaper' and 'pcre' were added
as totally new software, and the versions of the following software was
updated: 'flex', 'libtirpc', 'openblas' and 'lzip'. A 'run-parts.in'
shell script was added in 'reproduce/software/shell/' which is installed
with 'libpaper'.
- Even though we intentionally add the necessary flags to add RPATH inside
the built executable at compilation time, some software don't do it
(different software on different operating systems!). Until now, we were
simply checking if the operating system is macOS, but a more robust
check is to see if 'patchelf' (used for manually updating RPATH in
GNU/Linux systems and not installed on macOS) is present or not. Because
of this, 'patchelf' has been put as a top-level prerequisite, right
after Tar and is installed before anything else.
- In 'versions.conf', GNU Libtool is recognized as 'libtool', but in
'basic.mk', it was 'glibtool'! This caused many confusions and is
corrected with this commit (in 'basic.mk', it is also 'libtool').
- A new argument is added to the './project' script to allow easy loading
of the project's shell and environment for fast/temporary testing of
things in the same environment as the project. Before activating the
project's shell, we completely remove all host environment variables to
simulate the project's environment. It can be called with this command:
'./project shell'. Note that it doesn't yet have any prompt! all lines
just start with a '$'. We should add one later.
_______________________________________________________
Reply to this item at:
<https://savannah.nongnu.org/task/?15686>
_______________________________________________
Message sent via Savannah
https://savannah.nongnu.org/
- [task #15686] Removing original software URLs from Maneage?, (continued)
- [task #15686] Removing original software URLs from Maneage?, Raul Infante-Sainz, 2020/06/11
- [task #15686] Removing original software URLs from Maneage?, Mohammad Akhlaghi, 2020/06/11
- [task #15686] Removing original software URLs from Maneage?, Raul Infante-Sainz, 2020/06/11
- [task #15686] Removing original software URLs from Maneage?, Boud Roukema, 2020/06/12
- [task #15686] Removing original software URLs from Maneage?, Mohammad Akhlaghi, 2020/06/14
- [task #15686] Removing original software URLs from Maneage?, Boud Roukema, 2020/06/15
- [task #15686] Removing original software URLs from Maneage?, Mohammad Akhlaghi, 2020/06/15
- [task #15686] Removing original software URLs from Maneage?, Boud Roukema, 2020/06/16
- [task #15686] Removing original software URLs from Maneage?, Mohammad Akhlaghi, 2020/06/16
- [task #15686] Removing original software URLs from Maneage?, Mohammad Akhlaghi, 2020/06/19
- [task #15686] Removing original software URLs from Maneage?,
Mohammad Akhlaghi <=
- [task #15686] Removing original software URLs from Maneage?, Boud Roukema, 2020/06/25
- [task #15686] Removing original software URLs from Maneage?, Mohammad Akhlaghi, 2020/06/25
- [task #15686] Removing original software URLs from Maneage?, Raul Infante-Sainz, 2020/06/25
- [task #15686] Removing original software URLs from Maneage?, Mohammad Akhlaghi, 2020/06/25
- [task #15686] Removing original software URLs from Maneage?, Raul Infante-Sainz, 2020/06/26
- [task #15686] Removing original software URLs from Maneage?, Mohammad Akhlaghi, 2020/06/26
- [task #15686] Removing original software URLs from Maneage?, Mohammad Akhlaghi, 2020/06/27
- [task #15686] Removing original software URLs from Maneage?, Raul Infante-Sainz, 2020/06/27
- [task #15686] Removing original software URLs from Maneage?, Mohammad Akhlaghi, 2020/06/27