Re: [GNUnet-developers] packaging GNUnet for OpenWrt

[Christian Grothoff]

On 05/31/2015 02:57 PM, Daniel Golle wrote:
> Hi GNUnet folks,
> 
> I was working on packaging GNUnet for OpenWrt in the past couple of
> days. As a result, there now is a gnunet package which should work
> on all OpenWrt (hardware and emulator) targets.
> However, I'm quite new to GNUnet and could use some assistence on
> deciding how to split-up the package, which in it's current state could
> be the OpenWrt package with the most external dependencies ever ;)
> I also wonder whether using gnunet-arm is really what I want, given
> that we got procd to pretty much do the same job.

In theory you can replace 'gnunet-arm' with anything else that provides
systemd-like functionality, at the expense of having to document a
system-specific way to configure things (as the PORT/UNIXPATH
configurations from the GNUnet-configuration would have to be mapped 1:1
to the procd syntax, maybe with some conversion script?).

Still, while ARM is really tiny, for an embedded system where a few KB
of RAM/ROM can matter, this might be worth the effort.

> Also, some dependencies (like libcurl) do make sense on desktop systems
> where you may have them installed anyway. On smaller systems, you may
> want to use stuff like libuclient instead.

libgnurl is our attempt to remove the crud from libcurl, did you
actually 'configure' it with the right options? (So only using GnuTLS
and not any protocols other than http/https?)

> However, to have a clean
> state to start-off from, I also packaged gnurl (but would like to get
> rid of it, the sooner the better). We also don't have a good solution
> to build curl and libcurl in several variants with different SSL
> implementations -- this was easy to do for other packages, but seems
> to be harder for curl. So gnurl it is for now.

If one of your curl packages is guaranteed to use libgnutls and you're
short on space, you *can* re-use that package safely.  On an embedded
system that has libcurl already, this would make sense to me.

> See
> https://github.com/openwrt/packages/blob/master/net/gnunet/Makefile
> to get an impression of the current state which generates a single
> package called 'gnunet' which contains everything built and got a
> total size of about 3.8MiB, which is way to much if one wants to use
> e.g. only gns and cadet on a small router (not even speaking about
> the ~30MiB of dependencies being pulled-in right now)

Do you have a list of which dependencies are how costly (in terms of
size)? That might provide a useful way of determining how to cut it down.

> If you want to play with 'embedded GNUnet', go ahead and get OpenWrt
> https://dev.openwrt.org/wiki/GetSource
> 
> run 'make menuconfig' and select e.g. one of the emulator targets
> malta, realview or x86_kvm.
> 
> Now select the gnunet package inside the "Network" menu.
> 
> You can play with the "Global build settings" enabling various
> security, debugging and localization related build-time features.
> Generally, all packages should build and work independently of these
> settings. To be safe, start with the defaults, then change things
> and *make distclean* after changes made on toolchain options!
> Once you are done quit menuconfig and run 'make'.
> 
> See the README file corresponding to the target to learn how to
> run the resulting image in QEMU, e.g.
> https://dev.openwrt.org/browser/trunk/target/linux/malta/README

Great, thanks for the instructions, I'll try them once I have some free
time (maybe a few weeks off).

> It'd be great if some of you can confirm that this is basically
> working. I still had some trouble starting gnunet-arm as user
> 'gnunet' which is what the init-script currently does.
> When manually starting gnunet-arm -s as root, things seem to work
> from what I can tell -- however, I don't know what I should
> expect and you probably know better how well things should work
> and general and how to operate the beast.

Well, running as 'root' generally works, but of course given that we do
not need root, is really a bad idea from a security perspective. Also,
in that case the DNS interception for GNS will *not* work for the user
'root' himself, which may be problematic. If you run as user 'gnunet',
DNS interception will not be effective for the user 'gnunet', which is
typically what you want/need (so that gnunet-service-dns can forward DNS
queries to DNS and GNS queries to GNS).

> The next high-priority item is to split up the package into a bunch
> of small packages, e.g.:
> - libgnunet

Not sure there is utility in just having the libs.

> - gnunet-core
> - gnunet-utils

Not sure what 'utils' would naturally include.

> - gnunet-testing

Plenty here, makes definitively sense to keep apart.

> - gnunet-app-gns
> - gnunet-app-filesharing

For filesharing, you might have two versions:
- gnunet-app-filesharing
- gnunet-app-filesharing-extractor-plugins

The 2nd one would drag in the LE plugins (and their dependencies!),
while the first one would just ship with an empty
PREFIX/lib/libextractor/* directory.  As users are unlikely to run
'gnunet-publish' on the OpenWRT to publish files locally anyway, that
would remove a lot of unnecessary crud.

> - gnunet-app-conversation

Here a split between the service and the user-tools might make sense.
If the OpenWRT doesn't have a speaker and a microphone, it doesn't make
sense to drag in opus and pulseaudio.  So you could run the conversation
service and allow the user to connect to it from a TCP port, running the
command-line/GUI on the desktop.  That'd remove Opus and PA dependencies
from the OpenWRT.

> - gnunet-app-vpn

There's also what you might consider the 'experimental' group:
psyc, multicast, social, ATS plugins (especially the one needing glpk),
scalarproduct, rps, sensor* rest, env, dv, consensus: none of these are
useful for end-user applications today, so there is really no point in
installing them today on an embeded system, for anything but
development.  In the future, some might become part of 'core' or a new
package ("social"), but for now it's safe to just leave those separate
and generally not install them.

> As I said, I'm pretty new to GNUnet and that's really just a vague
> idea I got about how to do this properly, so feel free to suggest
> more accurate package names and/or boundaries.
> Have a look at other complex packages like strongswan or opensc to
> get an impression how the game works.
> 
> 
> Cheers
> 
> 
> Daniel
> 
> So, basically these are the files currently installed by the gnunet
> package and their size in bytes (on MIPS32r2 big-endian):

I've put comments on those that are more or less useless for
OpenWRT/embedded systems inline.


> 10055 ./usr/bin/gnunet-arm
> 11912 ./usr/bin/gnunet-ats
> 13208 ./usr/bin/gnunet-auto-share
> 10520 ./usr/bin/gnunet-bcd

Requires TexLive to actually function, I'd remove this.

> 12616 ./usr/bin/gnunet-cadet
> 4792 ./usr/bin/gnunet-config
> 15160 ./usr/bin/gnunet-conversation
> 5832 ./usr/bin/gnunet-conversation-test
> 4664 ./usr/bin/gnunet-core
> 4808 ./usr/bin/gnunet-datastore
> 5352 ./usr/bin/gnunet-directory
> 7896 ./usr/bin/gnunet-download
> 13338 ./usr/bin/gnunet-download-manager.scm
> 9032 ./usr/bin/gnunet-ecc
> 3735 ./usr/bin/gnunet-fs
> 8536 ./usr/bin/gnunet-gns
> 3014 ./usr/bin/gnunet-gns-import.sh
> 2075 ./usr/bin/gnunet-gns-proxy-setup-ca
> 5528 ./usr/bin/gnunet-identity
> 5384 ./usr/bin/gnunet-namecache
> 13128 ./usr/bin/gnunet-namestore
> 6152 ./usr/bin/gnunet-nat-server
> 4456 ./usr/bin/gnunet-nse
> 13096 ./usr/bin/gnunet-peerinfo
> 3544 ./usr/bin/gnunet-peerstore
> 14136 ./usr/bin/gnunet-publish
> 2749 ./usr/bin/gnunet-qr
> 4248 ./usr/bin/gnunet-resolver
> 9144 ./usr/bin/gnunet-revocation
> 8472 ./usr/bin/gnunet-scalarproduct
> 7592 ./usr/bin/gnunet-scrypt

For brute-force vanity keys, not useful for WRT, can be safely removed.

> 7511 ./usr/bin/gnunet-search
> 7496 ./usr/bin/gnunet-statistics
> 2808 ./usr/bin/gnunet-template

Template (example code), can be safely removed.

> 6584 ./usr/bin/gnunet-testbed-profiler
> 9480 ./usr/bin/gnunet-testing
> 21496 ./usr/bin/gnunet-transport
> 5496 ./usr/bin/gnunet-transport-certificate-creation
> 4760 ./usr/bin/gnunet-unindex
> 5720 ./usr/bin/gnunet-uri
> 6551 ./usr/bin/gnunet-vpn
> 33960 ./usr/lib/gnunet/libexec/gnunet-daemon-exit
> 33752 ./usr/lib/gnunet/libexec/gnunet-daemon-hostlist
> 7511 ./usr/lib/gnunet/libexec/gnunet-daemon-latency-logger
> 14360 ./usr/lib/gnunet/libexec/gnunet-daemon-pt
> 35560 ./usr/lib/gnunet/libexec/gnunet-daemon-regexprofiler

Only used for experimentation (performance evaluation), no need to ship.

> 6040 ./usr/lib/gnunet/libexec/gnunet-daemon-testbed-blacklist
> 9240 ./usr/lib/gnunet/libexec/gnunet-daemon-testbed-underlay
> 14824 ./usr/lib/gnunet/libexec/gnunet-daemon-topology
> 11224 ./usr/lib/gnunet/libexec/gnunet-dns2gns
> 28296 ./usr/lib/gnunet/libexec/gnunet-gns-proxy
> 13016 ./usr/lib/gnunet/libexec/gnunet-helper-audio-playback
> 12760 ./usr/lib/gnunet/libexec/gnunet-helper-audio-record
> 10008 ./usr/lib/gnunet/libexec/gnunet-helper-dns
> 8408 ./usr/lib/gnunet/libexec/gnunet-helper-exit
> 6984 ./usr/lib/gnunet/libexec/gnunet-helper-fs-publish
> 4600 ./usr/lib/gnunet/libexec/gnunet-helper-nat-client
> 5208 ./usr/lib/gnunet/libexec/gnunet-helper-nat-server
> 11752 ./usr/lib/gnunet/libexec/gnunet-helper-testbed
> 16872 ./usr/lib/gnunet/libexec/gnunet-helper-transport-bluetooth
> 10504 ./usr/lib/gnunet/libexec/gnunet-helper-transport-wlan
> 7512 ./usr/lib/gnunet/libexec/gnunet-helper-transport-wlan-dummy

For testing ('make check'), can be safely removed.

> 6744 ./usr/lib/gnunet/libexec/gnunet-helper-vpn
> 12888 ./usr/lib/gnunet/libexec/gnunet-namestore-fcfsd
> 19240 ./usr/lib/gnunet/libexec/gnunet-service-arm
> 22056 ./usr/lib/gnunet/libexec/gnunet-service-ats
> 92552 ./usr/lib/gnunet/libexec/gnunet-service-cadet
> 14456 ./usr/lib/gnunet/libexec/gnunet-service-conversation
> 34776 ./usr/lib/gnunet/libexec/gnunet-service-core
> 17736 ./usr/lib/gnunet/libexec/gnunet-service-datastore
> 42168 ./usr/lib/gnunet/libexec/gnunet-service-dht
> 25400 ./usr/lib/gnunet/libexec/gnunet-service-dht-whanau
> 55111 ./usr/lib/gnunet/libexec/gnunet-service-dht-xvine

-xvine and -whanau are pure research prototypes for comparing DHT
designs.  They're not "finished" and not used in production and
incompatible protocol-wise with the 'running' network. So definitively
remove.

> 12759 ./usr/lib/gnunet/libexec/gnunet-service-dns
> 63736 ./usr/lib/gnunet/libexec/gnunet-service-fs
> 40504 ./usr/lib/gnunet/libexec/gnunet-service-gns
> 11975 ./usr/lib/gnunet/libexec/gnunet-service-identity
> 6872 ./usr/lib/gnunet/libexec/gnunet-service-namecache
> 15480 ./usr/lib/gnunet/libexec/gnunet-service-namestore
> 16904 ./usr/lib/gnunet/libexec/gnunet-service-nse
> 16280 ./usr/lib/gnunet/libexec/gnunet-service-peerinfo
> 10408 ./usr/lib/gnunet/libexec/gnunet-service-peerstore
> 32584 ./usr/lib/gnunet/libexec/gnunet-service-regex
> 7992 ./usr/lib/gnunet/libexec/gnunet-service-resolver
> 13080 ./usr/lib/gnunet/libexec/gnunet-service-revocation
> 14344 ./usr/lib/gnunet/libexec/gnunet-service-scalarproduct-alice
> 14280 ./usr/lib/gnunet/libexec/gnunet-service-scalarproduct-bob
> 31031 ./usr/lib/gnunet/libexec/gnunet-service-set
> 12744 ./usr/lib/gnunet/libexec/gnunet-service-statistics
> 2824 ./usr/lib/gnunet/libexec/gnunet-service-template

No need to ship _template's, those are from example code.

> 74408 ./usr/lib/gnunet/libexec/gnunet-service-testbed
> 5944 ./usr/lib/gnunet/libexec/gnunet-service-testbed-logger
> 80296 ./usr/lib/gnunet/libexec/gnunet-service-transport
> 27095 ./usr/lib/gnunet/libexec/gnunet-service-vpn
> 36536 ./usr/lib/gnunet/libgnunet_plugin_ats_mlp.so

Experimental, drags in GLPK, known bugs, not required, can be safely
removed (or be part of 'experimental').

> 15092 ./usr/lib/gnunet/libgnunet_plugin_ats_proportional.so
> 26532 ./usr/lib/gnunet/libgnunet_plugin_ats_ril.so

Experimental, drags in GLPK, known bugs, not required, can be safely
removed (or be part of 'experimental').

> 4856 ./usr/lib/gnunet/libgnunet_plugin_block_dht.so
> 3812 ./usr/lib/gnunet/libgnunet_plugin_block_dns.so
> 4980 ./usr/lib/gnunet/libgnunet_plugin_block_fs.so
> 4692 ./usr/lib/gnunet/libgnunet_plugin_block_gns.so
> 5940 ./usr/lib/gnunet/libgnunet_plugin_block_regex.so
> 3196 ./usr/lib/gnunet/libgnunet_plugin_block_template.so

No need to ship _template's, those are from example code.

> 3692 ./usr/lib/gnunet/libgnunet_plugin_block_test.so

Used ONLY if users want to try out the DHT's command-line tools.  Could
be removed if 4k really matter without breaking 'real' applications, but
only after all of the others I mentioned ;-).

> 6092 ./usr/lib/gnunet/libgnunet_plugin_datacache_heap.so
> 16864 ./usr/lib/gnunet/libgnunet_plugin_datacache_sqlite.so
> 4236 ./usr/lib/gnunet/libgnunet_plugin_datacache_template.so

No need to ship _template's, those are from example code.

> 8788 ./usr/lib/gnunet/libgnunet_plugin_datastore_heap.so
> 26924 ./usr/lib/gnunet/libgnunet_plugin_datastore_sqlite.so
> 4956 ./usr/lib/gnunet/libgnunet_plugin_datastore_template.so

No need to ship _template's, those are from example code.

> 4264 ./usr/lib/gnunet/libgnunet_plugin_gnsrecord_conversation.so
> 11992 ./usr/lib/gnunet/libgnunet_plugin_gnsrecord_dns.so
> 6960 ./usr/lib/gnunet/libgnunet_plugin_gnsrecord_gns.so
> 14856 ./usr/lib/gnunet/libgnunet_plugin_namecache_sqlite.so
> 16772 ./usr/lib/gnunet/libgnunet_plugin_namestore_sqlite.so
> 11852 ./usr/lib/gnunet/libgnunet_plugin_peerstore_sqlite.so
> 2472 ./usr/lib/gnunet/libgnunet_plugin_test.so

Used during 'make check', can be safely removed.

> 20227 ./usr/lib/gnunet/libgnunet_plugin_transport_bluetooth.so

OpenWRT has bluetooth? If not, remove.

> 33516 ./usr/lib/gnunet/libgnunet_plugin_transport_http_client.so
> 33516 ./usr/lib/gnunet/libgnunet_plugin_transport_https_client.so
> 37676 ./usr/lib/gnunet/libgnunet_plugin_transport_http_server.so
> 41824 ./usr/lib/gnunet/libgnunet_plugin_transport_https_server.so
> 33628 ./usr/lib/gnunet/libgnunet_plugin_transport_tcp.so
> 4512 ./usr/lib/gnunet/libgnunet_plugin_transport_template.so

No need to ship _template's, those are from example code.

> 43764 ./usr/lib/gnunet/libgnunet_plugin_transport_udp.so
> 22640 ./usr/lib/gnunet/libgnunet_plugin_transport_unix.so
> 20212 ./usr/lib/gnunet/libgnunet_plugin_transport_wlan.so
> 16972 ./usr/lib/libgnunetarm.so.1.0.3
> 25728 ./usr/lib/libgnunetats.so.4.0.0
> 52500 ./usr/lib/libgnunetatstesting.so.0.0.0
> 4492 ./usr/lib/libgnunetblock.so.0.0.0
> 20932 ./usr/lib/libgnunetcadet.so.5.0.0
> 15432 ./usr/lib/libgnunetconversation.so.0.0.0
> 22644 ./usr/lib/libgnunetcore.so.0.0.1
> 7560 ./usr/lib/libgnunetdatacache.so.0.0.1
> 17084 ./usr/lib/libgnunetdatastore.so.1.0.0
> 18852 ./usr/lib/libgnunetdht.so.0.2.0
> 14476 ./usr/lib/libgnunetdnsparser.so.0.1.0
> 7560 ./usr/lib/libgnunetdns.so.0.0.0
> 8451 ./usr/lib/libgnunetdnsstub.so.0.0.0
> 14172 ./usr/lib/libgnunetfragmentation.so.2.0.0
> 6292 ./usr/lib/libgnunetfriends.so.0.0.0
> 167128 ./usr/lib/libgnunetfs.so.2.1.1
> 12648 ./usr/lib/libgnunetgnsrecord.so.0.0.0
> 9436 ./usr/lib/libgnunetgns.so.0.0.0
> 12604 ./usr/lib/libgnunethello.so.0.1.0
> 15932 ./usr/lib/libgnunetidentity.so.0.0.0
> 4052 ./usr/lib/libgnunetmicrophone.so.0.0.0
> 10672 ./usr/lib/libgnunetnamecache.so.0.0.0
> 21080 ./usr/lib/libgnunetnamestore.so.0.0.1
> 32492 ./usr/lib/libgnunetnat.so.0.1.1
> 6464 ./usr/lib/libgnunetnse.so.0.0.0
> 12612 ./usr/lib/libgnunetpeerinfo.so.0.0.0
> 12780 ./usr/lib/libgnunetpeerstore.so.0.0.0
> 6796 ./usr/lib/libgnunetregexblock.so.1.0.0
> 5612 ./usr/lib/libgnunetregex.so.3.0.1
> 6904 ./usr/lib/libgnunetrevocation.so.0.0.0
> 8052 ./usr/lib/libgnunetscalarproduct.so.0.0.0
> 10676 ./usr/lib/libgnunetset.so.0.0.0
> 4252 ./usr/lib/libgnunetspeaker.so.0.0.0
> 17152 ./usr/lib/libgnunetstatistics.so.0.1.3
> 7708 ./usr/lib/libgnunettestbedlogger.so.0.0.0
> 109720 ./usr/lib/libgnunettestbed.so.0.0.0
> 22660 ./usr/lib/libgnunettesting.so.1.1.0
> 39460 ./usr/lib/libgnunettransport.so.2.2.0
> 15964 ./usr/lib/libgnunettransporttesting.so.0.0.0
> 12540 ./usr/lib/libgnunettun.so.0.1.0
> 319916 ./usr/lib/libgnunetutil.so.13.0.0
> 8424 ./usr/lib/libgnunetvpn.so.0.0.0
> 1895 ./usr/share/gnunet/config.d/arm.conf
> 2310 ./usr/share/gnunet/config.d/ats.conf
> 514 ./usr/share/gnunet/config.d/cadet.conf
> 854 ./usr/share/gnunet/config.d/conversation.conf
> 515 ./usr/share/gnunet/config.d/core.conf
> 64 ./usr/share/gnunet/config.d/datacache.conf
> 652 ./usr/share/gnunet/config.d/datastore.conf
> 764 ./usr/share/gnunet/config.d/dht.conf
> 1249 ./usr/share/gnunet/config.d/dns.conf
> 2331 ./usr/share/gnunet/config.d/exit.conf
> 2229 ./usr/share/gnunet/config.d/fs.conf
> 1111 ./usr/share/gnunet/config.d/gns.conf
> 1117 ./usr/share/gnunet/config.d/hostlist.conf
> 487 ./usr/share/gnunet/config.d/identity.conf
> 438 ./usr/share/gnunet/config.d/namecache.conf
> 723 ./usr/share/gnunet/config.d/namestore.conf
> 1509 ./usr/share/gnunet/config.d/nat.conf
> 1336 ./usr/share/gnunet/config.d/nse.conf
> 678 ./usr/share/gnunet/config.d/peerinfo.conf
> 286 ./usr/share/gnunet/config.d/peerstore.conf
> 243 ./usr/share/gnunet/config.d/pt.conf
> 191 ./usr/share/gnunet/config.d/regex.conf
> 379 ./usr/share/gnunet/config.d/resolver.conf
> 483 ./usr/share/gnunet/config.d/revocation.conf
> 567 ./usr/share/gnunet/config.d/scalarproduct.conf
> 226 ./usr/share/gnunet/config.d/set.conf
> 429 ./usr/share/gnunet/config.d/statistics.conf
> 378 ./usr/share/gnunet/config.d/template.conf
> 4409 ./usr/share/gnunet/config.d/testbed.conf
> 297 ./usr/share/gnunet/config.d/testing.conf
> 178 ./usr/share/gnunet/config.d/topology.conf
> 3547 ./usr/share/gnunet/config.d/transport.conf
> 2373 ./usr/share/gnunet/config.d/util.conf
> 370 ./usr/share/gnunet/config.d/vpn.conf

Btw, it is technically 'safe' to combine all of the '.conf' files into
one big file (just using 'cat').  If your file system doesn't like tons
of files (inode overheads, etc.) that might help. You could also remove
comments (lines starting with '#').

> 105182 ./usr/share/gnunet/gns-bcd.html

That's for generating business cards with QR codes, to make this really
work you'll need to have TeXlive installed as well.  Definitively just
don't ship this one.

> 6851 ./usr/share/gnunet/gnunet-logo-color.png

Also unnecessary on OpenWRT, not sure it's used by anything (maybe
gnunet-gtk).

> 380 
> ./usr/share/gnunet/hellos/DSTJBRRKZ8TBW3FGK6B0M5QXWT9WYNZ45H5MCV4HY7ST64Q8T9F0


Happy hacking!

Christian

signature.asc
Description: OpenPGP digital signature