[DotGNU]Concept: Box files

[Peter Minten]

Hi,

I've did some more thinking about the .box concept, this is the result:

--- CONCEPT ---

Name: DotGNU box files standard

Description:

Java has jar files that make it possible to send a program in a nice small
package over the internet and to run it straight from that package without
unzipping or anything. This maybe useful to IL bytecode programs too, especially
since IL programs tend to have extras like resource files that need to be send
with the executable.

What I'm proposing are .box files which can:
* contain executables
* contain support files (resource files)
* contain configuration files
* contain a whole project directory with source code

The box files would enable a user to download a whole program + source in a
simple efficient package. The box files would have a description in them telling
what they contain and some details of using it (somewhat like the .la libtool
files). Programs wanting to link with the contents of a box file can do so
without complicated manoeuvres (executables are unzipped into temp dir and run,
libraries are only unzipped into a tempdir) simply by using a central package
management program that keeps track of all the boxes on a system and notes their
contents in a registry. This makes difficult linking commands superfluous.

Of course there is the matter of support files that need to be present
somewhere, these files are simply unzipped into a temp dir where they stay until
the last process of the program ends. The configuration files are simply put
into the DotGNU Registry where they stay even when the program is terminated.
This requires that a programmer knows what files are non-changing and what are
changing, support files should not change, configuration files could change.

The box files could be automatically signed with a GPG key for security, a sign
can also be added to an existing box. This can be used for distribution over
central servers: programmer A signs the box with his private key, server B has
A's public key and uses it to check the box A send in, Server B replaces key A
with key B which public part is known to user C, D, E, etc. Box files can also
contain an md5sum, in combination with signing this allows for secure downloads
from trusted parties.

The box files on a system are managed by an application called 'Warehouse'.
Warehouse keeps track of the packages installed on a system, every time a
package is removed or added Warehouse updates it's inventory, updating may also
be put in as a cron job. Warehouse also checks the requirements of new programs
and alarms the user if a new version of an existing or a completely new package
needs to be downloaded, if the user has specified downloading locations (I'll
tell more about these later) Warehouse can automatically download the
applications. It also checks automatically if the file is signed with a key in
it's trusted key ring and if the md5sum is correct.

Apps seldom have to deal with Warehouse if the runtime engine interfaces with it
(which is so easy that it shouldn't pose any problems). An app simply specifies
the libraries it want's to link to and Warehouse tells the runtime engine where
the files are.

The user will also seldom deal with Warehouse since there will be a graphical
interface to Warehouse called 'Warehouse Manager'. This program simply gives the
user easy access to all functions of Warehouse.

Warehouse will know (using information in the boxes and/or information on the
net) what version of a program/library is interface compatible with what other
versions. If a program requires a higher version of a file that is interface
compatible with the version on the machine Warehouse will suggest to the user to
upgrade the package, if it isn't interface compatible Warehouse will make sure
that both versions of the package live safely besides each other on the system.
Warehouse can also do a cleanup scan in which it can suggest the removal of
library boxes which no other box needs, bye DLL hell (note that the DLL hell 
can come to GNU/Linux with IL libraries carrying the .dll extension).

Warehouse can find needed boxes on the net, using a list of download locations
or by simply searching the web. It can also offer downloads or offer linking to
boxes on it's system (making it the DotGNU app server). This functionality is
based on p2p. Every warehouse knows a number of other warehouses, when it needs
something it asks the warehouses it knows, if they don't know it either they
will ask the warehouses they know, the search stops however at the first match
(so that the network won't get overloaded). Also a warehouse can offer boxes for
linking or for download.

A warehouse can also keep an inventory list of the warehouses it knows, the
number of inventory lists it has can be changed by the user, also a warehouse
can decide if it offers an inventory list. Some warehouses may even do nothing
else than keep inventory lists (becoming a webservice directory), a number of
these warehouses placed on big servers and with good interaction and
co-ordination can speed up the search a lot (a normal search could only ask a
webservice directory, for most of the software this would give good results).

The webservice directories could have some keyword tracking mechanisms to purge
the inventories from cracked software, unauthorized copied software will only
help the proprietary software firms in the long run (by increasing their
marketshare). Besides that, it avoids the BSA hunting the providers of the
webservice directories. This would by the way not prevent the distribution of
unauthorized copied software over the warehouse network since the p2p concept
makes that impossible, but it would keep that kind of software out of the search
results of most users.

--- END CONCEPT ---

And here are a few details on how I want to do it:

--- DETAILS ---

The box files are build up like this:
1. Make is called in a project dir to create the binaries
2. A manifest file is put up out of data in a box.manifest file and other data
gathered by the 'box' utility.
3. Binaries, source stuff (the whole project dir without files included
elsewhere), support files and configuration files are put into 4 separate temp
dirs.
4. The temp dirs are tarballed and gzipped.
5. The manifest file is gzipped.
6. The tempdir tarballs and the manifest file are tarred together.

When a user wants to run a program support files are unpacked, configuration
files are loaded into the local registry (DotGNU version) and the binary is
loaded into memory. If the user wants to work with the source code the box
utility uses the manifesto to create a project dir in which is places all the
files from the source, support and config packages in their original locations.

The manifesto is a simple text file. 

The 'box', 'warehouse' and 'warehouse-manager' programs are written in C
(because of speed matters and using C makes it easier to expand the use of
Warehouse beyond IL executables if that becomes feasible).

The programs will use the system or execve functions for using 'gzip' 'tar' and
'gpg', this is a lot easier than using library interfaces.

--- END DETAILS ---

In short: the warehouse system: 
* Will make it easier for users to work with IL programs. 
* Will encourage distributing source (by making binary + source distribution the
standard in the box app).
* Will enable secure downloads and usage of IL programs.
* Will be a DotGNU replacement for UDDI.
* Will be the DotGNU app server.

Greetings,

Peter