Unboxed package manager

[Lynn Winebarger]

Hi,
I want to write a small package "unboxed" that extends the built-in
package manager with packages whose elisp files are installed into a
common packages directory instead of one directory per package.  I've
done this manually on various systems I use with a significant
improvement in startup performance.  Of 2403 packages I've installed
on my current personal system, only 161 have a subdirectory.  Ignoring
"-pkg.el" and "-autoloads.el" files, 1971 of those 2403 packages have
a single elisp file.  I'm including a complete histogram table at the
end of this mail for reference.
I would like to just tweak the existing functionality so that, if a
predicate is satisfied, a given package will be locally administered
by "unboxed" rather than "package":
* mapping package contents to local paths for installation and removal
(e.g. info files go to a common info directory for unboxed packages,
themes to a separate theme directory, etc)
* creating autoloads for the common directory, preferably after an
entire set of packages has been installed/removed rather than
one-at-a-time
* providing package-desc objects to the package initialization routine
I can think of 3 predicates for determining when "unboxed" might be
assigned, depending on how risk-adverse the user is
1) A package contains a single elisp library
2) A package has no subdirectories
3) A package has subdirectories but it is referenced through some
variable set in one of the source files, which can be patched.
Most packages are amenable to the 3rd option - with the exception of a
small number of cases, of which none that I can recall have a
subdirectory anyway, the variable in question contains the only
reference to "load-file-name".
There are some packages that simply cannot be managed this way without
extraordinary effort, basically any package using "load-relative".
Packages that attempt to facilitate active development even after
installation can also do things that are problematic, like look for
git information in the installation directory.  An explicit list of
these cases (as a customization variable) can be maintained to ensure
they are handled the default way.
>From my perspective, package.el already supports 2 package tracking
mechanisms - one for "built-in" packages, and the usual one for
external packages or updates to the built-in packages.  However, if
there is no appetite for facilitating a more general "wiring in" of
additional local package management (which might be useful if you
wanted to enable backends with an explicit database of installed
packages), then my best option would appear to be to derive from the
package-menu-mode as a replacement, and either define variants of the
functions in the package-menu and package namespaces or use advice.
The latter goes against my taste in code, but is probably the most
expedient.
Can any maintainers of package.el advise?

The following is the promised histogram of the number of packages from
my sample with a specified number of elisp libraries.  2403 package
directories in load-path absolutely devastates locate-file.  Even the
most conservative approach above (option 1) would drastically reduce
this effect.
#libs  #packs   Percent
 1         1971       82.022
 2          148       88.181
 3           57       90.553
 4           37       92.093
 5           17       92.801
 6           49        94.84
 8           11       95.298
 9           17       96.005
10            8       96.338
11            5       96.546
12            8       96.879
13            7        97.17
15            7       97.462
16            4       97.628
17            5       97.836
18            6       98.086
19            2       98.169
20            2       98.252
21            2       98.335
22            6       98.585
23            1       98.627
24            3       98.752
26            2       98.835
27            9       99.209
29            1       99.251
30            1       99.293
31            4       99.459
32            2       99.542
35            1       99.584
36            2       99.667
37            1       99.709
41            2       99.792
45            2       99.875
66            1       99.917
77            1       99.958
95            1          100

For the curious, the last three are jdee, doom-themes, and lsp-mode
respectively.  They can be "installed" by simply copying them into a
common directory, updating the directory autoloads, and byte-compiling
without any issue.
Thanks,
Lynn