Keymap priorities - how do define a global, low-priority binding from a minor mode?

[David Reitter]

I have defined a global minor mode which gives me a bunch of key  
bindings, among them bindings for `up' and `down'.

Turns out that the global minor mode overrides local key maps, at  
least minibuffer-local-map, which defines its own bindings for these  
keys (which I would prefer).

How do I specify priorities for key bindings? Can I define a minor  
mode key map that does not override the minibuffer map? Is using the  
global key map from a global minor mode a clean solution??

In this particular case, defining a key for [remap 'next-line] does  
the job, but is semantically different - it doesn't allow me to  
distinguish between C-n and `down', so I'm trying to get rid of it.

Thanks,
David