Re: [Neurostat-develop] API proposal

[Joseph Rynkiewicz]

On Fri, 29 Mar 2002, Fabrice Rossi wrote:  
> Basically, I don't really understand what can be simplified thanks to these
> layer functions. At first I tought we could hide details to layer but in fact,
> it might be difficult because of one of our main assumptions: we allow a layer
> to receive inputs from any preceding layer. Therefore, to calculate the output
> of one layer, we must in general use parts of *act. 

True, but we have only to do the propagation in the right order, and allow
some layer to share partially the same entries.
A layer may be constructed with several matrices for each group of
entries, and one vector for the wheight of the bias and only one output.
The API have to be changed in  (note the double * before input) :

int layer_propagate(Layer *lay, double *weights, double ** input, 
double* output, activation * act, activation *dact)

the function layer propagate become :

copy vector bias in the output
loop on matrices relying entries "i" to output
  output <- multiplication of matrix[i] with vector[i] of entries + output
(this is a function of blas)
activation of the output

The order of the layers is natural,
the first one have for output the first layer,
(the entries aren't a layer)
the second one the second  layer, etc...

For the layer k the k possible vector of entries are the output of the
layer k-1,...,k-(k-1), entries of the MLP.

For the backpropagation the algorithm become :

loop on layers 
 layer_back_propagate

The API of layer_back_propagate is changed in

int layer_back_propagate(Layer *lay, double *weights, 
double *local_deriv_out,double **local_derive_in, activation *dact)

with layer_back_propagate :

receive derivative with respect to the output in local_deriv_out 
local_derive_out<-multplication of local_deriv_out with *dact
loop on matrices relying output of previous layer "i" to output
 local_deriv_in[i]<-transpose(matrix[i])*local_deriv_out

A real part of the complexity is in the layer, 
but The MLP have the not so easy task to compute the good adresses for all
the pointers used in the previous functions.

(remember that the parameters of weights are stocked in a one
dimensional vector without hole, even if we use sparse MLP, and that the
matrices and vector of the layers relies only to the "I hope good" address
inside the weights vector)

I think that not using the "layer" concept will be too hard,
moreover these function are only the bases, keep in mind that we
will have to construct sparse MLP and remove connections. 
In a sparse MLP, only one connection missing can change fundamentally the
architecture, so we have to carefully modularize our code to be able to
calcul the change of architecture...

I hope that I have been understandable... 

Joseph