Re: [help-GIFT] Histograms and Normalization

[Wolfgang Müller]

On Wednesday 04 June 2003 08:05, Sailesh Suvarna wrote:
> Hi all,
> Thanks for the answers to my last question.
>
> I have some more...........
>
> I am using just 4 images in my collection and am using only feature (1) -
> Colour Histograms. In  CWeightingFunction::subApply if I change-
>
> lRetVal=copysign(min(inDocumentFrequency,fabs(getTermFrequency())),getTermF
>requency());
>
> to
>
> lRetVal=fabs((inDocumentFrequency-getTermFrequency()));
> to find out the difference rather than similarity
>
> I get the following results:
> Similarity: inf
> Similarity: nan
> Similarity: inf
> Similarity: inf
>
> Also the lQueryScore always computes to 0
>   The following is what I get
>
>  Pruning used!
>  Pruning: I will evaluate 39 Features.
>
>  [keepScorePruning 0 - before adjusting - running total: 0.000000
>  In CWeightingFunction(C-IDF)-returning 0.000000
>  I got:subApply(C-IDF) return value: 0.000000
>  [keepScorePruning 0 -  after adjusting - running total: 0.000000
>
>  In CWeightingFunction(C-IDF)-returning 0.000000
>  I got:subApply(C-IDF) return value: 0.000000
>  [keepScorePruning 1 - before adjusting - running total: 0.000000
>  In CWeightingFunction(C-IDF)-returning 0.000000
>  I got:subApply(C-IDF) return value: 0.000000
>  [keepScorePruning 1 -  after adjusting - running total: 0.000000
>
> In CWeightingFunction(C-IDF)-returning 0.000000
> I got:subApply(C-IDF) return value: 0.000000
> [keepScorePruning 2 - before adjusting - running total: 0.000000
> In CWeightingFunction(C-IDF)-returning 0.000000
> I got:subApply(C-IDF) return value: 0.000000
> [keepScorePruning 2 -  after adjusting - running total: 0.000000
>         <snip>
>
> In CWeightingFunction(C-IDF)-returning 0.011765
> I got:subApply(C-IDF) return value: 0.011765
> In CWeightingFunction(C-IDF)-returning 0.000000
> I got:subApply(C-IDF) return value: 0.000000
> In CWeightingFunction(C-IDF)-returning 0.011765
> I got:subApply(C-IDF) return value: 0.011765
> In CWeightingFunction(C-IDF)-returning 0.031373
> I got:subApply(C-IDF) return value: 0.031373
> [keepScorePruning 38 - before adjusting - running total: 0.000000
> In CWeightingFunction(C-IDF)-returning 0.000000
> I got:subApply(C-IDF) return value: 0.000000
> [keepScorePruning 38 -  after adjusting - running total: 0.000000
>
> In CWeightingFunction(C-IDF)-returning 0.003922
> I got:subApply(C-IDF) return value: 0.003922
> In CWeightingFunction(C-IDF)-returning 0.000000
> I got:subApply(C-IDF) return value: 0.000000
> In CWeightingFunction(C-IDF)-returning 0.007843
> I got:subApply(C-IDF) return value: 0.007843
> In CWeightingFunction(C-IDF)-returning 0.019608
> I got:subApply(C-IDF) return value: 0.019608
> Out of keepScorePruning: 0.000000[Timestamp3.78]
> .query
> presort
> [Timestamp3.78]
> [Timestamp3.78]
> postsort
> Assembling a query result tree
> from a result of size4
>
> Why is the above progressively done 4 times?

4 query engines are queried :-) You can change that by changing the 
configuration file (cut some of the sub1...sub4 algorithms, and leave only 
the one that has not blocked the histogram features)

> Where is the normalization taking place 
CQInvertedFile.cc:949:
    (*i).second/=fabs(lQueryScore);

The score of the first result is divided by the score of the query applied on 
itself (i.e. the query does not need to be a document).

> and what factor is used? 

CQInvertedFile.cc:1056:
lQueryScore+=
        (*i)->applyOnThis();

> Is it
> possible to just turn off the normalization for experimenting.

Either you comment out line 949 (quick fix, interfering with everything!), or 
you make applyOnThis virtual in CWeightingFunction.h:243: and then overload 
it in CWFClassicalIDF to your needs somewhere in CWFClassicalIDF.cc:

double CWFClassicalIDF::applyOnThis()const{return 1;}

Even better would be if you hacked both CWeightingFunction.h:243:
CQInvertedFile.cc:142:
  (*this)["SaileshTest"]=
    new CWeighter(new CWFSaileshTest(0,0,0),
                  new CQNNoNormalization(0),
                  new CQNMaxDocumentFrequency(0));

and created your weighting function CWFSaileshTest (or whatever name you 
chose). If you find something worth demoing this won't interfere with the 
rest.


> Awaiting your reply,
I hope this helps,
> Sailesh
Wolfgang