Re: [Logs-devel] fixing (removing) dynamically bound variables

[Jim Prewett]

Hi Aashish,

Its nice to see you joined the mailing list. :)  Let me see if I can clear 
some things up for you.

Let me first say that LoGS isn't quite (at all?) like the other tools out 
there.  The point is to enable us to do just about anything we can dream 
up when analyzing our logfiles.  So, things can be a bit strange when 
moving from a tool like SEC or Logsurfer :)

On Mon, 8 Jan 2007, Aashish Sharma wrote:

> I am still quite unfamiliar with LoGS but I will definitely stress the 
> importance of having ENVIRONMENT variables storing the dynamically bound 
> variables.

LoGS, as is currently the latest in the CVS tree, uses what is known as 
"dynamic variable binding" (where the variables are treated as, 
essentially, global variables).  Lisp used to be known for using only 
dynamic variable binding until it was decided that that was not a very 
good idea and lexical binding was the way to go. :)

The current LoGS allows for functions to be defined where the value of a 
variable used in that function could be "special" or dynamically bound - 
its value would come, seemingly from nowhere.  The following would be a 
perfectly acceptable "action function" (and would cause no compiler 
warnings or anything).

(lambda (message)
  (declare (special server-name))
  (format t "misbehaving server: ~A~%" server-name))

this happens via a magical process where variables from a variety of 
places (stored in the matching rule and ruleset(s) and returned by their 
match functions) would be bound, again dynamically, and assigned values, 
then the "action functions" would be run.

One of my colleagues points out that using dynamic variable binding 
instead of lexical variable binding can cause debugging nightmares.  I 
didn't believe him until I experienced a debugging nightmare due, 
entirely, to this dynamic variable binding. 

Vijay has pointed out, privately (I believe), that the use of dynamic 
variable binding may present problems for multi-threaded applications - 
Clearly LoGS will want to be multithreaded if possible.  For now, it is my 
opinion that we won't need to go multithreaded until the large systems (as 
in top 10 on top 500) grow by several orders of magnitude, so we've got a 
while (but I still believe it could be worthwhile to make LoGS 
multithreaded for other reasons).

So, to make a long long long sad story short, :) these are the variables I 
meant by ENVIRONMENT variables.  And, I think it was actually pretty 
different from what you were thinking of.

I think what I just described needs to be serioiusly re-thought, 
re-worked, etc.  I would imagine that we could also get an efficiency win 
by getting rid of dynamic variable bindings if we do things right.  

> I work with SEC rules and was getting into issues of a cascade 
> modification problem where I would be going into each rule file and 
> trying to determine and change paths for scripts folders or such local 
> variables. I ended up initializing a lot of environment variables and 
> made sec read those values in rule set. That made things a little 
> simplified.

While this isn't the sort of thing that is built into LoGS per-se, it 
should be pretty easy to do since you've got a full programming language 
behind you.  You could easily define a variable like:
(defvar *MY-OUTPUT-DIRECTORY* #P"/var/log/myLoGS/")

Then you could freely use that variable throughout your ruleset.

> I think a conf file with default paths and other dynamic variables with 
> default values would be a good feature to have.

Well, LoGS is a sort of an interesting beast. :)

We don't really have that feature, but we already have that feature a 
little differently.  :)

IMO, Lisp provides plenty of perfectly good ways for working with paths, 
I/O, etc.  LoGS provides only a couple of convienance functions for 
writing messages and the contents of contexts, etc. out to files, 
specifically, the WRITE-TO-FILE methods in the LoGS package.

One could have a "ruleset" file which would contain only variable 
definitions.  ruleset files are loaded in the order listed on the 
command-line, so a command line like the following could be used to first 
load some path variables from path-definitions.lisp that are then used in 
my-ruleset.lisp

lisp -core /path/to/LoGS.core \
--ruleset /my/LoGS/dir/definitions/path-definitions.lisp \
--ruleset /my/LoGS/dir/rulesets/my-ruleset.lisp

path-definitions.lisp might contain something like:

(defvar *output-directory* #P"/my/output/dir/")

and my-ruleset.lisp might contain something like:

(LoGS::enqueue 
  LoGS::*root-ruleset*
  (rule named 'default-rule
        matching regexp ".+"
        doing
        (lambda (message)
         (write-to-file 
          (merge-pathnames *output-directory* "out.log")
          message))))

which would match any message that had one or more characters (a regexp is 
a less-than-optional choice here, but this is just for the purposes of 
demonstration) and would write matching messages to the file out.log in 
the directory specified in the variable *output-directory*.

Also, since LoGS is completely programmable, these ruleset files could 
also contain function defintions.  You could, in theory, write a function 
like:

(let ((filename "out.log"))
   (defun write-message-to-my-logfile (message)
     (write-to-file 
      (merge-pathnames *output-directory* filename)
      message)))

Then, your rule could look like:

(LoGS::enqueue
  LoGS::*root-ruleset*
  (rule named 'default-rule
        matching regexp ".+"
        doing
        (lambda (message)
         (write-message-to-my-logfile message))))

or even

(LoGS::enqueue
  LoGS::*root-ruleset*
  (rule named 'default-rule
        matching regexp ".+"
        doing
        #'write-message-to-my-logfile))

make sense?

Gosh, I sure hope this (coredump) helps...
Jim

> Aashish 
> On Sun, Jan 07, 2007 at 12:03:52AM -0700, Jim Prewett wrote:
> > 
> > > How would we pass the ENVIRONMENT to the actions?
Ã> > 
> > I was thinking of modifying run-actions in rule.lisp in partiucular to 
> > remove the IN-GIVEN-ENVIRONMENT references and instead funcall the action 
> > functions with an additional argument: This is the simplest thing I can 
> > think of to do.
> > 
> > I also think that such a change would make testing a ruleset simpler as, 
> > instead of having variables bound, they would be passed in in a 
> > predictable way.
> > 
> > It should also make all of that scoping stuff we were talking about 
> > resolve itself very nicely.  Variables bound by in/the Lisp code would be 
> > clearly separated from things bound by the match functions.  Lisp already 
> > does its scoping right and LoGS already maintains the ENVIRONMENT list 
> > correctly; My concern before was in how these "variables" could co-exist 
> > in the sensibly in the same namespace, which, possibly, they cannot in any 
> > sensible way.
…> > 
> > I've done a little playing with this and I think the modifications should 
> > be pretty small.
> > 
> > Jim
> > 
> > 
> > 
> > _______________________________________________
> > LoGS-devel mailing list
> > address@hidden
> > http://lists.nongnu.org/mailman/listinfo/logs-devel
> 
> 
> _______________________________________________
> LoGS-devel mailing list
> address@hidden
> http://lists.nongnu.org/mailman/listinfo/logs-devel
>