[GNUnet-developers] fshcron/fshcrontab

[jan marco alkema]

Hi All,

I want to test the programming process ("proof of concept") from java GUI to C-core via Jdbc and visa versa. Port it also to Windows.

What:

Purpose: Schedule jobs to meet predefined goals.

A lot of work load is not needed right away like "gnunet-insert/gnunet-download".

For example: "Before 15 may 2003 I want the Redhat 9.0 CD’s => job scheduler manage this goal for me (beside the other goals)."

If you look at http://www.mysql.com/doc/en/OPTIMIZE_TABLE.html you can see that "optimize table" locks the table. I don’t think that inserting of information is enabled as the optimize command runs. Schedule the optimize command of the tables should also be done by the job scheduler. Maybe a status "locked database" is needed in the "gnunet" protocol for the other gnunet servers.

In mine point of view there are two types of programs namely intern and extern programs. Intern programs are in the build version (like optimize table) and extern programs will be executed via the Operating System "exec" command.

A job contains of 1..N steps.

For instance:

(Job 1, step 1): /tmp/mysql.tar.gz <- file_from_http_server(http://www.server.com/mysql.tar.gz)

(Job 1, step 2): catalog(/tmp/mysql.tar.gz)

(Job 1, step 3): /tmp/mysql.tar.gz.encrypt <- encrypt(/tmp/mysql.tar.gz)

(Job 1, step 4): /tmp/mysql.tar.gz <- de-encrypt(/tmp/mysql.tar.gz.encrypt)

(Job 1, step 5): catalog(/tmp/mysql.tar.gz)

How:

I use http://fcron.free.fr/ for the core cron engine. Java for the user interface to the core cron engine via the Jdbc protocol. Because I use Jdbc a polling technique will be used. Recalculation of the scheduler can be done by a dummy job which will be executed every 300 seconds. If someone wants to fire an instant job tcp/ip programming should be preferred. N.B. I won’t use JNI because you get platform dependent code.

Implementation:

I ‘made’ two programs: fshcron.c and fshcrontab.c. I must port the current file system implementation of fshcron to MySQL/ODBC database. I only thing I will use in fshcrontab is the signal function to fshcron. A little bit strange fcron uses an extern program for the signal function.

# /usr/bin/fcronsighup

Modifications will be taken into account right now.

If in fshcore a new MySQL query has arrived (via Java GUI) and recalculation gives a new scheduler time then fshcore should signal fshcron for updating.

For job scheduler programs the same GUID (global unique identifier) concept as I mentioned for the telephone records could be used. Depending on the attributes of a (job scheduler) program you can make a hash (distribution) number. This hash distribution number should be programmed in the program like:

./download_file_from_webserver –d

"A317EACA667CBF38F356121765B40527A22638159890C3BB8A0AD1D3CFC9C193C8F214865094FF8C"

If the hash distribution number has a registration everyone knows what the program does.

The hash of the version of "download_file_from_webserver" can be obtained by the hash of the executable of the program "download_file_from_webserver ".

./download_file_from_webserver –r

"70D02E4EB48F1AD36ED52B3C67B73283898DC8193507BFAFFD3C05BA13F512C78456C4D3035E6948"

Note: I don’t think it is very simple to program the version hash in the code, because implementation of the hash in the code gives another hash to implement. The only solution I see is to insert the hash routines in the source code or call an external program for the computation of the version hash code.

Requirements:

· Java for GUI and C for cron core engine. In Java you can program visual time tables of years, months, day and hours.

· All log messages should be logged in database. It will be very simple to analyze the different log messages. N.B. In SQL you can count the number of messages of some kind in a certain period of time very easily.

· Don’t use a configuration file. All configuration items should be in database.

· Multiple job schedulers ought be used on the same Unix server. The different Tcp/Ip address can be obtained out of the configuration database.

· Synchronization technique between different job schedulers is an important issue. For example: Run transportation job of a file on Unix to Mainframe, if the transfer has succeeded start a SAS job on the Mainframe environment.

If someone has better ideas about this subject please let me know,

Greetings Jan Marco

 

Appendix A: number of lines of fshcron.c

# cat fshcron.c|wc -l

5877

Appendix B: make of fshcron.c

# make

gcc -O3 -Wall -DWITH_MYSQL -I/usr/local/fsh/var/mysql/include/mysql -I/usr/local/unixODBC/include -c -o fshcron.o fshcron.c

gcc -O3 -Wall -DWITH_MYSQL -I/usr/local/fsh/var/mysql/include/mysql -I/usr/local/unixODBC/include -o fshcron fshcron.o -L/usr/local/fsh/var/mysql/lib/mysql -lmysqlclient -ldl -lm -lnsl -lpthread -lgdbm -lcrypto -L/usr/local/lib -lmyodbc3

Appendix C: ./fshcron -f

21:53:17 Looking for jobs to execute ...

21:53:17 next sleep time : 299

21:55:00 updating configuration from /var/spool/fcron

21:55:00 adding new file root

21:55:00 User new.root Entry

pass 1 = 0

pass 2 = 0

pass 3 = 0

pass 4 = 0

21:55:00 Env: "SHELL=/bin/bash"

21:55:00 cmd date>>/usr/local/fsh/fcron/jma3.txt next exec 5/7/2003 wday:3 22:00:00

21:55:00 from last conf: date>>/usr/local/fsh/fcron/jma3.txt next exec 5/7/2003 wday:3 21:58

21:55:00 Saving root...

21:55:00 write_buf_to_disk() : written 126/126, 1 (re)try(ies)

21:55:00

21:55:00 Looking for jobs to execute ...

21:55:00 next sleep time : 196

Appendix D: number of lines of fshcrontab.c

# cat fshcrontab.c|wc -l

3366

Appendix E: make of fshcrontab.c

# make

gcc -O3 -Wall -DWITH_MYSQL -I/usr/local/fsh/var/mysql/include/mysql -I/usr/local/unixODBC/include -c -o fshcrontab.o fshcrontab.c

gcc -O3 -Wall -DWITH_MYSQL -I/usr/local/fsh/var/mysql/include/mysql -I/usr/local/unixODBC/include -o fshcrontab fshcrontab.o -L/usr/local/fsh/var/mysql/lib/mysql -lmysqlclient -ldl -lm -lnsl -lpthread -lgdbm -lcrypto -L/usr/local/lib -lmyodbc3

Appendix F: ./fshcrontab my-fcrontab

21:54:59 fcronconf=/etc/fcron.conf

21:54:59 installing file /usr/local/fsh/fcron/fshcrontab/my-fcrontab for user root

FILE /usr/local/fsh/fcron/fshcrontab/my-fcrontab

SHELL=/bin/bash

Env : 'SHELL=/bin/bash'

!mailto(root)

Opt : "mailto" "root"

@first(1) 300s date>>/usr/local/fsh/fcron/jma3.txt

Opt : "first" 60

Cmd "date>>/usr/local/fsh/fcron/jma3.txt", timefreq 300, first 60

Saving ...

21:54:59 write_buf_to_disk() : written 126/126, 1 (re)try(ies)

Modifications will be taken into account right now.

Appendix G: # cat my-fcrontab

SHELL=/bin/bash

!mailto(root)

# run every 5 minutes

@first(1) 300s date>>/usr/local/fsh/fcron/jma3.txt