.\" $Id: kicker.man,v 1.29 2012/08/23 17:13:12 ksb Exp $ .\" by Kevin Braunsdorf .\" $Compile: Display%h .\" $Display: ${groff-groff} -Tascii -man %f |${PAGER:-less} .\" $Display(*): ${groff-groff} -Tascii -man %f .\" $Install: %b -mDeinstall %o %f && cp %f $DESTDIR/usr/local/man/man8/kicker.8 .\" $Deinstall: ${rm-rm} -f $DESTDIR/usr/local/man/[cm]a[nt]8/kicker.8* .TH KICKER 8 LOCAL .SH NAME kicker - send recurring tasks to various batch queue .SH SYNOPSIS .ds PN "kicker \fI\*(PN\fP [\fB\-nv\fP] [\fB\-S\fP\~\fImnemonic\fP] [\fIhours\fP | \fIfiles\fP] .br \fI\*(PN\fP \fB\-B\fP [\fB\-nv\fP] [\fIgroup\fP] .br \fI\*(PN\fP \fB\-h\fP .br \fI\*(PN\fP \fB\-V\fP .br .SH DESCRIPTION This is a special purpose tool to inject recurring clean-up tasks into the \fBbatch\fP(1) queue system to process log files or related recurring system tasks. These tasks are executed as the owner of the injected file, under a \fIbatch\fP queue that might restrict the number of jobs run in parallel and the CPU nice value set for each task. .P A 25 line \fBcrontab\fP(5) control segment injects the present hour's tasks into any batch queue. The hours are usually \fB00\fP, \fB01\fP... to \fB23\fP, plus \fBtop\fP for the top of every hour and \fBeod\fP the end-of-day (the last daily time \fIcron\fP will schedule, usually 23:55). .P The superuser may inject commands, by name, at any time. In addition to that insertion the environment variables $\fBHOME\fP, $\fBLOGNAME\fP, and $\fBUSER\fP are replaced with values from the password file for the target login. .P The \fIhours\fP parameter may be a full path to a node under the spool directory (the \fI/var/kicker\fP prefix is always assumed in queue selection, redundantly stating it doesn't hurt anything). Any directory specified by \fIhours\fP is recursively searched for tasks to insert, the special directory \*(lqOLD\*(rq is not searched as it is used by \fBinstall\fP to keep back-out files. .P A batch queue is selected based on the name of each plain file in the \fIhours\fP subdirectory of \fB/var/kicker\fP. Any single letter followed by at least one of dot (.), slash (/) or dash (\-) specifies a queue for that file to run under. The default is the \fBZ\fP queue, which doesn't exist in the default configuration of the \fBbatch\fP(1) system. .P In addition to the suggested 25 time-based divisions the administrator might choose to implement a \fBboot\fP queue for tasks that are started by mortal users at system startup. The inverse queue (\fBshutdown\fP) has not been as useful, as the queued tasks may actually run after the system reboots, not as the system is shutdown. .P A final queue \fBmanual\fP might hold tasks that are only started \fIad hoc\fP, or by some other task, to complete a common operation in background. .SH RATIONAL .P Using \fBcron\fP directly has 4 down-sides: .TP Conflicts in control of a single crontab The \fBcrontab\fP command is an interactive interface, application of a change to a shared crontab across many hosts is error-prone, in that any unique lines in a hosts crontab might be deleted by the application of a "clean copy" from an up-stream source. .TP Contention for a time slot Multiple tasks want to run at exactly 12:00am. \fBcron\fP is happy to start many major tasks all in a single second, as it should. We'd rather leverage the \fBbatch\fP rules to keep the resource demand on a host more even. .TP Limits in product testing It is really hard to test a crontab, especially a single line from those (many) listed. .TP Contention for a resource Multiple tasks might be network intensive, or CPU intensive. If we put those in a sequenced batch queue we can improve through-put, at the cost of a little uncertainty in the exact start time of the task. .SH OPTIONS If the program is called as \fI\*(PN\fP then no options are forced. .TP \fB\-B\fP Build the directory structure \fI\*(PN\fP expects. .TP \fB\-h\fP Print a brief help message. .TP \fB\-n\fP Do not execute commands, trace only. .TP \fB\-S\fP \fImnemonic\fP Use an \fBop\fP(1l) access escalation rule to become the target login of each batch task submitted. The op rule needed is given under NOTES below. .TP \fB\-v\fP Be verbose, trace actions with pseudo shell commands. A comment in the shell output for the format "# uid.gid" explains the applied effective (and real) user and group ids. .TP \fB\-V\fP Show version information. .SH EXAMPLES .TP \fI\*(PN\fP 16 This command might be triggered from \fBcron\fP at 16:00 hours (tea time) to inject log processing commands into several batch queue. These commands may continue processing by re-batching cascaded processing, or wait for another \fB\*(PN\fP run to launch any follow-up actions. .TP \fI\*(PN\fP top 16 This command adds tasks in the \fItop\fP directory to the list of things to queue at tea-time. For tasks that run hourly this is a safe way to run them (as \fBbatch\fP won't run them in parallel). .TP find /var/kicker \-type f \-print |grep \-v OLD |xapply \-f 'ls \-ld' \- Outputs a nice table of the tasks in \fB\*(PN\fP's queue. .TP \fI\*(PN\fP \-S batch manual/b.squisher This assumes that the \fBop\fP rule "batch" is available, if so the mortal login \fBkicker\fP starts the "squisher" task in the batch queue from the spool of manual tasks as the owner of the queued file. See the NOTES below for the required \fBop\fP rule. .SH NOTES .P To run \fI\*(PN\fP as a mortal login (rather than the superuser) we need an \fBop\fP(1l) rule to allow the mortal login (viz. \fBkicker\fP) to run the batch program as another login. .sp This rule allows the login \fBkicker\fP to run the \fBbatch\fP(1) program as anyone that owns a task-script under \fB/var/kicker/\fP, with input taken from the file, a queue name that is a single letter, the credentials of the file's owner and group, with some environment variables passed in as well. .RS .nf batch /usr/bin/batch \-q $1 ; users=^kicker$ %f.path=^/var/kicker/.*$ stdin=<%f $1=^[a-zA-Z]$ uid=%f gid=%f initgroups=%f $KICKER_Q_TIME $HOME $USER $LOGNAME $SHELL $PATH .fi .RE More escalation restrictions may be placed on the \fBbatch\fP mnemonic, for example it we don't allow any task to be queued as the superuser include: .RS .nf !f.uid=^0$ .fi .RE A whole \fIpam_session\fP is not usually required (see \fBop\fP). .P The administrator must add the 25 lines of \fBcrontab\fP configuration to root's (kicker's) crontab -- once, this avoids breaking many cron tables over the life of the host: nothing is really free. .RS .nf # kicker support if run as root 0 0 * * * root /usr/local/sbin/kicker top 00 0 1 * * * root /usr/local/sbin/kicker top 01 \&\fI...\fP 0 22 * * * root /usr/local/sbin/kicker top 22 0 23 * * * root /usr/local/sbin/kicker top 23 55 23 * * * root /usr/local/sbin/kicker eod .fi .RE .P If you use the \fBop\fP escalation you might use this version in \fBkicker\fP's crontab (if you put this in \fBroot\fP's crontab add the mortal's login name before the path to kicker): .RS .nf # kicker support in kicker's /etc/crontab 0 0 * * * /usr/local/sbin/kicker \-S batch top 00 0 1 * * * /usr/local/sbin/kicker \-S batch top 01 \&\fI...\fP 0 23 * * * /usr/local/sbin/kicker \-S batch top 23 55 23 * * * /usr/local/sbin/kicker \-S batch eod .fi .RE .P Setting up multiple batch queues with different rules is not really hard work, see \fBqueuedefs\fP (in /etc/cron.d or /usr/lib/cron) as you need them. I've used as many as 5 before: the default 'b' queue, one for compression tasks ('Z') one for log processing ('L'), one for backups ('R'), and one for application tasks ('T'). One a large modern host with more than 16 threads of execution tuning this brings a lot of benefits without a lot of recurring costs. .sp .P To run a command on every Friday, not every day, one must code a \fIguard\fP at the top of the script. The script will be started every day, but the guard exits the program if it is the wrong day to run, e.g.: .RS .nf case `date +%a` in Fri*) ;; *) exit 0 ;; esac # Time to take action... .fi .RE This same tactic could for months or for arithmetic on Julian days, e.g. (in \fBksh\fP): .RS .nf # run this only every other day, oddly enough [ 0 \-eq $(expr `date +%j` % 2) ] || exit 0 .fi .RE .P A lock file might be used to avoid conflicts with applications: .RS .nf # Wait for the spool to be idle... flock \-EX /var/spool/cats/.lockfile critical-code .fi .RE .P Use the \fBhaveip\fP guard to protect scripts that must run on the active member of a \fBCARP\fP, DNS round-robin or other multi-homed service: .RS .nf # If we are not the web-server today stop now. haveip www.example.com || exit 0 .fi .RE .P To chain tasks based on a different resource, one should end the script with a call to \fI\*(PN\fP to kick off the next step in the process. That way we keep the batch queue invariant working for us: .RS .nf # we are done, let the data compressor have a shot \*(PN manual/b.compress .fi .RE The limit here is that the next task will run as the same login, unless the \fB\-S\fP option is allowed to the requesting login. Another use of \fBop\fP here would be to allow an escalation to the \fBkicker\fP login to enqueue the compression task, I would use a group or netgroup rule to allow any such access. .P A task designed to start a long running process (viz. run from \fBboot\fP) \fBmay not block\fP the batch queue. Two safety features are suggested: first put them in a unique queue (say \fBs\fP, for start), second limit them to running the real process in background. For example the task \fIboot/s.unixstats\fP might start the PEG data collector for a host as: .RS .nf #!/bin/sh /usr/local/libexec/unixstats \-s 171 \fB&\fP .fi .RE Or use the \fBdaemon\fP(8l) program. .SH ENVIRONMENT .TP $\fBKICKER_Q_TIME\fP This environment variable contains the time (in seconds since the epoch) that the given task was enqueued. It is most useful as input to \fBdatecalc\fP or a \fBperl\fP program. .TP $\fBHOME\fP If set in the process, and the process is run by the superuser, it is replaced with the target login's home directory from the password database. .TP $\fBLOGNAME\fP and $\fBUSER\fP Each is replaced by the login name associated with the target login, if the variable exists in the current environment. .TP $\fBSHELL\fP Is left as is. This is a feature, as an account with \fIfalse\fP or the message shell is still useful as a target login. .SH BUGS While \fI\*(PN\fP takes care to set groups correctly, it doesn't honor all the login class attributes. This might also be a feature, you can use \fBop\fP's pam access to work around this. .p This program depends on the \fBbatch\fP system to manage the workload on the host. Some \fBbatch\fP systems are less configurable than others. .P All task files \fBmust\fP be executable (or they are silently ignored). This prevents files created inadvertently (core files) from accidentally being executed from the spool. .P The default batch queue for a task is "Z", which doesn't exist in the default batch configuration (on Solaris, for example). Either make that queue, or be sure to prefix task file names with "b." or "b-". This is a side-effect of being able to configure lots of related queues and flows from queue to queue -- which is complex but can be really useful. .P When installing this program be sure to allow all the logins that might be in \fI\*(PN\fP's queue to use \fBat\fP and \fBbatch\fP. One might have to touch a zero length \fI/var/at/at.deny\fP, \fI/usr/lib/cron/at.deny\fP, and/or \fI/usr/lib/cron/cron.deny\fP. .P As the inheritor of \fIcron\fP's current working directory, which might be root's home directory, \fI\*(PN\fP may instance a target login's shell in a directory which has modes that are too restrictive. Errors about an inability to find the current working directory are common, and fatal. The crontab line for kicker may have to change directory to the root filesystem to start mortal processes. It might be argued that a \fBchdir\fP(2) to the home directory of the target login should be arranged by \fI\*(PN\fP, however some target logins have an intentional directory of "/nonexistent" to prevent other issues, and that would lead to even more madness. .P \fIKicker\fP disposes of \fIstderr\fP from the \fIbatch\fP command, which is verbose for no good reason on most UNIX systems. Use \fB\-v\fP to see this output. .P To work around the effective uid bug from \fBop\fP-driven commands you have to either put an \fBsu\fP in the op process path or be sure to set for euid and uid to the same thing. Now days setting \fBuid\fP sets \fIeuid\fP to the same value. See \fBop\fP(1l). .SH FILES .TP \fB/var/at/at.deny\fP Touch this to allow customers to use the batch queues. .TP \fB/var/cron/deny\fP Touch this to allow customers to use cron .TP \fB/var/cron/allow\fP Add any mortal login to this file if they need to schedule their own \fI\*(PN\fP instance. .TP \fB/var/kicker/00\fP .. \fB/var/kicker/23\fP Hours of the day spools. How do you allow Customers to install task files in these? Maybe you need \fBinstallus\fP(1l). .TP \fB/var/kicker/boot\fP You'll need a boot script to start tasks from this spool, if you have mortal processes you want to start at boot under mortal control. .TP \fB/var/kicker/manual\fP Daisy chain tasks from one \fI\*(PN\fP task to another through a directory like this one. By allowing access to the \fBop\fP \fImnemonic\fP for some logins you can even allow the login to change, as well as the CPU priority (from \fBbatch\fP's queuedefs configuration). .SH AUTHOR Kevin Braunsdorf, NPC Guild .br kicker Not-spaM ksb.npcguild dot org .SH "SEE ALSO" .hlm 0 sh(1), batch(1), at(1), crontab(4), op(1l), xapply(1l), ls(1), perl(1), touch(1), su(1), environ(7), login.conf(5), haveip(1l), nice(1), daemon(8l), datecalc(1l)