/***************************************************************************** 
  Copyright(C) 1996, 1997 Network Flight Recorder, Inc.
  All rights reserved.

  Use and distribution of this software and its source code
  are governed by the terms and conditions of the
  Network Flight Recorder Software License ("LICENSE.TXT") 

  File:   README
  Author: Kent Landfield <kent@nfr.net>
 
 *****************************************************************************/ 

SPACE MANAGEMENT:

It is important that the system be able to operate sanely when the system 
is at or near capacity and an event occurs that forces space management 
facilities to do the right things.  The space manager can choose to expire 
data to satisfy it's own constraints.  It might delete old data because a 
recorder is over it's disk quota or it might delete old data because it is 
too old.

The space management system consists of two different parts. First is the
archive_* library routines.  If backend authors wish to take advantage of
the space management capabilities supplied they must use the archive_*
routines to store their data.

The second part of the space management system is the space manager itself.
The space manager, spacemand, is a persistent process that on startup, reads
the space management configuration file building a table of archived database
areas to manage.  It scan the databases looking to see if any files need to 
be removed.  When a new file is created and closed, the archive_close() 
routine sends a message to spacemand.  Spacemand then checks to see if that 
file pushed  things over the limit.  In this manner, spacemand can do a 
better job of monitoring space conditions during times of high traffic.
 
------------------------
Changing Configuration:
------------------------

When you wish to add, delete or modify an existing entry in the spacemand
configuration you need to simply edit the file with your favorite editor. 
If there is a configuration change, send the signal SIGHUP to spacemand to 
force it re-read of the configuration files. This can be done from a command 
line by executing spacemand with the '-r' flag.

    # spacemand -r

This will signal spacemand that it needs to re-read the configuration file
and use the newly configured information from that point forward.
   
------------------------------
Starting and Stopping spaceman
------------------------------

Spacemand should be started up during the standard NFR startup.  If you
are starting spacemand outside of normal NFR procedures there are a couple
scripts supplied to assist, startspaceman and stopspaceman.

------------------------------
Spacemand command line options:
------------------------------

Syntax:

      spacemand [-cdrv] [ [-f configfile] [-t seconds]

Options:

   -c   Check syntax of the config file.  Without specifying a configuration
        file with the -f option, the default spacemand configuration file is
        checked.  If the -f option is used, the specified configuration file
        is checked. 

   -d 
        Turns on debugging.  When debugging is turned on all output that 
        was previously sent to alertd is displayed on standard error instead.
        Also additional information about spacemand execution is displayed.

   -f configfile  
        Path to spaceman configuration file. The default spacemand 
        configuration file is $NFRHOME/etc/spaceman.cf.

   -r 
        Tell the running spacemand to reread the configuration file.  If there
        is no spacemand running the command will print a warning.

   -t seconds 
        Specifies the amount of time, in seconds, that spacemand waits 
        before doing a complete expiration check.  The current default is
        900 seconds (15 minutes). 

   -v   Turn on verbose output. This is only applicable when used in
        conjunction with the -c option.

The '-r' option cannot combined with other options. If tried, only the '-r' 
processing is done.

------------------------------
Space Management Configuration
------------------------------

There are two different types of configuration file entries for spacemand,
a backend/recorder entry and a directory path entry.  The backend/recorder
entries are used to tell spacemand to read the index files generated by the
NFR archive_* library routines. The index files are used to determine which
files to expire and which to leave on disk.  The directory path entries list
a specific directory that you would like to see checked for diskspace and age
limits.  The directory-location listed is the actual path to the directory.
Files in the directory are then examined to determine which files to expire
and which to keep.

Record Syntax:
---------------

   TYPE:backend/recorder:QUOTA:AGE:PRIORITY:ACTION:COMMAND:LOCATION:
                 or
   TYPE:directory-location:QUOTA:AGE:PRIORITY:ACTION:COMMAND:LOCATION:

The following variables are used by the space management system to specify
boundarys used to expire old data from the disk.  In all cases, expiration 
is done on a file basis and not a record basis at present.  Certain fields 
are optional.  TYPE must be specified. Either a QUOTA or an AGE specification 
must exist. Both can exist in which case both are checked.  Data will be 
expired based on which ever threshold is reached first.  PRIORITY and ACTION 
are mandatory. COMMAND and LOCATION are optional depending on the expiration 
ACTION requested.  

`#` characters at the beginning of a line (such as this one) are considered
comment lines and are ignored.  Blanklines are also ignored.

Long lines can be continued on the next line provide the continuation 
character '\' appears at the end of the line and whitespace is used at 
the beginning of the next line such as:

R:webwatcher/urlhist:1m::7:Process+Purge:\
     /bin/mail -s 'URL History for %M %y" %f::

In all cases there must be the complete 8 set of fields even if one is
empty. 

Field Descriptions:
-------------------

TYPE:             The type of the configuration entry,
                       'R' for a backend/recorder
                       'D' for an actual directory to monitor

backend/recorder:   The name of the backend and recorder combination for
      or            which this entry applies. Both the backend and the
directory-location: recorder must be specified and the entry applies only 
                    to the data archived from that recorder.  

QUOTA:            The actual disk space limit placed on the backend and/or
                  recorder for the entire archived database.  The following
                  characters have special meanings.

                      'k' kilobytes
                      'm' for megabytes, 
                      'g' gigabytes

AGE:              How long to keep archived data on disk before expiring it

                      'm' minutes
                      'h' hours
                      'd' days
                      'w' weeks

                  Examples: 
                     30m - Keep only the last thirty minutes worth of data
                     3d  - Expire all data older than 3 days.
                     1w  - Expire week old data

PRIORITY:         A priority level, ranked from 0-10 that is used to 
                  determine which datasets are the most important to keep 
                  the longest.  When available space becomes limited and 
                  may begin affecting operations, the space management 
                  system will begin forced expiring of data earlier than 
                  is set in the backend/recorder entries files to assure the 
                  system has the needed space to continue operations.  When 
                  this condition occurs PRIORITY is checked and expires data 
                  with the higher priorities first.  When an acceptable level
                  of available disk space is reached, the early expiration 
                  processing is terminated.  All actions taken are logged.

                     Priority 10  # Highest priority and is removed first.
                      ...
                     Priority 0   # Lowest priority and is removed last.

ACTION:           This describes the type of action that the space management
                  system should take when data is to be expired.  

                      None - No expiration is done and this file is exempt 
                             from expiration.
                     Purge - Remove the file from disk thus freeing its 
                             diskspace.
                      Save - Save the file to some external media prior to 
                             removal.  Save implies purging of data after 
                             external storage.
                   Process - Send the data to an command external to the 
                             space management system to process the data 
                             with.  The syntax of the command is described 
                             in COMMAND below.  Process can be combined
                             with the Purge or Save directive using a '+' 
                             character.  In this case command processing of 
                             data will occur before the save/purge operations.
                             If there is no save/purge directive it is up to 
                             the external command to remove the file.  If the
                             command fails an alert is generated.  

             Process+Purge - Sends the data to the indicated process and 
                             then the data is removed when the process
                             completes. (See "Process" description above)

              Process+Save - Sends the data to the indicated process and 
                             then saves the data to the specified location.
                             When that succeeds, the data is removed. (See
                             "Process" description above)

COMMAND:          This is the command line to be used by the space management
                  to process the data via an external command or set of 
                  commands.  The expire commands lines can be given a 
                  substitution character as listed below.

                      %b - name of backend the file was archived with
                      %d - two digit day of month (1-28/30/31)
                      %f - name of file being expired 
                      %F - full path to the file being expired 
                      %m - two digit month of year (1-12)
                      %r - name of recorder the file was archived with
                      %y - four digit year (1990,..2001)
                      %D - Day of the week (Sunday, Monday, ..)
                      %M - Month of year (January, February, ..)

                  You can setup the command as either a 'filter' or as
                  a standalone command line to be executed.  If you specify
                  a '|' (pipe) symbol as the first character of a command
                  then the file is sent to the command on standard input.
                  If you do not specify the '|' symbol then the command 
                  you have specified is simply executed.

                  Examples: 
                     Following command line is expanded to fill in any
                     substitution characters and then the command is
                     executed.

                             COMMAND="/bin/gzip -9 %f"

                     Following command line is expanded to fill in any
                     substitution characters and then the command is
                     executed as a pipe.  The file to be expired is
                     written to the standard input of the command.
                 
                             COMMAND="| /bin/gzip -9 > /cdrom/%p/%m%d%y"

LOCATION:         This field is used if the data is to be copied to another
                  disk location. This can be an NFS mounted location. It is 
                  up to the administrator to assure that appropriate space is 
                  available and mounted at LOCATION. LOCATION fields
                  can use substitution characters as listed below.

                      %b - name of backend the file was archived with
                      %d - two digit day of month (1-28/30/31)
                      %f - name of file being expired 
                      %F - full path to the file being expired 
                      %m - two digit month of year (1-12)
                      %r - name of recorder the file was archived with
                      %y - four digit year (1990,..2001)
                      %D - Day of the week (Sunday, Monday, ..)
                      %M - Month of year (January, February, ..)

--------------------------
SAMPLE Configuration file:
--------------------------
#---------------------------
# Backend/Recorder Samples:
#---------------------------
# The data in these examples would be created by recorders using the
# NFR archive library.  There is an associated index file that is read
# as input to the expiration process.
#
# R:backend/recorder:QUOTA:AGE:PRIORITY:ACTION:COMMAND:LOCATION:
#---------------------------------------------------------------
#
# detect all new webservers
#
R:webwatcher/detect:400m::7:Process+Purge:\
	| /bin/gzip -c -9 > /tmp/data/%b/%m%d%y.%f.gz::

#
# Capture URLs accessed history
#
R:webwatcher/urlhist:250m::7:Process+Purge:\
	/home/kent/spaceman/spacemand/tarfile /tmp/data/%b/%m%d%y.tar %F::

#
# Running total of NNTP traffic 
#
R:news/newsstats:10m:1d:1:Purge:::

#
# Log what activity went through the firewall 
#
R:firewall/passthroughs:1g:1w:5:Save::/Bigdisks/1/%b/%y/%m/%d/%f:

#----------------------------
# Directory Location Samples:
#----------------------------
#
# The data in these examples would be created outside of the NFR system
# and would not have used the NFR archive library.  There is no associated
# index.  This method reads the directory and decides what files are due
# to expire based on filetimes and sizes.
#
# D:directory-location:QUOTA:AGE:PRIORITY:ACTION:COMMAND:LOCATION:
#-------------------------------------------------------------------
#

D:/var/log/syslogs/:10M:4w:8:Process+Purge:| logreport | /usr/local/bin/elm -s 'Syslog Reporting for %y/%m/%d' admin-reports::


-------------
Index Files:
-------------

Index files are not archived currently with the data.  Only the data
files themselves are.  When it becomes necessary to regenerate the index
files you can do so by using the 'rdheader' program.  If you have a
directory named saved-data-dir that contains only data files, you can
regenerate an index-file by typing something like the following.

   find saved-data-dir -type f -print | xargs rdheader -i > newindex-file

------------------------------------------
Taring all files to a single day tar file:
------------------------------------------

See/use 'tarfile' to append all archived data to a single dayfile.

A Cron job can compress the day's tarfile at midnight.

-----------------------------------------
General overview of Spacemand processing.
-----------------------------------------

Startup:
	- check to see if this is the only spacemand process running.
          if not, die
          if so, create file with pid in it.
 
	- create input fifo

        - read configuration file and setup memory structures.

        - Run through list looking for expired items

	- Setup signal handlers - timer alarm to reread config files
	
Main:
	- In a loop, read the fifo waiting for input
          if input is received, check that single archive and verify
          if the new file has pushed things over the specified threshold. 

        - if alarm goes off, run through archived list looking for files 
          to expire.

Stop:
	1. close open files
	2. Remove input FIFO
	3. free memory allocated
	3. Log termination
	4. Remove pid file
	5. Exit process
