/***************************************************************************** 
  Copyright(C) 1996, 1997 Network Flight Recorder, Inc.
  All rights reserved.

  Use and distribution of this software and its source code
  are governed by the terms and conditions of the
  Network Flight Recorder Software License ("LICENSE.TXT") 

  File:   README
  Author: Kent Landfield <kent@nfr.net>
 
 *****************************************************************************/ 
                                  Archiving Data

Terminolgy:

      Package - A set of backends

      Backend - A set of information that specifies an information processor. 
                A backend is usually a combination of a recorder process, a 
                query process, and options specific to that individual backend. 

      Recorder - The process that receives data directly from the engine and 
                must then process it further depending on the purpose of the 
                recorder. 

To explain this is tricky but maybe an example will help.

The 'package' "web" has the following backends.

     web/detect
     web/watchservers

The 'backend' "web/detect" has the following components:
      detect.cfg - configuration file
      detect.desc - User Interface strings
      detect.nfr - Filter N-Code

The 'backend' "web/watchservers" has the following components:
      watchservers.cfg - configuration file
      watchservers.desc - User Interface strings
      watchservers.nfr - Filter N-Code

The recorder is specific to a backend and in the case of the backend
web/detect, the recorder is detect. In the case of the backend
web/watchservers, the recorder is watchservers.

---------------------------------
Backend/Query Application needs:
---------------------------------

	1. Write new timestamped data to a file.
	2. Generate index entry for the data written
	3. Read raw data from an archive
	4. Read data from an archive within specified time boundaries
	5. Retrieve data based on a unique-id 

---------------
General Notes:
---------------

1. Stdio is be used for performance reasons.

2. Archive files are records-based.

3.  Format of archived files

   At the beginning of each archived file is a two-part file header record.  
   The first part of this record holds standard archive related information 
   and the second part is backend specific information.  The remaining
   content in the files are the individual records generated by the backend.

    ----------------------
    |  struct header     |
    ----------------------
    |  recorder blob     |
    ----------------------
    |    Data records    |
    |        ....        |
    |    Data records    |
    ----------------------

    The header structure is intended for use within the archiving library.

     struct header {
         char   backend[MAXNAMELEN];      /* backend name             */
         char   cfversion[CFVERSION_LEN]; /* Version of CF used       */
         char   ctype[DTYPE_LEN];         /* Data content type        */
         time_t starttime;                /* Start time delimiter     */
         time_t endtime;                  /* End time delimeter       */
         size_t archive_size;             /* File size                */
         long   datastart;                /* Data Start Seek position */ 
         long   maxreclen;                /* Size of largest record   */ 
     };

    The recorder blob is optional and contains information about the
    state of the recorder at the time of archiving. It is intended to 
    give the backend recorders a place to store things in the files.  
    The archive library treats this area as a simple character string. 
    Please keep that in mind when writing to it.  There is no specified
    limit on the size of the recorder specific information supplied during 
    an aopenwrite() call.

4. Data Record Format 

   There are three parts to a data record, the record header, an optional 
   unique identifier and the actual data.

    -----------------------------------
    | struct rec | ID |    Data       |
    -----------------------------------
  
    The record header contains a timestamp associated with the record
    that was supplied by the backend. It also contains other housekeeping 
    items specific to the archive library.

    ID is a unique-id that is variable to 1024 bytes. 1024 bytes is the
    largest an ID can be.  This is optional and up to the recorder to use
    as they see fit.
  
    The actual Data can be variable in size.


------------------------------
Archive Library Usage Overview
------------------------------

The archive subsystem stores/retrieves time stamped records for a backend.

WRITING DATA:

When you open an archive for writing, you specify these things in
a call to archive_openwrite:

    - a backend/recorder name [1]
        this is what recorder you are; it is used to identify your
        data, kind of like a file name.  This name is constructed
        as a single string such as "backend/recorder". See Notes
        below.

    - a content-type
        this string is additional file naming information.  It allows
        a single recorder to have more than one stream of data.  (Some
        recorders need this; if yours does not, use "")

    - a starttime
        time of the first record to be recorded.

    - a recorder specific blob
        This block of data is recorder specific data to be stored
        in the archive.  It is anticipated that the recorder might
        place metadata here for it's own reasons.  You can read this
        blob back when you open the archive for reading. [2]

Once the archive is open for writing, you place data in it with
calls to archive_write.  Parameters are:

    - the archive to write to
    - a buffer to write
    - the timestamp for this recorder
    - a "unique id"

The data in the buffer is treated as an atomic record.  It is written
to the archive, using the timestamp and unique id as indices.  When
you read the archive later the record, timestamp, and unique id will
be returned.

Note that the record is stored in the order it was received.  Records 
with identical timestamps will be read in the order they are written.  
This interface DOES NOT support random access.  Timestamps must be 
written in a non-decreasing sequence.  

The function archive_flush flushes buffers to disk so that a consistent
file is visible to other processes.  Call archive_close to close the
archive when you are done with it.

NOTE:

    [1] The "backend" parameter allows the archive_open* routines to lookup
        specific archive and rollover information from the appropriate config
        files.  A "Backend" name includes both the back end and the specific
        recorder name, so you have both available.  For example, 
        "webwatcher/urlhist" would be a package named webwatcher and a
        recorder named urlhist in that package.  This allows the
        libraries to get the file storage locations and rollover parameters
        that are specific to that instance of the recorder.

    [2] A recorder might close an archive and re-open it with new metadata.  
        This will have to rollover to a new file so they can see the new 
        metadata at the right point in the stream.


READING DATA:

Before reading data, you must call archive_openread with these 
parameters:

    - recorder name
    - content type
    - starttime 
    - endtime

The starttime/endtime limits what records will be returned to those records
that satisfy ( time >= starttime &&  time < endtime ).  You can specify
0 as a starttime/endtime time to indicate the end of the database in that
direction.

archive_read fetches a record from the archive.  You must give these 
parameters:

    - the open archive to read from
    - a buffer to store the record in
    - a maximum size of bytes to read into the buffer
    - a time to stop reading records. Can be zero read all records.
    - a unique id desired - if specified, only records that match this
      unique id will be returned

Any of the buffers for the record, timestamp, and unique id may be 
passed as NULL if you do not desire those particular data values.

This function returns records in the order of 
        1) increasing timestamp
        2) the order they were written to the archive

At any time you can fetch the recorder specific header information from
the open Archive structure.  Note that the recorder specific header might
change at any point while you read through the archive, depending what
the recorder did while it was writing.  This would be due to new archives
with different recorder headers being opened inside the library.  If this
information is important to the backend, the boolean recorder_info_read 
variable in the recorder header is available to the backend to indicate 
it has not read the current recorder-specific header.  It is the backend's 
responsibility to set the recorder_info_read flag to 1 when it accesses that 
information in the Archive. The flag will be set to 0 when a new archive 
file opens occur inside the archive_* routines.

Make sure to call archive_close() when you are done reading or writing the
archive.  This assures that all index files are updated and any memory that
was allocated within the libraries is freed.  It also assures that the
space management system is notified of any new files your backend/recorder
may have created.

-------------------------------
Archive Library Function Calls:
-------------------------------

**********************************************************************
** archive_openwrite - Open an archive file for writing.
**
** Parameters:
**       backend     - The unique name of the specific backend whose config
**                     file is to be read to see what directory it should
**                     open files in.
**
**     content_type  - Backend supplied option suffix for filename
**
**     starttime     - used to build filename
**
**    recorder_blob  - filled in at archive_openwrite() time
**
**
** Returns:            Pointer to struct Archive
**
** Notes:              Records do not have uniform sizes.
**********************************************************************

Archive *archive_openwrite(char *backend, char *content_type, 
                              time_t starttime, char *recorder_blob);

*************************************************************************
** archive_write
**
** Parameters:
**      archive - pointer to type Archive returned from archive_openwrite()
**
**     nbytes   - number of data bytes to write
**
**      buf     - pointer to buffer to read the data into
**
**     tstamp   - timestamp
**
**    unique-id - Unique identifier, recorder specific.
**
**
** Returns:       Number of bytes written, -1 on error
**
** Notes:         Writes to the archive and if it's past the rolltime
**                configured in the backend's configuration, close it, 
**                open a new archive file and write the data. (roll over)
*************************************************************************

int archive_write(Archive *rkive, size_t nbytes, char *buf, 
                   time_t tstamp, char *unique_id);

**************************************************************
** archive_openread - open a previously archived file for reading
**
** Parameters:
**       backend -   The unique name of the specific backend whose config
**                   file is to be read to see what directory it should open
**                   files in.
**
**    content_type - suffix used to distinquish between different datasets
**                   created from a single recorder.
**
**       starttime - used to delineate file contents started
**                   if starttime is 0 then read from beginning of
**                   backend's archive. If starttime is a time, start
**                   reading from that time.
**
**       endtime   - used to delienate when file contents ended
**                   if endtime is 0 then read to the end of the backend's
**                   archive. If endtime is a time, stop reading when
**                   records are encountered outside that time.
**
** Returns:          Pointer to struct Archive on successful open  
**                   Null on error
**************************************************************

Archive *archive_openread(char *backend, char *content_type,
                              time_t starttime, time_t endtime);

*****************************************************************************
** archive_read - read next record in the time period
**
** Parameters:
**      archive   - pointer to type Archive returned from archive_openread()
**
**      nbytes    - maximum number of bytes to read
**
**      buf       - pointer to buffer to read the data into
**
**      endtime   - time to stop reading records. Can be zero read all records.
**
**     unique_id  - tcp.hash or some indexable identifier that is unique to
**                  the content type. Only records matching that are returned.
**
** Returns:       A Pointer to a rec structure containing the the data bytes
**                read, the record's timestamp and unique-id is returned.
**                The rec structure is stored in as a static structure that
**                will be overwritten on subsequent calls. If there is a
**                need within the application to keep the information returned
**                in the rec structure, the application must do so itself.
**                Returns NULL when an error is encountered and archive_errno
**                is set.
**
** Notes:         Read the next record from the archive and if it hits EOF
**                it'll open the next file as long as endtime has not been
**                encountered.  Deals with the archive on a record basis
**                and will return at most nbytes to the calling application.
*****************************************************************************

struct rec *archive_read(Archive *rkive, size_t nbytes, char *buf,
                                     time_t endtime, char *unique_id);

***************************************************************
** archive_close - close the archive
**
** Abstract:   Close a specific archive.
**             This frees memory allocated during archive_open{read/write}().
**             This is the utility that backend writers need to call.
**
** Parameters:
**      archive - pointer to type Archive returned from archive_open*()
**
** Returns:       1 - success, 0 failure and  archive_errno set
**
***************************************************************

int archive_close(Archive *rkive);

*******************************************************************
** archive_flush - flush the stream to the disk
**
** Parameters:
**      archive - pointer to type Archive returned from archive_open*()
**
** Returns:       0 - flushed, EOF - not flushed
********************************************************************

int archive_flush(Archive *rkive);

***********************************************************************
** archive_rewind - rewind archive pointer to time specified
**
** Parameters:
**          rkive   - Archive struct returned by archive_openread()
**
**        starttime - time to rewind to.
**
** Returns:       1 - rewound, 0 - error
**
** Notes:         Reset archive access to the first record in the archive if
**                start_time is 0, else it rewinds back to the first record
**                matching starttime, and if supplied, content_type.
***********************************************************************

int archive_rewind(Archive *rkive, time_t starttime);

**************************************************************
** archive_header - Return the archive header of the file.
**                  rkive->fp file pointer is positioned on
**                  the first record ready for reading.
**
** Parameters:
**     path     - The path to the file
**
** Returns:       Pointer to Archive structure filled with the
**                files meta header information
**************************************************************

Archive *archive_header(char *path);

--------------------
Archive Index Files:
--------------------

A single index file is generated for each backend/recorder.  These are simple
index files with minumual information in them. It is not up to the archiving 
library to build special purpose index files. To do this this requires smarts 
that the recorder and associated query facilities need to agree on based on 
the function of the backend and the content of the data.

------------------------------
Where to store archived files:
------------------------------

Archived files are stored where you specify them to be.  The actual filename 
the data is stored in is currently setup as HH:MM:SS.  In each recorder
configuration file the 'archive_path' variable is required to be set to
tell the archive_openwrite() routine where to create the archive. 

Substitution characters can be specified in the 'archive_path' variable 
allowing you to specify the directory to the archive.  The supported 
substitution charaters are:

      %p - name of backend the file was archived with
      %b - name of recorder the file was archived with
      %d - two digit day of month (1-28/30/31)
      %m - two digit month of year (1-12)
      %y - four digit year (1990,..2001)
      %D - current date archived as %y/%m/%d

For example, the backend webwatcher/urlhist using the following archive_path
configuration line

   archive_path=data/%p/%b/%D

would archive data on Setember 30, 1997 in the directory

    data/webwatcher/urlhist/1997/09/30/
   
--------------
File Rollover:
--------------

File Rollover is built in to the archive_* routines to assure that a
single file being written by a backend/recorder does get too big or 
the data in the file get too old.  Configuration options are available
on a per-recorder basis that allows you to specify when it is time to go 
on to the next file.  File rollover can be done on a time basis (as in 
every x minute segments) or it can be done on a file size basis (as in 
file no bigger than x meg).


The following variables are used by the archive system to allow the 
archive_* library functions to roll over to new files based on size 
or time or to prevent roll over completely.

rollover_size=DO
rollover_size_val=600
rollover_time=DO
rollover_time_val=1m


rollover_time:     Do I roll over by time ? 
                      DO - rollover by time and ROLLOVER_TIME_VAL must be set.
                      DONT - dont rollover by time. 

rollover_time_val: Indicates that time should be used. The time is calculated
                   in seconds since file creation. 

                     's' - absolute seconds (nothing less than 60 seconds)
                     'm' - minutes
                     'h' - hours
                     'd' - days

rollover_size:     Do I roll over by file size ? 
                      DO - rollover by size - 'rollover_size_val' MUST be set.
                      DONT - dont rollover by size. 

rollover_size_val: Indicates the maximum size that the file should be before 
                   rollover.  This is a boundary for new writes. Records are 
                   not split in the middle so the size of a file can exceed 
                   this size by at most one record (-1 byte).  This can be 

                      'b' bytes
                      'k' kilobytes
                      'm' for megabytes, 
                      'g' gigabytes

If both 'rollover_time' and 'rollover_size' are set to DONT then no rollover 
is done. It is up to the backend to reuse this file's diskspace. This is 
useful for persistent statistics files that are completely overwritten when 
new statistics are generated.
          
-----------------------
Space Management Notes:
-----------------------

When a new file is closed, the archive_close() routine sends a message 
to spacemand.  Spacemand then checks to see if that file pushed things 
over the limit specified for that backend/recorder.  In this manner, 
spacemand can do a better job of monitoring space conditions during 
times of high traffic.
 
Independent of the back end, the space manager can choose to expire
data to satisfy it's own constraints.  It might delete old data
because a recorder is over it's disk quota or it might delete old
data because it is too old.

The back end cannot assume anything about which data will be 
deleted or when it will be deleted.  In particular, after spaceman
deletes something, any arbitrary record might be the first one
in the archive - you can't assume it will split in any particular 
place.

The only presumption is that older data will go first.
