| [Top] | [Contents] | [Index] | [ ? ] |
1. Installation Requirements 2. Installation Instructions 3. CVS 4. Y2K Considerations 5. Editing this Manual 6. General Support
| [ < ] | [ > ] | [ << ] | [ Up ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
The DBMS need not be on the same host as the web server, so long as it is network-accessible from the web server.
| [ < ] | [ > ] | [ << ] | [ Up ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
Here's an overview of the installation process. If you dislike reading, then you might be able to get by without reading the other sections of this chapter.
If the above conditions aren't satisfied except that the architecture is similar, then I expect that the Perl modules can still be copied over. On the other hand, it doesn't take much work to install the Perl modules, and you may wish to remove a source of uncertainty.
make, so you must also install that if not already done so.
PATH, install the
Perl `DBI' module, followed by the `DBD::Pg' and/or `DBD::Oracle'
modules.
sed scripts.
Each of the above is covered in more detail in following sections.
2.1 Installation of PostgreSQL or Oracle 2.2 Restarting the PostgreSQL Server 2.3 Installation of the DBI Perl Module 2.4 Customising for a New Host or DBMS 2.5 Initialising the Database
| [ < ] | [ > ] | [ << ] | [ Up ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
Even if you don't want to run the database from this machine (e.g. so that there's only one copy of the database to be maintained, or for security or performance concerns), you still need a basic client runtime environment (shared libraries), and building the DBMS-specific Perl modules (DBD::Pg or DBD::Oracle) requires a development environment (e.g. header files, and in Oracle's case ProC as well) (though it should be possible to build the Perl modules on a different machine).
This section deals only with installing PostgreSQL.
2.1.1 Installing GNU make2.1.2 Installing PostgreSQL
| [ < ] | [ > ] | [ << ] | [ Up ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
make
Compiling the PostgreSQL DBMS requires GNU make. You should
be able to just copy the binary that was installed on the www2 machine
in `/home/hydroda3/usr/bin/make'.
Otherwise: ftp to ftp://mirror.aarnet.edu.au/pub/gnu/make/ and download the latest version from that directory. Do something like:
i=$HOME/build; mkdir -p $i; cd $i gzip -cd /path/to/make-3.xx.tar.gz | tar xf - cd make-3.?? ./configure --prefix=$HOME/usr make make install cd .. rm -rf make-* |
| [ < ] | [ > ] | [ << ] | [ Up ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
First of all, get `postgresql-6.5.3.tar.gz' or more recent from ftp://mirror.aarnet.edu.au/pub/postgresql/.
cd ~/build
gzip -cd postgresql-6.5.3.tar.gz | tar xf -
cd postgresql-6.5.3/src
# Make sure that GNU |
Follow the instructions in the `INSTALL' file. On www2 we set the PostgreSQL root user to be `hydroda3' rather that `postgres' as is suggested in the PostgreSQL documentation; you probably want to do similarly (i.e. use some existing user-id) on the new machine (depending on what user-ids are available on this machine and who has access to them).
After you've run the regression tests, the server log will contain lots of errors. Kill the PostgreSQL server: do `ps -e | grep -i postmaster' and kill the pid you see; there should be only one there to kill. Remove the `pgserver.log' file from the home directory. Then start it up again (which the next section covers).
| [ < ] | [ > ] | [ << ] | [ Up ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
Whenever the computer is rebooted, the PostgreSQL server process will need to be restarted. (Use a command like `uptime' or `w' to find out if the computer has been rebooted lately; or do `ps -e | grep -i postmaster' to see whether the daemon is still/already running.)
Log in as the PostgreSQL root user (which on www2 is `hydroda3',
and will probably be the same on www1). cd to the home
directory.(1)
Use a pager to view the existing `pgserver.log' file that the
previous server process has accumulated. Usually it will be empty; any
lines that are there may indicate problems to be resolved.
Then restart the PostgreSQL server process by doing (assuming Bourne or Korn shell):
nohup postmaster -i > pgserver.log 2>&1 & |
| [ < ] | [ > ] | [ << ] | [ Up ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
DBI is a Perl module that tries to provide a common interface to any DBMS. (In the Streamgauging Catalogue software, the remainder of the differences between database backends are handled in the `postgresql.sed', `oracle.sed' etc. files.)
Download DBI-1.13.tar.gz from http://cpan.valueclick.com/modules/by-module/DBI/.
Make sure that a recent version of Perl (5.004_04 or later) is at the head of your path:
export PATH=/opt/perl5/bin:"$PATH" perl -v |
# Configure for installation into home directory. perl Makefile.PL prefix="$HOME/usr" make make test make install |
Once DBI and PostgreSQL are installed, DBD::Pg can be installed.
(Installation of DBD::Oracle is not covered here, as I've never
installed it. One thing to note is that it does require an Oracle
development environment to be available, including the "ProC"
preprocessor for C (command name proc).)
gzip -cd DBD-Pg-0.93.tar.gz | tar xf - cd DBD-Pg-0.93 export POSTGRES_LIB="$HOME/usr/lib/postgresql/lib" export POSTGRES_INCLUDE="$HOME/usr/lib/postgresql/include" perl Makefile.PL prefix="$HOME"/usr make make test make install |
| [ < ] | [ > ] | [ << ] | [ Up ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
The make process uses sed scripts to do the host-specific or DBMS-specific configuration.
The host-specific sed files have names of the form `nodename.sed', where nodename (throughout this section) means the result of `uname -n' (or rather `uname -n | sed 's/\..*//'', if that makes a difference; see the `all' target of the Makefile).
The DBMS-specific sed file has the name `dbsrc.sed', where
dbsrc is the setting of the @dbsrc@ variable in
`nodename.sed'; typically `postgresql.sed' or
`oracle.sed'.
These are just normal sed scripts run with `sed -f
filename'. Most of the lines are of the form `s|pattern
regexp'|replacement|g. (Do `man sed' for more detail.)
2.4.1 `nodename.sed' variables 2.4.2 `dbsrc.sed' Variables
| [ < ] | [ > ] | [ << ] | [ Up ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
The `nodename.sed' file is used to configure for various
host-specific settings. A good example file is `www2.sed'. (See
the rules target of the makefile, and the deploy target of
`rules.in', to see how the sed files get used.) See the last
paragraph of section `Customising for a New Host or DBMS' above for
brief description of the format (if it isn't obvious). The
`nodename.sed' file contains the following configuration
variables:
@PERL@
@BEGIN@
BEGIN subroutine called by Perl at the start of execution.
We use it to modify Perl's @INC variable (which contains a list
of directories in the order that Perl should search for modules). We
insert the place where the CGI scripts are being placed (see the
@CGIDIR@ variable below). If the Perl modules needed by this
script (see section 1. Installation Requirements) are not being
installed site-wide, then we also need to add the location of these Perl
modules.
@DBDIR@
@LOCKFILE@
@HTMLDIR@
@CGIDIR@
Currently used only by the @BEGIN@ and @DBDIR@
variables (see above), and the deploy target of `rules.in'.
@HCGIDIR@
@HTDOCDIR@
@DBHOST@
@DBUSER@
DBI->connect(...) call. See the
DBD::Pg or DBD::Oracle man page respectively (or run `perldoc
DBD::Pg' / `perldoc DBD::Oracle' if the man pages aren't
accessible). Once the Oracle database is up and running, we can
probably add a @SID@ variable to this list.
@dbsrc@
dbsrc_install target in `rules.in'. (The
dbsrc_install target is called every time you do
`make' or `make deploy'. For postgresql and
oracle, this target is empty.)
| [ < ] | [ > ] | [ << ] | [ Up ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
See section 2.4 Customising for a New Host or DBMS, for brief comments on this file's use & format.
The file contains the following configuration variables:
@CONNECTARGS@
DBI->connect(), viz. data
source, username, password. Do `perldoc DBD::Pg' or `perldoc
DBD::Oracle' (or whatever DBD driver this dbsrc.sed file is for) to see
what format the data source argument should take, and whether or not the
separate username and password arguments are necessary.
@TBLNAME@
@DISTINCT@
@UPPER@
@BETWEEN@
If you're using a new DBD driver, then: if the DBD driver supports a reasonable amount of SQL (which should be the case for any real DBMS as opposed to e.g. DBD::CSV or DBD::XBase), then copy these variables from `oracle.sed' (or `postgresql.sed', which has the same). If the DBD driver is more primitive, then look at `xbase.sed' instead.
| [ < ] | [ > ] | [ << ] | [ Up ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
If you're using PostgreSQL, then assuming PostgreSQL been installed (see section 2.1.2 Installing PostgreSQL), create a new database called `streamgauging' with:
createdb streamgauging |
(If you want to use a different database name, you can edit `postgresql.sed' and change the `@CONNECTARGS@' variable, specifically the `dbname=streamgauging;' part; see section 2.4.2 `dbsrc.sed' Variables.)
If you're using Oracle, then ask your DBA to create the empty stations table (see next paragraph) and to edit the `oracle.sed' file (see section 2.4.2 `dbsrc.sed' Variables).
The file `dclgen' contains SQL to create a table named `stations'
with the necessary fields. The table name `stations' is fine for the
private PostgreSQL database that we'll be using to start with, but if
you switch to using Oracle, then your DBA may ask you to fit in with
some corporate table naming standard. If you change the table name in
`dclgen', then also change the @TBLNAME@ variable setting
in your `dbsrc.sed' file (see section 2.4.2 `dbsrc.sed' Variables).
2.5.1 Populating (or Updating) the Database 2.5.2 Changing the Database Table Name
| [ < ] | [ > ] | [ << ] | [ Up ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
To populate the database from a DBF file, first convert to tab-separated value format, i.e. tab character as field separator, newline as record separator. The line endings ("newlines") can be either DOS/Windows format (CR LF) or Unix format (LF), but not Macintosh format (CR). (Macintosh format can be converted to Unix format with `tr \\r \\n < macfile > unixfile'.) Text fields (as with all fields) should appear verbatim between the tab characters, i.e. not wrapped in quotation marks.
Suppose this tab-separated-value file is called `stations.csv'. Do:
make csv2sql
# The `-d{' flag means to start with a statement to clear any}
# existing contents of the table.
./csv2sql -d < stations.csv > stations.sql
more stations.sql
# Check that it looks roughly correct.
|
Then feed that SQL into the database by doing:
make sql-feeder ./sql-feeder < stations.sql |
If there are no error messages, then skip the rest of this subsection.
If there are any constraint errors, then either there's a problem with the new data, or else the constraints need to be loosened. The "chunk" number referred to in the error message should be the line number of `stations.sql'. Have a look at that line (e.g. using `view' or `more') and check that they are all reasonable. If they are, then edit the `dclgen' file and change the appropriate `check' expression.
If it says `no such table' or `no such database' then it probably means
that you need to edit the DBMS sed file (`postgresql.sed' or
`oracle.sed') and change the @TBLNAME@ or
@CONNECTARGS@ variable setting (see section 2.4.2 `dbsrc.sed' Variables).
If you're using PostgreSQL, and the error message asks whether postmaster is accepting connections on port 5432, then (if you've already checked that postmaster is running with the `-i' flag; see section 2.2 Restarting the PostgreSQL Server) check that the `pg_hba.conf' file is allowing TCP/IP connections.
| [ < ] | [ > ] | [ << ] | [ Up ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
create table stations ( ... |
s|@TBLNAME@|stations|g |
| [ < ] | [ > ] | [ << ] | [ Up ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
CVS, "Concurrent Versions System", is a version control system; see http://www.cyclic.com/ for more information.
CVS keeps track of the different versions of the source code, and allows merging of changes made to the source code by different people.
At any one time, there should only be one CVS repository for the streamgauging catalogue source code; otherwise it becomes more difficult to merge changes.
At the time of writing, the repository is on www2. It can be accessed
from other hosts so long as www2 is accessible to those hosts by
rsh, ssh, or if the www2 sysadmin has added a CVS
line to the `/etc/inetd.conf' file.
See the `Remote repositories' section of the CVS documentation, for
information on how to do this.
(If you're just using rsh access, then just `export
CVSROOT=:ext:user@www2:/home/hydroda3/cvsroot', where
`user' is any user on www2 that is rsh-accessible to you and
which has access to the CVS repository; probably this means
`hydroda3'.)
3.1 Installing CVS Client Software 3.2 Using CVS 3.3 CVS Documentation 3.4 How to Move the CVS Repository to a Different Place
| [ < ] | [ > ] | [ << ] | [ Up ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
First of all, get the source code from
ftp://mirror.aarnet.edu.au/pub/gnu/cvs/cvs-1.10.tar.gz.
Like all GNU packages, it can be built much as follows:
gzip -cd cvs-1.10.tar.gz | tar xf -
cd cvs-1.10
./configure # You might want to give arguments to this,
# e.g. `--prefix=/some/where' if you want CVS to be
# installed in /some/where/{bin,man,...} instead of
# /usr/local/{bin,man,...}; try `./configure --help'
# for more info on what the options are.
make
make check
su -c "make install" # su isn't necessary if you already have
# write access to the appropriate directories.
cd ..
rm -rf cvs-1.10
|
That's it for installing the client software.
| [ < ] | [ > ] | [ << ] | [ Up ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
Once you've installed the CVS client software
(see section 3.1 Installing CVS Client Software), set your CVSROOT environment
variable. If the CVS repository is on the same host, then it's just the
name of the `cvsroot' directory; e.g. `export
CVSROOT=/home/hydroda3/cvsroot'. Otherwise, see section `Remote
repositories' in the CVS documentation.
Create a directory called, say, `~/wd' (for "working
directories"), and cd into it. Then use cvs to retrieve the
current version of the code from your CVS repository:
mkdir -p ~/wd cd ~/wd cvs checkout gauging |
The rest of this section deals with errors that may have occurred during that command.
Depending on what access method you're using (e.g. if your CVSROOT variable setting begins with `:pserver:'), it may have told you `use "cvs login" to log in first'. If that's the case, just type `cvs login', and enter the password of the user specified in the CVSROOT variable setting.
When you next want to update the code in `~/wd/gauging' to the latest version, do:
cd ~/wd/gauging cvs update |
(There's no need to set the CVSROOT environment variable because the information is stored in `~/wd/gauging/CVS/Root'. There's no need to redo `cvs login', because the password should still be available in `~/.cvspass' from the last time you did `cvs login'.)
If you make a change to one of the source files and wish to commit that back to the repository, then do:
cvs commit -m "my comment" my-file1 my-file2 ... |
See the next section for documentation on more advanced usage.
| [ < ] | [ > ] | [ << ] | [ Up ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
Basic documentation is available with `man cvs' of course. (If
that doesn't work, it's probably because you installed CVS into an
unusual place; try changing your MANPATH variable to include
`/some/where/man', replacing `/some/where' with whatever
prefix you told the configure script to use.)
More usable documentation is in Texinfo form. If you're familiar with the info browser, Try typing `info cvs' or accessing the cvs page in emacs (C-h i g (cvs) RET).
If you prefer a web interface over the info browser, then go to http://www.loria.fr/cgi-bin/molli/wilma.cgi/doc.847210383.html.
More information, including a link to a tutorial, is at
http://www.cyclic.com/cyclic-pages/books.html.
| [ < ] | [ > ] | [ << ] | [ Up ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
Reasons one might wish to do this are if www2 is too slow, or if it is firewalled off from some host from which you wish to be able to access the CVS repository, or if the directory it's in on www2 does not get backed up. (The latter problem is to some extent reduced by the fact that one typically has at least the current version of the source checked out onto both www1 and www2.)
If that doesn't apply then skip this section.
The main thing to note is that for each directory that the source has been checked out into, the location of the repository is stored in the `CVS' subdirectory, more specifically `CVS/Root'.
Update the contents of each `CVS/Root' file. E.g.:
find . -name Root -print # Read the above list to make sure it hasn't found # any files that you don't want to write over. for i in `find . -name Root -print`; do echo ':ext:hydroda3@www1:/home/hydroda3/cvsroot' > "$i" done # or whatever the new method, user, host, directory are, i.e. # whatever the value of the CVSROOT environment variable would be. |
[If you're familiar with CVS, then an alternative would be to go through each directory that has source checked out into (before moving the repository), doing `cvs update' in each. If there are no modifications in any of the directories (as signified by one of CVS's output lines beginning with `?' or `M' or `C'), then just `rm -rf' that directory, and check it out again after you've moved the repository.]
Once those working directories have been taken care of, then the repository can be moved with just:
cd /home/hydroda3 tar cf cvsroot.tar cvsroot rm -rf cvsroot |
and extracting that cvsroot.tar into its new location with
mv cvsroot.tar /new/location # (The above is probably done from Windows if the new location is # on a different host.) cd /new/location tar xf cvsroot.tar rm cvsroot.tar |
Once it's moved, check that everything is OK by doing `cvs update' in each of the working directories, and check that it doesn't say `can't connect' or `no such file or directory' or whatever. If it does give an error, then probably there's a typo in the `CVS/Root' file. Try creating a new directory, `export CVSROOT=whatever', and do `cvs checkout gauging'. If it works, then echo $CVSROOT over each of the `CVS/Root' files that don't work (or just get rid of those working directories and use the new one instead).
| [ < ] | [ > ] | [ << ] | [ Up ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
The short version of this section is that this software is believed by its authors to be Y2K-compliant (though there has been no formal Y2K process).
The information in the remainder of this section is provided in case BoM wishes to do more thorough checking.
4.1 External Dependencies
| [ < ] | [ > ] | [ << ] | [ Up ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
See section 1. Installation Requirements.
| [ < ] | [ > ] | [ << ] | [ Up ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
REGULATED field
DISCONT, OPENED, RECORDED fields
smallint). Never stored in 2-digit form.
$nowyear variable
my @tm = localtime; my $nowyear = 1900 + $tm[5]; |
The `$tm[5]' above refers to the tm_year field of libc's
struct tm, which contains the number of years since 1900
(see `perldoc -f localtime' or `man 3 ctime').
| [ < ] | [ > ] | [ << ] | [ Up ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
The OPENED, RECORDED and DISCONT fields have an SQL
check that they are greater than 1800 (or possibly equal to zero for the
latter two), but no upper limit.
Numbers entered in the search fields OPENED, RECORDED and
DISCONT must be either 4-digit numbers (1700-2199)(2) or be `0'. (See
numerical_pred_tail in `general.pl'.)
In `Sg_Detail.pm', the REGULATION field is considered to hold
a year if it's a number between 1800 and 2199. The SQL check on this
field tries to restrict it to the same range if it's numerical.
| [ < ] | [ > ] | [ << ] | [ Up ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
At the time of writing, the master form of this document is in Texinfo format, in `instmanual.texinfo' in the source code.
Texinfo is a markup language, i.e. one edits the file with a plain text editor (see below), and then uses a separate program to produce the HTML or Postscript output (which can be done by typing `make doc' if the appropriate transformation software is installed).
If you don't already use Texinfo for documentation purposes, then perhaps you switch to using the HTML as the master document, or even importing the HTML into MS Word. If you do this, then do `rm instmanual.{texinfo,ps} && cvs rm instmanual.{texinfo,ps} && cvs commit -m "The HTML file is now the master format." instmanual.{texinfo,ps}', so that no-one tries to edit or read the wrong file. You should also update this section of the manual, of course.
The remainder of this section discusses editing and using the Texinfo format.
You probably want to install at least Texinfo itself (which contains
documentation on the Texinfo language; and its makeinfo -html
command can be used as some sort of replacement for texi2html),
and the separate texi2html program.
Texinfo is available from ftp://mirror.aarnet.edu.au/gnu/texinfo/.
Texi2html is available from http://www.mathematik.uni-kl.de/~obachman/Texi2html/.
To produce postscript, you need to have TeX installed. (Try typing `type tex' to see if it's installed.) TeX is quite big, so you probably don't want to install it just to get slightly better printed output if you don't already have it installed.
The file can be edited emacs, vi, or even a Windows
text editor so long as it either understands Unix line endings (Notepad
doesn't, though pretty much everything else does) or you convert the
file to have DOS line endings. emacs and xemacs are
the best editors to use for the task, as they have special commands
helpful for editing Texinfo files (see section `Emacs Editing' of the
Texinfo documentation).
| [ < ] | [ > ] | [ << ] | [ Up ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
For questions about the database contents, talk to Ross James (r.james@bom.gov.au) or `Sri' Srikanthan (R.Srikanthan@bom.gov.au), who know most about the database, and are generally in charge of the project.
Rodney Davidson has been the main technical contact.
Outside of the Bureau of Meteorology:
Questions regarding the streamgauging catalogue software or CVS or PostgreSQL (but generally not Oracle) can be directed to Peter Moulder, pjm@bofh.asn.au, (03) 9882 0676. Peter is the principal author of the software and this manual.
Another person familiar with the streamgauging software and with CVS usage is Graham Bean, grbm@bigfoot.com.
| [Top] | [Contents] | [Index] | [ ? ] |
Any directory will do, but if other than your home directory then you should update this manual so that the next person knows where to find `pgserver.log'. See section 5. Editing this Manual.
The ranges given in this subsection are inclusive but not exhaustive: in many cases the comparison is a simple string-based comparison rather than comparing actual numerical values.
| [Top] | [Contents] | [Index] | [ ? ] |
make
| [Top] | [Contents] | [Index] | [ ? ] |
1. Installation Requirements
2. Installation Instructions
3. CVS
4. Y2K Considerations
5. Editing this Manual
6. General Support
| [Top] | [Contents] | [Index] | [ ? ] |
| Button | Name | Go to | From 1.2.3 go to |
|---|---|---|---|
| [ < ] | Back | previous section in reading order | 1.2.2 |
| [ > ] | Forward | next section in reading order | 1.2.4 |
| [ << ] | FastBack | previous or up-and-previous section | 1.1 |
| [ Up ] | Up | up section | 1.2 |
| [ >> ] | FastForward | next or up-and-next section | 1.3 |
| [Top] | Top | cover (top) of document | |
| [Contents] | Contents | table of contents | |
| [Index] | Index | concept index | |
| [ ? ] | About | this page |