Source code
Revision control
Copy as Markdown
Other Tools
<!doctype html public "-//w3c//dtd html 4.0 transitional//en">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta name="GENERATOR" content="Mozilla/4.7 [en] (X11; U; SunOS 5.8 sun4u) [Netscape]">
</head>
<body text="#000000" bgcolor="#FFFFFF" link="#0000EE" vlink="#551A8B" alink="#FF0000">
<h3>
<b><font face="Times New Roman,Times">Author Sonja Mirtitsch</font></b></h3>
<h3>
<b><font face="Times New Roman,Times">Last updated: 4/4/2001</font></b></h3>
<h1>
<b><font face="Times New Roman,Times">NSS 3.2.QA Wrapper</font></b></h1>
<p><br>The QA wrapper tests the nightly builds of NSS. The actual
tests are being run are called from the QA script all.sh. I will add documentation
for the actual QA soon. The main purpose of the wrapper is: find out which
build (NSS version, date, Build Platform) to test on which machine (OS,
OS version) and construct a summary report, which is then mailed to the
nss developers (aka mailing list nss-qa-report@netscape.com). Please see
also the <a href="#advertisement">feature</a> section.
<p><a href="#nssqa">nssqa</a> - the script that calls the actual
qa script all.sh
<br><a href="#qa_stat">qa_stat</a> - sends out status reports
<br><a href="#qaclean">qaclean</a> - if everything else fails
<p>Sample <a href="/u/sonmi/doc/publish/glob_result.html">global result</a>,
<a href="/u/sonmi/doc/publish/results.html">individual result </a>and <a href="/u/sonmi/doc/publish/output.log">log
files</a>
<p>The QA wrapper consistst mainly of scripts, most located in security/nss/tests
and subdirectories, but run from /u/sonmi/bin
<p>nssqa and qa_stat, the main scripts both include a common header (<a href="../header">header</a>)
and a common environment (<a href="../set_environment">set_environment</a>).
<br>Also used is <a href="../mksymlinks">mksymlinks</a> and <a href="../path_uniq">path_uniq</a>
and <a href="#qaclean">qaclean</a>.
<p>The scripts that are used on a daily basis are located in /u/sonmi/bin
and checked into security/nss/tests
<p>Parameters and Options are the same for most scripts.
<p><a NAME="Parameters"></a><b><u><font size=+1>Parameters</font></u></b>
<br> nssversion (supported: 30b, 31, tip, default tip)
<br> builddate (default - today, format mmdd)
<p><a NAME="Options"></a><b><u><font size=+1>Options</font></u></b>
<br> -y answer all questions with y - use at your own
risk... ignores warnings
<br> -s silent (only usefull with -y)
<br> -h, -? -help you guessed right - displays the usage
<br> -d debug
<br> -f <filename> - write the (error)output to filename
<br> -fcron writes resultfile in the same location as
would the -cron
<br> -m <mailinglist> - send filename to mailinglist
(csl) only useful
<br> with -f on nssqa
<br> -l <mozroot> run on a local build - does not
work at this time
<br> -cron equivalient to -y -s -d -f $RESULTDIR/$HOST.<scriptname>
<br>
<p>Please be aware that some iPlanet specific environments and features
are being used.
<p>-d Debug option might be removed from cron in a few weeks - or maybe
not
<br>-l QA on local build is not fully implemented yet - will not be implemented,
all.sh can be called directly instead
<p>Please do not use on Windows 95 and 98, ME platforms yet.
<p>use -d if script behaves strange or exits unexpectedly
<p><b><font size=+1>How to use the QA-wrapper</font></b>
<br>To test a build, first run nssqa on the required QA platforms (some
buildplatforms require QA to be run on additional platforms - for example
Solaris 2.6 has to be tested on 2.8 32 and 64bit) If QA has been run on
multiple or all required platforms it makes sense to run qa_stat on the
output of nssqa as well.
<br>Before used on a new system (even if the same platform has been tested
before) please use completely interactive, to see what the variables are
being initialized to, and read the warnings. Same is true if being run
from a different user account than svbld.
<p>In any case, if you are using it, please let me know the results.
<p><a NAME="nssqa"></a><b><u><font size=+1>nssqa:</font></u></b>
<p>the script that calls the actual qa script all.sh
<p>nssqa <a href="#Parameters">parameters</a> and <a href="#Options">options</a>
<p><a href="../nssqa">view the script</a>
<p><b><u><font size=+1>Pseudocode Description of nssqa</font></u></b>
<br>not quite up to date
<p> header:init (global)
<br> set flags and variables
to default values
<br> signal trap (for interupts
and kills)
<br> set HOST and DOMSUF variables
if running from cron
<br> parse parameters and options
<br> determine os and set up
the environment (espec. PATH)
<br> set the directories to run
in (influenced by parameters and -l option)<br>
set the directories for backward
compatibility testing
<br> set and initialize the tmp
/ debugging / output files
<p> nssqa:init (local)
<br> locking: if nssqa is already
running on this systems (yes-exit,
<br>
no-lockfile)
<br> set HOST and DOMSUF variables
if running interavtively
<br> set flag to kill remaining
selfserv processes during cleanup
<br> if QA platform different
from build platform create neccessary
<br>
symbolic links
<br> wait for the build to finish
(max of 5h)
<p> main:
<br> repeated per test (optimized,
debug, 32, 64 bit)
<br>
set flags for this run of all.sh (optimized, debug, 32, 64 bit)
<br>
set the DIST directory (where the binaries reside)
<br>
kill running selfservers (sorry - just don't use the svbld
<br>
account if you need to do your own testing... I will fix
<br>
selfserv as soon as I can - but it hangs too often and
<br>
disturbs all following QA)
<br>
run all.sh
<p> header:exit (global)
<br> remove temporary files
<p> kill remaining selfservers
<br> send email to the list
<br>
<p> errorhandling
<br> Option / Parameter errors:
Exit with usage information
<p> Severe errors: Exit wit errormessage
<br>
example: directory in which all.sh resides does not exist
<br>
can't create files or directories
<br>
build not done after 5 hours
<br>
is already running
<p> Other errors: User is prompted
with the "errormessage - continue (y/n)?"
<br>
example: local DIST dir does not exist (continues with next all.sh)
<br>
outputdirectory does not exist (user can specify other)
<p> Signals 2, 3, 15 are treated
as severe errors
<br>
<br>
<br>
<p><img SRC="clean.gif" height=129 width=92 align=LEFT><a NAME="qaclean"></a><b><u><font size=+2>qaclean:</font></u></b>/u/sonmi/bin/qaclean
<br>
<p>Use qaclean as user "svbld" to get the propper permissions. It is supposed
to clean up after a "hanging" QA and will also brutally kill, interupt
and disturb any other nss related test or performance meassurement on the
named machine. NT and 2000 might require an additional reboot, since the
ps is not so good about telling us the actual programmname - so we can't
kill them... Please note that this is a brute force script, it should not
be used on a regular basis, file a bug whenever you have to use it, since
hanging QA is nothing that should occur frequently
<p> <a href="../qaclean">view the script</a>
<p>What it does:
<ol>
<li>
see if there is a lockfile (/tmp/nssqa.$$ or $TMP/nssqa.$$)</li>
<br>if yes:
<ol>kill the process of the lockfile <font color="#666666">(future expansion
and if possible it's children )</font>
<br>rm the lockfile</ol>
<li>
kill selfservers</li>
<li>
kill whatever other qa related processes might be hanging</li>
<li>
clean up tmp files</li>
</ol>
<b>QAClean Parameters:</b>
<br> machinename.
<br> for example
<br> qaclean kentuckyderby
<br> started on any machine, will clean up on kentuckyderby
<p><a NAME="qa_stat"></a><b><u><font size=+2>qa_stat</font></u></b>
<p>qa_stat is the script that is being started from the svbld cron on kentuckyderby
every morning at 10:00 and runs some (very primitive) analysis on the qa
results.
<br>I'd like to rewrite the whole thing in perl, and in a few weeks I might
just do this...
<p> <a href="../qa_stat">view the script</a>
<p>qa_stat <a href="#Parameters">parameters</a> and <a href="#Options">options</a>
<p><a NAME="advertisement"></a><b><u><font size=+1>Why we need the QA wrapper</font></u></b>
<p>We need the new QA wrapper, because we have to test on so many platforms,
that running the tests and evaluating the results for the nightly builds
took about an average workday.
<p><b><font size=+1>New Features:</font></b>
<ul>
<li>
runs from <b>cron</b> / rsh or <b>interactive</b> if desired</li>
<li>
generates <b>summary</b> (no need to look through 60-90 directories)</li>
<li>
sends <b>email</b> about results</li>
<li>
automatically <b>recognizes common errors</b> and problems and conflicts
and corrects them</li>
<br>(or attempts to correct them :-)
<li>
automatically determines <b>which build </b>to test (waits if build in
progress, exits if no build)</li>
<li>
runs on <b>all required platforms</b> (Windows 98 and before not functional
yet)</li>
<li>
Windows version runs on <b>free Cygnus</b> as well as on MKS</li>
<li>
debug mode, normal mode and silent mode</li>
<li>
<b>locking</b> mechanism so it won't run twice</li>
<li>
<b>cleanup</b> after being killed and most errors (no remaining selfservers,
tmpfiles, lock files)</li>
</ul>
The 1st script is started via cron between 5:00 and 8:00 am on different
systems, and starts QA on the nightly build. At 10:00 the next script is
started, and sends a QA summary to the nss developers.
<p><b>Cygnus Advantages</b>:
<ul>
<li>
<b>free</b></li>
<li>
better handling of <b>processes</b> (background, processIDs, Signals)</li>
<li>
Unix / Linux <b>compatible</b> sh / bash</li>
</ul>
<b>Disadvantages</b>
<ul>
<li>
MKS functionality needs to be preserved (makes <b>8 Windows platforms</b>
instead of 4 for the QA suites - makes 32 testruns on Windows alone)</li>
<br>In certain functionality's <b>slow</b>
<br><b></b> </ul>
<b>Porting the windows QA to Uwin as well is also being considered</b>
</body>
</html>