INTRODUCTION Flash now comes packaged with its own test-suite, called FlashTest, which can be used to setup, compile, execute, and test a series of Flash simulations on a regular basis. What kinds of simulations are run? What input parameters are passed to Flash? What kinds of tests are carried out on the results? It is easy to control all of these variables in FlashTest, and easy to analyze the results of your test- suite simulations with FlashTestView, also included with the Flash release. FlashTest must be run under Unix/MacOSX and requires python2.3 or higher. CHAPTER I - FlashTest Navigate to the FlashTest root directory and follow the steps below to get FlashTest up and running. 1. Edit the "config" file Before you can run FlashTest, you have to enter some basic information into its config file, which lives in the top-level FlashTest directory. Open this file ("config") in your favorite text editor, and take a look. The comments at the top of the file contain information about the syntax of the "config" file, which variables can be set here, and how those variables are used by FlashTest. Take a minute to read through them. Variables are assigned values at the bottom of the "config" file according to the pattern: variable: value For now, you need only supply a value for the variable "pathToFlash". This should be an absolute path to the top-level directory of a working copy of the FLASH code. Ideally, it should be a copy which has already been used to successfully build and execute a Flash simulation. FlashTest sets up, compiles, runs, and tests Flash problems, so without FLASH, there can be no FlashTest. Go to the end of the comment section of the "config" file and uncomment the line for "pathToFlash". There is a suggested value included in "config", but you should replace it with whatever value is appropriate for your system. A value for "pathToFlash" is all the "config" file really needs, but to make the example below go more smoothly, uncomment the line after "pathToFlash". This is the line that begins "defaultParfile:" Once you've uncommented it, you're finished with this line. Specifically, you should not replace the keyword in angle-brackets "" with the real path to the FLASH code that you provided in the line above. TIP Keywords in angle-brackets are used throughout FlashTest as place-holders for real values which are filled in during the course of a FlashTest invocation. These real values can be determined in one of two ways: 1. The user defines the value in the "config" file or in "test.info" data 2. The value is filled in automatically by FlashTest In our definition of "defaultParfile", we have an example of the first way in which angle-bracket keywords are filled in, as we've defined a real value for "pathToFlash" on the line above. This feature can be very useful for defining keys whose values are specific to your particular system, and angle-bracket keywords can be used in the "config" file, the "test.info" file (see below), or any other file that you want to use in your own tests. END TIP There are other lines below "defaultParfile", some of which have been commented out, and others which have been left uncommented. The uncommented lines have keys with real values already filled in, to help you get FlashTest up and running as quickly as possible. Reading the comments at the top of the "config" file will tell you more about what these lines are for. When you're finished reading, save and close the "config" file. Don't exit your editor, yet, though! 2. Edit the "exeScript" file The code which controls the invocation of the Flash executable, as defined in lib/flashModule.py, looks for a file called "exeScript" for instructions on just how to perform the invocation. This will be a call to "mpirun" with different options depending on your platform. Open the file "exeScript" in your editor and check to see if your platform is represented. At the time of writing, there are suggested scripts in "exeScript" only for Linux and SGI (IRIX) machines, but please insert the necessary instructions to run an mpi application on your machine. Uncomment one of the suggested lines and/or make whatever other changes are necessary, and re-save. For example, if you are running FLASH on a Linux platform, the relevant part of your "exeScript" file might look like this: # Linux mpirun -np <numProcs> <pathToFlashExe> -par_file <parfile> TIP Note again the keywords in angle-brackets. Here we see examples of the second way in which these keywords can be given real values. These keywords are not hard-coded in "config" or anywhere else. Rather, they are automatically filled in, often with a succession of different values, by FlashTest iteslf during the course of a FlashTest invocation. For exmaple, if you use FlashTest to run a series of two FLASH simulations, one on a single processor and the other on two processors, will be replaced first by "1" when the first executable is invoked, and then by "2". will be replaced first with the absolute path to the first Flash executable, and later with the absolute path to the second. will be filled in with the name of the file that specifies runtime parameters for a single FLASH run. A list of all "magic" keywords filled in automatically by FlashTest is given at the end of this chapter. END TIP Save and close the "exeScript" file. Exit your editor. 3. Populate the "test.info" file The last ingredient we need to run FlashTest is a populated "test.info" file. This is just a text-file that uses a simple xml-like heirarchy to define how a given FLASH problem will be set up, compiled, and executed, as well as how the results of the execution will be tested. For now, for the sake of seeing FlashTest in action, invoke the python script "createInfoFile.py", which lives in the top-level flashTest directory, to set up a very simple "test.info" file: $ ./createInfoFile.py Now examine the contents of the file "test.info" that was created in the top- level FlashTest directory $ more test.info setupName: unitTest/Grid/UG setupOptions: -1d -auto +noio numProcs: 1 parfiles: In place of the words "NAME_OF_SITE", the "createInfoFile" script will have entered the name of your computer, determined by taking the first element of the fully-qualified hostname as returned by the unix "hostname" command. This is the name that will be used to identify "test.info" data intended for this installation of FlashTest and to identify output from this installation. If this name is unsatisfactory to you for any reason, perhaps because you are not connected to the internet or because you are connected through a wireless network (which can cause the "hostname" command to return a non-intuitive name) or because you are running multiple installations of FlashTest on one computer and want to distinguish their output, you can set the value manually by giving a value to the "site" keyword in the "config" file like this: site: NAME_OF_SITE Throughout these instructions, the word "site" will refer to this value. That is, to the first element of your computer's fully-qualified hostname or to the value assigned to the "site" keyword in the "config" file. If are satisfied with the "test.info", you are ready to try out FlashTest 4. Try it out From the top-level FlashTest directory, invoke the flashTest.py script with no options: $ ./flashTest.py Read through the brief usage notes to give yourself a better idea of how the flashTest.py script is invoked. Then, when you're ready, invoke FlashTest with the following options. It should take under a minute to run. Note that the slash at the end of "UnitTest/" is important: $ ./flashTest.py -v UnitTest/ While the program is running, keep reading to learn about what exactly that word "UnitTest/" - called a "node-path" - means to FlashTest: In general, the main FlashTest script is invoked like this: $ ./flashTest.py [options] [node-path-1] [node-path-2] ... [node-path-N] The first element of a node-path is called the "test-species" and indicates the broadest categorization of test known to FlashTest. The elements of the node-path that follow the test-species imply one of two courses of action: 1. A node-path that contains any slashes ("/") indicates a path through the nodes of the "test.info" file, terminating in the information necessary to run a given FLASH simulation. 2. A node-path that contains no slashes (and that therefore consists of a single word, which itself is the test-species) indicates that the test requires no "test.info" data. In practice, most of your node-paths will be paths to data in the "test.info" file. Our example is one of these. That's why it was important that the node- path "UnitTest/" end in a slash. The slash tells FlashTest that it needs data from the "test.info" file that we made in step 3 in order to run this test. It happens that in our case, the path was only one element long, but it was still a path to data in the "test.info" file. TIP Ordinarily, the node-path does not need to end in a slash. It is the presence or absence of slashes *anywhere* in the path that tells FlashTest whether it is looking for "test.info" data or not. Most node-paths contain more than one element and therefore need slashes to separate these elements. It was only in our special case of a node-path with length 1 that we needed to put a slash at the end. END TIP Occasionally, however, a test will be simple enough that it does not require any data from the "test.info" file. The built-in test called "Default" is an example of this kind of test. An invocation of a Default test (with verbose output) would look like this, with no slash following the test-species: $ ./flashTest.py -v Default Please note that although the outermost element of our "test-info" file is named for our site, the first element of the node-path is not the site, but the test-species. When FlashTest needs data from the "test.info" file, it knows to start looking in the element named for the site. All right, FlashTest is probably finished by now. If all went well, you saw a message "FlashTest complete. End of Logfile." at the end of the verbose output that you chose by including the "-v" option. Now let's examine the output that FlashTest generated. 5. Examine the output In section 2, we'll use FlashTestView to examine FlashTest output quickly and efficiently, but let's look at the output of this first invocation "by hand" before we get involved in the installation of a second tool. From the top- level FlashTest directory, do an "ls". You'll see that FlashTest has created a new directory called "output". If you navigate inside you'll see that this new directory contains another directory named for the site. "cd" into this site-directory and "ls" again. You should see yet another directory, this one named for today's date. This is called an "invocation directory", and contains all the output generated by a single invocation of the flashTest.py script. "cd" inside this one as well. Now things get a little more interesting. Doing an "ls" reveals two files and a directory. The file "errors" will be empty if nothing went wrong, and "flash_test.log" will contain the same information that you saw printed in real time by choosing verbose output when you invoked flashTest.py. That leaves the directory "UnitTest". This is called a "build directory". Each invocation directory will contain as many build directories as there were node-paths listed on the flashTest.py command-line (or in the file named after the "-f" option to flashTest.py). "cd" into the build directory and "ls" again. Now you'll see a lot of files, including the FLASH executable that FlashTest compiled, and a single directory. This bottom-most directory is called a "run-directory", and there is one for every parfile that was run against this executable. We only used one parfile, called "flash.par", and so there is only one run-directory, called "flash", inside the build-directory. Try this: $ ls flash Run-directories hold all output generated by the running of the executable against a single parfile. The "ls" shows that this one contains a bunch of text files, including several written by FlashTest and anything written by the Flash executable. The most interesting ones are "flash_output", which captures all output generated by the Flash code itself during runtime, and "test_output", which captures anything written by the user's testing code. In the case of a unit-test like the one we just ran, success or failure is decided by the presence of the string "all results conformed with expected values" in a certain file or files written by Flash. The testing phase of a unit-test checks for this string and writes the results of its check in "test_output". Digging through directories and files is the most basic method of analyzing the results of simulations made with FlashTest, but there is a better way. CHAPTER II - FlashTestView FlashTestView is a browser-based tool for viewing the results of FlashTest invocations. It requires Python 2.3 or higher and a web-server configured to execute python code. Follow these steps to get FlashTestView running on your computer. 1. Test the python code The first step is to place the directory "flashTestView" in a location where your web-server can see it. Once you have done this, open a browser and navigate it to the "home.py" script inside "flashTestView". For example, if your server administers "mydomain.com", the URL might be: http:///www.mydomain.com/flashTestView/home.py A couple of different things might go wrong at this point. If your web-server does not know to execute the python code, but prints it instead, you'll see something like: #!/usr/bin/env python import sys, os import cgi, pickle, tempfile import purgeTmp sys.path.insert(0, "lib") import invocations, littleParser ... If you see this, you need to move the directory to a location for executable scripts (like "cgi-bin") or otherwise configure your web-server to execute files ending in ".py". You may have to talk to your sysadmin about this. Make sure that he or she also gives the web-server write permissions on the top- level FlashTestView directory. If you don't, you'll see an error message that starts with: "The web-server does not have write permissions..." Once you've got these issues worked out, point the browser at the URL again. If all goes well, you will see the following error message (in this case, the error is a good thing!): File "config" either does not exist or does not contain any values. Create a "config" file if necessary and add the following text: pathToOutdir: [path/to/outdir] where [path/to/outdir] is an absolute path to a top-level FlashTest output directory. Then reload this page. When you successfully see the above error message, move on to step 2. Don't close your browser, though. 2. Write a "config" file Open a terminal and navigate to the flashTestView directory. Open the file "config" in your editor of choice. Read through the very brief comments about the two variables that can be set here, "pathToOutdir" and "pathToInfo". The syntax for the FlashTestView "config" file is exactly the same as for the FlashTest "config". Variables are assigned values according to the pattern: variable: value The first variable to which you should assign a value is 'pathToOutdir'. This should point to the "output" directory which was generated by FlashTest when we worked through the UnitTest example in Chapter 1. Enter the following into "config", substituting the absolute path to the top-level directory of your FlashTest installation for "[path-to-FlashTest]": pathToOutdir: [path-to-FlashTest]/output Save the file and exit your editor. Then reload the page that gave you the error message in step 1. If all went well, you should see something like this: SCREEN-CAPTURE COMING SOON, ALONG WITH MORE DOCUMENTATION