This tutorial is intended to walk you through the steps of setting
up a magnitude estimation experiment using WebExp on the UCSD Linguistics
Department servers. Some of this information is general for WebExp users,
and some specific to the UCSD set-up.
This tutorial will assume you:
- have read (or at least have downloaded) the WebExp
user manual
- have installed (and know how to use) SFTP software and an SSH client
(more info)
- have access to a text editor (NOT Microsoft Word. Notepad is sufficient.
I recommend SciTE.)
- have a basic understanding of HTML
and XML
- have an account on quiz.ucsd.edu (contact computerhelp if you need
one.)
Setting up your experiment files
To create an experiment, you will need 3 types of files:
- an experiment template file, which sets up the
structure for the experiment itself (defining training stages, experiment
stages, and any other data collection).
- resource files, which feed your stimulus lists
into the experiment template. (Separating form and content in this
way allows you to reuse your experiment template file.)
- an html file, which embeds the webexp
java applet (the bit of
code that serves up your experiment).
Experiment Template File
See the Implementing an Experiment section of the WebExp
user manual. The tags used in the experiment template files are
not always informative, and it can take a while to get oriented in the
file. If you're good at that sort of thing, check out the demos
included on the WebExp download. If you're not, your best bet is to
adapt a file that already does what you want it to do. This sample
file describes the structure for a magnitude estimation experiment.
Feel free to use this file as your starting point.
This example experiment template file is divided into stages as follows:
- Stage 1 collects some basic participant information
like age and gender and whether the participant has signed a consent
form.
- Stage 2 gives instructions for practicing magnitude
estimation using line length.
- Stage 3 lets the participants practice estimating
line length.
- Stage 4 gives instructions for doing magnitude
estimation with sentences.
- Stage 5 gives some practice with sentences.
- Around line 533 there is a tag for "repetitions".
Here you need to specify how many practice sentences you will
be using. I have used 3.
- Check the modulus sentence in this stage (around
line 562). You will probably want to use the same modulus in practice
as in the actual experiment. Your modulus should be a middle of
the road sentence. Not perfect, but not incomprehensible. Keep
in mind that this will be your baseline, so you will probably
want your sentence to be uncontroversially "grammatical".
- Check the modulus rating (around line 579).
It was suggested to me that 100 is a good rating, as it encourages
people to use a broader scale. Left to set their own modulus,
some participants may use a range from 2 to 12, as opposed to
20 to 120, leaving less room for fine grained estimates. You should
determine what makes a good modulus rating for your experiment
and use it in the practice sections and in the experiment.
- Notice that around line 599, you reference a variable called
"prac", which names the resource file
containing your practice sentences.
- Stage 6 introduces the experiment.
- Stage 7 provides the format for the experiment
itself
- Check the number of repetitions (around line
676) for your experiment. This should be the number of sentences
each participant sees. I have used 56.
- Check the modulus sentence (around line 705)
to be sure it matches your practice session.
- Check the modulus rating (around line 722).
- Notice that around line 743, you reference a variable called "stim",
which names the resource file containing your experiment
sentences.
- Stage 8 is set up for collecting data after the
experiment. I have included questions to assess native speaker status.
(I like to administer these after the experiment-- otherwise some
participants assume this is a test of differences between native and
non-native speakers and they shift into hyper-prescriptive mode.)
I have also included a debriefing question, just to check whether
the experimental manipulations are too obvious.
Resource files
Be sure to read the Randomization of Resources section of
the WebExp
user manual. There are several options available to you for ordering
the stimuli you feed into the experiment template. Because I often interleave
stimuli from different experiments, along with fillers, I like to have
control over the exact order of presentation. I generally create 4 or
more stimulus "lists" or "blocks" and each participant
sees a different list (or block). In order to manage this through WebExp,
I use the "between" subjects randomization setting so that
WebExp serves up a different block to each subject. In order for this
to work though, two things need to happen:
- First, you need the number of repetitions designated
in the experiment template file to match the actual number of stimuli
in the stimulus blocks of the resource file. (See lines 533 and 676
of the template file.) That way, Webexp will cycle through an entire
block of sentences for each participant.
- Second, the template file needs to call the correct resource
file with the correct parameter setting. (See lines 17 and
18 of the template file.)
- Down on lines 599 and 743 of the template
file, you call the variables which refer to your practice
sentence resource file and your experimental stimuli resource
file.
- These should match the variable names in lines 17 and 18.
- The tags containing those variable names have an "attribute"
which determines the randomization method. That attribute should
be set to "between" for a between-subjects randomization
scheme. This way, webexp will cycle through the blocks, showing
one per participant, as opposed to presenting all the blocks to
each participant.
- Next you will need to create resource files with names that
match the file names in lines 17-18.
- An example practice sentence file is here.
- An example experimental stimulus file is here.
- Within these files, you will need to specify blocks.
- And then within each block, you will need to list a "resource"
(stimulus), including its ID and the sentence itself.
- In the example files I have included, stimuli have different
IDs like "107b" or "F15".
- If you run the experiment using these demo files, you can observe
the randomization behavior directly.
- If you have especially long stimuli, you will need to insert a line
break in the stimulus in your resource file. A line break is indicate
by "\n". Take a look at line 8 of the experiment
stimulus file for an example.
HTML file
Next you'll need to set up an html file with the WebExp Applet. Be
sure to read the Running the Client Applet section of the WebExp
user manual. An example html file is here.
(This experiment has been disabled, so the applet will not load. Choose
"file | view source" after the click to see the relevant code.)
You need to check the following settings in this file:
- codebase="http://quiz.ucsd.edu/"
- archive="webexp2.jar"
- <PARAM name="port"
value="2xxxx">
This port value is for your development server, and will change when
you move to the production server.
(See below.) Everyone has their own port values (that's how we avoid
stopping and starting the server on each other), but your development
port should start with a 2, and your production port with a 1.
You should also set up an html file with an introduction to the experiment,
and maybe some instructions. See the Writing supporting web pages
section of the WebExp
user manual. An example supporting page is here.
When you click the button at the bottom of the page, it loads the html
page containing the applet.
Uploading files and starting the server
Once you have created your files, you need to upload them to the quiz.ucsd.edu
server.
Testing your experiment and moving to the production server
Once you have uploaded all of your files, you need to fire
up your webexp server. Note than any time you make a change
to your experiment files, you will need to restart your server. Ezra
has very graciously written a series of scripts and prompts which make
this much easier for you. They work by calling the webexp script and
then feeding it two arguments.
The first argument tells the script what you want to do (e.g. start
the server). The second argument tells the script which server you are
interested in (either development (testing) or production (the real
thing)). The list below is just for your information. Instructions begin
again after this list.
- 'status' shows the prod and/or dev server you're
running - this spits out a list of servers you are currently running.
- 'stop' {dev|prod} stops the relevant server -
you will lose any data from participants that are in progress
- 'watch' {dev|prod} starts the relevant server
in interactive mode - the script will send back relevant info to you
for troubleshooting. to get out of interactive mode, you need to type
"w". (this will also stop your server.)
- 'start' {dev|prod} starts the relevant server
in background mode - that is, without all the interactive information.
this script will dump info to a file called "nohup.out"
in the relevant server directory (e.g. /local/home/student/kertz/webexp_prod/).
you can consult it for troubleshooting.
- 'restart' {dev|prod} stop the relevant server
and starts it again in background mode
Once you have loaded all of your files into the proper directories
(as above), you need to start your development server.
- ssh to quiz.ucsd.edu
- at the prompt, type webexp
start dev
- if you are successful, you will see a display like
this:
# User : Type : Port : PID
kertz : dev : 2xxxx : 00000
- if you don't see this display, you have some kind of error
- you can go to the nohup.out file and try to sort it out. review
the instructions above carefully to see if you can fix the error.
Here's an example of an error reported in the nohup.out file (with
my comments):
INFO Java 1.5.0_10 from Sun
Microsystems Inc.
INFO Finished reading file data/experimenters.xml with 1 objects
the above line shows that
the experimenters file loaded correctly, with one experimenter (me)
listed
INFO Loading experiments configuration file for kertz
INFO Finished reading file data/kertz/experiments.xml with 3 objects
the above line shows that
the experiments file loaded correctly,
with three experiments listed
File data/kertz/elme01/elme01.xml cannot be found
here's the problem. either my experiment template file was
missing, or it had syntax errors.
Exception in thread "main" java.lang.NullPointerException
- In the example above, I had a problem with my template file. So
I went to fix it, uploaded the revised copy to the server, then restarted
the server by typing webexp
restart dev in my ssh window.
- Once your experiment loads correctly, you should be able to load
your instructions page in a web browser, then click on the link to
the page containing the applet. The applet should load, and you can
test out your experiment.
- You should run through as many practice sessions as you have blocks
(plus more to be sure). You want to look out for typos, font
encoding errors, and formatting errors. Once you make a change
to correct an error, you need to restart your server and test it again.
You should consult your results files (see below) that participant
responses are being recorded properly.
- Once you have tested your experiment and you are ready to go live,
you need to copy your files to your production server.
- In the production data directory (e.g. /local/home/student/kertz/webexp_prod/data/)
you should see a folder with your username. You need to upload
a copy of your experimenters.xml file here.
- Then, inside your personal data directory (e.g. /local/home/student/kertz/webexp_prod/data/kertz/)
you will need to upload a copy of experiments.xml.
- In that same directory (e.g. /local/home/student/kertz/webexp_prod/data/kertz/)
you need to create a new directory with a name
that matches the name of your experiment, e.g. MEdemo.
- In that directory, you should upload you experiment
template file and your resource files.
- Next, navigate back to your public_html directory e.g. (/local/home/student/kertz/public_html/)
- You don't need to change the instructions html file, but you
do need to edit the html file containing the java applet.
You need to change the port value to match the production server:
<PARAM name="port" value="1xxxx">
- Next you need to start your production server. Follow the instructions
above to start/troubleshoot, but replace the dev
argument with prod
in all cases.
- When you run your experiment, webexp will automatically assign your
participants a code. The first participant will be 00000. I recommend
running one test on your production server. Then when you collect
your results, discard participant 00000 before analyzing your data.
Analyzing your data
- Within your experiment directory (e.g. /local/home/student/kertz/webexp_dev/data/kertz/MEdemo),
webexp will create three subdirectories: logs, results, subjects.
- Your participant responses are recorded in individual files within
the results directory.
- You will need to comb through those files to pull out your data.
I do not recommend doing this by hand. I have written a python script
that works with files that have been formatted according to the specifications
above. If you'd like to try to use it with your data, let me know,
and I'll pass it along. (Please note though that the script comes
with no guarantees.)
- Once you have your data neatly formatted, you may need to transform
your results. To do this, first divide by your modulus so that you
get a ratio score for each response. Then take the common log of
the ratio. One easy way to do this is to throw your data in an excel
file, then use excel
functions to convert all of your data at once.
- Now you can use the transformed results as your dependent variable
for statistical analysis.
Have fun and good luck! |