|My tutorial for helping you set up |
acceptability rating experiments
using Microsoft Excel 2008 (Mac) and WebExp
A standard is the book by Cowart (1997): Experimental Syntax
Selected papers I found helpful:
J. Myers 2009a: Syntactic judgment experiments
J. Myers 2009b: The design and analysis of small-scale syntactic judgment experiments
- I am assuming that you have a list of of experimental sentences (stimuli)
- In a document, have all your item sets listed
- Each item set should include all your conditions in the same order
- It would be good to have each subject see each condition at least 3-4 times; that is, you will need to write 3 to 4 times as many item sets as you have conditions; in the example case below with four conditions, we would have 12 or 16 item sets.
Example item set:
a) Which politician supported the bill in the caucus?
b) Which politician did not support the bill in the caucus?
c) Which bill did the politician support in the caucus?
d) Which bill did the politician not support in the caucus?
- I am assuming that you have a list of filler sentences (fillers)
- Usually, I use 1.5 to 2 times (see also Cowart 1997) as many filler sentences as there are stimuli
to conceal my manipulations
- My filler sentences are similar to my experimental sentences in some sense.
For instance, if my stimuli are interrogatives, my fillers will also be interrogatives.
- Filler sentences can be used to balance out imbalances in the stimuli:
o Grammatical vs. ungrammatical sentences (e.g. 50% grammatical, 50% ungrammatical)
o Negative vs. positive sentences
o Types of sentences
- I like to create my unacceptable filler sentences with blatant grammatical violations such as subject-verb agreement mistakes, aux-verb mismatches, word-order rearrangements as to distract from my experimental manipulations
- If you are unsure how to create medium to bad filler sentences, you could consult the literature and look for sentences that were rated with e.g. '??'. You can also consult papers with rating studies to get ideas.
|Distributing your stimuli into lists (Latin-Square design)|
|Create your experimental lists|
- Open as many Sheets in your workbook as you have lists
- Rename the different Sheets: List1, List2, List3 etc. (this will make your life easier later)
- From the first Sheet that contains all your lists (rename to All_Lists), copy each list individually (all sentences that are marked with 1 build List1, all sentences that are marked with 2, put into List2 etc.) with its ID numbers into the corresponding sheet into the SECOND column (column B)
- Open an additional sheet in your workbook to copy in your filler sentences and provide them with individual IDs that provides you with the possibility to identify each individual filler by means of its ID.
- Copy your filler list with the IDs onto the bottom of each of your experimental lists. Make sure, all your IDs (stimuli and filler IDs) are in the same column and all your sentences are in the same column.
Ordering your lists|
- Now we need to get your stimuli and filler sentences into an order that helps conceal your manipulation(s):
You might want the filler sentences and the experimental stimuli intermixed.
- I like to avoid having multiple experimental stimuli directly following one another; I prefer having one or more fillers between two experimental stimuli
- It might be a good idea to let subjects get used to the task: Start each list with filler sentences, good and bad ones, so subjects get a feel for the task while rating fillers.
- Use the Excel function RANDOMIZE:
In the first column of each list sheet, enter in the first cell left beside the first sentence =RAND and hit enter.
This should give you a random number.
- Click on that same cell once to highlight it. Then go with your cursor to the right bottom corner of that cell. A cross, filled white appears. Wait until a completely black cross appears. Then you are ready to pull it all the way down until you are in the cell right beside your last sentence. Every sentence is now assigned a different random number.
- Mark/Highlight all columns: random number, ID and sentence. Sort by the random number column. Do this for all your lists.
- Now you have to go through every list by hand and see where you have to make manual changes such that there are no two experimental sentences in a row. Remember to pay attention to the beginning of the lists: It's good to not start out with a stimulus but with a filler sentence.
- In order to avoid order effects on acceptability ratings, I usually present each list in at least two different orders: Forward and backwards
- For each list that you brought into a particular order both manually and with help of Excel, preserve that order by adding a column that numbers each row 1,2,3,4,5,6,7 etc..
- You can get your list in its reversed order by sorting it decending (click on Data in Menu, select Sort; choose decending)
Format your lists so you can us WebExp|
Now you can consult the WebExp Manual to find out how to set up your experiment on-line.
designed by Simone Gieselman