My tutorial for helping you set up
acceptability rating experiments
using Microsoft Excel 2008 (Mac) and WebExp
Recommended literature:
A standard is the book by Cowart (1997): Experimental Syntax

Selected papers I found helpful:
J. Myers 2009a: Syntactic judgment experiments
J. Myers 2009b: The design and analysis of small-scale syntactic judgment experiments
Experimental Sentences:
  • I am assuming that you have a list of of experimental sentences (stimuli)
  • In a document, have all your item sets listed
  • Each item set should include all your conditions in the same order
  • It would be good to have each subject see each condition at least 3-4 times; that is, you will need to write 3 to 4 times as many item sets as you have conditions; in the example case below with four conditions, we would have 12 or 16 item sets.

  • Example item set:
    a) Which politician supported the bill in the caucus?
    b) Which politician did not support the bill in the caucus?
    c) Which bill did the politician support in the caucus?
    d) Which bill did the politician not support in the caucus?

    Filler Sentences:
  • I am assuming that you have a list of filler sentences (fillers)
  • Usually, I use 1.5 to 2 times (see also Cowart 1997) as many filler sentences as there are stimuli
       to conceal my manipulations
  • My filler sentences are similar to my experimental sentences in some sense.
       For instance, if my stimuli are interrogatives, my fillers will also be interrogatives.
  • Filler sentences can be used to balance out imbalances in the stimuli:
  •       o   Grammatical vs. ungrammatical sentences (e.g. 50% grammatical, 50% ungrammatical)
          o   Negative vs. positive sentences
          o   Types of sentences
  • I like to create my unacceptable filler sentences with blatant grammatical violations such as subject-verb agreement mistakes, aux-verb mismatches, word-order rearrangements as to distract from my experimental manipulations
  • If you are unsure how to create medium to bad filler sentences, you could consult the literature and look for sentences that were rated with e.g. '??'. You can also consult papers with rating studies to get ideas.
Distributing your stimuli into lists (Latin-Square design)
  • Open a new Excel Workbook
  • Copy all your item sets into the third column of your Excel sheet (part of your Excel workbook)
       (Important: each item set has to contain the different condition in the same order)
  • Assign IDs to your stimuli. I like to have my first item set (see example above) labeled like this: 01a, 01b, 01c, 01d. The second item set: 02a, 02b, 02c, 02d etc.
  • Once you are done with the IDs for your stimuli, you can start sorting your stimuli into lists. You will have as many lists as you have conditions (four in the example above).
  • In the first column of your sheet, start numbering your item sets into lists. The idea is that each list contains exactly one sentence from each item set and you want each list to contain the same number of instances of each condition. You start out with the first item set: 1,2,3,4; then for the second item set: 2,3,4,1; the third: 3,4,1,2; fourth: 4,1,2,3; fifth: 1,2,3,4 ..

  • Example:

    List assignment           ID
    1                                 01a
    2                                 01b
    3                                 01c
    4                                 01d
    2                                 02a
    3                                 02b
    4                                 02c
    1                                 02d
    3                                 03a

  • Once you are done with the numbering schema above, you can sort your stimuli by list.
  • In Excel, mark all three columns you have created.
  • Click the Data menu and select Sort
  • Sort by the List assignment column
  • Now you have your stimuli distributed over lists
Create your experimental lists
  • Open as many Sheets in your workbook as you have lists
  • Rename the different Sheets: List1, List2, List3 etc. (this will make your life easier later)
  • From the first Sheet that contains all your lists (rename to All_Lists), copy each list individually (all sentences that are marked with 1 build List1, all sentences that are marked with 2, put into List2 etc.) with its ID numbers into the corresponding sheet into the SECOND column (column B)
  • Open an additional sheet in your workbook to copy in your filler sentences and provide them with individual IDs that provides you with the possibility to identify each individual filler by means of its ID.
  • Copy your filler list with the IDs onto the bottom of each of your experimental lists. Make sure, all your IDs (stimuli and filler IDs) are in the same column and all your sentences are in the same column.
Ordering your lists
  • Now we need to get your stimuli and filler sentences into an order that helps conceal your manipulation(s):
       You might want the filler sentences and the experimental stimuli intermixed.
  • I like to avoid having multiple experimental stimuli directly following one another; I prefer having one or more fillers    between two experimental stimuli
  • It might be a good idea to let subjects get used to the task: Start each list with filler sentences, good and bad ones, so subjects get a feel for the task while rating fillers.
  • Use the Excel function RANDOMIZE:
          In the first column of each list sheet, enter in the first cell left beside the first sentence =RAND and hit enter.
         This should give you a random number.
  • Click on that same cell once to highlight it. Then go with your cursor to the right bottom corner of that cell. A cross, filled white appears. Wait until a completely black cross appears. Then you are ready to pull it all the way down until you are in the cell right beside your last sentence. Every sentence is now assigned a different random number.
  • Mark/Highlight all columns: random number, ID and sentence. Sort by the random number column. Do this for all your lists.
  • Now you have to go through every list by hand and see where you have to make manual changes such that there are no two experimental sentences in a row. Remember to pay attention to the beginning of the lists: It's good to not start out with a stimulus but with a filler sentence.
  • In order to avoid order effects on acceptability ratings, I usually present each list in at least two different orders: Forward and backwards
  • For each list that you brought into a particular order both manually and with help of Excel, preserve that order by adding a column that numbers each row 1,2,3,4,5,6,7 etc..
  • You can get your list in its reversed order by sorting it decending (click on Data in Menu, select Sort; choose decending)
Format your lists so you can us WebExp
  • If you want to use WebExp, you need to get your lists into a format that WebExp can read. You can use this template.
  • Copy the content of the template file into your list workbook beside each list in each sheet and make sure that the template fills as many rows as your list does (remember the black cross that you can drag/pull downwards to fill more cells with the same content or formula).
  • See the two empty columns in the template?
  • Copy the column with your stimuli and filler IDs and paste it into the first empty column.
  • Copy the column with your sentences and paste it into the second empty column.
  • Now we need again a formula from Excel. The goal is to get each row into one cell
  • <>resource id=" 01a "> Which <>/resource>"

  • In the column right beside your template complete with IDs and sentences,
        enter = to tell Excel that you want to use a formula
  • Then click once on the first cell of your template (the first one with resource in it).
  • Put: &double-quotes space double-quotes&
  • Click next cell: id=
  • Put: &
  • Click next cell: 01a
  • Put: &
  • Click next cell: >
  • Put: &
  • Click next cell: Which politician
  • Put: &
  • Click next cell: /resource>
  • Hit enter
  • Now use the black cross again to apply your formula to every row that contains a part of your list.
  • Your lists are in a format now that WebExp can read.
  • Copy each of your newly created columns (the ones that contain the template complete with IDs and sentences) in the following way into this file:
  • Open the attached file with e.g. emacs or taco html edit
  • Copy each of your lists between:
  •       block id="listA1"> [here goes your list 1] /block>
          block id="listB1"> [here goes your list 12] /block> ...
Now you can consult the WebExp Manual to find out how to set up your experiment on-line.


designed by Simone Gieselman