22-Mar-2022

Manipulation Check With Chi Square

Hello, this is Dr. Kay Sweeter at San, Diego, State, University, and I am going to show you how to check your manipulation check from Qualtrics into SPSS. So any experiment that you do you have to have a manipulation check the manipulation check is a question. Typically, at the end of the post-test that basically asks the participants, whether they can correctly identify the particular cell that they were in without knowing that there were different versions of the stimulus. And so the example that I'm. Going to do this on is actually a little of a complex type of us setup.

And what I have here is a 2 by 4 factorial design. So that means that there are two different independent variables, which means in order to do a manipulation check I need to do then two different manipulation, check questions, one for each independent variable. And one of the independent variables has two levels that's the organization in this case and the other independent variable has four levels. And that is the type of. Apology that the organization is issuing in this case. So if you just have a regular experiment where you only have one independent variable, then you're probably only going to have one manipulation check question. And this process will be a little less complex than what I'm going to show you in here.

So first off in setting up our experiment on Qualtrics. We have everything in different blocks. And so I have the consent form block right here. I have the first stimulus where I'm telling people.

You're going to read an article, and then I have stimulus cell number one, and then I have the timer and I won't down lock. If I have the timer in here, which is shown as question, 23, automatically generated as hold how long somebody stayed on this page. Then I go into the next block. You can see here. This is a whole new block. I'll, go ahead and expand this. The next block I have the same set-up the introduction that you have to read this article below.

This is a different version of the story, where the. Only thing that's different here is the quote and the attribution. And then again, I have a timer on here, and I can see that this one was numbered 24. And so, so on and so forth, I have all of my different cells in here. And this was again, a 2 by 4 factorial design. So that means that there are eight different treatments, and then I also had two control groups.

So I have 10 different cells in here, that's a lot to keep. And you know, a track of in this process. So I went ahead and on our stimulus paperwork, That we submitted to the IRB I sort of made some cheat sheets for us. And so here are all of our stimulus articles, and it shows what the change is, but I have gone in here, and I have identified for the information that I'm going to see in Qualtrics. When I do the download into SPSS I have identified.

What variable name is going to tell me that people were in that particular cell. And so let me show you how I did this again, I'm going to go into this is the Navy response. Navy is the organization versus.

Carnival Cruise, and then the type of apology that they gave was a concession apology. So I scroll down into here, and I see that question, 23 is uniquely identified as only being in this cell. And so I just wrote down the timer question, let's, see, 23, concession response by US Navy. So that I have a little guide in here. So that I can go ahead and make sure that I know what each of these cells are. So in order to download, you go into that particular Qualtrics file that Qualtrics Project, you go into data. Analysis, you go export, you export data you're going to do it as SPSS.

And then you press download, and then it's ready for you to open, so I have it already open on my computer. And this is what Qualtrics has given me in SPSS. So as I look at the date of view, you can see in here, there are a lot of holes.

And if you don't, remember how your study was set up, you might get a little nervous about these holes. Well, this is exactly the way that it should be. Because that means that this person right here went.

Through one particular stimulus, he did not experience if we look across and yeah, the other stimuli of the other nine versions of the stimuli. And then he finished up. And he answered the questions on the manipulation check at the end. So this section I'll, go ahead and highlight it for you. So you can see it more clearly.

This section from there to there is actually broken up into our ten different experimental cells, and so I need to kind of clean this up to make this a little easier to understand. So that I can actually run my manipulation check in here, if I go into variable view in here, then I can see exactly what the questions are named. And this is where I'm going to match it up with what I have in here. So here I see question, 23, here, I, see, ah, here's, the block for everybody that participated in question, 23. And so what I'm going to need to do is I'm going to use this as an opportunity to say, well, I know that these people were all in cell number five, so I'm going to anytime I see data in.

Any of these columns I'm going to mark that they were cell number five. So first thing I need to do is I'm just going to go to the very beginning, and I'm going to insert a new variable, regardless of whether you have a complex data set like I have or you have a really easy one. If you have an X, you have to do this. You have to put all of your cells in a single variable. So that we know which one it is so here's. My variable I'll, go ahead and give it a name I'll, make sure that we know that this is the. Experimental cell and so perfect, and then I'll go back over again to the data side.

And now I could go through, and I could say, oh look, here's. One. So that's going to be, oops, that's, actually cell five. If you remember on my little cheat sheet, here cell five I'm, going to put a five there and I could scroll down, and I could look for all of them when they came up, and then I could make them all fives, but ain't, nobody got time for that. So what we're going to do instead is let's just make sure that this is 23 and.

On my cheat sheet, 23, perfect, and I'm going to sort descending, and it looks like only one person got sound number five. So I would want to let this. This particular pilot test run out a little more because I don't want to base it on one person's data, but for the purposes of moving on let's, go ahead. And now let's go to the second one.

So this guy is let's open him up number 24. So 24 is going to be cell number one. You can see right here cell number one. So again, I'm going to sort descending and. Then these guys are going to be cell number one, and then I'm going to move them over, and I'm going to do this ten different times total so that I can oops I have them in the wrong spot. Sorry, that was just a visual error for me.

Nothing, nothing bad happened. Okay. So then this guy is number 25. Let's. Look in here, where's 25.

25 is cell number 6, so I'm going to give these guys sixes all right and then going to move it to the end. And next one sort descending. So this one is cell number 26. And that is cell 2.

So now I'm going to type in a 2, and then I'm going to move it. Oops, sort do you sending? And then this guy's name is 27. So that is going to be cell 7 for me, again, I figured out which cells these were by doing this legwork before I, even started the video.

And so I took a little of time and I went back to my stimulus and I just made sure that I had sort of like a master I. So that I could run through this very quickly on the video 28 28 links up with 3 so perfect. And we are so close to being almost.

Done here, descending and 29 29 is cell. Number 8 when it's 34 34 is cell number 9. This one is 35 35 is cell. Number 10, I could also put this on my clipboard if I wanted to and copy it down that works as well.

And it looks like this. One is probably going to be the last one do you sending number 30, 30 is cell number 4 all right. So in here I can go ahead and see is there anybody who was not even assigned a cell, so I'm going to sort a sending to get those if there were any zeros or missing data up to the top.

There are none? Let me just do descend to make sure that there are no problems. Yep, there are no issues.

So I can see that everybody was assigned a cell and I can also see in here that these are my manipulation check questions, right here. So going all the way down the line. It looks like everybody actually answered the manipulation check question now in general Qualtrics, if you in the survey flow check that you want it to show your stimuli in the randomization equally it's going to, but if.

Somebody goes to your consent form. And then they press next it's going to count all of a sudden that they showed that particular cell. And so your cells are not going to be a hundred percent, equal it's, just going to attempt to show them in that in as equal of a manner as it can.

And so that's, why you're going to have slightly unequal cell sizes, but they should be grouped around the same size, um. So in general, we do have everything grouped around the same size, except for that, very first one that me. Did where we only had one person. So people obviously did not like the article that they saw, and they just didn't want to continue on when they were when they were getting that one.

Okay. So I have my cells all figured out it's going to be easier for me if I can actually write in here, what type of cell it is. So if I come into experimental cell I can say that one is this one press add cell two is this one, and it's helpful. If you do get this set up in here into this new variable that you've created. Because otherwise you're just going to have to be going back and forth between your notes, trying to figure out exactly what you have. So I'm, not going to go through the whole thing.

I'll, just do these first couple and then say, okay, so I have experimental cell I'm. Just going to drag him down to the bottom to my actual questions, which was, you know, what organization was involved was a carnival cruise or was it the Navy in there. And then the other question, the article I read. And then it has all of these. Different things they took the blame or the responsibility. They provided an excuse.

They offered a solution or remedy. Furthermore, they said that they were extremely sorry, or there was no quote whatsoever. So we want to see if people understood the difference between these four quotes or no quote at all that they saw in the shell of the story that we gave everybody. So because cell experimental cell is a nominal or categorical variable, again, I don't, have it all filled out, but I have my cheat sheet here and.

Because what was the organization is a nominal or categorical, variable I have two nominal variables. And in order to see if they're how they're how they're mixing together I'm going to use I am chi-square so analyze descriptive statistics and cross tabs. And for me today, it doesn't matter which one I put where I'm going to put cell in the rows, just because there are 10 cells, and then I'm going to put which organization was involved. That's, good BY. 43 you can kind of see it in the background here over. There and I'm going to press cells I'll do all of these, and then I'm also going to go into chi-square. And the first thing I do when I look at a chi-square is I go all the way to the bottom and I see is the p-value good.

Yes. The p-value is good. So whatever people thought they were in up here is exactly what they thought so concession by Carnival Cruise. So of the people who got the carnival cruise 1/3 of the people who got this particular cell said that this article had Carnival Cruise, none of the. People said, it had Navy. So that means this one worked because look this is Carnival Cruise, excuse by chronicle, cruise for the people that got this one said, it was Carnival Cruise 0 of the people that got this one said, it was Navy, they're, correct, Carnival Cruise and so on and so forth. So let's, go ahead and let's.

Look at cell six. Let me just pull over my screen would not be crazy here for a minute I could show you so cell number six is Navy as you can see back on the back end here cell number six. Is Navy. So when I go into cell number six, I should see zero people said, it was carnival cruise zero people.

And then the people that said, it was US Navy. There were five people. So this one was correct as well. So I would go through each of these iterations of my independent variable to make sure that whatever people got which is represented for me over here is what they identified, which is represented over here.

And then I would repeat this for the other independent variable, which. For me is a little more involved because I have ten different cells. So this is how you run a manipulation check and make sure that indeed, you have the differences that you think you are going to have if you find that there's a problem with one of your manipulations likes, maybe people can't tell the difference between a concessional response and an emotional display. Then you need to figure out by looking at the two different, the two differences in those stimuli. What you can do to.Further, emphasize that one is emotional and one is a concessional response, and then try the pretest the pilot test again. So good luck with your experiments and enjoy.