Inquiry Project: Psychometric Testing

The inquiry project was a tough one for me.  I had difficulty coming up with a topic because I'm in a state of life right now where it would be cool to learn something completely new (like quilting, for example.... I would like to learn to quilt) but if I am going to dedicate 6 hours to a project, right now it better be to contribute to my learning on a project I must complete -- because there are many of them on the go right now.  There were several home improvement projects that fell on that list, but my significant other is fussy.  He loves doing that kind of thing so much, to the point that he wants to be the one that does it all. Hence our conversation by text the first night of class about what I could build for this project.  Sigh......   I love him.  (P.S. I'm a feminist, really, I am).

But in doing my reading for a research project I am currently working, it struck me that I not only could I, with some self-study, replicate some of the tests done on the questionnaire described in the dissertation I was reading, but I could also use it for my project.  And so the psychometrics project was born. 


Learning the statistical procedures turned out to be not that difficult but it did take me most of a work day to put it all together.  The presentation had it's own challenges because it was hard to describe what was going on in each slide in such a way that it would only take 15 seconds to read.  I feel I make a few leaps of understanding in my descriptions but I did the best I could to keep it brief but still make it understandable and interesting. 


Interestingly enough, I have opportunity to use this format again at the Manitoba Cycling AGM, as we were invited to do a presentation on women and cycling but were only given 5 minutes.  No problem!




Below is the script to the video that is imbedded above.  Please enjoy. 




1. Psychometrics are the statistical tests performed on a questionnaire to assess if that questionnaire is performing consistently and measuring what it says it is measuring.


2. Usually when I do research, I hire a statistician. Here is my statistician, Tom Harrigan. Statisticians don’t have time to think about petty things like office cleaning when they are performing mathematics on data so here is Tom on a day that he cleaned his office and we were all amazed

3. The questionnaire I developed was an instrument used to measure Writing Self-Efficacy.  It uses a 4-point Likert scale measuring agreement to disagreement on 10 items related to confidence in writing ability.  The lowest possible score on this questionnaire is 10 and the highest is 40.

4. The inspiration for this project came when I was reading a dissertation describing the psychometrics of a similar scale designed for Adult Basic Education Students.  I felt that I could replicate many of the statistical procedures explained in this dissertation.

5. One of the key psychometric testing procedures is Factor Analysis. I took Biostatistics in 2002 when I was in the masters of nursing program.  I found my old textbook and notes hoping there would be information on Factor Analysis …… but there wasn’t.

6. So I asked statistician Tom what he thought about my project and he told me I needed more than 6 hours to learn Factor Analysis. And given the complexity of the formulas for factor analysis, I could see what he meant.  So I performed some of the other tests described in the dissertation instead.

7. A program like SPSS is much more efficient for statistical analysis but for this project, I had to go with what I had, which was Excel. By using you tube videos, I learned how to draw graphs, perform correlation, and create binned tables.  

8. So, I pooled the data collected from 231 participants over two past studies. I reordered the data so that the total scores from the questionnaire were ranked from lowest total writing self-efficacy to the highest. The lowest score in the sample was 15 and the highest was 40.

9. Many of the statistical tests I performed required analyzing each question by comparing the top 25% to the bottom 25% of the sample. Strong questions would show low writing self-efficacy students disagreeing, and high self-efficacy students agreeing with the statements on the questionnaire.

10. For example: Question 1: “I feel I have the skills to write a scholarly paper” can be considered a successful question, as can be seen in this plot mapping the number of strongly disagree, disagree, agree or strongly agree responses provided by both the low and high writing self-efficacy participants.

11. I also experimented with using an online stats calculator from a QuickCalcs website to calculate mean, standard deviation, and perform T-tests both on each question, comparing the highest and lowest scoring students, as well on the total sample of 231.

12. Using question 10 as an example: The independent group T-test successfully showed that all the individual questions demonstrated statistically different means between the low and high writing self-efficacy group which shows the questionnaire could correctly identify these opposing groups.

13. The dissertation I followed suggested that each individual question should have a mean score between 2 and 3 and a standard deviation between 0.5 and 1, when these tests are performed on the total sample of 231.

14. You’ll see by the arrows that question 3 met the criteria for the mean but did not meet the criteria for Standard deviation “SD” where it scored below .5.  Question 4, however, met the criteria for both mean and standard deviation.

15. Now for the hard part….the gratuitous selfie!  What does this all mean?  And who cares? It required a lot of thinking on my part.

16. Most questions faired like question 8:  The mean fell between 2 and 3, the standard deviation fell between 0.5 and 1 and high self-efficacy students (in blue) were more likely to agree or strongly agree with the question, while low self-efficacy students (in red) were more likely to disagree or strongly disagree.

17. Compare that to question 3 which, as already indicated, had a low standard deviation of .45.  The graph shows that low self-efficacy students (in blue) were just as likely to “agree” with the statement as high self-efficacy students (in red). Suggesting there was not enough variability in the data.

18. A similar observation can be made with question 9 where the mean score was greater than 3. The graph shows that low and high self-efficacy students were both likely to agree with the question presented.  Subjective analysis is then important to suggest why these questions were not as strong as the others.

19. Question 3 may be measuring a general ability to overcome difficulties while question 9 may be measuring a general ability to be on time, rather than measuring these behaviours as specific to writing. Likely both these items need editing or removal from the questionnaire.

20. In conclusion. I gathered some important information about my questionnaire. And it has made me more determined to eventually learn Factor Analysis.  This image shows me brainstorming the possible factor categories for my questionnaire. Thank you.