By Mark Moulton, Ph.D., Director of Research, Evaluation, and Psychometric Development
I recently returned from the excellent International Objective Measurement Workshop (IOMW) 2018 conference (www.iomw.org) in New York City, overlooking the famous Washington Square Park, where I had a chance to present on “objectivity.” IOMW has been around since the early 1980s and is, in my opinion, one of the premier world measurement conferences with a specialty in the Rasch Model. Educational Data Systems (EDS) has been a major sponsor of IOMW for some years and was central in organizing the 2014 (Philadelphia) and 2016 (Washington, DC) conferences. This year’s conference was ably organized by Andy Maul (University of California, Santa Barbara) and Ronli Diakow (NYC Department of Education). I helped organize the software workshops.
What I like about IOMW is its spunky, intimate, international flavor—unburdened by the academic and professional pretensions of larger conferences. Presenters fly in from all over the world, everyone shares, and little gets held back. This year, the conference was bursting at the seams with presentations. Skye Barbic (University of British Columbia) led it off with talking about measuring the mental health needs of young adults—helping them track their progress on a finely calibrated mental health yardstick. There was a great session on “Rasch in International Development,” organized by William Fisher (BEAR Center, University of California, Berkeley), who is spearheading a crusade to get economists and world leaders to adopt Rasch-based measurement methods for global economic variables. There also were presentations on the larger field of “Metrology” and how to apply physical definitions of measurement to the social sciences.
My favorite was a session on “Models and Reality,” which was about building a philosophical and measurement framework for defining “objectivity,” the unifying theme of all the Rasch measurement methods, including my multidimensional model developed at EDS—NOUS (Use of a Non-Unidimensional Scaling) model. This provided a nice launching pad for me to present my topic, “Toward an Objectivity Statistic.”
My goal is to derive a universal statistic for assessing the “objectivity” of the models behind measurement, machine learning, neural networks, and artificial intelligence. My starting point—a built-in feature of NOUS—is to define Objectivity as a combination of Accuracy (ability to predict missing cells), Stability (ability to replicate model parameters with different subsets of data), and Independence (the statistical independence of each “object”). These properties are tied closely to the philosophical definition of what an “object” is, and the resulting statistic makes it possible to evaluate not just numerical methods but even the “objectivity” of the real world we are trying to model. Deep waters indeed, my friends!
The third day of the conference was given over to software workshops, including Winsteps, RUMM, jMetrik, and others. EDS recently performed a successful replication study that compares the open-source jMetrik package (Patrick Meyer, University of Virginia) with the now-defunct industry standard, Multilog. I presented the open-source “Damon on Python.” Damon was written to implement NOUS and the Rasch Model in the context of Python’s fabulous numeric computing ecosystem, and it is at the heart of some of our coolest work at EDS. I also gave a lunch presentation and hosted a panel on “Measurement and Big Data,” where I had an opportunity to talk about pain points relating to the handling of big data and the crazy workflows that psychometricians routinely encounter. I was able to share some of my projects at EDS for simplifying data storage and retrieval and building an all-purpose, open-source psychometrics platform in Python.
All told, it was a lovely conference, spring-fed by the secret vitality thrumming away in the heart of Manhattan. The next IOMW will be in San Francisco in 2020. Hope to see you there!