Evaluating UX
Research Tools for CCMS Design
the challenge
In an independent study with Dr. Tharon Howard at Clemson University, I researched how we could use and evaluate user experience methodologies on Component Content Management Systems (CCMS). The intent of this study was to develop a set of standards to rank how well a CCMS performs, compare how current UX methods hold up to those standards, and then consider how and what the field of technical communication may need as CCMS has become standard in most industries.
at a glance
Audience: Technical Communicators/UX Designers
My role: UX Researcher
Duration: 4 months
Tools:
​
​
Research questions
-
How do we teach technical and professional communicators:
-
To "invent" content,
-
To "arrange" information, and
-
To adopt an appropriate "style" for CCM environments?​
-
-
How can UX methods be used to evaluate the performance of a CCMS?​
-
What are the gaps in UX for CCMS design?
the approach
Reviewing XML, DITA, and CCMS Literature
The research began with understanding the groundwork of mark-up languages to organize and sort content topics and modules. I then turned to the existing literature on CCMS to find out what had been explored in the field and by what researchers, what the key concepts and terms being used were, to identify other sources being used, and to ground our own research in the field. Dr. Howard notified me heading into this research that there were few academics investigating CCMS, and provided me articles from the main scholars in the field, such as Tatiana Batova and Rebekka Andersen. I developed a rather informal annotated bibliography, documenting the readings assigned by Dr. Howard.
I was also tasked with finding out if there were additional resources, academic or otherwise, writing on CCMS’s or CCMS design/usability. The sources that I found on my own were never formally summarized, as I instead took notes in a way that was more helpful for me as I moved forward in my research, and talked through what I found each week with Dr. Howard in our meetings.
​
Proposal
As the semester moved forward, Dr. Howard recommended that we submit a proposal of our research.We were accepted into the 2019 Symposium for Communicating Complex Information in Shreveport, LA. The theme was “Speaking with, not at,” which ultimately focused on speaking with others about research in, the teaching of, and applications of usability and user experience design. Unlike a typical conference presentation, the presenter sessions at SSCI were intended to help individuals plan the next steps of his/her project, share ideas about future research options, discuss opportunities for collaboration, and ask questions about approaches.
​
Defining CCM Heuristics & UX Research Methods
With the foundation of CCMS established, I then started to define what makes a good CCMS. Although CCMS and CCM authors can have vastly different needs across businesses or industries, we tried to stay as general and objective as possible to assess what CCM users value in a CCMS. For example, users should be concerned with how a system will ensure that components are written at a level appropriate for an end user, or how accessible is a system to multiple authors? This stage of research went through several rounds of revision, as I defined the heuristics and brought them to Dr. Howard each week to discuss and revise.
​
Because we lacked a metric to evaluate a CCM, we turned to Jacob Nielson and Marie Tahir's Homepage Usability, which developed a set of heuristics on the homepages of websites based on the evaluations from user tests, design experience, and general observation of what makes them work well or not. We adopted a similar role of producing heuristics that aimed at the specific functions of a CCM, without being specific to a particular industry or set of users. Based off of our own experiences and studies, we would also have to craft new heuristics to examine a CCM.
​
As I put together the CCMS heuristics, Dr. Howard and I chose UX research methods to evaluate their usability in a CCM. Because there are so many UX research methods available, this project was really just a starting point for future research, and we were limited to a semester of time, we narrowed down our UX methods to manageable total of 15 methods. To refresh myself on the UX research methods and sketch out my thoughts on how they might perform in a CCM environment, I developed a document that summarized the method and speculated on how it might work in a CCM.
​
Evaluating UX Research Methods to CCM Heuristics
In this final phase of the research, Dr. Howard and I first revised and finalized our set of CCMS heuristics, ensuring that the wording and intent of each one was clear. At one point, I had developed 12 heuristics, but after examining the list, Dr. Howard and I agreed that the list could be narrowed to 10. With this complete, I then was tasked with ranking each of the criteria against the UX methods on a scale of 1-4 (1 – poor, 2 – fair, 3 - good, 4- excellent). I used a Microsoft Excel spreadsheet to list the heuristics, UX methods, and my rank of each one. I then calculated the average of each method to show how it fared against the others. Over the course of two weeks, Dr. Howard and I went through the criteria and debated the rankings on each UX method. We typically were able to compromise if we disagreed, but for the most part, we did agree on the rankings given. The first spreadsheet contains the rankings that I gave to the UX methods. The final spreadsheet is the rankings after Dr. Howard and I went through each heuristic together.
​
Presenting Research
Going into the presentation, I knew the SCCI goals were to facilitate discussion and encourage working through early research ideas. With the audience and time-limitations in mind, I created three handouts for the presentation. I transferred the spreadsheet into a more readable, cleanly designed InDesign file. To engage the audience before I presented how Dr. Howard and I evaluated each method, I also printed blank copies of the spreadsheet so that we could ask the audience to consider how they might rank each method. Alongside these spreadsheets, I provided a handout for our presentation that listed the ten CCMS heuristics.
constraints & solutions
The main constraint of this project was my total lack of experience using a Component Content Management System (CCMS). While I was familiar with what a Content Management System (CMS) was, and had used one like WordPress before, I did not have any concept of what a CCMS was prior to this study. I also had little knowledge of the history leading up to CMS and CCMs, such as XML, DITA, and Lightweight DITA, and the first weeks of my research were spent understanding this groundwork. This made the research feel quite abstract, and a greater challenge than I what I expected. I also had experience using some of the UX methodologies I considered in my research, but there were many I had not yet used in my usability research. This was also a constraint as I pursued my research. However, with the guidance of Dr. Howard, I had numerous resources to complement my lack of experience in actually employing certain UX methods, and meetings each week to discuss the direction of the research each step of the way.
With 15-20 minutes to share the research, and the 10-15 minutes dedicated to discussion among the attendees, my primary consideration of the presentation of the research and handouts was to condense the research appropriately and facilitate discussion. My primary consideration for this project was the end target audience of the research and presentation (i.e. those attending SCCI), and my deliverables (e.g. PowerPoint presentation, handouts) reflect that decision.
the outcome
This research was truly challenging in ways I did not expect, but also very rewarding. I had the experience of conducting UX research on a management system that I’ve never used. In my own technical communication education, there was clearly a gap between the concepts and writing strategies taught versus how industries now expect technical communicators to write and adapt to environments like a CCMS. I already felt like I was playing catch-up on learning CCM concepts, although it is an industry standard and has been pretty well established over the past 15 or so years.
My research felt well-suited for Batova and Andersen’s call in “A Systematic Literature Review of Changes in Roles/Skills in Component Content Management Environments and Implications for Education” that to ensure longevity in technical communication, that we must begin to bridge the gap between academia and industry for the changes of roles and skills in CCM (175).
I would measure this research project as a success, considering the further into the research I went, the more the research made itself important. The main outcome of our research was that:
​
We see that current UX methods work well for certain criteria, but we should be developing tools that people need as designers and authors in CCM environments to better help them make the decisions that they need to.
​
After receiving the feedback from the conference, it would have been great to move forward with the research on a larger-scale if I weren’t set to graduate in May. It would be interesting for future research to ask actual CCM authors and designers to rank UX methods using our set of heuristics, and compare the results against our own to see how they hold up. Perhaps too, our research project can pave the way for better bridging the gap between academia and industry in CCM environments.
Ultimately, through this project I have gained invaluable knowledge into CCM environments from a UX perspective, which I believe will be greatly useful to me as I enter the workforce and manage my expectations of what technical communication can now look like.