top of page

Mixed Methods Research

Case Study

Relevance

This case study demonstrates the importance of conducting exploratory mixed methods user experience research to test assumptions and identify opportunities prior to investing significant resources in design and delivery. This particular example debunked assumptions in user behavior and motivation, saving time and resources for the organization. 

Context and Objective

XSEDE is a national advanced computing ecosystem funded by the National Science Foundation to provide free compute resources and training to users. XSEDE would like to know more about its users’ pathways and product engagement patterns to identify opportunities and test assumptions regarding user behavior and motivation.

Key Research Questions

  1. To what extent does increasing online training registrations boost utilization of compute resources?

  2. How does activity vary across groups, particularly those underrepresented in advanced computing (e.g. women and members of underrepresented racial and ethnic groups)? 

Role

Co-led the study design, instrument development, data mining (SQL database), analysis, reporting, and presentation of results. 

Methodology

Exploratory quantitative and qualitative longitudinal study to build personas and user pathways.

Study Participants

Demographic: Online training registrants with access to XSEDE compute resources, 6 months following last training event. 

Sample: Census / population study

Data Sources

Behavioral - portal activity data measuring training registrations and compute resource usage.

 

Attitudinal – qualitative survey to gauge user motivations for pursuing online training.

​

Findings

  1. This study contradicted previously held assumptions that training registrations precede activity on computational resources when in fact 77% of registrants who had accessed a computational resource did so before ever registering for training. ​

  2. Self-reported survey data combined with behavioral portal data demonstrated that 44% of all training registrants never utilize their newly gained skills on XSEDE resources but instead apply them to local or regional systems.

Insights

Based on these data, the following insights were generated.

  1. Faculty and senior researchers do not participate in training due to their minimal role in conducting day-to-day research activities, particularly coding. 

  2. Graduate students are the primary users of both training and compute resources. This group takes advantage of training resources in order to maximize their limited time on shared or allocated systems. 

  3. Undergraduate students make up a considerable portion of trainees but are not necessarily interested in utilizing the research computing systems. These students take training to develop marketable skills for future paid work.

Personas

Data and insights were used to generate three personas and corresponding pathways.​

Opportunities

Insights identified the following opportunities:

  1. How can XSEDE adjust its training content to include topics that are most useful to graduate students looking to optimize their allocation time on its systems?

  2. What training topics will help build marketable skills for undergraduate students and how can these be incorporated into the XSEDE training suite?

Research Impact

Research insights were used to refine program theory and expand training content to include more industry focused topics and content applicable to resources beyond XSEDE. The former assumption that XSEDE training led to increased utilization of compute resources was refined. Since the study, annual online trainings registrations have increased, particularly among those self-identifying as women or members of underrepresented racial and ethnic groups. 

bottom of page