On the Identification of Accessibility Bug Reports in Open Source Systems

Published at the 19th International Web for All Conference (W4A’22)

Manual inspection of a large number of bug reports to identify accessibility-related ones is time-consuming and error-prone. Prior research has investigated mobile app user reviews classification for various purposes, including bug reports identification, feature request identification, app performance optimization etc. Yet, none of the prior research has investigated the identification of accessibility-related bug reports, making their prioritization and timely correction difficult for software developers. To support developers with this manual process, the goal of this paper is to automatically detect, for a given bug report, whether it is about accessibility or not. Thus, we tackle the identification of accessibility bug reports as a binary classification problem. To build our model, we rely on an existing dataset of manually curated accessibility bug reports extracted from popular open-source projects, namely Mozilla Firefox and Google Chromium. We design our solution to learn from these reports the appropriate discriminative features i.e., keywords that properly represent accessibility issues. Our trained model is evaluating using stratified cross-validation, along with a comparison with various baselines models using keyword-based matching as a solution. Findings show that our classifier achieves high F1-scores of 93%.

More specifically, we investigated the following research questions:

RQ1. Can we accurately detect accessibility - related bug reports?

RQ2. What is the size of the training dataset needed for the classification to effectively identify accessibility bug reports?

Finding the Needle in a Haystack: On the Automatic Identification of Accessibility User Reviews

Published at the International Conference on Human-Computer Interaction (CHI'21)

In recent years, mobile accessibility has become an important trend with the goal of allowing all users the possibility of using any app without many limitations. User reviews include insights that are useful for app evolution. However, with the increase in the amount of received reviews, manually analyzing them is tedious and time-consuming, especially when searching for accessibility reviews. The goal of this paper is to support the automated identification of accessibility in user reviews, to help technology professionals in prioritizing their handling, and thus, creating more inclusive apps. Particularly, we design a model that takes as input accessibility user reviews, learns their keyword-based features, in order to make a binary decision, for a given review, on whether it is about accessibility or not. The model is evaluated using a total of 5326 mobile app reviews. The findings show that (1) our model can accurately identify accessibility reviews, outperforming two baselines, namely keyword-based detector and a random classifier; (2) our model achieves an accuracy of 80.7% with relatively small training dataset; however, the accuracy improves as we increase the size of the training dataset.

Our empirical study focused on investigating whether the developer perception of quality improvement (as expected by developers) aligns with the real quality improvement (as assessed by quality metrics).

In particular, we addressed the following research question:

RQ1. To what extent machine learning models can accurately distinguish accessibility reviews from non-accessibility reviews?

RQ2. How effective is our machine learning approach in identifying accessibility reviews?

RQ3. What is the size of the training dataset needed for the classification to effectively identify accessibility reviews?

Automatic Classification of Accessibility User Reviews in Android Apps

Published at the 7th International Conference on Data Science and Machine Learning Applications (CDMA'22)

In recent years, mobile applications have gained popularity for providing information, digital services, and content to users including users with disabilities. However, recent studies have shown that even popular mobile apps are facing issues related to accessibility, which hinders their usability experience for people with disabilities. For discovering these issues in the new app releases, developers consider user reviews published on the official app stores. However, it is a challenging and time-consuming task to identify the type of accessibility-related reviews manually. Therefore, in this study, we have used supervised learning techniques, namely, Extra Tree Classifier (ETC), Random Forest, Support Vector Classification, Decision Tree, K-Nearest Neighbors (KNN), and Logistic Regression for automated classification of 2,663 Android app reviews based on four types of accessibility guidelines, i.e., Principles, Audio/Images, Design and Focus. Results have shown that the ETC classifier produces the best results in the automated classification of accessibility app reviews with 93% accuracy.

In particular, we addressed the following research questions:

RQ. To what extent can machine learning models accurately distinguish different types of accessibility reviews?

Learning Sentiment Analysis for Accessibility User Reviews

Published at the 36th IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW'21)

Nowadays, people use different ways to express emotions and sentiments such as facial expressions, gestures, speech, and text. With the exponentially growing popularity of mobile applications (apps), accessibility apps have gained importance in recent years as it allows users with specific needs to use an app without many limitations. User reviews provide insightful information that helps for app evolution. Previously, work has been done on analyzing the accessibility in mobile applications using machine learning approaches. However, to the best of our knowledge, there is no work done using sentiment analysis approaches to understand better how users feel about accessibility in mobile apps. To address this gap, we propose a new approach on an accessibility reviews dataset, where we use two sentiment analyzers, i.e., TextBlob and VADER along with Term Frequency—Inverse Document Frequency (TF-IDF) and Bag-of-words (BoW) features for detecting the sentiment polarity of accessibility app reviews. We also applied six classifiers including, Logistic Regression, Support Vector, Extra Tree, Gaussian Naive Bayes, Gradient Boosting, and Ada Boost on both sentiments analyzers. Four statistical measures namely accuracy, precision, recall, and F1-score were used for evaluation. Our experimental evaluation shows that the TextBlob approach using BoW features achieves better results with accuracy of 0.86 than the VADER approach with accuracy of 0.82.

In particular, we addressed the following research questions:

RQ1. How do users express their sentiments in their accessibility app review?

RQ2. How effective is our proposed sentiment analysis based approach in the identification of accessibility reviews?

I Cannot See You—The Perspectives of Deaf Students to Online Learning during COVID-19 Pandemic: Saudi Arabia Case Study

Published at the Education Sciences Journal

The COVID-19 pandemic brought about many challenges to course delivery methods, which have forced institutions to rapidly change and adopt innovative approaches to provide remote instruction as effectively as possible. Creating and preparing content that ensures the success of all students, including those who are deaf and hard-of-hearing has certainly been an all-around challenge. This study aims to investigate the e-learning experiences of deaf students, focusing on the college of the Technical and Vocational Training Corporation (TVTC) in the Kingdom of Saudi Arabia (KSA). Particularly, we study the challenges and concerns faced by deaf students during the sudden shift to online learning. We used a mixed-methods approach by conducting a survey as well as interviews to obtain the information we needed. Our study delivers several important findings. Our results report problems with internet access, inadequate support, inaccessibility of content from learning systems, among other issues. Considering our findings, we argue that institutions should consider a procedure to create more accessible technology that is adaptable during the pandemic to serve individuals with diverse needs.

In particular, we addressed the following research questions:

RQ. What are the challenges and concerns that deaf and hard-of-hearing students are having with an online education during COVID-19 pandemic?