Benjamin Paassen
Last changed: 2021-02-15

Evaluation of Hints in Computer Programming Tutors

What is this study about?

This study concerns the evaluation of automatically generated hints for programming students. The motivation for this study is to automatically generate helpful hints for students. We hope that you are willing to provide your expertise to help us identify hints that are helpful to students.

You have been invited to participate in this study because of your expertise in teaching in the programming domain. This Participant Information Sheet tells you about the research study. Knowing what is involved will help you decide if you want to take part in the research. Please read this sheet carefully and ask questions about anything that you don’t understand or want to know more about.

Participation in this research study is voluntary.

By giving your consent to take part in this study you are telling us that you:

Who is running the study?

The study is being carried out by the University of Sydney (School of Computer Science, Faculty of Engineering). The investigators are Dr Benjamin Paaßen (Visiting Researcher from the Humboldt-University of Berlin, Germany), Ms Jessica McBroom (PhD student), A/Prof Kalina Yacef, A/Prof Irena Koprinska.

The database for developing the methods that will be presented in this study has been provided by Grok Learning. The examples shown in the study are also taken from programming courses taught by Grok Learning. All researchers have signed a data sharing agreement with Grok Learning. As part of this agreement, Grok Learning reserves the right to formally approve any resulting publication of this research.

What will the study involve for me?

You will be shown three programming tasks as well as solution attempts for these tasks. In each case, you will be asked whether you believe a hint would be helpful for this solution attempt, which hint you would give, and whether any hint from a selection of automatically generated hints is relevant and/or useful. You will also be asked about your years of programming/teaching expertise.

How much time will the study take?

We estimate that the study will take less than 30 minutes to complete.

Who can take part in the study?

This study is open to any participant over 18 with experience in teaching students how to program.

Do I have to be in the study? Can I withdraw from the study once I’ve started?

Being in this study is completely voluntary and you do not have to take part. Your decision whether to participate will not affect your current or future relationship with the researchers or anyone else at the University of Sydney. At any time during the completion of the questionnaire, you can withdraw from the study and no data will be transmitted. Once you have completed the questionnaire, your data will be anonymously transmitted to us and a withdrawal is not possible anymore because we can not tell which of the submitted responses are yours.

Are there any risks or costs associated with being in the study?

We do not expect any risks or costs arising from taking part in the study.

Are there any benefits associated with being in the study?

By taking part you support the development of automatic hint generation systems for programming education by providing your expertise. However, you will not receive financial or other compensation for taking part in this study.

What will happen to information about me that is collected during the study?

We do not collect personal information about you. Your responses will be collected anonymously and, in this anonymized form, will be made available to the scientific community to support the development of future hint generation mechanisms. A summary of the results will also be submitted as an article to a scientific journal or conference. Under no circumstances will your responses be identifiable.

Can I tell other people about the study?

Yes, you are welcome to tell other people about the study.

What if I would like further information about the study?

If you have further questions, you can contact the principal investigator Dr. Benjamin Paaßen at

Will I be told the results of the study?

If you sign up at this link, you will receive a summary of the results as soon as they are available.

What if I have a complaint or any concerns about the study?

Research involving humans in Australia is reviewed by an independent group of people called a Human Research Ethics Committee (HREC). The ethical aspects of this study have been approved by the HREC of the University of Sydney, Protocol no. 2020/655. As part of this process, we have agreed to carry out the study according to the National Statement on Ethical Conduct in Human Research (2007). This statement has been developed to protect people who agree to take part in research studies. If you are concerned about the way this study is being conducted or you wish to make a complaint to someone independent from the study, please contact the university using the details outlined below. Please quote the study title \enquote{Evaluation of Programming Hints} and the protocol number 2020/655.

The Manager, Ethics Administration, University of Sydney:

Telephone: +61 2 8627 8176


If you wish to keep a copy of this statement, please print it out and/or store it on your disk.

Go to the survey.