Questionnaire design and sampling procedures for business and economics students: a research-oriented, hands-on course

Introductory undergraduate statistics courses widely focus on statistical concepts or software-based data analysis. Despite the fact that the analysis of real data has shown to enhance students' engagement, the step of data collection is often neglected. Once students know the challenges of data collection, they are more aware of potential imperfections, such as a lack of representativeness, during data analysis. In this paper, we present a course that closes the gap allowing Business and Economics students to conduct a full survey under realistic conditions including questionnaire design, sampling, and data analysis. It entangles theory and application by combining course-based research experiences with cooperative learning and a flipped classroom approach. Students do not only obtain competences in the field of statistics, they also gain experiences and self-confidence for future research projects because the lecturer acts as a mentor guiding the students throughout the project. Although statistics is usually an unpopular field for Business and Economics students, their motivation was high throughout the semester as they acted as researchers who analysed a specific research question. This is in agreement with student feedback, which highlights the promotion of research-related competences and self-efficacy within the course.


Introduction
Statistical analysis is a key method in the field of Business and Economics defining the discipline of Econometrics.This comprises, for example, the analysis of financial data for new copula models (Krupskii & Joe, 2020) or the application of a new estimator to simulated and real-data from the Oregon Health Insurance Experiment (Bach et al., 2018).Data from the German Socio-Economic Panel Study and the United States Census Bureau is analysed by Tutz and Berger (2020) to quantify the influence of explanatory variables on the income.All these publications use data stemming from a random sample.The thoughtful conduction of such analyses requires knowledge on data generation and the correction of potential imperfections in the data sets, for example, caused by a lack of representativeness.Such competences are relevant not only in the field of Business and Economics but also in each discipline analysing data sets generated from sampling (for instance Gallardo et al., 2017;Oliveira et al., 2020).Consequently, they need to be a part of the curricula in undergraduate programmes to properly train future generations of researchers and practitioners.
In the last decades, higher education has transformed from frontal teaching to competence-oriented, student-centred learning.For statistics education, the 2005 Guidelines for Assessment and Instruction in Statistical Education (GAISE) report (American Statistical Association, 2005) and its update (American Statistical Association, 2016) fuel the process of enhancing student learning by re-designing traditional courses.This is further strengthened by the Making Statistics More Effective in Schools of Business (MSMESB) conferences which aim to improve the effectiveness of statistics in the field of Business and Economics (Love & Hildebrand, 2002).Multiple statistics introductory courses have been implemented trying to promote the development of conceptual understanding and foster active learning by using a large variety of didactic concepts.For instance, the flipped classroom approach has gained some popularity.Loux et al. (2016) re-designed a former traditional introductory lecture on biostatistics shifting knowledge acquisition to the self-learning phases by using video lectures and online quizzes.They observed increased satisfaction by the students compared to the lecture.However, effects, such as students' perceived efficacy in performing basic statistical methods or understanding statistical claims in media and research, remained unaffected.With a different flipped course design, Carlson and Winquist improved students' attitudes towards statistics and statistics retention (Carlson & Winquist, 2011;Winquist & Carlson, 2014).In their course, self-learning phases prior class consisted of reading chapters from the textbook and answering questions online.Wilson (2013) experienced improved performance on the departments' standardized assessment with a partially flipped course, in which short lectures were still part of the in-class activities.In general, the ways to flip a course are as versatile as their impact on student learning and engagement (Butt, 2014;Peterson, 2016;Reimann et al., 2019;Strayer, 2012).Most flipped courses make also use of other beneficial didactic concepts, such as student-centred learning, cooperative learning, or small-group learning environments (Kalaian & Kasim, 2014;Keeler & Steinhorst, 1995;Knypstra, 2009).
These approaches can also be implemented in inquiry-based or research-oriented course concepts.For example, Li et al. (2018) designed a case-study assignment with students assuming the role of a loan officer at a bank.The students assess the probability of default by using logistic regression to decide on the application for approval.Li et al. observe high student engagement and increased student awareness that statistics is essential for business decision making, which is in agreement with further statistics courses using case-study assignments (Nolan & Speed, 1999;Parr & Smith, 1998;Smith & Bryant, 2009).While in inquiry-based courses the students often tackle practical challenges with a result that is to some extent known by the lecturer, research-oriented courses or course-based undergraduate research experiences (CURE) involve the investigation of a research question with unknown outcome (Auchincloss et al., 2014).For the field of statistics education, Nolan et al. (2020) summarize multiple benefits such as the exposure to statistical research, the strengthening of students' connection to the discipline of statistics, and the development of confidence as a statistician.Such challenging yet satisfying learning experiences can also improve students' attitude towards statistics -something most introductory statistics courses are struggling with (Bateiha et al., 2020;Berenson et al., 2018;Garfield et al., 2002;Hollis, 1997;Schau & Emmioglu, 2012).In doing so, research-oriented courses can increase the chances that students apply gained competences in their lives outside of the classroom (Garfield et al., 2002).
Despite their variety, these undergraduate statistics courses focus widely on fundamental statistical concepts and software-based data analysis.Some of them generate own data or use available data sets to perform the analysis on real data as suggested by the GAISE report (American Statistical Association, 2005;Berenson et al., 2018;Brophy & Hahn, 2014;Hollis, 1997;Li et al., 2018).Fellers and Kuiper (2020) designed learning materials for undergraduate students to train the analysis of survey data.However, the process of data collection is often neglected -especially having statistics as a minor subject -although being an essential aspect in research and industry.In addition, knowing the challenges of data collection is important for a reflective analysis and discussion of data, for instance, when differences between sample and population in survey data need to be adjusted (Gelman, 2007).
Within this paper we discuss a course concept that aims at bridging the gap between introductory courses in statistics and advanced statistics courses as well as the Bachelor thesis for Business and Economics students.By using a hands-on approach, the students analyse the acceptance of rental bikes by performing an own research project together as a team.The focus is not only on questionnaire design and sampling due to the restricted prior knowledge of the students but also software-based descriptive data analysis is included to limited extent.According to the flipped classroom approach, in-class sessions are dominantly used for progressing the research project, peer review, and group discussions.Theoretical concepts are widely shifted to extended self-learning phases and only briefly summarized in class.The lecturer gives guidance throughout the project by defining the schedule, acting as a critical friend, or moderating discussions, but does not affect the project fundamentally.Consequently, learning within the course also involves practical experiences, critical reflection on these experiences, as well as personal development including soft skills such as team work or self-management.The effectiveness of the course concept is discussed based on our experiences and student feedback.As both aspects were widely positive with comparably strong learning progress by the students, we encourage lecturers to implement the course concept on their own.

Boundary conditions and learning objectives
Before we initiated the course, there was no advanced statistics course in the Bachelor program 'Business and Economics' at the Technische Universität Dresden (TUD), Germany, beyond the introductory course.The students gain competences in descriptive statistics, correlations, stochastics, and statistical tests.The course also provides an introduction on random variables.However, the statistical concepts taught remain widely theoretical and abstract to the students and are neither connected to research activities nor to realistic economic problems.This might create a pronounced competence gap with respect to the Bachelor thesis or everyday work.This was further promoted by the fact that other concepts required for Business and Economics related research such as questionnaire design or software-based statistical analysis were widely missing within the curriculum of the Bachelor program.
According to these boundary conditions, the students attending the course only have limited prior knowledge and no experiences on performing a survey.Still, the course aims at preparing students to plan and conduct a survey on their own and puts focus on questionnaire design and sampling.The learning objectives of the course are formulated in accordance to the competence levels by Anderson et al. (2001).They use a hierarchy of six levels with increasing complexity which can be summarized by distinctive tasks: remembering (level 1), understanding (level 2), applying (level 3), analysing (level 4), evaluating (level 5), and creating (level 6).The learning objectives can be split into the three key tasks of the research project (cf. Figure 1): • Questionnaire design: The students know and understand general principles to design a scientific questionnaire (level 2).They conduct a specific questionnaire based on a given research question (level 3).They evaluate the quality of questionnaires and give constructive feedback (level 5).• Sampling: The students know and understand common sampling methods (level 2).
They apply two methods to a given research question (level 3).The results are used to evaluate and compare these methods in terms of representativeness (level 5).• Data analysis: The students analyse the collected data concerning the research question by using the software R (R Core Team, 2019) descriptively (level 4).
It is apparent that the learning objectives are restricted up to competence level 5, which is related to the minor prior knowledge by the students.Still, as for example highlighted by Bonwell and Eison (1991), higher-order thinking tasks such as analysis or evaluation are required to propel active learning.
In addition to these field-specific competences, it is our strong belief that the quality of research projects and project work in general is related to the quality of the communication within the project team.Thus, students work on the survey in small teams of four students and in-class time is dominantly used either for collaborative work on the research project or for plenary discussions on the different aspects of survey design and conduction.This, for example, includes an evaluation of the suitability of different sampling methods with respect to the research question.
We implemented the course within the mandatory module 'Bachelor seminar', which consists of seminars on different disciplines depending on the research activities of the individual chairs of the faculty.Its main learning objective is the training of students in research techniques, such as literature survey or data analysis and presentation, which is in agreement with our learning objectives.Most of the students are in the fourth semester, which means that they attended the basic courses of the Bachelor program, meeting the requirements of the course.The module is restricted to five ECTS (European Credit Transfer and Accumulation System) credits which represents 150 h of work à 45 min.Usually, this is split into 30 h in class (15 sessions with 90 min each), 30 h of self-learning, and 90 h to conduct the research project including the preparation of a seminar paper for examination.We reduced the time in class to 10 h and put a stronger focus on the self-reliant conduction of the survey by the students aiming to promote their self-efficacy.

Course design
As already briefly described, the course is characterized by an alternation of in-class sessions and out-of-class activities including self-learning phases and, to a large extent, the performance of the research project as depicted in Figure 2. Detailed schedules for the inclass sessions are given in the Supplementary Materials (SM).As the course was conducted before the COVID-19 pandemic, the course concept is designed without online elements.Adjustments for switching from analogue to digital are discussed in the SM.
The course follows a hands-on approach with fundamental concepts being briefly introduced and discussed with the students and directly applied to the research project.This is in agreement with suggestions, for example, by the GAISE report, American Statistical Association (2005, 2016) or Hollis (1997).The course design is inspired by the flipped classroom approach: literature survey on concepts of survey design is shifted to the selflearning phases leaving more time for their discussion and application in class.In doing so, the lecturer acts as an expert mentor guiding discussions and asking questions to reveal the collective knowledge of the students.This type of discussion-based learning is also described by Bateiha et al. (2020) and Lugosi and Uribe (2020) to promote active learning during the in-class instruction process.We want to create an atmosphere, where the research project is owned by the students while the lecturer acts as 'safety guard'.It needs to be highlighted that students within the fourth semester are usually not used to flipped, inquiry-based, or research-oriented learning scenarios at the TUD.Instead, they mostly attend frontal lectures and seminars.Accordingly, the students need to adjust their learning habits because the learning progress and the success of the research project depend on their dedication and self-organization.
Within the first in-class session, we thus include an extended and transparent introduction to the course concept and its requirements.In addition, a trustful learning atmosphere is promoted by a short personal introduction of the lecturer and the individual students.In contrast to the introductory courses with often relatively large and anonymous classes, this shall demonstrate the student-centred approach of the seminar.It is also emphasized that they will make mistakes throughout the research project and that this is a crucial part of the learning process, especially, by constructively discussing them with their peers.Critical reflection of the individual phases of a survey and a constructive and appreciative communication are important to correct misunderstandings and essential when conducting a research project.
The introductory part also includes a presentation of the guidelines for the preparation of the seminar paper.In contrast to a research paper, it represents a chronological description of the conduction of the research project with an evaluation of each step.To some extent the seminar paper acts as a learning journal for the students because most of the writing process is done as an out-of-class activity simultaneously to the progress in the seminar.However, the students are encouraged to use a scientific writing style and reasonable presentation of the research data.
A further part of the first session is the collection of quality criteria for the design of scientific questionnaires by using the method of brainstorming.This is in agreement with the intention of the course design to collect and promote the knowledge of the students while the lecturer moderates the process.Afterwards, missing aspects are highlighted by the lecturer and shifted to the following out-of-class phase for review.This especially includes criteria required for achieving representative results.Accordingly, the students need to perform a literature survey on the concept of representativeness as well.
Afterwards, the research objectives for the project are discussed and sufficiently specified.The general topic is already revealed within the announcement of the course.To increase the participation of the individual students, they are intended to work according to the collaborative learning strategy think-pair-share.First, the students individually analyse the topic and propose improvements.Second, they share their ideas with a partner.Finally, the lecturer collects all ideas on the white board.The aspects with the most votes are considered leading to the final research objectives for the survey.With the quality criteria defined and the research question finalized, each student creates a first draft for the questionnaire within the following out-of-class phase.
After two weeks, the second in-class session focuses on the questionnaire drafts.After a brief summary of the previous in-class session and an outline of the learning objectives, the concept of representativeness is discussed altogether.Both the design of the questionnaire and the sampling method determine whether representative results are obtained or not.Keeping this in mind, the quality of the questionnaires is evaluated using the method of peer review.This method uses the collective intelligence and is known to enhance students' competences, especially, considering their self-reflection and communication skills (Herzog & Katzlinger, 2017;Katzlinger-Felhofer et al., 2018;Moulder et al., 2014).The method shifts the responsibility of the learning process and the quality of the research project from the lecturer to the students.However, for optimal performance, the lecturer needs to sensitively guide the students throughout the review process.After an introduction to the review process by the lecturer, students distribute the drafts among each other.Each student reviews the received draft with respect to the formulated quality criteria for questionnaire design as well as the research question.After 15 min, the students share their feedback, whereby the lecturer observes the discussion and supports in case of overlooked important aspects.The students identify most suited questions and create a first draft of a joint questionnaire.
At home, they revise the joint questionnaire individually.The students focus on the questions: Does each item contribute to the research question?Is the questionnaire suitable for the application within the intended target group?Can the questionnaire be answered within an appropriate time frame?In addition, they review sampling methods using literature given by the lecturer.The course focuses on simple random sampling, stratified random sampling, and cluster sampling, as they are widely used in research.For each method the students need to identify main characteristics, strengths and weaknesses, as well as examples for application.
The third in-class session starts with another review and discussion of the joint questionnaire by the students.The resulting optimized joint questionnaire is submitted to the lecturer, who evaluates its quality after the in-class session and sends feedback to the students via email.In addition, the students contact the data protection office of the TUD for reviewing the questionnaire.Accordingly, the students not only learn how to formulate scientifically correct questions, but also conduct a questionnaire in agreement with laws on data protection.Based on the feedback by the lecturer and the data protection office, the final joint questionnaire is created.
The third in-class session is also dedicated to the evaluation and application of the sampling methods.For each sampling method, a flip chart is designed by the students including its characteristics, strengths, and weaknesses.Uncertainties are discussed and clarified in plenary while the lecturer acts as a moderator.Based on this analysis, the application of two sampling methods is planned altogether and performed by the students during the following out-of-class session.The application of multiple sampling procedures allows students to gain experiences in sampling complementing their theoretical knowledge from literature.
In order to evaluate the collected data, the fourth session trains students in data analysis using R (Version 3.6.0)and R Studio (Version 1.0.143)(R Core Team, 2019; RStudio Team, 2016).Not only multiple studies recommend software-based statistical analysis for improved learning but also it is the common standard in industry and research (American Statistical Association, 2016;da Silva & Moura, 2020;Sigal & Chalmers, 2016).The session takes place in a PC pool.It follows a more traditional learning design: The lecturer presents the most essential concepts and commands in R, while the students apply them to small exercises.After the course, the students are able to read csv-data in R as well as to apply a restricted set of commands for descriptive data analysis.Most importantly, they are able to extend their knowledge independently using help routines and the internet during the following out-of-class phase.This process is guided by individual consultations with the students discussing challenges with the lecturer on demand.
In the last in-class session, the analysed data are discussed and the entire survey design is evaluated.In particular, the sampling methods are assessed with respect to the theoretical concepts discussed in the third in-class session.Based on selected items, the representativeness of the samples is analysed.In addition, the students are asked for feedback on the course design.For that purpose, an evaluation sheet is designed (see Supplementary Materials for details).It includes closed and open questions due to time constraints, still allowing students to give detailed responses with respect to key aspects of the course design such as the discussion atmosphere or the schedule of the course.After the last in-class session, the students have three weeks to complete the seminar paper.During this time, they are supported by optional consultations with the lecturer.
After the submission and the evaluation of the seminar papers, each student receives a personal feedback including his/her performance during the course and the quality of the seminar paper.This helps the students to identify strengths and weaknesses and supports their further progress beyond this particular course.In addition, strengths and weaknesses of the course concept can be discussed personally.

Course experiences
Within the trial of the course concept in the summer term 2019, only three students attended the seminar because the module is split into a large set of courses and the main part of the students chose well-established courses.This led to slight modifications in the didactic concept because learning scenarios such as think-pair-share are not effective for such a small group.Still the experiences made with the course as well as feedback were positive with students highlighting the satisfying promotion of their competences as discussed below in detail.We therefore believe that the course concept is beneficial for the community of Business and Economics lecturers.Within this section we discuss our experiences and describe adjustments to small audiences.
The analysis of the 'acceptance of rental bikes at the TUD' was selected as the research topic for the course.The company 'nextbike' is the dominant distributor of rental bikes in Dresden and conducts several stations around the campus.The first 60 min are free for students leading to regular use (StuRa TU Dresden, n.d.), which is further promoted by the fact that the facilities of the TUD are spread over the city.Indeed, the topic arouse the students' interest and lead to a motivated working atmosphere.The students who attended the course emphasized the research topic as an essential criterion for registering for the course.It can be speculated whether a different topic could have generated a larger audience.The course concept can be applied to a large set of research questions without modification.We recommend a strong connection to the interests of the students still also being connected to the field of Business and Economics.For example, an analysis of electric scooters could nowadays generate a larger audience because it is controversially discussed by students and the media.
In general, the students got easily accustomed to the course concept, which appeared uncommon to them in the beginning.This was supported by the extended introduction phase during the first in-class session.All students appeared open-minded already during their self-introduction.The small number of students was certainly beneficial.The working atmosphere was comparable to a motivated research team who received professional guidance by a supervisor.
The first in-class session also included the brainstorming session about quality criteria of questionnaire design.It was surprisingly fruitful considering the restricted prior knowledge by the students and the small group size.Its result is depicted in Figure 3.The students gave diverse answers and contributed their ideas without an effort by the lecturer.They widely focused on the design of the items and considered the perspective of the interviewer.This changed after the revision at home.Including the concept of representativeness raised the students' awareness for the global concept of the questionnaire and they added the perspective of the interviewees as a substantial quality criterion.They understood that the questionnaire needs to be tailored with respect to the requirements of the target group.One reason for this change in the perspective might be that the brainstorming during the first in-class session occurred before the specification of the research objectives.Initially, the students only knew that the seminar would focus on the 'acceptance of rental bikes at the TUD' without mentioning the target group of the survey.Such aspects needed to be clarified by the students.As the students had no prior knowledge in the conduction of a survey and due to time constraints, it was not possible to formulate scientifically correct research objectives or a research question.Instead, they collected aspects they assumed to be relevant for the design of the questionnaire, e.g. the awareness of the rental bike system or its costs.In addition, the survey was restricted to matriculated students of the university.
At home, the students designed first questionnaires, which were the basis for the second in-class session.The peer review process revealed shortcomings in the drafts, especially, being ambiguous and superficial.For example, the students forgot to ask whether the interviewee belongs to the target group or not.Such weaknesses were only partially addressed by the peer review process, i.e. the students analysed the drafts relatively superficially.On the one hand, the students were not used to this method.On the other hand, collective mistakes can hardly be corrected by this method further pronounced by the small student group.Such aspects require support by the lecturer as the collective knowledge by newcomers in study design is limited.For instance, the lecturer highlighted the consideration of the perspectives of the interviewees -an aspect already initially ignored during the brainstorming of the quality criteria.This included an evaluation of each item concerning its comprehensibility and the time required for answering.
Still, the first joint questionnaire (see SM for details) represents already an improvement with respect to the individual drafts and includes a reasonable framework with respect to the research objectives.However, further weaknesses needed to be remedied: The questions are relatively imprecise and leave room for interpretation, a large scale is applied to several questions that lacks validity, and the answer categories for the age are too detailed.The students wanted to be exact but considered neither the level of detail required with respect to the research objectives nor the level of detail that can be reliably distinguished by the interviewees.Aside from the quality of the joint questionnaire, it needs to be highlighted that the students formed a hierarchy to design it: One student guided the process by moderation while the others added ideas and comments.
Due to the shortcomings of the first draft of the joint questionnaire, it was further revised by the students at home.The improved version, which was created during the third inclass session, is also presented in the SM with all changes being highlighted.The students discussed the draft in depth leading, for example, to an adjustment of the scales or a specification of the wording.While the first version of the joint questionnaire only contained the items, the revised version also includes an introduction and guiding phrases.The students submitted the improved joint questionnaire to the lecturer and the data protection office of the university for feedback, which led only to minor modifications to the layout of the questionnaire or the introductory paragraph (see SM for details).
Considering the restricted prior knowledge by the students, they achieved a reasonable questionnaire with only minor external support.Regarding our experiences within the course, three aspects appear essential to achieve such a satisfying learning progress: • Questionnaire design needs time and systematic promotion of the students' competences: It took almost three in-class sessions and a considerable effort within the self-learning phases to achieve the final questionnaire.With the hands-on approach used, the students had an early draft which they optimized incrementally.The relatively long selflearning periods of two weeks between the in-class sessions allowed the to reflect upon the questionnaire draft including a literature survey.We believe that this was beneficial for the quality of the questionnaire and the consolidation of competences.
Still, an extension of the course concept should also include a pretest of the questionnaire which was omitted here due to time constraints.This would enhance the quality of the questionnaire and give the students further insights in the individual steps required for a scientific survey.• Self-reliance and self-efficacy: The students appeared motivated throughout the entire process of questionnaire design and reacted inquisitive with respect to the comments by their peers as well as by the lecturer.They appreciated the freedom and trust they received for conducting the survey which is not usual for the study programme at such an early stage.• Guidance by the lecturer -only where needed: As the students lack knowledge and experience in questionnaire design, they focus on individual quality criteria instead of considering them altogether.Initially, the students tried to construct unambiguous questions improving the reliability.However, they forgot about a proper definition of the scales.A slight reminder by the lecturer was sufficient to extend the scope of the students' review of the questionnaire.Still, this only led to an extension by this particular aspect while, for example, forgetting to create an appealing setting for the interviewees.
The third in-class session included the design of the sampling as well.As already observed for the questionnaire design, the hands-on approach was challenging but also beneficial for the students.This is related to the application of their theoretical knowledge obtained during the self-learning phase to the research project.In contrast to the questionnaire design, more guidance had to be provided by the lecturer because of the restriction of the course to only five in-class sessions.In addition, the literature survey on sampling methods lacked in quality as reflected by a superficial characterization of the sampling methods.While the students reproduced the steps required for the application of the sampling methods, they missed potential challenges and examples.This corresponds to a mismatch in the learning objectives: Instead of evaluating the procedures, the students were only able to reproduce essential aspects.Usually, within the concept of flipped classroom, such shortcomings are not compensated by the lecturer.However, due to the tight schedule, the lecturer determined the sampling methods for the survey and, despite their shortcomings, the students were able to reasonably plan their application.
The simple random sampling was selected because of its ease of application.The students quickly realized that it is nearly impossible to apply it in practice.They decided to collect data at three different locations at the campus at the same time.This stipulation included an extended, lively discussion of the advantages and disadvantages of data collection at the same time versus at different times and of appropriate locations.It was satisfying to see the seriousness and the complexity of the discussion.The achieved sample included 75 valid questionnaires.
In addition, cluster sampling was applied.It allows data collection at ease and its results are expected to differ from the ones obtained with the simple random sampling.Consequently, it was used to demonstrate the relevance of the selection of a proper sampling method and to correct the shortcomings from the theoretical analysis of the sampling methods.The cluster sampling was performed in a course at the faculty of Business and Economics.For this purpose, the students collected all courses with more than 50 students and randomly drew a course.It needs to be mentioned that the first lecturer rejected their request to distribute the questionnaires after the lecture, which led to a second draw.With the agreement by the lecturer, the students collected data from 150 students during the out-of-class phase.The students experienced challenges they did not initially expect during their survey.Such experiences are essential as they may lead to a more realistic time management during future surveys.Empirical data sets are subject to practical constraints, which needs to be considered during data analysis.One constraint includes that the simple random sampling was based on oral interviews, while the interviewees filled out the questionnaires in the cluster sampling.Consequently, the data from the simple random sampling could potentially be biased by the so-called interviewer-effect (Biemer & Lyberg, 2003, pp. 170).
The introductory hands-on course to descriptive data analysis using R and R Studio during the fourth in-class session included methods such as frequency tables, mean values and standard deviations, as well as the scientific visualization of data.In addition, the analysis of subgroups from the samples was briefly demonstrated.The application of more advanced statistical tests would have been desirable but was excluded due to time constraints and missing experiences with R by the students.It was expected that challenges in the application of the software would occur during the data analysis, which was performed during the self-learning phase.Surprisingly, the students did not demand for consultations, which were offered by the lecturer, but requested for help via email.Problems by the students were widely restricted to technical questions, e.g. the input of the questionnaire, which could be easily solved this way.Surprisingly, they had no queries with respect to the descriptive data analysis.
In preparation for the final in-class session, also the lecturer performed an analysis of the obtained data sets.This allowed a discussion of the most valuable information.The discussion in the in-class session focused on the effect of the sampling methods on the quality of the results, which aimed to strengthen the students' awareness towards practical challenges.Representative results are shown in Figure 4.Although the discussed results were selected by the lecturer, the students owned the discussion, identified sources for the differences, and proposed improvements for future surveys.
The commitment by the students was also reflected in the quality of the seminar papers, which was well above our expectations for undergraduate students, especially considering that it was their first project documentation prepared at all.Still, the seminar papers revealed shortcomings as well: It appeared that the students were overwhelmed by the extent of the collected data.They focused on the statistical analysis and the demonstration of the data.The results were only partially discussed with respect to the research objectives.They also forgot to consider the concept of representativeness.Such aspects as well as their performance during the entire course were discussed personally in feedback consultations.While these conversations were constructive and appreciative in two cases, one student demonstrated a lack of understanding considering the shortcomings of his/her seminar paper because he/she invested much effort for data analysis.Unfortunately, the student was not willing to accept the feedback with respect to the discussion of the results, although appreciation was shown for this specific aspect of the seminar paper and despite the fact that it was stated before that critical discussion of the results is more important than the sheer mass of analysed data.Despite one unsatisfactory feedback conversation, we are convinced that relatively demanding research-oriented learning concepts -as presented here -should provide personal feedback to further propel the students' learning progress.This is also supported by positive experiences we gained within a former course concept (Schellhammer & Cuniberti, 2017).
Although the students started the course with only minor prior knowledge with respect to the preparation and conduction of a scientific survey and scientific work in general, all of them were able to create a questionnaire, plan and perform the sampling, and use R for descriptive data analysis after attending the course.They not only know and understand the necessary steps and quality criteria, but also experienced practical challenges and learned how to deal with them.However, their competences require further consolidation in more advanced courses in the following semesters.Especially further training in data analysis and discussion as well as scientific writing is required.Due to time constraints, these competences were consciously not or only to some extent taken into account within the learning objectives.

Student feedback
Two different methods were applied to collect student feedback considering the course design and its effect on the learning progress.At the end of the second in-class session, the Minute Paper classroom assessment technique was used as feedback method.Accordingly, the students answered the questions: • What did you like about today's session?
• What needs improvement?
• Is there something else you want to note?
The feedback included predominantly positive aspects.All students appreciated the open and uninhibited discussion culture allowing them to contribute (e.g.'everyone gets a chance to speak and express their opinion').In addition, they considered positive the connection between theoretical knowledge and practical implementation with respect to the design of the joint questionnaire.This was further specified by one student who highlighted the motivating effect of directly applying new knowledge to the project without shifting it to a session 'perhaps sometime in the future'.The feedback reports included one negative remark, which was related to the seminar paper: Despite the introduction to the requirements in the first in-class session, a student was still feeling uncertain.This led to a short repetition of the requirements and a clarification of questions in the beginning of the following in-class session.
A more extensive evaluation was conducted at the end of the last in-class session (see SM for details).The questions were designed to guide further improvements of the course design.However, the feedback by the students was widely positive with one student being very satisfied and two students being satisfied according to their personal conclusion (question 8).All students would also recommend the course to fellow students (question 9: twice 'rather yes', once 'definitely yes').The evaluation of the personal learning progress was quite diverse (question 7): One student acknowledged the practical relevance, but wished for more guidance with respect to the theoretical concepts.Another student appreciated that the course allowed him/her to immerse into the field of statistical research and analysis.The third student remarked that he/ she obtained strongly advanced competences in sampling, but questions his/her ability to perform statistical analysis using R.The questions with respect to the schedule of the course (questions 2-5) revealed that all students felt overstrained with the application of R. As the students had no prior knowledge, a single in-class session was too little to accustom to the software (e.g.'I felt overwhelmed analzing data by using R because I have never before worked with R and 90 min were too short to apply it on my own').To further promote students' competences in software-based statistical data analysis, the hands-on course could be shifted to the first in-class session (Schellhammer & Cuniberti, 2017).In the following weeks, the competences could be further promoted by using regular problem sets.These could be solved in the self-learning phases with discussions of the solutions in the following in-class session.This would continuously train the students in the application of R and potentially promote the learning transfer.Such a modification is accompanied by an extension of the course concept by at least one additional in-class session to include the regular discussion of the problem sets.Still, all students appreciated both the design of the individual in-class sessions with discussion and working sequences (question 2: twice 'completely', once 'mainly') and the prescribed course concept (question 5: twice 'I prefer the current concept', once 'I would prefer a slightly tighter schedule with more obligations').Further positive feedback included the open and relaxed discussion culture (question 1), where mistakes can occur without negative consequences and are considered as an essential part of the learning process.One student also positively mentioned the introduction to the topics and prudent interventions by the lecturer.The mixture of theoretical basics and their implementation in a survey (question 6) were positively evaluated by all students.The students enjoyed the application of their theoretical knowledge on a research project under realistic conditions and the freedom given while still receiving support by the lecturer, where required ('I have never experienced such an application-oriented course allowing me to exercise my knowledge from the introductory statistics course','the course encouraged independence because we were responsible for the organization').However, one student also remarked that an extended theoretical part would be beneficial because it would allow a more detailed analysis and discussion of the data.

Conclusions
Within the Bachelor program Business and Economics at the TUD, students obtain the theoretical background in statistics but have only minor opportunities to practically apply these competences under realistic conditions in terms of a research project.Accordingly, they are neither satisfactorily prepared for conducting a survey, e.g. for their Bachelor thesis nor sufficiently recognize the relevance of applying an appropriate sampling method in order to receive representative data.As was demonstrated within the paper, the discussed course concept can bridge this gap to a large extent, especially with respect to competences required for questionnaire design and sampling.An extension of the schedule by one or two in-class sessions can further promote the student performance in statistical data analysis using R and data discussion.As the presented course was implemented in an existing module with a fixed workload of five ECTS credits, we decided to shift the improvement of these competences to more advanced courses.
It was our aim to design a course, where students widely own the research project still being guided by the lecturer.According to the flipped classroom approach, also the literature survey on theoretical concepts was shifted to the self-learning phases, which leaves more room for discussions and an application of the concepts onto the research project in the in-class sessions.These aspects were highly appreciated by the students as demonstrated by the student feedback.This is also reflected in the high motivation by the students throughout the course, the surprisingly vivid and detailed discussions, and the relatively high effort the students made, for instance, with two samplings and the analysis of the data sets.
Still, we also faced several challenges, for instance, the small group of students, the extended time required to design a sufficient questionnaire, or the demanding application of R. Based on our experiences, we expect the following aspects to be essential for a satisfactory implementation of the course: • Attractive research topic: As the students should own the research project and spend much effort in the individual stages, the research objectives should take into account the students' interests for enhanced attractiveness.As the course is mandatory and in competition to other seminars, we previously defined the research objective and used it for promoting the course.Still, the specification of the research objectives was performed by the students in the first in-class session.In an obligatory course, the research objectives could offer more freedom, e.g. the usage habits of transport.However, this would require more time and an agreement on the research objectives appears less straightforward.• Become familiar with each other: An extended part of the first in-class session was used for personal introduction.The lecturer did not only demonstrate an approachable and trustful attitude, the students also introduced themselves, which created an open discussion culture right from the start.During the course, every contribution to the discussion was appreciated and constructively debated, even if it exhibited considerable deficiencies.
• Balance between autonomy and guidance: Throughout the research project, the lecturer observed the group activities and gave guidance, where required.Thereby he acted as a critical friend, raising the students' awareness towards a certain aspect by asking questions.Only, if this proceeding did not cause a considerable progress, more guidance was provided.However, it was not our aim to perform the best possible survey.According to the minor prior knowledge of the students, the focus was more on the individual learning progress.Still, this approach created satisfying results with personal opportunities for improvement being discussed individually in the feedback sessions at the end of the course.• Research-oriented learning needs time: It was beneficial for the students to have two weeks for self-learning and research activities between the in-class sessions.Despite their responsibilities in relation to other courses, they had enough time for literature survey and development of the research project.Although it is not uncommon that research-oriented learning leads to increased stress two students evaluated the presented course neither as insufficiently nor exceedingly challenging.Only one student felt overstrained with respect to the application of R, which leads to adjustments of the course concept.In addition to the extended out-of-class phases, also relatively much time was spent in the in-class sessions for the design of the questionnaire.This was necessary because the students were initially not able to consider all defined quality criteria.With several iterations of improvement they were able to include more and more aspects and finally create a sufficient result.• Progressive development of competences in statistical analysis using R: The course should start with an introduction and hands-on session to the fundamental application of R in the first in-class session.Based on problem sheets that are solved in the selflearning phases and discussed in the respective in-class sessions, the competences can be incrementally advanced and preserved.
With these considerations in mind, we encourage other lecturers to test and further develop the presented course as we strongly enjoyed experiencing the pronounced learning progress and the evolution of the students' competences.It did not only train students in questionnaire design, sampling, and data analysis but it also teaches them that empirical research is always affected by deviations from pure theoretical concepts, i.e. the generation of representative results, the needs of the target group, or practical challenges along the way.

Figure 1 .
Figure 1.Learning objectives of the course 'Questionnaire design and sampling methods'.Students' performance is evaluated based on a seminar paper at the end of the semester, where students reflect on the individual steps of survey design and analysis.

Figure 2 .
Figure 2. Design of the course 'Questionnaire design and sampling methods'.Organizational tasks are highlighted by a question mark, theoretical tasks by a mortar board, and practical tasks by gearwheels.In-class sessions take 90 min and out-of-class phases usually last two weeks.

Figure 3 .
Figure 3. Collected quality criteria for questionnaire design after the in the first in-class session and its extension in the second in-class session.

Figure 4 .
Figure 4. Selected results of the survey discussed within the last in-class session.(A) Distribution of the age of the interviewees.(B) Distribution of the gender of the interviewees.(C) Use of the rental bike system 'nextbike'.