Myartspace: Design and evaluation of support for learning with multimedia phones between classrooms and museums

This paper presents a description and evaluation of Myartspace, a service on mobile phones for inquiry-led learning that allows students to gather information during a school field trip which is automatically sent to a website where they can view, share and present it, back in the classroom or at home. The evaluation focused on three levels: a micro level, concerned with issues of usability; a meso level, which examined educational effectiveness in terms of learning breakthroughs and breakdowns; and a macro level, which looked at the impact of the new technology on school museum visits practice. Issues of implementing mobile learning across school and museum settings are identified and cross-level influences are discussed. The study showed that the service was effective in enabling students to gather information in a museum and this provided resources for effective construction and reflection in the classroom. Minor usability problems did not detract from the learning, however there are significant issues concerning how to structure the visit and on the viability of Myartspace as a regular service.


Introduction
The class trip to a museum or gallery 1 is a long-established activity for many schools. Although museums have been conducting classes, delivering lectures, and designing special programmes for schools for over a century (Hein 1998), it is only in recent decades that museums have received recognition for their value to formal education (Black 2005). The UK National Curriculum emphasises the "direct use by pupils of a range of sources of evidence [where] museums and heritage sites [are] seen as a readily accessible means of delivery" (Black 2005:157). Museum visits can expose students to subject-matter that cannot be effectively covered in the classroom, introduce them to resources in their community, and provide a varied social experience (Anderson & Zhang 2003;Kisiel 2005;Rennie & McClafferty 1995;Storksdieck 2006). Furthermore, museum visits are memorable events for the students, allowing the teacher to call upon them later in appropriate learning situations (Rennie & McClafferty 1995). Museum visits can offer rich learning experiences facilitated by authentic objects and structured into inquiry-led learning. The museum can be a place to learn across curriculum topic areas (DCMS & DfEE 2000) in an engaging environment (Johnsson 2003).
Thus, a school museum visit will preferably include a pre-visit lesson where questions are formulated, continue during the visit with the collection of evidence and information, and conclude post-visit with interpretation of evidence and drawing of conclusions: "…making the links between school and museum learning explicit, genuine, and continuous affords real opportunities for school students to have enjoyable learning experiences in both settings" (Griffin 2004:S67).

Technology-enhanced museum learning
The last decade has seen increased research into the use of digital, mobile tools to support school museum visits. Moving beyond the audio or multimedia museum guide (e.g. Abowd, Atkeson, Hong, Long, Kooper & Pinkerton 1997), these tools facilitate inquiry activities in the museum such as exploration, information search, communication, and experience documenting (Hsi 2002). While a number of mobile learning applications have been proposed (see for example Cabrera, Frutos, Stoica, Avouris, Dimitriadis, Fiotakis et al. 2005;Galani, Chalmers, Brown, MacColl, Randell & Steed 2003;Hsi & Fait 2005;Mulholland, Collins & Zdrahal 2005;O'Hara, Kindberg, Glancy, Baptista, Sukumaran, Kahana et al. 2007;Papadimitriou, Komis, Tselios & Avouris 2006), none have been designed to support activities that span pre-, during-and post-visit learning in the museum and the classroom.

Myartspace evaluation aims
This paper describes our evaluation of Myartspace, a mobile service that supports learning in the museum and connects it with learning in the classroom. The design of Myartspace arose from an initial proposal by a multimedia company to a UK funding agency to develop mobile phone and web technology to support school visits to museums and galleries 2 . The aim of the service is to support a process of inquiry learning, beginning with goal-setting at school, followed by 'collection' and annotation of artefacts in the museums during a school visit. The visit is then followed by synthesis back at school (or during homework) and then presentation and sharing of what students have learned and created. We present the results of our evaluation of Myartspace at three pilot sites. The aim of the evaluation is to examine the effectiveness of the service for enhancing learning between classrooms and museums. Section 2 of this paper describes the Myartspace service, section 3 outlines the evaluation framework and methods and section 4 presents the evaluation results. The paper concludes with a discussion of the results and their implications for future systems.

Overview of Myartspace
Myartspace is a service that uses both mobile phones and a website to support inquiry learning between classrooms and museums. Development started in April 2005 and trials were conducted in three UK museums from February 2006 to March 2007. During this period over 3000 children engaged with the service. Myartspace combines interactions in three spaces: the physical space of the museum and classroom, the personal space that students create on their mobile phones and personal computers (PCs), and the virtual space containing 'collected' artefacts, in the form of online 'stores' and 'galleries'. The recommended mode of use includes three lessons, preceded by customisation of each lesson by the teacher. Participating teachers receive a Teacher's Pack and Lesson Plans, specific to the museum. The Lesson Plans contain ideas for designing pre-, on-and post-visit lessons related to that museum. Teachers are however free to customise and use the service as they wish, to suit their teaching practices and the specific needs of their students. During the pre-visit lesson the teacher explains the concept of 'collecting', introduces the museum, sets the students one or more inquiry questions, and optionally allows the students hands-on experience with the Myartspace website. The students' task is then to investigate the inquiry question during their museum visit, by using the phone to collect relevant evidence and information, then use these during a post-visit lesson to create a 'personal gallery' on the Myartspace web site. For example, students on a school visit to the D-Day museum were set the question: "Was D-Day a triumph or disaster for Britain?" The website hosts three repositories of digital objects: a museum 'store' (maintained by the museum), a class 'store' (maintained by the teacher and students), and personal student and teacher 'stores' (maintained by each individual user). It also hosts personal student and teacher 'galleries', which are ordered collections of digital objects similar in style to a Powerpoint® presentation that users create by selecting objects in their personal store and defining a sequence in which they will be presented. Although users have no access to each other's stores, they can view each other's galleries. All content in personal galleries is automatically made available in the class store, where all users can access and use it (see Figure 1). The content of the museum store is provided by the museum and comprises digital content such as photos, illustrations, and text representing physical museum objects and exhibits. Decisions over how much and what museum content to digitise were guided by the need to have enough curated content to support a wide range of learning tasks. Input came from all stakeholders (developers, funding body, educational consultants) and agreed targets were negotiated with individual museums as part of their contractual agreement. The museums involved in the pilot either had no available digitised content at all; or where digital content existed it was not appropriate for the target age groups. The task was seen by museum staff as an extension of their curatorial and archival services. A content specification and style guide was developed by the project team to ensure content quality and suitability for the audience. The digitisation was carried out by the museums' educational and curatorial teams. The process involved the selection of objects to digitise, taking of digital photographs, and writing of textual information 3 . Digital content was then imported into the museum store (content database) using a bulk upload facility and a spreadsheet that enabled the association of different items so as to create multimedia museum objects. After upload, all the objects could be edited and updated by museum staff as required. The process took place over the design and implementation phases of the project. Before the museum visit, the museum store is available only to the teacher, who can select some of that content to copy into their personal store and publish it in their gallery. This will make the selected content available to the students through the class store for the pre-visit lesson. Students can fill their (initially empty) personal stores with digital content as the lessons progress, while their objective throughout the lessons focuses on the creation of a personal gallery to demonstrate their work on the learning task.

Figure 1:
Organisation of stores and galleries in Myartspace. Solid lines indicate deliberate copy of content by users and dashed lines indicate automatic copy of content by the service. Users can copy content deliberately (a) from the museum / class stores into their personal (student/teacher) store (lines d1, d3), and (b) while publishing (copying) content from their personal store into their personal gallery (line d2). Automatic copy of content takes place (a) from the mobile phone in the museum onto the user's personal store (line a1) and (b) from the user's personal gallery onto the class store (line a2). The class store is initially empty; it is automatically populated as users copy items from their personal stores onto their personal galleries. The museum store is a repository available throughout the lessons and its content cannot be altered or added to by the users. Users can view each other's galleries but cannot copy content directly from them and have no access to each other's stores.
The Myartspace experience is: • Prior to the lessons, the teacher may create a teacher's gallery using objects in the museum store, both as an example to students of a gallery and to familiarise themselves with the web site. Content displayed in the teacher's gallery is made available in the class store and students can select and copy objects from it into their personal store during the pre-visit lesson. Students, however, have no access to the museum store in the pre-visit lesson. • During the visit, students are loaned a Nokia 6680 multimedia mobile phone, with the Myartspace service pre-loaded as a Java application. The phone is locked into running the service and cannot be used for other functions (such as making calls). They may create new content by using the phone to take photos, record sound and input text (see Figure 2). They may also 'collect' objects from the museum store by typing in a two-letter code displayed on a label next to the exhibit. Having collected an item they are prompted to type in the reason why they have chosen to collect it. They can also select to view the usernames of other people who have collected that object. All the content they capture is automatically sent by GPRS from the phone to their personal store on the Myartspace website; • In the post-visit lesson, students can access their personal store on the website and copy into it objects from (a) the museum store, to which they now have full access, and (b) the class store (which contains items held in other students' galleries). Their task is to create personal galleries. This is the culmination of the inquiry, where students review, enrich and synthesise their evidence to answer the inquiry question. At the end of the post-visit lesson the students can present their galleries in the classroom or share them with friends and family outside school (some schools have organised parents' evenings where these galleries are presented) using the Myartspace website (see Figure 3), access to which is password protected and the content moderated to ensure privacy protection and appropriate use.

The M3 evaluation framework
The evaluation was based on the Lifecycle approach (Meek 2006), which places evaluation at the centre of the technology development process, from the very early stages of design to the final assessment of the technology in a learning context. Similarly, the Myartspace evaluation took place throughout the course of the project: from conception, through design and implementation, to deployment. The main intention was to evaluate the potential and effectiveness of the Myartspace service to enhance learning between classrooms and museums. The M3 evaluation framework (Vavoula & Sharples 2009) was used. M3 comprises three levels: Micro level: examines the individual activities of the technology users and assesses usability and utility. In the case of Myartspace the activities included collecting objects through exhibit codes, making notes, contacting people who have collected a particular item, recording audio, and taking pictures. Meso level: examines the learning experience as a whole, to identify learning breakthroughs and breakdowns; it also examines how well the learning experience integrates with other related learning experiences. In the case of Myartspace, evaluation at this level involved exploring whether there was a successful connection between learning in the museum and in the classroom as well as identifying critical incidents that reveal new patterns and forms of learning or where learning activity is impeded. Macro level: examines the longer term impact of the new technology on established educational and learning practice. For Myartspace this related to the organisation of school museum visits. The evaluation at this level looked, for example, at the appropriation of the new technology by teachers, the emergence of new museum practices in supporting school visits, and how they related to the original project visions. To establish the value of the service at each level, the evaluation explores the gap between expectations and reality and also unforeseen processes and outcomes. This takes place in two stages of data collection followed by one stage of analysis: Stage 1: collect data about what is supposed to happen at a level -user expectations are captured through interviews with users (teachers, students and museum staff) and by analysing user documentation, training sessions and materials Stage 2: collect data about what actually happened at a level -observations, video and audio recordings substantiate the reality of technology use for the different users. Stage 3: examine the gaps between user expectations and reality through a combination of reflective interviews with users and critical analysis of the data collected in stages 1 and 2. The development of Myartspace comprised four broad phases: (1) Requirements analysis, to establish the requirements for the socio-technical system (the users and their interactions with technology) and specify how it would work, through consultation with the different stakeholder groups; (2) Design of the user experience and interface; (3) Implementation of the service; and (4) Deployment of the service. These are compatible with an Agile Development approach (Beck, Beedle, van Bennekum, Cockburn, Cunningham, Fowler et al. 2001) where requirements can evolve throughout the development process to take account of the evaluations of usability, learning effectiveness and institutional adoption. Thus, the requirements analysis persisted throughout the project lifecycle, and covered all three levels of analysis (micro, meso and macro). Figure 4 illustrates the gradual introduction of evaluation activities at the three framework levels over all project phases. The horizontal axis in Figure 4 depicts time; the change of focus development phase over time is shown, as is the persistence of requirements analysis throughout the project lifecycle. The shaded areas represent activities for requirements analysis (dark grey) and evaluation at the three levels during design, implementation and deployment (all other shades). Note that the figure is not intended to indicate the relative durations of the project phases, only the sequence in which they came into focus. The emphasis on level of requirements analysis changes during the development process. At the start of a project, the requirements analysis must take account of all levels to set initial requirements for an educational experience that integrates technology, effective learning and institutional support. As the project progresses, the technology matures so that changes to requirements become focused on the learning context and institutional adoption. At the end of the project, the requirements have been finalised and are evaluated at all levels. The emphasis on level of evaluation also changes during the development process. Early evaluations at micro level inform the user interface and human-technology interactions. Once the technology is robust enough to allow assessment of educational value, evaluation activities at the meso level are introduced during the implementation phase. Similarly, the macro level requires that the technology is in place and used for long enough to establish its effects on e.g. school museum visiting practice, so evaluation activities at the macro level may be introduced during the deployment phase.

Evaluation activities and methods
The evaluation team comprised the authors of this paper, and was independent of the development team. The evaluation took place throughout the project lifecycle, from April 2005 to March 2007, and was planned as a continuous, formative and informative activity rather than a final 'success or failure' test. Three test sites were involved: the Study Gallery in Poole, the Urbis museum of urban life in Manchester, and the D-Day museum in Portsmouth. The evaluation framework presented in section 3.1 guided the design of requirements analysis and evaluation activities and the selection of methods. Evaluation began with a series of scoping studies and consultation workshops to establish the educational and user requirements for the service. During the design, evaluation was concerned with the micro level, focussing on usability. The main technique used was Heuristic Evaluation (Nielsen & Molich 1990), carried out on low-fidelity, paper prototypes. These evaluation activities continued into the implementation phase. As more robust functional prototypes became available, into the first part of the implementation phase, evaluation at the meso level was added, up to full-scale user trials with user observations, interviews, focus groups and questionnaires. Such user trials continued into the deployment phase. Later into the deployment phase summative evaluations took place, where the macro level was also introduced, including stakeholder interviews and surveys. Table 1 summarises the evaluation techniques used at the three levels and the associated three-stage data collection and analysis. As shown in Table 1, multiple methods were used, including interviews, observations, questionnaires, and expert reviews. Interviewing included one-to-one semi-structured interviews with teachers and museum educators, and focus groups with students. Observations were facilitated by paper-based reporting sheets and videos of the student focus groups, the teachers, and the museum educators during all lessons, where observers were asked to focus on key areas, e.g. "describe how the teacher introduces the learning task". Questionnaires incorporating both open and closed questions were administered to students in paper format before and after each lesson, and to Myartspace teachers in the form of an online survey. Expert reviews in the form of Heuristic Evaluations were carried out on paper and functional prototypes. The mixed-methods approach enabled triangulation. For example, data collected through teacher interviews before a lesson were checked against data collected during teacher observations for that lesson and post-lesson interviews with teacher and students; outcomes of Heuristic Evaluations were tested during full-scale user trials; and data collected during full-scale trials were compared against interviews with teachers who had not participated in the trials. The data analysis was guided by the M3 framework presented above and employed Grounded Theory (Glaser & Strauss 1967) and the Critical Incident method (Flanagan 1954). We used an adaptation of the Critical Incidents method to analyse the students' experience in the museum. Right after the visit, the evaluators watched videotapes of a student group to identify observable critical incidents that appear to be breakthroughs (indicating productive new forms of learning or important conceptual change) or breakdowns (where a learner is struggling with the technology, is asking for help, or appears to be labouring under a clear misunderstanding). These incidents were then assembled into a compilation tape and were reviewed with the students for further elaboration. Open coding was applied to data collected through different sources in stages 1 (expectations) and 2 (reality), to extract key themes and emerging issues pertinent to the micro, meso, and macro levels. Sources of data included interview transcripts, observation sheets, video recordings, questionnaires, heuristic evaluation reports, and the outcomes of the critical incident analysis. Issues emerging from stage 1 (expectations) were mapped onto issues from stage 2 (reality), enabling us to identify gaps between expectations and reality.

Myartspace evaluation results
In this section we summarise the evaluation results of the final full-scale user trial in November 2006, conducted at the D-Day museum in Portsmouth, UK (which commemorates and interprets the 1944 Normandy Allied landings). We include references to findings of previous evaluation activities where relevant.
Page 10 Design Implement Deploy S1 • Analyse descriptions in service promotion materials, original proposal, minutes of early project meetings • Interviews with stakeholders to elicit initial expectations for impact of service S2 • Review of press coverage and interviews with stakeholders to document impact/transformations effected by the service Macro Requirements Interviews S3 • Reflective analysis of expectations-reality gaps in terms of service impact Macro S1 • Analyse description of educational experience based on Teacher's Pack and Lesson Plans • Interview teachers and museum educators prior to lessons about what they have planned for the students' learning experience • Observe teachers and museum educators while presenting learning experience to students in the classroom/museum S2 • Observe educational experience in museum/classroom o Note critical incidents that show new forms of learning or educational interaction o Note breakdowns • Interviews/focus groups with teachers, museum educators, students on educational experience in museum/classroom

S3
• Capture expectations-reality gaps in terms of educational experience through reflective interpretation of documentation analysis and observations; and also through interviews/focus groups with teachers, students, museum educators.

S1
• Scoping study of previous projects and related recommendations • Consultation workshop on 'User Experience' to establish requirements S2 • Data supplied by evaluation analysis S3 • Workshop to finalise educational and user requirements • Revisions of requirements in light of evaluation findings S1 Data from system requirements S1 • Examine system documentation (Teacher's Pack and Lesson Plans, online help) for descriptions of functionality • Interview teacher prior to lesson to assess level of knowledge and expectations for functionality. • Observe training sessions at museum and school to document how functionality is described to teachers/students.

S2 System performance tests
Full-scale user trials S2 • Observe lesson to establish actual teacher and student experience of functionality • Interview teacher after the lesson to clarify experience of functionality • Questionnaire and focus groups with students to capture experience of functionality Technical testing

S3
Analysis of performance tests against requirements S3 • Capture expectations-reality gaps in terms of user experience of functionality through reflective interpretation of documentation analysis in the light of observations; and also through interviews and focus groups with teachers/students.

S1
Established heuristics S2 Expert reports

S3
Analysis of expert reports Micro Design Implement Deploy Table 1: Requirements analysis and evaluation activities and methods used at each level, for each project phase

Overview of Myartspace user journey for the final user trial
The trial involved a class of 23 students in Year 9 at Key Stage 3 (13-14 years old), their history teacher, a teaching assistant, and a museum educator. The Myartspace experience for the observed user group involved three lessons. A pre-visit lesson took place at school on Friday, November 3 rd . During the lesson the teacher introduced the topic of D-Day to students, along with sets of questions that students would work with in the museum, each set dealing with a D-Day issue (e.g. "Life at the home front"). No history modules relevant to D-Day were planned in that school at that time of year, so the teacher had to introduce D-Day out-oforder, specifically for the Myartspace visit. Students were divided into self-selected groups of three with each group assigned a set of questions to work on. Students were then told that in the museum they would use Myartspace to collect objects, take pictures and record sounds related to the questions on their worksheet. The teacher demonstrated Myartspace galleries and introduced the concept of 'collecting' to students through additional source work. The museum visit lesson took place the following Monday. Students arrived at the museum with their sets of questions on paper. The museum Education Officer gave them an introduction to Myartspace, explaining how the phones worked and what they could do with them. Students were given one phone each and told to explore the museum in their groups, collecting items relevant to their set questions. The Myartspace web site was not shown. The post-visit lesson took place on the next day. Students worked individually at a PC to create personal galleries. The teacher and the classroom assistant walked around the room helping students.
Our evaluation activities took place before, during and after each of the three lessons. The teacher chose two groups of three students for the focus group interviews: one group of girls and one of boys. Each group was interviewed after each lesson, and were followed closely and recorded on video in the museum. The footage from the girls' museum tour was reviewed by the evaluators after the museum visit to create a video-diary of incidents we considered important to the evaluation. The girls' group was then shown that diary in the interview following the post-visit lesson and was asked to describe the incidents. In addition, questionnaires were distributed to the whole class after each lesson. The teacher was interviewed before and after all three lessons. The museum Education Officer was also interviewed in a more general reflection on the Myartspace service that the museum had been running from February to November 2006. The findings from the three lessons are described below at the micro and meso levels. The analysis has highlighted the significance of preparation work, mainly on behalf of the teacher but also the museum, which takes place before the three lessons. Thus, findings relating to this preparation work before the lessons are described separately. Finally, findings at the macro level are described.

Myartspace -before the lessons
No teaching unit relevant to the D-Day is taught at this school at the time of year when the study took place, so the teacher ran an introductory lesson on the D-Day landings prior to the pre-visit lesson. The Teacher's Pack steps through the Myartspace process, including registering students, pre-visit lesson, visit to the museum, and post-visit lesson. However, it does not describe the functionality of either the website or the phones, or what the students can do with them. The main additional sources of information about the service were members of staff at the museum who deliver handson taster sessions; and the Myartspace web site.
Myartspace venues encourage teachers to visit the museum themselves before they come with their class. However, practical limitations often prevent this. This problem is not specific to Myartspace: "this is not a new problem … teachers don't normally attend the museum before a visit, though it would be beneficial if they did." (Interview, Education Officer). Nor is it specific to this museum, as other studies suggest (Anderson & Zhang 2003;Finkelstein 2005;Fisher 2001;Johnsson 2003;Kisiel 2003;Michie 1998;Storksdieck 2001).

Before the lessons -micro level
Teacher and students have access to an online general help page on the website which contains instructions for each task a user can perform (create gallery, view/copy from stores, etc.). However, no detailed, dedicated set of instructions (e.g. a manual) is provided, nor is there context-specific help (i.e. the help button only takes the user to the above mentioned general help page). Similarly, there is no online guide on how to use the Myartspace phones. These were conscious design decisions, aiming at designing a service that is easy to 'pick up and go' without the need for lengthy tutorials and manuals. Although the need for support materials was identified in previous evaluations, design efforts had instead focused on further simplifying the user interface.
The teacher in this particular study had used Myartspace previously, which provided her with firsthand knowledge of the functionality of the phones and the website. Other teachers surveyed indicated that training sessions at the museum did not cover the use of the website in full, leaving them uncertain as to how to use it back in their classrooms. Thus, although the teacher in this trial knew what to expect at each Myartspace lesson in terms of service functionality, the same cannot be assumed for the typical teacher.

Before the lessons -meso level
At the interview before the pre-visit lesson the teacher indicated that the students in general had enjoyed previous visits to museums, but that these consisted of filling in a prepared worksheet. The museum Education Officer, too, thought that worksheets were not adequate to support school visits: "The museum had a lack of resources to support secondary school age children. Myartspace was a way of attracting secondary schools into the museum".

The pre-visit lesson
During the lesson the teacher demonstrated the website and an example gallery on a class screen, using one of the galleries from her pervious visit as an example, showing pictures and collected objects and playing voice recordings, to introduce the capabilities of Myartspace. Emphasis was placed on the option to collect photos and voice recordings. The teacher explained the Myartspace process and phones to the students. Students were not allowed time to have hands-on experience of the website. The D-Day topic was introduced. Different groups of students were given different handouts, each containing several questions, on an inquiry topic, such as "how did people respond to outbreak of war?" and "would we respond in the same way?" The students were then asked to perform source work, looking at printed sources (photos and text) relevant to their group's worksheet and filling in a "who-what-why-when-where" sheet for them. They were told that their job in the museum was to carefully select relevant objects to collect, just as a curator of a museum would do.

The pre-visit lesson -micro level
The only technical problem we observed at the micro level was that the teacher experienced being timed out (she had logged onto the website prior to the lesson). The issue of the lack of a guide to using Myartspace came up during teacher interviews and student focus groups. This issue was also raised in previous evaluation activities, in this and other trials. The Education Officers in two of the participating museums have written their own support material, which they distribute to teachers.

The pre-visit lesson -meso level
Pre-visit lessons were not standard practice for this teacher, due to lack of time. The teacher's expectation of this pre-visit lesson and the introduction of Myartspace was that it would help students to be more focused at the museum visit. Following the lesson, the teacher was very positive about the students' level of interest. In summary, the learning focus of the pre-visit lesson was on: a. the History topic, i.e. D-Day; b. the process of analysing and selecting sources/objects, i.e. collecting; and c. Myartspace, i.e. new technology.

The museum visit
At the museum, the teacher began by reiterating the learning task and the significance of the students' inquiry questions. The phones were distributed and the museum's Education Officer gave a brief description of their main functions (see section 2). No mention was made of how many objects to collect. Specific points made included: • when collecting an object, giving a reason will help later when creating their gallery, • a list is available of people who collected the same object -they could discuss their reasons for collecting it, • for the audio recording they could read out an exhibit's label.
The students then set off in their groups to explore the museum.

The museum visit -micro level
After the pre-visit lesson, the student questionnaire asked how confident they were about using Myartspace in the museum. Results were 17% 'very confident', 50% 'fairly confident' and 33% 'not sure', with none indicating 'Not very confident' or 'Not at all confident'. When asked in advance what they expected to be able to do at the museum, the focus group's responses included: • Take pictures • Record what pictures are of • Make [sound] recordings • Copy information that is available at the museum • Gather information to make their own website • Be able to make changes to collected items, with no limits on how much they can collect.
These expectations were generally realistic, apart from "Record what pictures are of"; although Myartspace allows the user to take a photo and then record an audio or text comment about the photo, the two items are not linked together by the software. This caused a problem in the post-visit lesson, and will be discussed later. During the visit there were minor usability problems with the phones, but these did not distract from the activity. The main problems were a loss of signal in parts of the museum, the camera flash causing problems with reflection in glass (which students could overcome by covering the flash with a finger), and the 15 second limit on audio recordings that was sometimes insufficient for reading text from an object's label into the phone. In general, the students in the focus group reported that Myartspace was easier to use than they expected. One student mentioned that she liked the fact that the phone looked like a phone, and not like a usual museum handheld audio guide. One student commented that recording audio was quicker than typing. They found that Myartspace made the visit less boring, "more modern" than traditional visits to a museum. Taking pictures and recording audio were highlighted as the functions they enjoyed most. In a questionnaire administered immediately after the visit, 48% of the students found Myartspace 'very easy to use', 30% 'somewhat easy', and 22% 'average', with none indicating it was 'difficult' or 'very difficult'. Taking photos was judged the 'easiest' thing they could do, recording sound was 'best' and 'writing comments' was hardest. In total, 637 objects were collected or captured by the 23 students during this class's visit: 364 photographs, 121 sound recordings, 77 written comments, and 75 collected objects.

The museum visit -meso level
Both teacher and students were fairly confident and enthusiastic about using Myartspace prior to the museum visit. At an interview just before the visit, the teacher anticipated that Myartspace would lead to "more interaction of pupils with exhibits -more questioning of them, not just taking what they see at face value". She expected that the opportunity to practice collecting in a real context would mean the students learned more about the process of collecting and thought students would be quite discerning and critical about what they collected. Although the students exhibited generally high confidence levels regarding the use of Myartspace before the visit, they were reserved in their expectations of the Myartspace experience: 17% were 'Excited about using it', 66% 'Not sure' and 17% thought that 'It sounds rather boring'. The girls' focus group provides a good example of collecting behaviour. The group began collecting using the codes. One girl asked "how do you collect" and the other members of the group were able to help. As they progressed around the museum the dominant behaviour was to copy the text on the museum exhibit labels into the phone either as text or, more commonly, as an audio clip. Sometimes a student would take a photo of an exhibit and then read the associated text into the phone (sometimes in addition to collecting using the object code). Although reading text from a label was a frequent activity for this class, it was probably prompted by the Education Officer in her initial briefing; in other museum visits we have observed more students created their own audio descriptions Observation in the museum of girls from the focus group showed them discussing among themselves what they should collect. For example, one girl asked "Why do we want to collect this?" with another replying "It shows women working". The group negotiated what to write or, when recording, what to say. At times they cooperated in deciding who would collect which item; at other times they all seemed to compete to collect the same object. There was confusion as to how they would share the objects when they returned to school. One member of the group indicated they should all collect everything so they would all have it available in their personal stores, while another thought that they would all have access to each other's objects back at school. This issue also appeared in the answers to the questionnaire after the museum visit (e.g. "[The Myartspace store is] good, but confusing because working in groups people can't make just one").
Competitive collecting was repeatedly observed in one group. In the focus group following the museum visit the students referred to "cooperating" to have only one person take a photo (and all use it later), but "competing" to decide who should actually take the photo: "argue over who would take the photo, cooperating but competing to take the photo". In the focus group one of the students said he wanted to go around by himself, because of arguments with his friends over who was collecting what. Taking photos was a popular activity. In the girl's group for example, one girl started to take a photograph and another group member pointed out the 2-digit code for the object; the first girl took the photograph nevertheless, commenting "I want to take my own picture". A snapshot of activity in one museum room towards the end of the visit found half of those in the room were looking at their phones and half were looking around the room. That does not indicate that children were engrossed in their devices rather than the museum, but rather that the technology mediates and extends the museum visit. The activity of collecting multimedia presentations encourages students to linger at designated exhibits and the recording of notes and pictures enables them to create and preserve a personal perspective. Teachers in all Myartspace trials noted that students spent more time than on previous school visits and engaged more with the exhibits. According to one museum Education Officer, Myartspace extends considerably the average time students spend in the museum: "A normal visit lasts approximately 20 minutes; a Myartspace visit is an hour and a half, so that students have more time working in the venue, looking at the material." In the focus groups following their tour in the museum the students agreed that the process of reading out the label beside an exhibit as an audio recording was a good way of learning, and gave them a purpose for their activity. They said they related their collections directly to the specific topic they had been asked to concentrate on, although one student mentioned that she collected some items only because they 'looked good'. The students expected to sift through their collections and select items suitable for their galleries during the post-visit lesson.
In the questionnaire after the visit students were asked if they would visit a Myartspace museum again. Responses were 57% "Yes", 30% "Maybe" and 13% "No". Responses for whether they would recommend Myartspace to other students were 56% "Yes", 35% "Maybe" and 9% "No". Interviewed after the visit, the teacher was asked how effective the museum visit was. Her comment was: "students really enjoyed it and had lots of related questions after the visit". When asked how it differed from usual museum visits she commented: "normally a museum visit is a little passive with limited interaction". Asked if there were any surprises, she said she was surprised at: "just how enthusiastic the students were".

The post-visit lesson
The teacher guided them to log on to the website and demonstrated the main features, recapping the use of stores and instructing them to create their own galleries to present the answer to one of their group's questions. She did not mention that placing an object in a personal gallery makes that object available for all in the class store. Although they worked in groups in the museum, in this lesson they worked individually to create their own galleries. Each student selected one question from their group's list as the focus for their gallery. Students were allowed to choose the same question as other members in their group. The teacher and assistant walked around the class answering mostly 'how-to' questions and helping the students mainly with navigation problems such as "how do I see the museum store?" In some cases the students were prompted by the teacher to look in the museum store for items relating to the topic they were working on.

Post-visit lesson -micro level
While the students had been shown the website during the pre-visit lesson, they had no hands-on experience or detailed knowledge of how to create galleries. In the questionnaire after the museum visit students were asked "How confident do you feel about using the Myartspace website tomorrow?" Responses were 30% 'very confident', 61% 'fairly confident' and 9% 'unsure', with none indicating 'Not very confident' or 'Not at all confident'. In general, the students appeared to use the website successfully throughout the lesson. There were many queries about spelling and the availability of a spell-checker. Myartspace deliberately did not include such a feature, to encourage students to work on their spelling. The main micro level problems observed were: • Listening to audio recordings was problematic, as audio could not be turned up enough on the PCs. Earlier trials suggest school PCs may often have problems with audio. • Several of the photographs that students had taken in the museum were not easy to identify. This relates to the students' expectation (see section 4.4.1) to "Record what pictures are of". Audio and text comments made at the time a photograph is taken are not linked to the photograph, and do not therefore serve as annotations. • The absence of a student from the post-visit lesson meant that their group could not access the items that student had collected in the museum. • The gallery editing tool differed from other presentation tools such as PowerPoint®. In particular, deleting items is done by 'hiding' individual items, while changing their order involves moving individual items one place at a time (requiring page reloads), which students found counter-intuitive and time-consuming. During the teacher's interview following the lesson, when asked how confident she had felt about using the website in the lesson she replied "Ok although a little concerned. Instructions on use would have helped".

Post-visit lesson -meso level
A questionnaire after the museum visit asked students whether they thought the post-visit lesson would be different to lessons after other school trips. Eighteen students answered "Yes" and two students answered "No". The use of technology was the most commonly cited reason for this expected difference ("using computers", "using Myartspace", "pictures and phones"). Students also cited class discussion ("we'll talk about it"), appreciation of the practical experience offered by Myartspace ("exciting because it's not just theory") and the specified learning task ("know what to do"), and consolidation of their work ("draw all pictures and information together"). When asked how critical the teacher thought students had been while collecting in the museum, the reply was: "Not very -they thought they were at the time but have changed their minds since". There was clearly not enough time to work with all the pictures collected (some students collected over 40 items), which prompted the teacher to schedule a second lesson for students to complete their galleries (although this lesson was not observed). The teacher estimated that a student can effectively process 5 to 10 items during one lesson. When asked how effective the post-visit lesson was the teacher responded: "It was good and constructive, valuable learning took place as they reviewed their work". Focus group students said they learned more about D-Day from the post-visit lesson as it gave them the opportunity to reflect on what they had collected ("thinking over it all again") -they do not want to "read it all and forget it" and remember it more because "it's more modern" rather than just "reading it all and then writing an essay".

Myartspace -at the macro level
The macro level examined the impact of Myartspace in terms of the effect it has on the educational practice for school museum trips: the impact on museums, and on teaching and learning. Key stakeholders (six teachers, three museum education officers, two members of the design team, and one representative from the funding agency) were surveyed or interviewed with respect to their experiences with the project and how they matched their initial expectations. Some of the interviews took place as part of the final full-scale trial and others as part of a survey of teachers who had used Myartspace outside evaluations. This section summarises the main results.

Impact on museums
Myartspace requires museums to digitise their contents and offer visitors additional material to work with. Visitors then have the opportunity to access this content after they leave the museum. One museum representative noted that "Potentially [Myartspace] has made our collections much more accessible." As the teacher in this trial suggested, and participants in previous trials have noted, Myartspace also encourages visitors to go back to the museum. Museum representatives see Myartspace as a means of attracting secondary level students: "The museum had a lack of resources to support secondary school age children. Myartspace was a way of attracting secondary schools into the museum." The main drawback mentioned by museum representatives for adopting Myartspace was costs, in particular the purchase of phones. The pilot phase of Myartspace began in February 2006; less than a year later some participating students found the phones "ok but not cutting edge". Thus, to retain the 'coolness' effect on the students, museums may need to upgrade the phones regularly. Museum staff members are concerned about the time and resources required to support such a service: "Our reasons for seriously considering whether to continue are the basics of finance and staff time. It is difficult to see how we could make it sustainable due to the running costs." "We would have liked to continue with the service but it is not financially viable for us within our current budgets. We would only consider continuing with Myartspace if it could be funded again." The costs of digitisation were not mentioned as a drawback by museum representatives. This is not surprising, as digitisation was part of the venues' contractual agreement and was compensated by the project budget for all the venues that participated in the study. But, like phone purchase costs, we expect digitisation costs to be an issue where external funding is not available.

Impact on teaching and learning
One museum representative summarised the benefits of Myartspace for schools: "It is more work for the teachers involved but those who have experienced Myartspace say that it is worth the effort. It enables the school to focus on the museum, with the subject area and the technology." Myartspace offers a better bridge between the museum and the classroom than traditional worksheets by allowing students to select artefacts from the museum and have them readily available back in the classroom, along with the museum's whole digital collection. One teacher commented on how Myartspace meant that the children's work would not be lost on the bus on the way back to school. Myartspace students have been observed to work with exhibits, asking questions "why do we want to collect this", deconstructing objects, and reflecting on their relevance to the learning task. These benefits are also indicated in the following quotes: "Using Myartspace, children have to engage with the exhibit and communicate with each other. It is important because children learn through engaging" (Interview, Museum representative) "Made me look at artwork more." (Interview, Student) "It was really enjoyable -students were extremely focused on task and got much more out of the visit than I had anticipated." (Interview, Teacher) It might have been possible to design a student-visitor experience with the educational benefits of Myartspace without the use of digital technology, for example by replacing object collection through the phones with collecting small leaflets about each exhibit, or replacing the gallery creation with the creation of paper-based collages. Such a non-technological approach would have advantages and disadvantages over the current implementation of Myartspace. However, a major advantage of the use of technology is the high motivation of the students, an effect appreciated by teachers, students and museum representatives, and summarised by a student as: "Most people think going to galleries is boring, but when you put ideas on a web site and use the phones it's much more fun." (Interview, Student)

Discussion
The evaluation described in the previous section has raised a number of issues about the deployment of Myartspace, many of which will be relevant to the deployment of other technologies that support school museum visits. These issues are now discussed. We should first consider possible effects of the evaluation activities on the collected data. Teachers participating in the trials were approached in advance of the visit and interviewed with respect to their expectations about, perceptions of, and plans for the use of the service -an act that may have prompted them to reflect further and may subsequently have positively affected the quality of the teaching. Nevertheless, the trials enabled us to document the potential of Myartspace, to observe problems that occur even with well motivated teachers, and offer some advice for improvements.

Adoption
A significant problem facing Myartspace for wider adoption was the additional load placed on the school and the teacher. Firstly, teachers commented that the cost of attending a training session plus the costs for a replacement teacher for the day can be too expensive for a school, while running sessions after school hours requires teachers to give up their personal time. More significantly, most teachers who used the Myartspace service outside evaluation trials did not run a pre-visit lesson, and some did not run a post-visit lesson. The most commonly cited reason for this was costs, both time (integrating Myartspace with their classroom teaching) and financial (attending museum training sessions on the use of the website), as well as teacher's ICT skill level and thus their ability to support their classes using Myartspace back at school. Improving the Teacher's Pack may help here. Nonetheless, increased levels of engagement and motivation of the students in the museum were still noted, but the opportunity to link the museum and school work directly was lost. It should be noted that teachers who performed more than one visit suggested that subsequent visits take a lot less time to organise and plan. The problem appears to lie primarily in the usability of the website. The results at the micro level indicate that the website is not intuitive to use (see section 4.5.1), especially for teachers with lower ICT skills, necessitating teacher training which is only available on-site. This presents an example of how usability of the service at the micro level affects its adoption and institutional use at the macro level.
A pre-visit lesson is important because the Myartspace experience requires a combination of skills, e.g. art, history, cross-curricular ICT, speaking and listening skills, group work and presentation skills. Preparing students is thus complicated, as it requires introducing them not only to the subject matter (e.g. D-Day), but also to the concept and processes of collecting and the Myartspace technology. These all need to be addressed in the pre-visit lesson to enable students to get the most out of their experience. We expect that the introduction to the learning topic will not normally have to be part of the pre-visit lesson, if the Myartspace visit is booked according to the school subject programme. However, even the remaining two teaching goals (to introduce the process of analysing sources and objects, and to introduce the Myartspace technology) are challenging for a single lesson. Ideally, teachers would be encouraged to plan more than one pre-visit lesson to cover these topics fully; given that in practice most teachers do not run any pre-visit lesson, it becomes critical that at least the technology is easy to 'pick up and use'.

The phones
A main attraction of Myartspace in the museum was the use of mobile phones. Mobile phones are popular among this age group, and the students can easily relate to their use. This may put mobile phone-based systems like Myartspace at an advantage in relation to other types of handheld museum guide for this age group. However, in assessing the motivational aspects of the technology, it should be borne in mind that this technology 'ages' perhaps even faster than other modern digital technologies. Most mobile phone service providers in the UK offer their contract customers the option to upgrade their phones every 18 months. This means that the 'lifespan' of a phone (in terms of desirability) is somewhat less than this 18 month contract period, possibly even less than a year.
The implications of such short-lived appeal can be substantial in terms of upgrade costs, and may affect the relationship between user satisfaction at the micro level and student motivation at the meso level.

Student collecting
The 562 objects created by the students, against the 75 collected using codes, suggest that students appreciate the sense of creativity and ownership that comes with creating their own representations of exhibits. However, it created another micro-level problem, this time affecting the learning experience at the meso level. Myartspace does not support the interlinking of objects other than presenting them in chronological order of collection, causing students to complain that they could not remember what certain photos they had taken were about, or relate photos to sound recordings or text comments. Given the students' appreciation of creativity and ownership, technical solutions at the micro level (e.g. enable interlinking of collected items) are more appropriate than constraining student activity at the meso level (e.g. by confining the use of photo taking). In addition, students in all trials thought that they did not have enough time in the post-visit lesson to process all the items they had collected in the museum. The teachers, too, acknowledged this by scheduling further lessons. The teacher in the final trial suggested 5 to 10 items as a realistic maximum for students to process in one lesson. Most students had more than this in their personal store, with some having over 40 items. It is possible that students were uncritical in collecting at the museum. However, this does not seem to have been the case: the teachers thought students interacted with the exhibits more, the students themselves were observed to discuss "why should we collect this item", and museum representatives noted that students spend a lot more time on their museum tour with Myartspace than without it. Another possibility is that the time required to critically examine and assess an item in the museum is less than that required to meaningfully process the item on the web site (i.e. difference in time required for interactions at the meso level in the museum and the classroom); and also that the time required to interact with the mobile phone application to collect an item is less than the time required to manipulate that item on the web site (i.e. differences in time required for interactions at the micro level in the museum and the classroom). Thus, although students could be very critical in collecting 40 items in one hour in the museum using the mobile phone, the same amount of time was not adequate to meaningfully process them on the website in the classroom. This discrepancy led to student frustration, had a negative effect on the learning experience, and may have put some users off Myartspace. While a technical fix at the micro level (improving the web site, or restricting the number of objects a student can collect) is possible, it should be noted that learning to regulate the collection of data to simplify analysis is a valuable skill. It may be more effective to place greater emphasis in the pre-visit lesson on the need for students to be selective in creating and collecting objects.

Group work
Another example of problems at the micro level affecting learning at the meso level is in group work. The focus groups showed that students within a group worked in either coordination or competition mode. In coordination mode, the group would agree who was going to collect an object / take a photo / record a sound, and what it would be. When in competition mode, each group member would perform the function individually. In cooperation mode they had more opportunities to negotiate meanings and tasks and to challenge and complement each other's views. Switching from collaborative work in the museum to individual work in the classroom generated a number of problems. The failure of Myartspace to provide tools for group work at the micro level in the classroom (e.g. in the form of group stores and galleries or multi-user accounts) jeopardises collaborative learning at the meso level.

The experience
One of the focus groups considered the process of writing or dictating an exhibit label into the phone a good way of learning (see section 4.4.2), suggesting that students did grasp the educational purpose of collecting items and rationalised the capturing of the text in the exhibit labels. However, capturing the content of the exhibit label differed from the service's objective for students to capture their own interpretations and reactions to exhibits alongside those offered by the museum through digitised content and print labels. This was not explicitly explained to the students. Instead, a combination of the teacher's emphasis on picture and sound capturing and the museum Education Officer's suggestion that the two may replace collecting through the codes, reinforced the capturing of the museum's interpretation. It is not clear which of these two ways of use would maximise the learning outcomes for students.
To an extent, this can be considered evidence of Myartspace's flexibility and adaptability. Nevertheless, informing the students about how the use of individual functions is expected to benefit their learning may broaden their options and give them choice and control.

Conclusions
The three-level evaluation framework that was presented in section 3.1 provided us with an efficient way to structure both the data collection and analysis for the evaluation of Myartspace. Successes and failures of Myartspace at all levels, micro, meso and macro have been identified, along with inter-level influences. Some of these were issues relevant to school visits to museums generally, such as the increased demands on teacher time and effort, and the division of costs to schools and museums. Others were specific to the implementation of the service, such as the usability problems.
Nevertheless, and despite the needed improvements, we were able to document the potential of such a service. The trials have shown that the Myartspace experience can successfully bridge the museumclassroom gap by facilitating the teacher's design of pre-and post-visit lessons, enabling students to create artefacts in the museum and have them readily available for further work in the classroom, and extending the museum context into the classroom through personal and museum digital collections. The students in the trials appreciated the use of modern technology and teachers remarked on how enjoyable the experience was for students, increasing their levels of motivation.
Observations showed students working with exhibits and asking questions about their relevance, deconstructing objects, and reflecting on their relevance to the learning task. Often learners adopt a piece of new educational technology in ways that the designers and educators did not expect. New tools that enable learners to perform new activities may change the way they perceive and carry out old activities. Thus, it is only through continuous evaluation and fine-tuning of the new technology with the learning practice (including adjustment of peripheral and contextual support, like lesson planning, IT support, and activity planning), that an educational innovation like Myartspace will reach its full potential in transforming educational practice.