MoMA.org is in the process of being redesigned. While it’s not unusual for a museum to tweak and even overhaul a website, it is the first time MoMA is using agile evaluation to help inform the redesign. Perhaps even more unique to this process, is the input from a group of “Audience Advocates” representing various departments at MoMA (including Digital Media, Education, Membership, Visitor Services, Management Information, and Marketing). Read more
Posts by Jackie Armstrong
In 2014, MoMA’s education, curatorial, graphic design, exhibition design, and marketing departments collaborated to develop an interactive learning space adjacent to the exhibition The Paris of Toulouse-Lautrec: Prints and Posters. This is the third interactive space we’ve developed in relation to an exhibition, following the success of Performing John Cage and the Polke Pop-Up Activity Space. This café-like space offers activities and resources to connect participants with Lautrec’s life and artistic process using both unfacilitated and facilitated approaches. Read more
If you happen to visit the exhibition Alibis: Sigmar Polke 1963–2010 on Tuesday afternoons you will notice something different: the sight of Museum visitors making art inspired by Sigmar Polke’s processes, in close proximity to his works of art. This shift toward more hands-on learning experiences is not something that happened overnight. Read more
Twice daily, from February 7 to 20, MoMA staff and invited artists performed John Cage’s score 4’33” in an area just outside the exhibition There Will Never Be Silence: Scoring John Cage’s 4’33”. Over the course of those two weeks, 28 renditions of 4’33” were performed by 20 staff members and eight guest artists. Read more
One thing you’ll never hear me say about working in MoMA’s Department of Education is “I’m bored.” In fact, what I love most about my role as a researcher and evaluator in this department is the constant interest on the part of my colleagues to experiment, innovate, and try new things. There’s always the desire to find ways to improve and/or to assess the current offerings. No one is ever comfortable with just leaving things as they are. It’s this collective dynamism that drives a lot of what the Department of Education does.
Recently, two of my colleagues in Family Programs expressed interest in trying out some iterations of their successful Tours for Fours program to see what tweaks would make that experience even more engaging for four-year-olds and their caregivers. Kristin Roeder, one of our amazing MoMA educators, was also keen to be a part of the experimentation and came up with variations on the more typical Tours for Fours tours. All of us were interested in looking at length of tour, number of works included, art works chosen, theme, activities, and variables in the group (size, ratio of children to adults, inclusion of younger siblings, etc.) to see if there was an ideal mix for this particular age group. In February we tried out three different variations of MoMA’s Tours for Fours.
Using Materials and Techniques as a theme and focusing on artists’ gestures, the three variations were:
1) Comparing and contrasting two works in the galleries
2) Focusing on one work in the galleries and doing a complementary activity in the classroom
3) Engaging with three works in the galleries and receiving a related activity at the end that families can do together at MoMA
Each of these was documented through observational notes and photographs. Prior to the start of the tour, I collected e-mail addresses from parents so that we could send a follow-up survey to find out what they thought about their experience. Following each tour, Family Program staff, the lead educator on the tour, and myself sat down to debrief (what worked, what didn’t) and to consider what might be worth trying going forward.
What we know so far based on observations, educator reflections, and feedback from parents is that:
• Tours should include at least three works during the 45 minutes. Adults expressed an interest in exposing their children to a variety of works and four-year-olds lost interest if too much time was spent on one work. For future iterations of this tour we may experiment by including four or five works during the 45 minutes.
• Including works that are familiar and unfamiliar to most adults is a good way to keep families engaged
• Themes that inspire curiosity and enable families to get into the head of the artist/be an artist are really effective (e.g. gesture)
• Hands-on activities are appreciated and present teaching opportunities that children will not only enjoy, but also remember
• For gallery-based programs like Tours for Fours, activities that take place in the galleries are best as getting to the classroom and getting an activity underway takes too much time away from the tour; families also expect to be in the galleries as classrooms do not offer the unique or immersive experience they are looking for
• Involvement of the adult caregivers is key to a successful tour; ideally, adults and children should be thinking and talking about art together
• Younger children (particularly siblings) who come along on the tour often distract the four-year-olds and/or cause disturbances that upset the tour, however we realize that families generally like to stick together so we need to find more effective ways of addressing these realities
• Providing a little activity for families to take away at the end of the tour is an appreciated gesture, and for families that do the activities it really adds to their time at MoMA
In March, we are planning to test out more variations of Tours for Fours. We hope that this loop of iterative testing will help us create a tour that best matches the needs and abilities of four-year-old visitors (and their adult caregivers).
Have you been to a MoMA Tours for Fours program? We’d love to hear from you!
Following over a year of research and development by a cross-departmental team, MoMA Audio+ mobile guide debuted in July 2013, replacing the handheld audio guides that the Museum had been distributing for many years. MoMA Audio+ enables visitors not only to listen to audio commentaries and read interpretive texts about works of art, but also offers the ability to take and share photographs, locate works of art and facilities, and e-mail yourself a record of your visit using the My Path feature.
Testing and assessment has been key to the process of developing MoMA Audio+. Built using a method called agile development, in the months leading up to launch, the development team released and tested functional versions of the app as it was being built. Based on testing and feedback, we could iteratively implement improvements to the user interface and features to better enhance visitor experience. After launch, we conducted assessment from a variety of perspectives: visitors returning the mobile guide were interviewed about their experience, Antenna staff distributing the mobile guide were surveyed, and user-testing sessions were conducted.
How Does User Testing Work?
Nine individuals were recruited to participate in user testing. Participants represented a mix of ages and genders, as well as varying interests, work experiences, and technical abilities. User testing took place in December over the course of three mornings in the American Modern exhibition. Three participants were scheduled for each 60- to 90-minute session, and for every participant there was a lead facilitator and a cofaciliator. The lead facilitator guided the participant through the directed-testing instrument, prompted and asked for clarification as needed, and took notes. The cofaciliator also prompted as needed, asked questions, and took notes. For each session, there was also at least one additional staff person floating between groups and observing. Participants were asked to think out loud and voice any questions or concerns as much as much as possible during the testing.
So, what did we learn?
A lot! I’ve been working in museum evaluation for a few years and am constantly amazed by the amount and depth of information that even a small group of user-testing participants can provide. While some of the insights offered confirmed our own thoughts about the mobile guide, there were also plenty of things they pointed out that we did not realize were barriers to a smooth user experience. Along with critical feedback, it was also great to hear what people enjoyed about using MoMA Audio+.
Here are some of the overall findings from the user testing sessions:
What did users really like about MoMA Audio+?
• The amount and variety of content available for them to access
• The high quality images presented on the device
• The camera feature, My Path feature, and other functions that allowed them to personalize and share their experience at MoMA
What aspects of MoMA Audio+ did users find difficult or confusing?
• Locating the How To use MoMA Audio+ instructions
• Understanding and locating all the layers of content and features available
• Locating specific artworks using the map
We really appreciated all the critical feedback these user testing participants offered because it helps us find ways to improve on what we currently offer visitors. Of course, we also loved that 89% (N=9) felt that MoMA Audio+ is something that would enhance their experience in the galleries. A few comments from participants included:
• “I feel that MoMA Audio+ enhances what I came to see here. Thousands of things on view and this helps me to shape my visit.”
• “It added another dimension. Nothing replaces the act of looking, but knowing more about what I’m seeing enriches the experience. This achieves that goal. Devices are not too heavy, not burdensome, not ugly, nice MoMA graphic.”
• “Usually, I just want to be with the art and wander around but I really like the ability this device gives you to document what you see during your visit. Personalizes the experience”
Have you tried MoMA Audio+ on one of your visits to the Museum yet? If you did, we’d love to know what you thought!
Since I began working at the Museum, every MoMA Studio has undergone a complete evaluation. Evaluation strategies include interviewing visitors, surveying participants, observing/tracking/timing visitors, using prompts to encourage responses on comment boards, facilitator reflections, and a few other participatory forms of data gathering. Read more
Recently, Family Programs staff were interested in testing out some of the activities under consideration for MoMA Art Lab: Movement before it opened (on October 10, 2013). Formative evaluation is a “try it out” method that is less formal than other evaluation types. Read more
When visiting a museum, especially in New York City, it’s easy to wander around without pausing to look at specific works of art. After all, there’s so much to see and crowds to contend with. Read more