Brian Magerko
As both a cognitive scientist and a computer scientist, I have always been keenly interested in the formal representations of thought. Cognitive processes, knowledge, artificial intelligence—these are the phenomena that fascinate me.
What I had not noticed as a student was how consistently, when left to my own devices, my studies in cognition and computation veered off the steady path and swirled together in projects with a heavy humanities focus. Senior year: a robot comedy improv troupe; Advanced Artificial Intelligence (AI) course: a neural network for recognizing chorales written by J.S. Bach; dissertation work: an AI approach to creating interactive narratives.
EarSketch is a multi-disciplinary research project, funded by the National Science Foundation, that explores how computational thinking can be taught through authentic artistic practices.
There is a sublime beauty in the intersection between cognition and computation I had not realized in my early years. I had been too focused on games being the “it” medium in which to work, rather than looking at games—and digital media in general—as a means to an end—one of many trees in the forest. Three years into my career at Georgia Tech, when required to provide narrative to describe my career path for peer review, I had an opportunity to look beneath the surface and genuinely consider why I do what I do. Upon reflecting on my past and future work, it became obvious that creativity was the glue holding everything together: Understanding human creativity better, formally representing human creativity in computational systems, and supporting human creativity to support engagement and learning in digital media artifacts.
Creativity as a formal concept is the siren call of a phenomenon ridiculously hard to study and even harder to formally represent in computational systems. Still, creativity is one of those parts of the human experience that makes life so incredibly rich on a daily basis. It leads to surprise, to emotion, to connecting with others in a visceral and sacred way. It seems only natural to explore creativity as a first class phenomenon in the kinds of systems—cognitive systems—in which it exists. How are people creative? How can people be creative with computing to improve their cognition and learn? How can computers be creative? How can computers and humans be creative together to shape new creative paradigms and domains? Answering questions regarding the overlap between computing, cognition, and creativity is at the heart of the work we do in the Adaptive Digital Media (ADAM) Lab at Georgia Tech.
The research we engage in runs the spectrum from studying human creativity, creating computational representations of creative cognitive processes, to doing practice-based research that engages human creativity and cognition while becoming computationally literate. We study improvisational actors and pairs of people playing “pretend” as a way to better understand how people generate shared meaning together and how they collaboratively perform and unravel a story through their interactions over time. For example: We built a movement-installation piece built on the back of contemporary movement theory in dance and theater. And even our educational media work relies heavily on the potential of computing as a creative form to inspire and engage learners, like our EarSketch project described below.
EarSketch is one of the most surprising and oddly successful works to come out of the ADAM Lab. Rather than focusing on understanding human creativity to computationally represent it, this work engages learners in a creative practice—remixing music—by writing programming code. EarSketch is a multi-disciplinary research project, funded by the National Science Foundation, that explores how computational thinking can be taught through authentic artistic practices. We use computational music remixing (i.e., using code to manipulate loops, beats, and effects) as a means of captivating high school students’ interest in traditional introductory computer science practices. This involves a collaboration between academics; Atlanta high school teachers and policy makers; evaluators; and music industry professionals, such as Young Guru (Jay-Z’s DJ and producer) and Richard Devine (internationally known electronic musician and sound designer).
The EarSketch environment consists of a Python code editing environment, a pedagogically appropriate Application Programming Interface (API) for remixing, a digital audio workstation, and a social media sharing site where students can share their projects and remix with each other’s code. Students apply core computing concepts such as iteration, recursion, user-defined functions—in a domain that is both technical and expressive. And they learn basic aesthetic concepts in musical composition hand-in-hand with computing concepts.
This work has been piloted at multiple Atlanta-area schools and has been shown to both effectively teach computing principles as well as serve as a highly engaging learning domain for students who are traditionally underrepresented in computer science. It has also been used in an massively-online course on music technology, with over 10,000 students registered. While EarSketch has succeeded in teaching computing and increase student interest in computing as a discipline in general, success rates were even higher for underrepresented populations across gender and ethnicity. Women in particular are strongly motivated by the EarSketch experience and often change their attitudes about computing and their role in it.
EarSketch wants to offer an innovative, authentic learning experience for high school students that is both effective and appealing to students of different gender, ethnicity, and class backgrounds. We believe it has great potential to be used nationwide as a tool for learning computation and music technology. The curriculum and software are all freely available at http://earsketch.gatech.edu.
I would contend that EarSketch is the most symbolic work to come out of the ADAM lab in terms of showing the relevance of the arts and humanities to computing. By tightly integrating music production and programming, computing becomes more relevant to learners, especially those who may be turned off by it otherwise. There is something powerful about that combination that cannot be captured by music or programming alone; treating the computer as an expressive medium is as significant a mental turn as thinking of the typewriter as a medium for writing stories or canvas for painting. Computing has the potential for being a new humanistic domain, in which creative practice involves computation as a way to communicate vision, as a collaborator, and as a means for forcing us to better understand ourselves, to formally consider why and how we engage in our various creative activities.