Science AMA Series: We are Stanford neuroengineers who created a neural
prosthesis for monkeys to type Shakespeare with their “minds”. AUA!
Abstract
We are Paul Nuyujukian MD PhD (Postdoc, soon to be Bioengineering
faculty at Stanford) and Jonathan Kao PhD (Postdoc, soon to be
Electrical Engineering faculty at UCLA), neuroengineers in the Stanford
Neural Prosthetics Systems Laboratory, which is directed by Professor
Krishna Shenoy PhD. We just published a paper in the Proceedings of IEEE
in which we demonstrated a high-performance neural prosthesis where
monkeys transmitted text one character at a time at a rate of up to 12
words per minute. Video of monkey typing Before we get ahead of
ourselves, let us assure you that monkeys don’t understand English. In
the video above, the monkeys only saw the green and yellow dots, and not
the black letters (which were added afterwards in post-production as a
visual aid). The game engine prompted the green targets in a specific
sequence that if the monkeys got correct, would spell out words and
sentences that we can all understand. The video above was a selection
from Hamlet, but the primary data of the paper were articles from the
New York Times that the monkeys were asked to “transcribe.” All they
are doing though is navigating the white cursor to the green target at
every trial, and earning a liquid reward for each success. The ability
of the monkeys to control the cursor with their brain was accomplished
via a brain-machine interface (BMI). BMIs are systems that record from
the brain and translate these measurements to useful control signals,
which could be used to control a robotic limb, wheelchair, or, as was in
this case, a computer cursor. In this case, the BMI has similar
functionality as a one-button computer mouse: it can move in two
dimensions and click. The hardware interfaces used in the BMI, neural
electrodes (the one we used was the Utah mulitelectrode array), are not
new. They have been around for decades. What is new are the algorithms
that translate (or, as we refer to them, “decode”) the brain signals
into movement of the cursor. The machine learning decoding algorithms
used in this study were ones that we developed recently (cursor movement
and click decoders) that significantly improve the performance of
communication BMIs, enabling our monkeys to achieve rates that are 2-3
times faster than rates achievable with prior algorithms. There is tons
more we could write about (algorithm details, clinical trials that these
findings have resulted in, what other medical conditions BMIs may help
with, etc), but we’ll stop here and open it up to you all for questions.
We look forward to answering as many of your questions as possible! 2PM
PT - We are live! 6PM PT - We are done, thank you for the great
questions! More videos: Dwell typing Click typing Media coverage:
Stanford press release IEEE Spectrum NPR KQED - Future of You The Verge
Wired UK