1. Title of the paper
Evolving Deep Neural Networks


2. Authors
Risto Miikkulainen, Jason Liang, Elliot Meyerson, Aditya Rawal, Dan Fink, Olivier Francon, Bala Raju, Hormoz Shahrzad, Arshak Navruzyan, Nigel Duffy, Babak Hodjat

Sentient Technologies, Inc., One California St., Suite 2300, San
Francisco, CA 94111, +1 415-422-9886, firstname.fastname@sentient.ai.
Risto Miikkulainen, Jason Liang, Elliot Meyerson, and Aditya Rawal also at
The University of Texas at Austin, Austin TX 78712


3. Corresponding author
Risto Miikkulainen; risto@cs.utexas.edu


4. Abstract

The success of deep learning depends on finding an architecture to fit
the task.  As deep learning has scaled up to more challenging tasks,
the architectures have become difficult to design by hand. This paper
proposes an automated method, CoDeepNEAT, for optimizing deep learning
architectures through evolution. By extending existing neuroevolution
methods to topology, components, and hyperparameters, this method
achieves results comparable to best human designs in standard
benchmarks in object recognition and language modeling. It also
supports building a real-world application of automated image
captioning on a magazine website. Given the anticipated increases in
available computing power, evolution of deep networks is promising
approach to constructing deep learning applications in the future.


5. Criteria satisfied
B, C, D, F, G, H


6. Justification of criteria satisfied

The entry shows that evolution can be used to design complex learning
systems such as deep learning (DL) neural networks better than humans
can.  Starting from a well-known such an architecture, the "Show and
Tell" network for generating image captions, evolution optimized its
topology, components, and hyperparameters, and improved its
performance in all metrics in the standard MSCOCO dataset.

In terms of the criteria:

(B) The network architecture designed by evolution performs better
than the "Show and Tell" network (Vinyals et al. 2015) in the image
captioning task across all metrics. The best evolved network is 5%
better; an ensemble of three best evolved networks is 10% better.

(C) The results of the 2015 MSCOCO image captioning competition are at
http://mscoco.org/dataset/#captions-leaderboard; the "Show and Tell"
result is the best of those.

(D) New and better DL architectures are routinely published in venues
such as CVPR, NIPS, ICML, ICLR, etc. The evolved architecture in this
entry is innovative because it utilizes repetitive components and
multiple parallel pathways.

(F) The "Show and Tell" architecture tied for 1st place in the 2015
image captioning competition (which was a significant achievement).

(G) Designing DL architecture is increasingly difficult. Many recent
papers in CVPR, NIPS, ICML, ICLR etc. focus on new such designs.

(H) The entry in this paper would have won the 2015 MSCOCO image
captioning competition with 15 entries from various deep learning
research groups around the world.


O. Vinyals, A. Toshev, S. Bengio, and D. Erhan. (2015). Show and tell: A
neural image caption generator. In Proceedings of CVPR. 3156–3164.


7. Citation
Miikkulainen, R, Liang, J., Meyerson, E., Rawal, A, Fink, D., Francon,
O., Raju, B., Shahrzad, H., Navruzyan, A., Duffy, N., Hodjat,
B. (2017). Evolving Deep Neural Networks.   CoRR abs/1703.00548,
http://arxiv.org/abs/1703.00548.


8.
Prize money, if any, will be divided equally among the co-authors.


9. Why this entry is the best

This entry highlights a new genre of human-competitive results where
the object of design is itself a learning system. As pointed out by
Holger Hoos in his GECCO'16 keynote, many systems have become too
complex for humans to optimize; automated methods such as evolution
are necessary to obtain full benefit from them. With recent advances
in deep learning (DL), machine learning has also reached this limit.
DL systems now have hundreds of layers, repetitive structures
consisting of dozens of components, complex connectivities, and many
component types, all to be configured with hundreds of
hyperparameters.  They have reached the limit of human design and
optimization.

As this entry shows, such systems can be configured successfully by
evolutionary algorithms, beyond human ability to do so. This approach
is computationally extremely demanding. Training each network takes
days on a state-of-the-art GPU, and during the course of evolutionary
optimization, thousands (possibly millions) of such networks need to
be evaluated. Such computational power is only now becoming available,
and it already makes human-competitive results like the one in this
entry possible. Interestingly, as computational power increases
further, there are very few approaches that can take advantage of such
power---but evolution of DL systems can!  The entry thus demonstrates
that "AI designing AI" is the future of AI.


10. Type of EC used
Genetic Algorithms