For new readers and those who request to be “好友 good friends” ,  please read my 公告栏 first.

Recently, a reader asked me this question and myresponse:

Quote

[17]LSun777  2015-8-4 10:46

Dear Prof. Ho,

I just saw this piece (http://people.idsia.ch/~juergen/deep-learning-conspiracy.html) written by a seemingly angry computer scientist which  pointed that you were among the first to invent backpropagation but was not properly credited by LeCun, Bengio, Hinton's recent review on Nature.  I know assigning credit is usually a distraction from really important science, but I'm still glad to know your earlier insight could be of this enduring impact.  So are you aware the recent development in neural networks/deep learning?  It would be fantastic if you'd liketo comment on their possible connection to optimal control.  Thankyou very much.

Best,
Lei

博主回复(2015-8-422:51)：I just found the article via Google search. Let us  take some time to write an article about your query. This is not  something that can be answered by a couple ofsentences.

End of Quote

I am thus fulfilling my promise here.

Let us start at the beginning. The fact that we need to make decisions and  that we wish to make good (or optimal if possible) decisions all the time  are age old. We also learned early that to make good decisions you  need to utilize all the relevant information available. Furthermore, because  decision often have long term or long range (space and time) consequences,  we need information on such longterm/range (LTR)consequences. This brings in a severe complication since the LTR consequences for a problem may not always be simple. If each time when we make a decision we must carefully account for such LTR consequences, the labor may become excessive. Thus, not many optimal decision problems in the real world were solved until the middle of last century when computers came on the scenes. People also beginning to look for mathematical ways of reflecting future consequences  to the present so that they can be traded off with other current  information to form optimal decisions. The idea of “Backward Propagation” was born.The term was invented to explain the process of backward propagating  future consequences to the present. Many people  are involved with the “invention” of this idea. In connection with the method  of dynamic programming, people generally credit with the name of R. Bellman.  However, R. Isaacs deserves at least mention for independent discovery  (the situation here is very much similar to Darwin and Wallace concerning the  discovery  of evolution). From the viewpoint of optimal control and calculus of variation,  several people discovered Backward Propagation via the so called  “adjoint or multiplier differential equations”. The names are as reported in  the above “Nature”reference. However, my association was mainly due to my co-author Bryson in our book classic “Applied Optimal Control”.It is more proper to  say I was among the first user rather than discoverer. However, I do take  credit to give the BP idea the popular one sentence explanationas given in http://blog.sciencenet.cn/blog-1565-209522.html and above.

Once the idea of BP and of reflecting distant consequences to current time  and position was discovered people begin to realize that this general  idea is not limited to decision and control. You can apply it to any structure  or calculation that has an independent variable such as distance , time, or layers. It is in the area of calculation involving neural networks with many layers that the BP idea became “deep learning”.  More specifically deep learning has to do with the problem of training a many layered neural network to do AI tasks. BP or deep learning simplified the calculation task.

In this sense, the complaint by JürgenSchmidhuber  http://people.idsia.ch/~juergen/deep-learning-conspiracy.html   about the origin of deep learning  is basically correct. But I am not a researcher in neural networks to comment on other aspects of the subject and its successes/failures.

I always maintain that there are only a few simple but good  ideas under the sun.Different disciplines often discover or rediscover such ideas  and rename them to suit the use in their own disciplines. Over time,  such discovery and priority issues get forgotten or obscured and became  proper subject of research in the history of sciences.