Towards Integrating Information and Control Theories: From Information-Theoretic Measures to System Performance Limitations 

信息論與控制論之融合：從信息論測度到系統性能局限

链接：https://scholars.cityu.edu.hk/en/theses/towards-integrating-information-and-control-theories-from-informationtheoretic-measures-to-system-performance-limitations(8f21be24-127e-495d-a0cc-4a6e728450ee).html

---

其中引用了不少有意思的 quotes（也可参考：http://blog.sciencenet.cn/blog-286797-1022865.html 以及 http://blog.sciencenet.cn/blog-286797-1021452.html）：

Frontispiece

• When one submerges the gourd bowl in water, there floats the gourd ladle.

— Chinese proverb

Chapter 1

• There is an obvious analogy between the problem of smoothing the data to eliminate or reduce the effect of tracking errors and the problem of separating a signal from interfering noise in communications systems.

— R. B. Blackman, H. W. Bode, and C. E. Shannon, “Data Smoothing and Prediction in Fire-Control Systems,” 1946

• (We) become aware of the essential unity of the set of problems centring about communication, control, and statistical mechanics, whether in the machine or living tissue... We have decided to call the entire field of control and communication theory, whether in the machine or the animal, by the same Cybernetics.

— N. Wiener, “Cybernetics,” 1948,

• Fundamental limits are actually at the core of many fields of engineering, science and mathematics... Firstly, they evolve from basic axioms about the nature of the universe. Secondly, they describe inescapable performance bounds that act as benchmarks for practical systems. And thirdly, they are recognized as being central to the design of real systems.

— M. M. Seron, J. H. Braslavsky, and G. C. Goodwin, “Fundamental Limitations in Filtering and Control,” 1997

Chapter 2

• The sciences do not try to explain, they hardly even try to interpret, they mainly make models. By a model is meant a mathematical construct which, with the addition of certain verbal interpretations, describes observed phenomena. The justification of such a mathematical construct is solely and precisely that it is expected to work.

— John von Neumann

• The only way of discovering the limits of the possible is to venture a little way past them into the impossible.

— Arthur C. Clarke

• A theory is the more impressive the greater the simplicity of its premises, the more different kinds of things it relates, and the more generalized its area of applicability. Therefore the deep impression that classical thermodynamics made upon me. It is the only physical theory of universal content which I am convinced will never be overthrown, within the framework of applicability of its basic concepts.

— Albert Einstein

Chapter 3

• The idea of a statistical message source is central to Shannon’s work. The study of random processes had entered into communication before his communication theory. There was a growing understanding of and ability to deal with problems of random noise... Wiener had dealt extensively with the extrapolation, interpolation, and smoothing of time series. Although Wiener’s book was published in 1949, it had been available earlier in a wartime version known as the Yellow Peril (the cover was yellow). Shannon and Bode took considerable pains to put Wiener’s work in a form more directly useful to them (and to many others).

— J. R. Pierce, “The Early Days of Information Theory,” 1973

• We said before: “It feeds upon negative entropy,” attracting, as it were, a stream of negative entropy upon itself, to compensate the entropy increase it produces by living and thus to maintain itself on a stationary and fairly low entropy level.

— Erwin Schrodinger, “What is Life,” 1944

• If one has really technically penetrated a subject, things that previously seemed in complete contrast, might be purely mathematical transformations of each other.

— John von Neumann

Chapter 4

• However, by building an amplifier whose gain is deliberately made, say 40 decibels higher than necessary (10000 fold excess on energy basis), and then feeding the output back on the input in such a way as to throw away that excess gain, it has been found possible to effect extraordinary improvement in constancy of amplification and freedom from nonlinearity.

— H. S. Black, “Stabilized Feedback Amplifiers,” 1934

• In control and communication we are always fighting nature’s tendency to degrade the organized and to destroy the meaningful; the tendency, as Gibbs has shown us, for entropy to increase.

— N. Wiener, “The Human Use of Human Beings,” 1950

• All stable processes we shall predict. All unstable processes we shall control.

— John von Neumann

Chapter 5

• What’s in a name? In the case of Shannon’s measure the naming was not accidental. In 1961 one of us (Tribus) asked Shannon what he had thought about when he had finally confirmed his famous measure. Shannon replied: “My greatest concern was what to call it. I thought of calling it ‘information’, but the word was overly used, so I decided to call it ‘uncertainty’. When I discussed it with John von Neumann, he had a better idea. Von Neumann told me, ‘You should call it entropy, for two reasons. In the first place you uncertainty function has been used in statistical mechanics under that name. In the second place, and more importantly, no one knows what entropy really is, so in a debate you will always have the advantage.’”

— M. Tribus, E. C. McIrvine, “Energy and information,” 1971

• The bottom line for mathematicians is that the architecture has to be right. In all the mathematics that I did, the essential point was to find the right architecture. It’s like building a bridge. Once the main lines of the structure are right, then the details miraculously fit. The problem is the overall design.

— Freeman Dyson

• Far waters fail to quench near fires.

— Chinese Proverb

Chapter 6

• I like to think of Bode’s integrals as conservation laws. They state precisely that a certain quantity—the integrated value of the log of the magnitude of the sensitivity function—is conserved under the action of feedback. The total amount of this quantity is always the same. It is equal to zero for stable plant/compensator pairs, and it is equal to some fixed positive amount for unstable ones... This applies to every controller, no matter how it was designed. Sensitivity improvements in one frequency range must be paid for with sensitivity deteriorations in another frequency range, and the price is higher if the plant is open-loop unstable.

— G. Stein, “Respect the Unstable,” 2003

• The average performance of any pair of algorithms across all possible problems is identical. This means in particular that if some algorithm’s performance is superior to that of another algorithm over some set of optimization problems, then the reverse must be true over the set of all other optimization problems.

— D. H. Wolpert, W. G. Macready, “No Free Lunch Theorems for Optimization,” 1997

• We know the past but cannot control it. We control the future but cannot know it.

— Claude Shannon

Chapter 7

• Consider the case where you are the controller and you observe samples of the process output whose average has been satisfactorily close to set point and that suffers only from white noise disturbances. Should you make an adjustment to the control output upon observing a sample of the process output that is not on set point? If the average of the process output is indeed nearly at the set point then any deviation, if it is really white or unautocorrelated, will be completely independent of the previous value of the control output and it will have no impact on subsequent disturbances. Therefore, if you should react to such a deviation, you would be wasting your time because the next observation will contain another deviation that has nothing to do with the previous deviation on which you acted. You, in fact, may make things worse... A feedback controller cannot decrease the standard deviation of the white noise riding on the process output. At best it can keep the average on set point.

— D. M. Koenig, “Practical Control Engineering,” 2009

• In respect of military method, we have, firstly, Measurement; secondly, Estimation of quantity; thirdly, Calculation; fourthly, Balancing of chances; fifthly, Victory.

— Sun Tzu, “The Art of War”

• All the evidence shows that God was actually quite a gambler, and the universe is a great casino, where dice are thrown, and roulette wheels spin on every occasion. Over a large number of bets, the odds even out and we can make predictions... But over a very small number of rolls of the dice, the uncertainty principle is very important.

— Stephen Hawking

Chapter 8

• The world is continuous, but the mind is discrete.

— David Mumford

• Time is defined so that motion looks simple.

— John Wheeler

• If everything seems under control, you’re just not going fast enough.

— Mario Andretti

Chapter 9

• Essentially, all models are wrong, but some are useful.

— George. E. P. Box

• In theory, theory and practice are the same. In practice, they are not.

— Albert Einstein

• Any one who considers arithmetical methods of producing random digits is, of course, in a state of sin.

— John von Neumann

Chapter 10

• An understanding of fundamental limitations is an essential element in all engineering. Shannon’s early results on channel capacity have always had center court in signal processing. Strangely, the early results of Bode were not accorded the same attention in control.

— K. J. Astrom, in G. Stein, “Respect the Unstable,” 2003

• If I turn toward a science not for external reasons such as earning an income, or for ambition, and also not—at least not exclusively—for the mere sportive joy and the fun of brain-acrobatics, then I must ask myself the question: what is the final goal that the science I am devoted to will and can reach? To what extent are its general results “true”? What is essential and what is based only on accident in its development?

— Albert Einstein

• As a mathematical discipline travels far from its empirical source, or still more, if it is a second and third generation only indirectly inspired by ideas coming from “reality” it is beset with very grave dangers. It becomes more and more purely aestheticizing, more and more purely I’art pour I’art. This need not be bad, if the field is surrounded by correlated subjects, which still have closer empirical connections, or if the discipline is under the influence of men with an exceptionally well-developed taste. But there is a grave danger that the subject will develop along the line of least resistance, that the stream, so far from its source, will separate into a multitude of insignificant branches, and that the discipline will become a disorganized mass of details and complexities. In other words, at a great distance from its empirical source, or after much “abstract” inbreeding, a mathematical subject is in danger of degeneration. At the inception the style is usually classical; when it shows signs of becoming baroque, then the danger signal is up... In any event, whenever this stage is reached, the only remedy seems to me to be the rejuvenating return to the source: the re-injection of more or less directly empirical ideas. I am convinced that this was a necessary condition to conserve the freshness and the vitality of the subject and that this will remain equally true in the future.

— John von Neumann