Relative Information

1. Relative Information - What For?.- 1.1 Information Theory, What Is It?.- 1.1.1 Summary of the Story.- 1.1.2 Communication and Information.- 1.2 Information and Natural Language.- 1.2.1 Syntax, Semantics, Lexeme.- 1.2.2 Information, Learning, Dislearning.- 1.3 Prerequisites for a Theory of Information.- 1.3.1 Relativity of Information.- 1.3.2 Negative Information.- 1.3.3 Entropy of Form and Pattern.- 1.3.4 Information and Thermodynamics.- 1.3.5 Information and Subjectivity.- 1.4 Information and Systems.- 1.4.1 A Model of General Systems.- 1.4.2 A Model of Relative Information.- 1.4.3 A Few Comments.- 1.5 How We Shall Proceed.- 1.5.1 Aim of the Book.- 1.5.2 Subjective Information and Relative Information.- 1.5.3 Minkowskian Observation of Events.- 1.5.4 A Unified Approach to Discrete Entropy and Continuous Entropy.- 1.5.5 A Word of Caution to the Reader.- 2. Information Theory - The State of the Art.- 2.1 Introduction.- 2.2 Shannon Measure of Uncertainty.- 2.2.1 The Probabilistic Framework.- 2.2.2 Shannon Informational Entropy.- 2.2.3 Entropy of Random Variables.- 2.3 An Intuitive Approach to Entropy.- 2.3.1 Uniform Random Experiments.- 2.3.2 Non Uniform Random Experiments.- 2.4 Conditional Entropy.- 2.4.1 Framework of Random Experiments.- 2.4.2 Application to Random Variables.- 2.5 A Few Properties of Discrete Entropy.- 2.6 Prior Characterization of Discrete Entropy.- 2.6.1 Properties of Uncertainty.- 2.6.2 Some Consequences of These Properties.- 2.7 The Concept of Information.- 2.7.1 Shannon Information.- 2.7.2 Some Properties of Transinformation.- 2.7.3 Transinformation of Random Variables.- 2.7.4 Remarks on the Notation.- 2.8 Conditional Transinformation.- 2.8.1 Main Definition.- 2.8.2 Some Properties of Conditional Transinformation.- 2.8.3 Conditional Transinformation of Random Variables.- 2.9 Renyi Entropy.- 2.9.1 Definition of Renyi Entropy.- 2.9.2 Meaning of the Renyi Entropy.- 2.9.3 Some Properties of the Renyi Entropy.- 2.10 Cross-Entropy or Relative Entropy.- 2.10.1 The Main Definition.- 2.10.2 A Few Comments.- 2.11 Further Measures of Uncertainty.- 2.11.1 Entropy of Degree c.- 2.11.2 Quadratic Entropy.- 2.11.3 R norm Entropy.- 2.11.4 Effective Entropy.- 2.12 Entropies of Continuous Variables.- 2.12.1 Continuous Shannon Entropy.- 2.12.2 Some Properties of Continuous Entropy.- 2.12.3 Continuous Transinformation.- 2.12.4 Further Extensions.- 2.13 Hatori's Derivation of Continuous Entropy.- 2.14 Information Without Probability.- 2.14.1 A Functional Approach.- 2.14.2 Relative Information.- 2.15 Information and Possibility.- 2.15.1 A Few Prerequisites.- 2.15.2 A Measure of Uncertainty Without Probability.- 2.16 Conclusions.- 3. A Critical Review of Shannon Information Theory.- 3.1 Introduction.- 3.2 On the Invariance of Measures of Information.- 3.3 On the Modelling of Negative Transinformation.- 3.3.1 Classification of Terms.- 3.3.2 The Problem of Modelling "True" and "False".- 3.3.3 Error-Detecting Codes.- 3.4 On the Symmetry of Transinformation.- 3.4.1 A Diverting Example.- 3.4.2 Application of Information Theory.- 3.4.3 On the Symmetry of Transinformation.- 3.4.4 On a Possible Application of Renyi Entropy.- 3.5 Entropy and the Central Limit Theorem.- 3.5.1 The Central Limit Theorem.- 3.5.2 An Information Theoretical Approach to the Central Limit Theorem.- 3.5.3 Relation with Thermodynamics.- 3.5.4 Continuous Entropy Versus Discrete Entropy.- 3.6 On the Entropy of Continuous Variables.- 3.6.1 The Sign of the Continuous Entropy.- 3.6.2 A Nice Property of Continuous Entropy.- 3.7 Arguments to Support Continuous Entropy.- 3.7.1 On the Negativeness of Continuous Entropy.- 3.7.2 On the Non-invariance of Continuous Entropy.- 3.7.3 Channel Capacity in the Presence of Noise.- 3.8 The Maximum Entropy Principle.- 3.8.1 Statement of the Principle.- 3.8.2 Some Examples.- 3.9 Arguments to Support the Maximum Entropy Principle.- 3.9.1 Information Theoretical Considerations.- 3.9.2 Thermodynamic Considerations.- 3.9.3 Axiomatic Derivation.- 3.9.4 A Few Comments.- 3.10 Information, Syntax, Semantics.- 3.10.1 On the Absolute Nature of Information.- 3.10.2 Information and Natural Language.- 3.11 Information and Thermodynamics.- 3.11.1 Informational and Thermodynamic Entropy.- 3.11.2 Thermodynamic Entropy of Open Systems.- 3.12 Conclusions.- 4. A Theory of Relative Information.- 4.1 Introduction.- 4.2 Observation, Aggregation, Invariance.- 4.2.1 Principle of Aggregation.- 4.2.2 Principle of Invariance.- 4.2.3 A Few Comments.- 4.3 Observation with Informational Invariance.- 4.4 Euclidean Invariance.- 4.4.1 Orthogonal Transformation.- 4.4.2 Application to the Observation of Probabilities.- 4.4.3 Application to the Observation of Classes.- 4.5 Minkowskian Invariance.- 4.5.1 Lorentz Transformation.- 4.5.2 Application to the Observation of Probabilities.- 4.5.3 Application to the Observation of Classes.- 4.6 Euclidean or Minkowskian Observation?.- 4.6.1 Selection of the Observation Mode.- 4.6.2 Application to the [Uncertainty, Information] Pair.- 4.7 Information Processes and Natural Languages.- 4.7.1 On the Absoluteness of Information.- 4.7.2 The Information Process.- 4.7.3 Natural Language.- 4.7.4 Information and Natural Language.- 4.8 Relative Informational Entropy.- 4.8.1 Introduction to the Relative Observation.- 4.8.2 Informational Invariance of the Observation.- 4.8.3 Relative Entropy.- 4.8.4 Comments and Remarks.- 4.9 Conditional Relative Entropy.- 4.9.1 Relative Entropy of Product of Messages.- 4.9.2 Composition Law for Cascaded Observers.- 4.9.3 Relative Entropy Conditional to a Given Experiment.- 4.9.4 Applications to Determinacy.- 4.9.5 Comparison of H(?/?) with Hr(?/?).- 4.10 On the Meaning and the Estimation of the Observation Parameter.- 4.10.1 Estimation of the Observation Parameter.- 4.10.2 Practical Meaning of the Observation Parameter.- 4.10.3 On the Value of the Observation Parameter u(R).- 4.11 Relative Transinformation.- 4.11.1 Derivation of Relative Transinformation.- 4.11.2 Some Properties of the Relative Transinformation.- 4.11.3 Relative Entropy and Information Balance.- 4.11.4 Application to the Encoding Problem.- 4.12 Minkowskian Relative Transinformation.- 4.12.1 Definition of Minkowskian Relative Transinformation.- 4.12.2 Some Properties of Minkowskian Relative Transinformation.- 4.12.3 Identification via Information Balance.- 4.13 Effect of Scaling Factor in an Observation with Informational Invariance.- 4.14 Comparison with Renyi Entropy.- 4.14.1 Renyi Entropy and Relative Entropy.- 4.14.2 Transinformation of Order c and Relative Transinformation.- 5. A Theory of Subjective Information.- 5.1 Introduction.- 5.2 Subjective Entropy.- 5.2.1 Definition of Subjective Entropy.- 5.2.2 A Few Remarks.- 5.3 Conditional Subjective Entropy.- 5.3.1 Definition of Conditional Subjective Entropy.- 5.3.2 Application to Determinacy.- 5.3.3 A Basic Inequality.- 5.4 Subjective Transinformation.- 5.4.1 Introduction.- 5.4.2 Subjective Transinformation.- 5.4.3 A Few Properties of Subjective Transinformation.- 5.4.4 Application to Independent Random Experiments.- 5.5 Conditional Subjective Transinformation.- 5.5.1 Definition.- 5.5.2 A Few Properties of Subjective Conditional Transinformation.- 5.6 Information Balance.- 5.6.1 Optimum Conditions for Information Balance.- 5.6.2 Non-optimum Conditions for Information Balance.- 5.7 Explicit Expression of Subjective Transinformation.- 5.7.1 Discrete Probability.- 5.7.2 Continuous Probability.- 5.8 The General Coding Problem.- 5.8.1 Preliminary Remarks.- 5.8.2 The General Coding Problem.- 5.8.3 On the Problem of Error Correcting Codes Revisited.- 5.9 Capacity of a Channel.- 5.9.1 The General Model.- 5.9.2 Channel with Noise.- 5.9.3 Channel with Noise and Filtering.- 5.10 Transinformation in the Presence of Fuzziness.- 5.10.1 On the Entropy of a Fuzzy Set.- 5.10.2 Application of Subjective Transinformation.- 5.10.3 The Brillouin Problem.- 5.11 On the Use of Renyi Entropy.- 5.11.1 Renyi Entropy and Subjective Entropy.- 5.11.2 Transinformation of Order c and Shannon Transinformation.- 6. A Unified Approach to Discrete and Continuous Entropy.- 6.1 Introduction.- 6.2 Intuitive Derivation of "Total Entropy".- 6.2.1 Preliminary Definitions and Notation.- 6.2.2 Physical Derivation of He (X).- 6.3 Mathematical Derivation of Total Entropy.- 6.3.1 The Main Axioms.- 6.3.2 Derivation of Total Entropy.- 6.3.3 On the Expression of the Total Entropy.- 6.4 Alternative Set of Axioms for the Total Entropy.- 6.4.1 Generalization of Shannon Recurrence Equation.- 6.4.2 A Model via Uniform Interval of Definition.- 6.5 Total Entropy with Respect to a Measure.- 6.6 Total Renyi Entropy.- 6.6.1 Preliminary Remarks About Renyi Entropy.- 6.6.2 Axioms for Total Renyi Entropy.- 6.6.3 Total Renyi Entropy.- 6.6.4 Total Renyi Entropy with Respect to a Measure.- 6.7 On the Practical Meaning of Total Entropy.- 6.7.1 General Remarks.- 6.7.2 Total Entropy and Relative Entropy.- 6.7.3 Total Entropy and Subjective Entropy.- 6.8 Further Results on the Total Entropy.- 6.8.1 Some Mathematical Properties of the Total Entropy.- 6.8.2 On the Entropy of Pattern.- 6.9 Transinformation and Total Entropy.- 6.9.1 Total Entropy of a Random Vector.- 6.9.2 Conditional Total Entropy.- 6.9.3 On the Definition of Transinformation.- 6.9.4 Total Kullback Entropy.- 6.10 Relation Between Total Entropy and Continuous Entropy.- 6.10.1 Total Shannon Entropy and Continuous Entropy.- 6.10.2 Total Renyi Entropy and Continuous Renyi Entropy.- 6.10.3 Application to an Extension Principle.- 6.11 Total Entropy and Mixed Theory of Information.- 6.11.1 Background to Effective Entropy and Inset Entropy.- 6.11.2 Inset Entropy is an Effective Entropy.- 7. A Unified Approach to Informational Entropies via Minkowskian Observation.- 7.1 Introduction.- 7.2 Axioms of the Minkowskian Observation.- 7.2.1 Definition of the Observation Process.- 7.2.2 Statement of the Axioms.- 7.2.3 A Few Comments.- 7.3 Properties of the Minkowskian Observation.- 7.3.1 Learning Process.- 7.3.2 Practical Determination of u.- 7.3.3 Cascaded Observation of Variables.- 7.4 Minkowskian Observations in ?3.- 7.4.1 Derivation of the Equations.- 7.4.2 A Few Comments.- 7.5 The Statistical Expectation Approach to Entropy.- 7.5.1 Preliminary Remarks.- 7.5.2 Entropy and Statistical Expectation.- 7.6 Relative Entropy and Subjective Entropy.- 7.6.1 Comparison of Hr with Hs.- 7.6.2 Comparison of T(?/?) with J(?/?).- 7.7 Weighted Relative Entropy.- 7.7.1 Background on Weighted Entropy.- 7.7.2 Weighted Relative Entropy.- 7.7.3 Weighted Relative Entropy of a Vector.- 7.7.4 Total Weighted Relative Entropy.- 7.7.5 Observation and Subjectivity.- 7.8 Weighted Transinformation.- 7.8.1 Some Preliminary Remarks.- 7.8.2 Weighted Relative Conditional Entropy.- 7.8.3 Weighted Transinformation.- 7.8.4 Relation to Renyi Entropy.- 7.9 Weighted Cross-Entropy, Weighted Relative Entropy.- 7.9.1 Background on Kullback Entropy.- 7.9.2 Weighted Kullback Entropy.- 7.10 Weighted Relative Divergence.- 7.11 Application to Continuous Distributions.- 7.11.1 The General Principle of the Derivations.- 7.11.2 Continuous Weighted Relative Entropy.- 7.11.3 Continuous Weighted Kullback Entropy.- 7.11.4 Continuous Weighted Relative Divergence.- 7.12 Transformation of Variables in Weighted Relative Entropy.- 7.13 Conclusions.- 8. Entropy of Form and Pattern.- 8.1 Introduction.- 8.1.1 Entropy of Form and Fractal Dimension.- 8.1.2 Entropy and Pattern Representation.- 8.1.3 Randomized Definition or Geometrical Approach?.- 8.2 Total Entropy of a Random Vector.- 8.2.1 Definition of the Total Entropy of an m-Vector.- 8.2.2 Some Properties of the Total Entropy of m-Vectors.- 8.3 Total Entropy of a Discrete Quantized Stochastic Process.- 8.3.1 Preliminary Remarks.- 8.3.2 Axioms for the Total Uncertainty of a Discrete Trajectory.- 8.3.3 Expression for the Total Entropy of the Trajectory.- 8.3.4 Some Properties of the Total Entropy of the Trajectory.- 8.3.5 A Generalization.- 8.4 Total Renyi Entropy of Discrete Quantized Stochastic Processes.- 8.4.1 Total Renyi Entropy of a Vector.- 8.4.2 Derivation of the Total Renyi Entropy.- 8.5 Entropy of Order c Revisited.- 8.5.1 Preliminary Remarks.- 8.5.2 A New Definition of Total Entropy of Order c.- 8.6 Entropy of Continuous White Stochastic Trajectories.- 8.6.1 Notation and Remarks.- 8.6.2 A List of Desiderata for the Entropy of a White Trajectory.- 8.6.3 Towards a Mathematical Expression of the Trajectory Entropy.- 8.6.4 Entropy of Continuous White Trajectories.- 8.7 Entropy of Form and Observation Modes.- 8.7.1 Relative Uncertainty via Observation Modes.- 8.7.2 Classification of Observation Modes.- 8.7.3 Trajectory Entropy via White Observation.- 8.8 Trajectory Entropies of Stochastic Processes.- 8.8.1 Trajectory Shannon Entropy.- 8.8.2 Renyi Entropy of a Stochastic Trajectory.- 8.8.3 A Few Comments.- 8.9 Trajectory Entropy of a Stochastic Process Under Local Markovian Observation.- 8.9.1 Markovian Processes.- 8.9.2 Non-Markovian Processes.- 8.9.3 Transformation of Variables.- 8.9.4 A Few Remarks.- 8.10 Trajectory Entropy of a Stochastic Process Under Global Markovian Observation.- 8.10.1 Markovian Processes.- 8.10.2 Non-Markovian Processes.- 8.10.3 Transformation of Variables.- 8.10.4 A Few Remarks.- 8.11 Trajectory Entropies of Stochastic Vectors.- 8.11.1 Notation and Preliminary Results.- 8.11.2 Trajectory Entropies in ?n.- 8.12 Transinformation of Stochastic Trajectories.- 8.12.1 Definition of Transinformation of Trajectories.- 8.12.2 Transinformation Measures of Stochastic Trajectories.- 8.12.3 Application to the Derivation of Conditional Trajectory Entropies.- 8.13 On the Entropy of Deterministic Pattern.- 8.13.1 Background on Some Results.- 8.13.2 On the Entropy of Deterministic Pattern.- 8.13.3 Dependence of the Entropy upon the Observation Mode.- 8.14 Entropy of a Differentiable Mapping.- 8.14.1 Entropy with Respect to a Family of Distributions.- 8.14.2 Maximum Entropy of a Differentiable Mapping.- 8.14.3 Entropy of a Mapping Defined on a Finite Interval.- 8.14.4 Entropy of a Mapping with an Incomplete Probability Distribution.- 8.15 Entropy of Degree d of Differential Mappings.- 8.15.1 Trajectory Entropy of Degree d.- 8.15.2 Some Properties of Trajectory Entropy of Degree d.- 8.15.3 Practical Meaning of Trajectory Entropy of Degree d.- 8.16 Transinformation Between Differentiable Mappings.- 8.16.1 The Trajectory Entropy of Compositions of Functions.- 8.16.2 Application to Transinformation Between Mappings.- 8.17 Trajectory Entropy of Degree d in Intrinsic Coordinates.- 8.18 Trajectory Entropy of ?n??n Mapping.- 8.19 Trajectory Entropy of Degree d of a Discrete Mapping.- 8.20 Trajectory Entropy and Liapunov Exponent.- 9. A Theory of Relative Statistical Decision.- 9.1 Introduction.- 9.2 Background on Observation with Information Invariance.- 9.2.1 Euclidean or Minkowskian Invariance.- 9.2.2 On the Practical Meaning of the Model.- 9.2.3 Syntax and Semantics.- 9.3 Relative Probability.- 9.3.1 Definition of Relative Probability.- 9.3.2 On the Practical Meaning of Relative Probability.- 9.3.3 Semantic Entropy of a Real-Valued Variable.- 9.3.4 Relative Probability of Deterministic Events.- 9.3.5 Relative Probability of the Impossible Event.- 9.4 Composition Laws for Relative Probability.- 9.4.1 Relative Probability of the Complement of an Event.- 9.4.2 Relative Probability of a Deterministic Event.- 9.4.3 Relative Probability of Intersection of Events.- 9.4.4 Relative Probability of the Union of Events.- 9.5 Generalized Maximum Likelihood Criterion.- 9.5.1 The Problem.- 9.5.2 Generalized Maximum Likelihood Criterion.- 9.5.3 Practical Meaning of the Generalized Maximum Likelihood Criterion.- 9.5.4 A Simplified Criterion.- 9.5.5 Another Simplified Decision Criterion.- 9.5.6 Comparison with Another Criterion.- 9.6 Generalized Bayesian Criterion.- 9.6.1 Pattern Recognition.- 9.6.2 Formulation of the Statistical Decision Problem.- 9.6.3 Sharp Decision Rule.- 9.6.4 Soft Decision Rule.- 9.7 Generalized Bayesian Decision and Path Integrals.- 9.7.1 Formulation of the Problem.- 9.7.2 The Main Assumptions and Solution.- 9.8 Error Probability and Generalized Maximum Likelihood.- 9.8.1 Definition of the Problem.- 9.8.2 Application of the Generalized Maximum Likelihood Criterion.- 9.8.3 Application of the Generalized Bayesian Rule.- 9.9 A Pattern Recognition Problem.- 9.9.1 An Illustrative Example.- 9.9.2 Application of the Sharp Decision Rule.- 9.9.3 Application of the Soft Decision Rule.- 9.10 Concluding Remarks.- 10. Concluding Remarks and Outlook.- References.