Information Theory for Data Communications and Processing

This book, composed of the collection of papers that have appeared in the Special Issue of the Entropy journal dedicated to “Information Theory for Data Communications and Processing”, reflects, in its eleven chapters, novel contributions based on the firm basic grounds of information theory. The book chapters [1,2,3,4,5,6,7,8,9,10,11] address timely theoretical and practical aspects that carry both interesting and relevant theoretical contributions, as well as direct implications for modern current and future communications systems.

Information theory has started with the monumental work of Shannon: Shannon, C.E. “A Mathematical Theory of Communications”, Bell System Technical Journal, vol. 27, pp. 379–423, 623–656, 1948, and it provided from its very start the mathematical/theoretical framework which facilitated addressing information related problems, in all respects: starting with the basic notion of what is information, going through basic features of how to convey information in the best possible way and how to process it given actual and practical constraints. Shannon himself not only fully realized the power of the basic theory he has developed but further in his profound contributions addressed practical constraints of communications systems, such as bandwidth, possible signaling limits (as peak limited signals), motivating from the very start to address practical constraints via theoretical tools, see, for example: Jelonek, Z. A comparison of transmission systems, In Proc. Symp. Appl. Commun. Theory, E.E. Dep., Imperial College, Buttenvorths Scientific Press, London, September 1952, pp. 45–81. Shannon has contributed fundamentally also to most relevant aspects as source coding under a fidelity (distortion) measure, finite code lengths (error exponents) as well as network aspects of information theory (the multiple-access channel), see: Sloane, N.J.A. and Wyner, A.D., Eds., Collected Papers of Claude Elwood Shannon. IEEE Press: New York, 1993.

While at its beginning and through the first decades, information theory, as is reflected in the basic 1948 work of Shannon, was a mathematical tool that pointed out the best that can be achieved (as channel capacity for point-to-point communications), which with past technology could not even be imagined to be approached. Now, the power of information theory is way greater as it is able to theoretically address network problems and not only point out the limits of communications/signal processing, but with current technology, those limits can, in general, be decently approached. This is classically demonstrated by the capacity of the point-to-point Gaussian channel, which is actually achieved within fractions of dB in signal-to-noise (snr) ratio by advanced coding techniques (Low-Density-Parity-Check, Turbo and Polar codes). In our times, current advanced technology turns information theory into a practical important tool that is capable also to provide basic guidelines how to come close to ultimate optimal performance.

Modern, current and future communications/processing aspects motivate in fact basic information theoretic research for a wide variety of systems for which we yet do not have the ultimate theoretical solutions (for example a variety of problems in network information theory as the broadcast/interference and relay channels, which mostly are yet unsolved in terms of determining capacity regions and the like). Technologies as 5/6G cellular communications, Internet of Things (IoT), Mobile Edge Networks and others place in center not only features of reliable rates of information measured by the relevant capacity, and capacity regions, but also notions such as latency vs. reliability, availability of system state information, priority of information, secrecy demands, energy consumption per mobile equipment, sharing of communications resource (time/frequency/space) and the like.

This book focuses on timely and relevant features, and the contributions in the eleven book chapters [1,2,3,4,5,6,7,8,9,10,11], summarized below, address the information theoretical frameworks that have important practical implications.

The basic contributions of this book could be divided into three basic parts:

(1)

The first part Chapters [1,2,3,4,5] considers central notions such as the Information Bottleneck, overviewed in the first chapter, pointing out basic connections to a variety of classical information theoretic problems, such as remote source coding. This subject covering timely novel information theoretic results demonstrates the role information theory plays in current top technology. These chapters, on one hand, provide application to ‘deep learning’, and, on the other, they present the basic theoretical framework of future communications systems such as Cloud and Fog Radio Access Networks (CRAN, FRAN). The contributions in this part directly address aspects such as ultra-reliable low-latency communications, impacts of congestion, and non-orthogonal access strategies.

(2)

The second part of the contributions in this book Chapters [6,7,8] addresses classical communications systems, point-to-point Multiple-Input-Multiple-Output (MIMO) channels subjected to practical constraints, as well as network communications models. Specifically, relay and multiple access channels are discussed.

(3)

The third part of the contributions of this book Chapters [9,10,11] focuses mainly on caching, which, for example, is the center component in FRAN. Information theory indeed provides the classical tool to address network features of caching, as demonstrated in the contributions summarized below (and references therein).

Chapter 1: “On the Information Bottleneck Problems: Models, Connections, Applications and Information Theoretic Views” provides a tutorial that addresses from an information theoretic viewpoint variants of the information bottleneck problem. It provides an overview emphasizing variational inference, representation learning and presents a broad spectrum of inherent connections to classical information theoretic notions such as: remote source-coding, information combining, common reconstruction, the Wyner–Ahlswede–Korner problem and others. The distributed information bottleneck overviewed in this tutorial sets the theoretical grounds for the uplink CRAN, with oblivious processing.

Chapter 2: “Variational Information Bottleneck for Unsupervised Clustering: Deep Gaussian Mixture Embedding” develops an unsupervised generative clustering framework that combines the variational information bottleneck and the Gaussian mixture model. Among other results, this approach that models the latent space as a mixture of Gaussians generates inference-type algorithms for exact computation, and generalizes the so-called evidence lower bound, which is useful in a variety of unsupervised learning problems.

Chapter 3: “Asymptotic Rate-Distortion Analysis of Symmetric Remote Gaussian Source Coding: Centralized Encoding vs. Distributed Encoding” addresses remote multivariate source coding, which is a CEO problem and, as indicated in Chapter 1, connects directly to the distributed bottleneck problem. The distortion measure considered here is minimum-mean-square-error, which can be connected to the logarithmic distortion via classical information–estimation relations. Both cases—the distributed and joint remote source coding (all terminals cooperate)—are studied.

Chapter 4: “Non-Orthogonal eMBB-URLLC Radio Access for Cloud Radio Access Networks with Analog Fronthauling” provides an information-theoretic perspective of the performance of Ultra-Reliable Low-Latency Communications (URLLC) and enhanced Mobile BroadBand (eMBB) traffic under both Orthogonal and Non-Orthogonal multiple access procedures. The work here considers CRAN based on the relaying of radio signals over analog fronthaul links.

Chapter 5: “Robust Baseband Compression against Congestion in Packet-Based Fronthaul Networks Using Multiple Description Coding” also addresses CRAN and considers the practical scenario when the fronthaul transport network is packet based and it may have a multi-hop architecture. The timely information theoretic concepts of multiple description coding are employed, and demonstrated to provide advantageous performance over conventional packet-based multi-route reception or coding.

Chapter 6: “Amplitude Constrained MIMO Channels: Properties of Optimal Input Distributions and Bounds on the Capacity” studies the classical information theoretic setting where input signals are subjected to practical constraints, with focus on amplitude constraints. Followed by a survey of available results for Gaussian MIMO channels, which are of direct practical importance, it is shown that the support of a capacity-achieving input distribution is a small set in both a topological and a measure theoretical sense. Bounds on the respective capacities are developed and demonstrated to be tight in the high amplitude regime (high snr).

Chapter 7: “Quasi-Concavity for Gaussian Multicast Relay Channels” addresses the classical model of a relay channel, which is one of the classical information theoretic problems that are not yet fully solved. This work identifies useful features of quasi-concavity of relevant bounds (as the cut-set bound) that are useful in addressing communications schemes based on relaying.

Chapter 8: “Gaussian Multiple Access Channels with One-Bit Quantizer at the Receiver” investigates the practical setting when the received input is sampled and here it employs a zero-threshold one-bit analogue-to-digital converter. It is shown that the optimal capacity achieving signal distribution is discrete, and bounds on the respective capacity are reported.

Chapter 9: “Efficient Algorithms for Coded Multicasting in Heterogeneous Caching Networks” addresses crucial performance–complexity tradeoffs in a heterogeneous caching network setting, where edge caches with possibly different storage capacities collect multiple content requests that may follow distinct demand distributions. The basic known performance-efficient coded multicasting schemes suffer from inherent complexity issues, which makes them impractical. This chapter demonstrates that the proposed approach provides a compelling step towards the practical achievability of the promising multiplicative caching gain in future-generation access networks.

Chapter 10: “Cross-Entropy Method for Content Placement and User Association in Cache-Enabled Coordinated Ultra-Dense Networks” focuses on ultra-dense networks, which play a central role for future wireless technologies. In Coordinated Multi-Point-based Ultra-Dense Networks, a great challenge is to tradeoff between the gain of network throughput and the degraded backhaul latency, and caching popular files has been identified as a promising method to reduce the backhaul traffic load. This chapter investigated Cross-Entropy methodology for content placement strategies and user association algorithms for the proactive caching ultra-dense networks, and demonstrates advantageous performance.

Chapter 11: “Symmetry, Outer Bounds, and Code Constructions: A Computer-Aided Investigation on the Fundamental Limits of Caching” also focuses on caching, which, as mentioned, is a fundamental procedure for future efficient networks. Most known analyses and bounds developed are based on information theoretic arguments and techniques. This work illustrates how computer-aided methods can be applied to investigate the fundamental limits of the caching systems, which are significantly different from the conventional analytic approach usually seen in the information theory literature. The methodology discussed and suggested here allows, among other things, to compute performance bounds for multi-user/terminal schemes, which were believed to require unrealistic computation scales.

In closing, one can view all the above three categories of the eleven chapters, in a unified way, as all are relevant to future wireless networks. The massive growth of smart devices and the advent of many new applications dictates not only having better systems, such as coding and modulation on the point-to-point channel, classically characterized by channel capacity, but a change of the network/communications paradigms (as demonstrated for example, by the notions of CRAN and FRAN) and performance measures. New architectures and concepts are a must in current and future communications systems, and information theory provides the basic tools to address these, developing concepts and results, which actually are not only of essential theoretical value, but are also of direct practical importance. We trust that this book provides a sound glimpse to these aspects.