Multivaluedness in Networks: Shannon’s Noisy-Channel Coding Theorem

Recently, a necessary and suﬃcient condition for multivaluedness to be implicitly exhibited by counter-cascaded systems was presented. Subsequently, several systems that exhibit multivaluedness were reported. This brief interprets a general information transmission system as a counter-cascaded system with Shannon’s noisy-channel coding theorem providing the necessary and suﬃcient conditions for multivaluedness and is therefore a particular instance of the counter-cascaded network framework


I. INTRODUCTION
In networks, multivaluedness refers to situations where a single stimulus (cause) is associated with multiple distinct responses (effects), resulting in cause-effect relations that cannot be analyzed using existing network theories.Surprisingly, multivaluedness appears to have been confronted for the first time only recently in [1], where inverse modeling in microwave filters using neural networks was studied.Subsequently, [2] proposed a counter-cascaded systems analysis framework, which led to a necessary and sufficient condition for the manifestation of multivaluedness.Further analytical results were also presented there for the characterization of multivaluedness.
Several examples of counter-cascaded systems configurations were presented in [3] where it was demonstrated that the manifestation of multivaluedness could be either adverse, benign or even essential for a system to function, depending on the particular system considered.Importantly, it was emphasized there that multivaluedness could easily be misdiagnosed as modeling errors, exogeneous disturbances entering the system or as apparent inherent uncertainty in the system's behavior.Lastly, [3] showed that even though a multivalued relation can never be reduced to a mapping, it could be approximated (with nonzero error) by one but that this might not be particularly useful.However, for such cases, probabilistic methods offer alternatives.Li Ping is with the Department of Electrical Engineering, City University of Hong Kong, Hong Kong, China (e-mail: eeliping@cityu.edu.hk).
Guanrong Chen is with the Department of Electrical Engineering, City University of Hong Kong, Hong Kong, China (e-mail: eegchen@cityu.edu.hk).This brief interprets Shannon's noisy-channel coding theorem [4] [5] from the perspective of counter-cascaded systems and multivaluedness.

II. COUNTER-CASCADED SYSTEMS
Figure 1 depicts what [2] refers to as the counter-cascaded configuration of the systems M and N • T .The symbols M , T and N serve the dual purpose of representing the nonlinear systems shown as well as their respective mathematical descriptions as nonlinear operators [6] M : U → W , T : U → V and N : V → X that map from and into the signal spaces indicated.The symbol S ⊂ W × X represents an implicit relation, If the image under T of every equivalence class [8] induced on U by the preimage of M , is contained in an equivalence class induced on V by the preimage of N , then T is said to be immanent with respect to the ordered pair (M, N ) or simply (M, N )-immanent.If T is not (M, N )-immanent, then T is said to be transcendent with respect to the ordered pair (M, N ) or simply (M, N )-transcendent.
A necessary and sufficient condition for the relation S to be multivalued, now follows [2]: Theorem II.1.(Multivaluedness) In Fig. 1, the relation S is multivalued if and only if the mapping T is transcendent relative to (M, N ).
An equivalent interpretation of this result is that S is multivalued if and only if T is (M, N )-transcendent.

III. SHANNON'S THEOREM AND MULTIVALUEDNESS A. Shannon's Noisy-Channel Coding Theorem
First, we introduce the system setup, necessary terminology and present a brief overview of Shannon's noisy-channel coding theorem.Following this, we show that the setup for this theorem can be interpreted as a counter-cascaded systems configuration introduced in the previous section.
For this purpose, u ∈ U is an information carrying signal (message) that is transmitted over the channel T to arrive at the decoder N in Fig. 1.The decoder then decodes the channel output signal v ∈ V to produce a signal (message) x ∈ X that approximates the originally transmitted signal u.This means that the encoder together with the channel input processing, the physical channel and the channel output processing, up to but not including the decoder, form the channel T .Since encoders and decoders do not satisfy the linear superposition principle, they are necessarily nonlinear and hence T and N are both nonlinear operators.Here, M represents the identity operator I (i.e. in Fig. 1, W = U ) which is a linear operator.
The above setup amounts to S (in Fig. 1) representing the implicit relation that relates the original unencoded signal u to the decoded signal x.
In the context of Shannon's work [4], let 0 < P e < 1 represent the channel probability of error, i.e. the probability that bit errors occur during the transmission of the signal u.By reliable transmission is meant that, for an arbitrary value of P e specified, there exists an encoding scheme E which guarantees that the decoded signal x recovers the transmitted u to within an arbitrarily small end-to-end probability of error.Finally, channel capacity refers to the maximum information transmission rate at which reliable transmission of information can be achieved over the channel T , while the entropy rate of a signal source refers to its mean information transmission rate and we can now state Shannon's theorem [4]: Theorem III.1.(Shannon Noisy-Channel Coding Theorem) Consider a noisy channel T with capacity C and a fixed channel signal-to-noise ratio.In Fig. 1, let U be a signal space consisting of signals generated by a source that has an entropy rate of H.
For an arbitrary but fixed channel probability of error P e , if H < C then reliable transmission is achieved.
Conversely, if H > C then reliable transmission is unachievable.
Consequently, at least theoretically, it is possible to transmit information over a channel almost without error at any entropy rate H strictly less than C. Typically, encoders that yield lower end-to-end error probabilities operate on longer sequences of signal samples.Furthermore, in order to maintain a specified end-to-end error probability, a decrease in signal-to-noise ratio necessarily requires an encoder that operates on even longer sequences of signal samples, increasing end-to-end latency.
On the other hand, for entropy rates H strictly greater than C the end-to-end error probability cannot be made arbitrarily small, irrespective of the encoding scheme that is used.Furthermore, the strictly positive lower bound on the achievable end-to-end error probability increases as the violation H − C increases or the channel signal-to-noise ratio decreases.

B. Multivaluedness Interpretation of Shannon's Theorem
For the case H < C, Theorem III.1 implies that there exists an encoding scheme such that the output of the source can be transmitted over the channel T with an arbitrary small error probability by selecting the appropriate encoder E. In fact, for a fixed source entropy rate and signal-to-noise ratio, reliable transmission guarantees that there always exists a sequence (E n ) ∞ n=1 of encoding schemes with encoder E n operating on a sequence of N n consecutive signal samples.Furthermore, their associated error probabilities P e,n := P e (E n ) form a monotonically decreasing sequence that converges to zero.Consequently, for the asymptotic encoder lim n→∞ E n , the associated relation S (in Fig. 1) is single-valued.Then, from Theorem II.1, it immediately follows that the channel T is (I, N )-immanent.
For the case H > C, discounting the statistically insignificant event of immanence resulting as a consequence of the channel-decoder combination N • T forming some fixed permutation, the occurrence of an error means that two or more decoded signals are associated with some input signal u.It follows immediately that the relation S is multivalued and therefore Theorem II.1 yields the (I, N )-transcendence of T .
The preceding reasoning leads to the conclusion that the systems setup for studying information transmission is an instance of the counter-cascaded systems framework where multivaluedness adversely affects system performance [3].Furthermore, for this class of systems, perfect data recovery after transmission is synonymous with single-valuedness of S. It is important to point out that, as a particular instance of this framework, the rigorous proof of Shannon's noisy-channel theorem's first performed in [5], is an indication of the inherent complexity of the analysis of counter-cascaded systems.

IV. CONCLUSION
In this brief, we showed that Shannon's noisy-channel coding theorem describes conditions for an information transmission system, viewed as a counter-cascaded system configuration, which is single-valued for sub-capacity transmission rates and multivalued for super-capacity transmission rates.The class of systems and problems considered in Shannon's work, is of an ever increasing importance as emphasized by the Industrial Revolution 4.0.
Currently, work is in progress to extend the results of [2] to scenarios where both measurement noise and exogenous noise propagating through the system will be accounted for.Once this undertaking has been completed, this would enable one to also consider the individual encoders E n as well.

Fig. 1 .
Fig. 1.Two counter-cascaded paths N • T and M with the common cause, u ∈ U .The implicit relation S (in gray) takes M 's output as its input and produces N 's output as its output.