Information Theory

Originofthename

Informationtheoryconsidersthetransmissionofinformationasastatisticalphenomenonandgivesamethodtoestimatethecapacityofthecommunicationchannel.Informationtransmissionandinformationcompressionaretwomajorareasintheresearchofinformationtheory.Thesetwoaspectsarerelatedtoeachotherbytheinformationtransmissiontheoremandthesource-channelisolationtheorem.

Shannonisknownasthe"FatherofInformationTheory."PeopleusuallytakeShannon'spaper"AMathematicalTheoryofCommunication"publishedin"JournalofBellSystemTechnology"inOctober1948asthebeginningofmoderninformationtheoryresearch.ThisarticleisbasedinpartonpreviousworkbyHarryNyquistandRalphHartley.Inthisarticle,Shannongavethedefinitionofinformationentropy(hereinafterreferredtoas"entropy"):

ThisdefinitioncanbeusedtocalculateandtransmitThechannelbandwidthrequiredbytheoriginalinformationafterbinaryencoding.Entropymeasurestheamountofinformationcontainedinamessage,whichremovesthepartdeterminedbytheinherentstructureofthemessage,suchastheredundancyofthelanguagestructureandstatisticalcharacteristicssuchasthefrequencyofuseoflettersandwordsinthelanguage.

Theconceptofentropyininformationtheoryiscloselyrelatedtothermodynamicentropyinphysics.BoltzmannandGibbsdidalotofworkonentropyinstatisticalphysics.Entropyininformationtheoryisalsoinspiredbyit.

MutualInformation(MutualInformation)isanotherusefulmeasureofinformation,whichreferstothecorrelationbetweentwosetsofevents.ThemutualinformationoftwoeventsXandYisdefinedas:

whereisthejointentropy(JointEntropy),whichItisdefinedas:

Mutualinformationiscloselyrelatedtothemultivariatelog-likelihoodratiotestandthePearsonχ2test.

Abriefhistoryofdevelopment

Informationtheoryisadisciplinesummedupfromlong-termcommunicationpracticeinthelate1940s.Itspecializesinthegenerallawsofeffectiveprocessingandreliabletransmissionofinformation.science.

ECCherryoncewroteanearlyhistoryofinformationtheory.Hestartedfromstonehieroglyphs,wentthroughmedievalenlightenmentlinguistics,anduntilthe16thcenturyGilbert(ENGilbert)etal.Academicwork.

Inthe1920s,H.NyquistandL.V.R.Hartleyfirststudiedtheabilityofcommunicationsystemstotransmitinformationandtriedtomeasurethesystem'schannelcapacity.Moderninformationtheorybegantoappear.

Thepaper"TheMathematicalTheoryofCommunication"publishedbyClaudeShannonin1948wasthefirstpaperintheworldthatestablishedamathematicalmodelforthecommunicationprocess.Apapertogetherlaidthefoundationofmoderninformationtheory.

Duetotherapiddevelopmentofmoderncommunicationtechnologyandthecross-penetrationofotherdisciplines,theresearchofinformationtheoryhasexpandedfromthenarrowscopeofShannon’smathematicaltheorylimitedtocommunicationsystems,andhasbecomewhatisnowcalledinformationscience.Hugesystem.

Basiccontent

Traditionalcommunicationsystemssuchastelegraph,telephone,andpostareusedtotransmittextinformation,voiceinformationandtextinformationrespectively;whilebroadcasting,telemetry,remotesensingandremotecontrolsystemsItalsotransmitsallkindsofinformation,butthetypeofinformationisdifferent,soitalsobelongstotheinformationsystem.Sometimes,informationmustbetransmittedinbothdirections.Forexample,telephonecommunicationrequirestwo-wayconversation,andremotecontrolsystemsrequiretransmissionofcontrolinformationandreversemeasurementinformation.Thiskindoftwo-wayinformationsystemisactuallycomposedoftwoinformationsystems.Allinformationsystemscanbesummarizedintothemodelshowninthefiguretostudyitsbasiclaws.

Informationsource:Thesourceofinformationortheentitythatproducestheinformationtobetransmitted,suchasthespeakerinthetelephonesystem.Forthetelecommunicationssystem,amicrophoneshouldalsobeincluded,andtheelectricalsignaloutputbyitservesasthecarriercontainingtheinformation.

Informationdestination:thedestinationorrecipientofinformation.Inthetelephonesystem,thisisthelistenerandheadset.Thelatterconvertsthereceivedelectricalsignalintosoundforthelistenertoextracttherequiredinformation.

Channel:Thechannelfortransmittinginformation,suchasthecoaxialcablesystemincludingrepeatersintelephonecommunication,thetransceiver,antennaandtransponderonsatelliteinsatellitecommunication,etc.

Encoder:Ininformationtheory,itreferstoallequipmentthattransformssignals.Infact,itisthetransmittingpartoftheterminal.Itincludesallequipmentfromthesourcetothechannel,suchasquantizers,compressionencoders,modulators,etc.,toconvertthesignaloutputbythesourceintoasignalsuitableforchanneltransmission.

Decoder:Itistheinverseconversiondeviceoftheencoder,whichconvertsthesignalsentfromthechannelintoasignalthatthesinkcanaccept,whichcanincludeademodulator,decoder,digital-to-analogconverter,etc.

Whenthesourceandsinkhavebeengiven,andthechannelhasbeenselected,theencoderanddecoderdeterminetheperformanceoftheinformationsystem.Whendesigninganinformationsystem,inadditiontoselectingchannelsanddesigningitsancillaryfacilities,themaintaskistodesignacodec.Ingeneral,themainperformanceindexofaninformationsystemisitseffectivenessandreliability.Effectivenessistotransmitasmuchinformationaspossibleinthesystem;andreliabilityisthattheinformationreceivedbythesinkisasconsistentaspossiblewiththeinformationsentbythesource,orthedistortionisassmallaspossible.Thebestcodecistomakethesystemthemosteffectiveandreliable.However,reliabilityandeffectivenessareoftencontradictory.Moreeffectiveoftenleadstounreliability,andviceversa.Inaquantitativesense,thesystemshouldbemadetotransmitthemaximuminformationrateunderthespecifieddistortionorbasicallynodistortion;orunderthespecifiedinformationrate,thedistortionshouldbethesmallest.Calculatingthismaximuminformationrateandprovingtheexistenceofacodecthatreachesorclosetothisvalueisthebasictaskofinformationtheory.ThetheorythatonlydiscussessuchissuescanbecalledShannon'sinformationtheory.Generallyspeaking,thecontentofinformationtheoryshouldbemoreextensive,thatis,thetheoryofextractinginformationandensuringinformationsecurity.Thelatterareestimationtheory,detectiontheoryandcryptography.

Informationtheoryisformedonthebasisofprobabilitytheory,whichisbasedontheprobabilitycharacteristicsofsourcesymbolsandchannelnoise.Thistypeofinformationisusuallycalledgrammaticalinformation.Infact,thebasiclawsofinformationsystemsshouldalsoincludesemanticinformationandpragmaticinformation.Grammaticalinformationisthatthestructureofthesource'soutputsymbolsortheirobjectivecharacteristicshasnothingtodowiththesubjectiverequirementsofthesource,whilethesemanticsshouldconsiderthemeaningofeachsymbol.Thesamemeaningcanbeexpressedindifferentlanguages​​ortexts.Itiscontainedinvariouslanguages.Thegrammaticalinformationcanbedifferent.Generallyspeaking,thesemanticinformationratecanbelowerthanthegrammaticalinformationrate;theinformationrateofthetelegramcanbelowerthantheinformationrateofthevoiceexpressingthesamemeaningisanexample.Furthermore,therecipientortherecipientoftheinformationoftenonlyneedstheinformationthatisusefultohim.Thelanguagehedoesnotunderstandismeaningful,butitisuselesstohim.Therefore,thepragmaticinformation,thatis,theinformationusefultothedestinationisgenerallysmallerthanthesemanticinformation.Ifonlytheinformationsystemisrequiredtotransmitsemanticinformationorpragmaticinformation,theefficiencywillobviouslybehigher.Underthecurrentcircumstances,regardinggrammaticalinformation,asystematictheoryhasbeenestablishedonthebasisofprobabilitytheory,formingadiscipline;however,semanticandpragmaticinformationisnotyetmatureenough.Therefore,thediscourseonthelatterisusuallycalledinformationscienceorgeneralinformationtheory,anddoesnotbelongtothecategoryofgeneralinformationtheory.Insummary,thebasiclawsofinformationsystemsshouldincludeinformationmeasurement,sourcecharacteristicsandsourcecoding,channelcharacteristicsandchannelcoding,detectiontheory,estimationtheory,andcryptography.

Application

Scope

Encoding

cryptographyandcryptanalysis

datatransmission

Datacompression

Detectiontheory

Estimationtheory

Politics(politicalcommunication)

OverviewofInformationTheory

Informationtheoryisasciencethatusesmathematicalstatisticstostudythelawsofmeasurement,transmissionandtransformationofinformation.Itismainlytostudythecommonlawsofinformationtransmissionthatexistincommunicationandcontrolsystems,andtostudythebasictheoriesthatbestsolvetheproblemsofinformationacquisition,measurement,transformation,storageandtransmission.

ResearchScope

Theresearchscopeofinformationtheoryisextremelybroad.Generally,informationtheoryisdividedintothreedifferenttypes:

(1)Specialinformationtheoryisasciencethatappliesmathematicalstatisticstothestudyofinformationprocessingandinformationtransmission.Itstudiesthecommonlawofinformationtransmissionthatexistsincommunicationandcontrolsystems,andacommunicationtheoryofhowtoimprovetheeffectivenessandreliabilityofeachinformationtransmissionsystem.

(2)Generalinformationtheorymainlystudiescommunicationproblems,butalsoincludesnoisetheory,signalfilteringandprediction,modulationandinformationprocessing.

(3)Generalinformationtheoryincludesnotonlytheissuesofspecialinformationtheoryandgeneralinformationtheory,butalsoallinformation-relatedfields,suchaspsychology,linguistics,neuropsychology,semantics,etc.

Thedomainofinformationdefinition

Informationisaincreasedcertainty----inverseShannoninformationdefinition;

Informationisthelabelofsubstance,energy,information----theinverseofWienerinformationdefinition;

InformationisthingThecollectionofitsattributeidentifiers.

InformationandCommunication

Informationisakindofmessage,whichiscloselyrelatedtocommunicationissues.In1948,ShannonoftheBellInstitutesystematicallyputforwardanexpositionaboutinformationinapaperentitled"TheMathematicalTheoryofCommunication"andcreatedthetheoryofinformation.Wiener'smathematicalformulaformeasuringtheamountofinformationopenedupawiderangeofapplicationprospectsforinformationtheory.In1951,theAmericanSocietyofRadioEngineeringrecognizedthesubjectofinformationtheory,andithasdevelopedrapidlysincethen.The1950swasaperiodwheninformationtheoryimpactedvariousdisciplines.Inthe1960s,informationtheorywasnotaperiodofmajorinnovation,butaperiodofdigestionandunderstanding,andaperiodofmajorconstructionontheexistingfoundation.Theresearchfocusisoninformationandsourcecodingissues.Bythe1970s,duetothewidespreadapplicationofdigitalcomputers,thecapabilitiesofcommunicationsystemshavealsobeengreatlyimproved.Howtouseandprocessinformationmoreeffectivelyhasbecomeanincreasinglyurgentproblem.Peoplearemoreandmoreawareoftheimportanceofinformationandrealizethatinformationcanbefullyutilizedandsharedasaresourcelikematerialsandenergy.Theconceptsandmethodsofinformationhavepenetratedwidelyintovariousscientificfields,anditurgentlyrequiresabreakthroughinthenarrowscopeofShennongInformationTheory,sothatitcanbecomethebasictheoryofinformationproblemsencounteredinvarioushumanactivities,therebypromotingmanyotheremergingdisciplines.Furtherdevelopment.Peoplehaveappliedthelawsandtheoriesrelatedtoinformationestablishedearlierinphysics,chemistry,biologyandotherdisciplines.Aninformationsciencethatstudiesthegeneration,acquisition,transformation,transmission,storage,processing,display,recognitionandutilizationofinformationistakingshape.

InformationScience

Informationscienceistheprocessofexpandingpeople’sunderstandingandutilizationofinformationininformationtheory,electronics,computerscience,artificialintelligence,systemsengineering,automationAmarginalnewdisciplinedevelopedonthebasisoftechnologyandothermulti-disciplines.Itstaskismainlytostudythenatureofinformation,tostudythegenerallawsofacquisition,transformation,transmission,processing,utilizationandcontrolofvariousinformationbymachines,organismsandhumans,todesignanddevelopvariousinformationmachinesandcontrolequipment,andtorealizeoperationautomation,Inordertoliberatethehumanbrainfromtheconstraintsofnaturalforcesasmuchaspossible,andtoimprovetheabilityofhumanbeingstounderstandandtransformtheworld.Informationsciencealsohasimportantapplicationsintheresearchofsecurityissues.

Informationtheoryhypothesis

Englishtranslation

Materialism

Detaileddefinition

Matter,energyandinformationmakeuptheworldThethreemajorelements.Peoplehaveadeepunderstandingofmatterandenergy,andtheunderstandingofinformationhasjustbegun.So,whatisinformation?Inwhatwaydoesitexist?Whatroledoesithave?Thefollowingismyconjecture,Ihopeitwillbehelpfulformankindtofurtherunderstandtheworld.

1.Definitionofinformation

Informationdefinitionofnon-worldthreeelements:informationisathinganditsattributeIdentifiedcollection.Informationdefinitionwithoutthethreeelementsoftheworld.

Informationdefinitionwiththreeelements:

1.Informationisincreaseofcertainty----inverseShannoninformationdefinition;

2.Informationisinformation,andinformationisthelabelofsubstance,energy,informationanditsattributes----theinverseofWienerinformationdefinition.

Informationisauniversalformofthestateandmovementcharacteristicsofobjectivethings.Alargenumberofinformationexpressedinthesewaysexist,produceandtransmitintheobjectiveworld.However,thisisonlyforthethree-dimensionalspacewelivein,andinformationhasadeeperessence.Then,doesinformationstillexistinfour-dimensionalspace(thefour-dimensionalspacementionedheredoesnotincludetime,butthefour-dimensionalstateofspace)?Yes,buttobeclear,informationonlyexistsinthefour-dimensionalspace,andtheinformationinthethree-dimensionalspaceisonlytheshadowoftherealinformationinthefour-dimensionalspace.Alargeamountofinformationexistsinthefour-dimensionalspace,anditsessenceistheregulararrangementofakindofinformation(informer,hypotheticalbasicunitofinformationthatexistsinthefour-dimensionalspace)inthefour-dimensionalspace.

Informationistherootcauseoftheincident(corritor),whichwillbeanalyzedindetailinthethirdsection.

Second,thenatureofinformation

Informationhasthefollowingproperties:objectivity,extensiveness,completeness,andspecificity.Firstofall,informationexistsobjectively,itisnotdeterminedbywill,butithasaninevitableconnectionwithhumanthought(thefourthsectionwillanalyzeitindetail).Atthesametime,informationiswidespread,andthefour-dimensionalspaceisfloodedwithalargenumberofinformationsub-spaces.Animportantpropertyofinformationiscompleteness.Eachinformationsubcannotdetermineanyevent.Theremustbetwoormoreinformationsub-rulesarrangedascompleteinformation,sothattheenergyreleasedissufficienttomakesurethattheeventoccurs.Informationalsohasspecificity.Eachinformationdeterminesacertainevent,buttheinformationofsimilareventsalsohassimilarities.Theexplanationofthereasonrequiresfurtherdiscoveryofinformationsub-categoriesandarrangementcryptographictheory.

3.Informationtheorymechanism

Inthenormalstate,theinformationsub-distributionsaredisorderlydistributedinthefour-dimensionalspace.Whenthemoleculesinthethree-dimensionalspacecollidewithfriction,theenergyinthemescapesintothefour-dimensionalspace,andtheregulararrangementofinformationquantumisactivated.Thearrangedinformationquantumreleasestheenergyandentersthethree-dimensionalspace,causingfrictionandcollisionofothermolecules.,Andsoon.Ifthemoleculethatiscausedbyfrictioncollisionhappenstobeajujube(decider,thefactorthatdeterminestheevent,suchassodiumandpotassiumionsthatcausenerveimpulses,andtheelectricchargethatcauseslightning),andacertainamountofsubstanceiscausedbyfrictionandcollision,theeventhappen.Ofcourse,theenergiesgeneratedbythefrictionandcollisionofdifferentmoleculesaredifferent,andthetypesofpheromonearrangementscausedbythemarealsodifferent,sotheeventsthataredeterminedarealsodifferent.

However,beforetheexplosionoftheuniverse,onlyinformationexisted.Adeterminantfactor(Idon’tknowwhatthisfactoris)ledtotheaccidentalarrangementofinformationelements.Someinformationelementsweretransformedintoenergy.Therearecertainconditionsforenergy,whichcanonlybeachievedbeforeorintheinitialstageoftheexplosionoftheuniverse),andenergywillbetransformedintomatterundercertainconditions,andcontinuetotransferandtransform,andfinallyformourcurrentuniverse.Therefore,theorderlyarrangementofinformationelementsistherootcauseoftheevent,thefrictionandcollisionofmatteristhedirectcauseoftheevent,andthetransmissionofenergyisanecessaryconditionfortheeventtooccur.

Four.ExamplesofInformationTheoryHypothesis

1.Thoughtandmemory:Thoughtissomethingwehavebeenelusive,butaccordingtotheinformationtheoryhypothesis,thoughtisactuallyakindofinformation.Thefrictionandcollisionofcertainspecificmoleculesinthebraincausetheregulararrangementofcertaininformationelements.Theperformanceinthethree-dimensionalspaceistogenerateelectriccurrentsandcausebraincellactivity.Thisistheessenceofthought.Ofcourse,differentinformationshowsDifferentthoughts.However,doesn'tthismeanthatourthinkingisalreadydefined?Infact,that'sit.It'sjustthatthenumberofmoleculesinourbrainishuge,andthetypesofarrangementsthatcancauseinformationareextremelynumerous,andourthinkinghasonlydevelopedasmallpart.Inreality,ourso-calledthoughthasanotherfactor,thatis,itneedstobeexpressedthroughacompleteandcomplexregulatorymechanism.Thisregulatorymechanismisthenervoussystemforhumans,soonlywecanexpresscomplexthoughts..Memoryisaspecializationofthought.Whenthemoleculesthatcausefrictionandcollisionofinformationhappentobethemoleculesthatproducedthoughtsbefore(memoryjujubes),thepreviousthoughtswillbeexpressedagainthroughtheregulararrangementofspecificinformationelements.Inthisway,ourthoughtsarecontinuous,andthethoughtsofthepreviousmomentdirectlydeterminethethoughtsofthenextmoment,butwehavenotnoticeditandhavenomeanstodiscoverit.

2.Lifephenomenon:humanbirth,oldage,sicknessanddeathcanalsobeexplainedbytheinformationtheoryhypothesis.Peoplegettingsickareactuallyinformationcausedbyfrictionbetweenmolecules(bacteriaorviruses)inthebodythatarenotfused.Growthisactuallytheinformationcausedbyvariousexternalmolecules(suchascalciumions)enteringthehumanbodyandrubbingagainstthemoleculesinthebody.Senescenceanddeathareinformationcausedbymolecularfrictionincells,whicharemacroscopicallymanifestedascellsenescenceandapoptosis,whichinturnaffectspeople.

3.Premonitionandcoincidence:Premonitionisaveryspecialformofthought.Whensomemolecularfrictioninthebraincausesinformationsub-arrangement,theinformationdoesnotreleasealltheenergy,butonlypartoftheenergyisreleasedfirst,causingapremonitiondecision.Thefrictionoftheson,theremainingenergyisreleasedatanothermoment,andbecauseitisthesameasthepreviouspartoftheenergy,itjustcausesthefrictionoftheeventson,thusconfirmingthepremonition.Coincidenceisalsoaveryspecialphenomenon.Itsessenceisthattheenergyreleasedbyinformationdividesintotwohalvesandentersdifferentplacesinthree-dimensionalspace,triggeringfrictionofthesamemolecules,whichcausesthesamethingstohappenindifferentplaces.Thisusuallyoccursinthesameegg.Intwins,becauseofthesimilarityoftheirgenes,theyhaveagreaterchanceoffrictionwiththesamemolecules.

4.Dreamsanduntrueimpressions:Dreamsarethoughtsproducedunconsciously,andtheiressenceisalsoinformation.Weusuallyhavefalseimpressions.Seeingacertainsituationseemstohavehappenedbefore,butithasnothappenedbefore.Infact,thisisbecausethemolecularfrictioninthebraincausesinformation,andtheinformationdoesnotreleaseenergyimmediately,buttemporarilystoresit.Whenthesamemolecularfrictionoccursatanothermoment,itsenergyisactivatedanddoubleenergyisreleased.Halfoftheenergymakesusthink,andtheotherhalfmakesusproduceimpressions.Thisistheessenceoffalseimpressions.

5.Chemicalreaction:Theessenceofallchemicalreactionsisinformation.Severalkindsofmolecularfrictionscausespecificinformationandothermolecularfrictions.Inthefriction,chemicalbondsarebrokenandformed,completingchemicalreactions.

6.Destinyandsoul:Theancientsbelievedindestiny,perhapsbecausetheyfeltthatwewerealreadyarrangedindividualsinanotherdimension,sotherewashumanimaginationaboutsoulsandgods.

V.Significanceoftheinformationtheoryhypothesis

Theinformationtheoryhypothesisunifiesmatterandthought.Itisanecessarystepinthedevelopmentofmaterialism.ItexplainsthathumanbeingshavebeenunabletoClarifytheproblem.Ititselfisjustahypothesis,itneedsalongtimeforhumanstoexploreandprove,italsohasitsownflaws,itneedscontinuousdiscoverybyhumans.Maybeitwasoriginallyamistake,butitisatestimonyofhumangrowthandagreatspiritualwealthofmankind.

Lookingattheproblemfromthepointofviewoftheinformationtheoryhypothesiscanmakemankindrealizeawholenewworldandhelptoexplorethedeepernatureoftheworld.Itprovidesmankindwitharichexperienceandisamodelformankindtolookbeyondtheinherentthinking.Inshort,whetheritiscorrectornot,itistheimmortalworkofmankind.

BookInformationOne

Overview

BookTitle:InformationTheory

Publisher:HarbinEngineeringUniversityPress

Author:TangShiweiandLiuXianmei

ISBN:9787811332780

Format:16

Pages:217

Price:25.00yuan

Introduction

Thebookisdividedintosevenchapters.Thefirstchapteristheintroduction,whichintroducesthebasicconceptsanddefinitionsofinformation,theorigin,developmentandresearchcontentofinformationtheory;thesecondchapteristhesourceandInformationsourceentropy,introducestheconcepts,properties,theorems,etc.ofvariousentropy;Chapter3isdistortionlesssourcecoding,introducesthefixed-lengthandvariable-lengthcodingtheoremsandmethodsofthesource,aswellasseveralpracticaldistortion-freesourcesCoding;Chapter4isdistortion-limitedsourcecoding,whichintroducesthedefinition,properties,calculationsofinformationrate-distortionfunctionandpredictivecodingofspeechandimagesignals;Chapter5ischannelandchannelcapacity,whichintroducessingle-symboldiscretechannel,multipleThechannelmodelofsymbolicdiscretechannelandmulti-userchannelandthecalculationofchannelcapacity;Chapter6ischannelcoding,introducingthebasicconceptsofchannelcoding,channelcodingtheorem,linearblockcodesandcycliccodes;Chapter7isnetworkinformationsecurityandCryptography,introducesthebasicconceptsofcryptography,variousencryptionalgorithmsanddigitalsignaturetechnologies.

BookInformationII

Title:InformationTheory

Author:FuZuyun

Press:ElectronicIndustryPress

Publicationtime:May2007

ISBN:9787121042737

Format:16open

Price:38.00yuan

Introduction

"InformationTheory:BasicTheoryandApplication"(SecondEdition)systematicallydiscussedthebasictheoriesofShannon’sinformationtheoryandsomeapplicationissues,basicallycoveringallaspectsofinformationtheory.Thecontentincludes:thedefinitionandmeasurementofinformation;theinformationentropyofvariousdiscreteandcontinuoussources;thechannelcapacityofmemory,nomemory,discreteandcontinuouschannels;thethreebasictheoremsofShannon’sinformationtheory:distortion-freedatacompression(ieDistortion-freesourcecoding)practicalcodingalgorithmsandmethods,aswellasthebasiccontentandanalysismethodsofchannelerrorcorrectioncoding."InformationTheory:BasicTheoryandApplication"(2ndEdition)finallybrieflyintroducestheapplicationcontentofinformationtheoryandotherdisciplinessuchasthermodynamics,optics,statistics,biology,andmedicine.

Contents

Chapter1Introduction

Chapter3DiscreteChannelsandChannelCapacity

Chapter4WaveformSourcesandWaveformsChannel

Chapter5LosslessSourceCodingTheorem

Chapter6NoisyChannelCodingTheorem

Chapter7SourceCodingunderFidelityCriterion

Chapter8LosslessSourceCoding

Chapter9ChannelErrorCorrectionCoding

Chapter10NetworkInformationTheory

Chapter11BasicInformationTheoryofSecuritySystem

Chapter12RelationshipandApplicationofInformationTheoryandOtherSubjects

Appendix

BibliographyandLiterature

……

BookInformationThree

BasicInformation

Title:FundamentalsofInformationTheory

Author:TianBaoyu,YangJie,HeZhiqiang,WangXiaoxiang

Publisher:People’sPostsandTelecommunicationsPublishingHouse

ISBN:9787115177902

Publishingtime:August20081stedition

Formation:16open

Pages:275pages

Price:29.8yuan

Title:Answerstobasicexercisesofinformationtheory

Author:TianBaoyuYangJie,HeZhiqiang,XuWenjun,WangXiaoxiang

Publisher:People'sPostsandTelecommunicationsPublishingHouse

ISBN:9787115224552

Publishingtime:October2010firstedition

Format:16open

Numberofpages:242pages

Price:29yuan

Briefintroduction

"InformationTheoryFundamentals"Itistheaccumulationoftheauthor'smanyyearsofteachingandscientificresearchpractice.Itisbasedonabsorbingtheadvantagesofexcellentteachingmaterialsathomeandabroad,furtheroptimizingandintegratingtheteachingcontent,andimprovingandsupplementingit.Thebookisdividedinto12chapters,including:thebasicconceptsofinformation,thecontentandprogressofShannon'sinformationtheoryresearch,themeasurementofdiscreteinformation,discretesources,continuousinformationandcontinuoussources,distortionlesssourcecoding,discretechannelsandtheircapacities,Noisychannelcoding,waveformchannel,informationrate-distortionfunction,constrainedchannelanditscoding,preliminarynetworkinformationtheory,informationtheorymethodsandapplications,etc.

"InformationTheoryFundamentalExercisesAnswers"isasupportingteachingmaterialfortheundergraduatetextbook"InformationTheoryFundamentals".Themainpurposeistoprovidestudentswithmoredemonstrationsofbasicinformationtheoryproblemsolving,andtobroadenstudents’problem-solvingideas.Improvestudents'abilitytosolvebasicorcomprehensiveproblemsrelatedtoinformationtheory,andfurtherimprovethequalityoftheoreticalteachingofinformationtheory.Manyexercisesinthisbookcomefromtraditionalorclassictextbooksathomeandabroad,anditalsocontainsaconsiderablenumberoftypicalproblemsthathavebeenrefinedandverifiedthroughyearsofteachingpracticebyfrontlineteachers.Thisbookhasthesamestructureasthemaintextbook,anditalsocontains12chapters.Thecontentincludes4parts:"KnowledgeEssentials","ExemplarySolutions","ExercisesSolutions",and"SupplementaryProblemSolutions".

Contents

Chapter1Introduction

Chapter2MeasurementofDiscreteInformation

Chapter3DiscreteInformationSource

Chapter4ContinuousInformationandContinuousSources

Chapter5DistortionlessSourceCoding

Chapter6DiscreteChannelsandTheirCapacity

Chapter7NoisyChannelCoding

Chapter8WaveformChannel

Chapter9InformationRateDistortionFunction

Chapter10ConstrainedChannelandCoding

Chapter11PreliminaryNetworkInformationTheory

Chapter12InformationTheoryMethodsandApplications

……

BookInformationIV

InformationTheoryBasicCourse(SecondEdition)

Author:LiMei,editedbyLiYinongPress:BeijingUniversityofPostsandTelecommunicationsPress

PublishedTime:2008-10-1

Words:310000

Edition:2

Pages:217

Printingtime:2008-10-1

Format:16open

Numberofimpressions:1

Paper:Offsetpaper

ISBN:9787563518685

Packaging:Paperback

Category:Books>>IndustrialTechnology>>ElectronicCommunication>>Communication>>CommunicationTheory

Chapter1Introduction

1.1ConceptofInformation

1.2ResearchObject,PurposeandContentofInformationTheory

Chapter2MeasurementofInformation

2.1Self-informationandmutualinformation

2.1.1Self-information

2.1.2Mutualinformation

2.2Averageself-information

2.2.1Theconceptofaverageself-information

2.2.2Propertiesofentropyfunction

2.2.3Jointentropyandconditionalentropy

2.3Averagemutualinformation

2.3.1Theconceptofaveragemutualinformation

2.3.2Thenatureofaveragemutualinformation

2.3.3Dataprocessingtheorem

Exercises2

Chapter3InformationSourceandEntropyofInformationSource

3.1ClassificationofInformationSourceandItsMathematicalModel

3.2Discretesingle-symbolsource

3.3Discretemulti-symbolsource

3.3.1Discretestationarynon-memorysource

3.3.2DiscretestationarywithmemoryInformationsource

3.3.3Markovinformationsource

3.3.4Correlationandresidualdegreeofinformationsource

3.4LinkContinuedinformationsource

3.4.1Differentialentropyofcontinuousinformationsource

3.4.2Maximumentropyofcontinuousinformationsource

3.4.3EntropyofcontinuousinformationsourcePower

Exercise3

Chapter4ChannelsandChannelCapacity

4.1ClassificationofChannels

4.2DiscreteSingleSymbolChannelanditschannelcapacity

4.2.1Mathematicalmodelofdiscretesingle-symbolchannel

4.2.2Theconceptofchannelcapacity

4.2.3SeveralspecialchannelsChannelcapacity

4.2.4Channelcapacityofdiscretesymmetricalchannel

4.2.5Channelcapacityofgeneraldiscretechannel

4.2.6Channelcapacitytheorem

p>

4.2.7Iterativealgorithmofchannelcapacity

4.3Discretemulti-symbolchannelanditschannelcapacity

4.4Combinedchannelanditschannelcapacity

4.4.1Independentparallelchannel

4.4.2Cascadechannel

4.5Continuouschannelanditschannelcapacity

4.5.1ContinuousMutualinformationofrandomvariables

4.5.2ChannelcapacityofGaussianadditivechannel

4.5.3Channelcapacityofmulti-dimensionalGaussianadditivechannel

4.6Waveformchannelanditschannelcapacity

Exercise4

Chapter5LosslessSourceCoding

5.1RelatedConceptsofSourceCoding

5.1.1Encoder

5.1.2Classificationofcodes

5.2Fixed-lengthcodesandfixed-lengthsourcecodingtheorems

5.3Variable-lengthcodesandvariable-lengthsourcecodingtheorems

5.3.1Kraft'sinequalityandMcMman'sinequality

5.3.2Theonlydecodablecriterion

5.3.3Boundtheoremofaveragecodelengthforcompactcodes

5.3.4Undistortedvariable-lengthsourcecodingtheorem(Shannon'sfirsttheorem)

5.4Variable-lengthcodesCodingmethod

5.4.1Shannoncoding

5.4.2Shannon-Fano-Elliscoding

5.4.3BinaryHuffmancode

5.4.4r-aryHuffmancode

5.4.5Fenaucode

5.5Practicallosslesssourcecodingmethod

5.5.1Run-lengthcoding

5.5.2Arithmeticcoding

5.5.3LZWcoding

Exercises5

Part6ChapterNoisyChannelCoding

6.1Relatedconceptsofchannelcoding

6.1.1Errorprobabilityanddecodingrules

6.1.2ErrorprobabilityandCodingmethod

6.2Noisychannelcodingtheorem

6.3Upperboundoferrorprobability

6.4Errorcorrectioncoding

6.4.1Classificationoferrorcorrectioncodes

6.4.2Basicconceptsoferrorcorrectioncodes

6.4.3Linearblockcodes

6.4.4ConvolutionCode

Exercises6

Chapter7Distortion-limitedsourcecoding

7.1Distortionmeasurement

7.1.1Distortionfunction

7.1.2Averagedistortion

7.2Informationrate-distortionfunction

7.2.1Ddistortionpermissionchannel

7.2.2InformationDefinitionofrate-distortionfunction

7.2.3Thepropertiesofinformationrate-distortionfunctionR(D)

7.3Theoremofdistortion-limitedsourcecoding

7.4Calculationofinformationrate-distortionfunction

7.4.1ApplicationofparameterexpressiontocalculateR(D)

7.4.2Iterativealgorithmofrate-distortionfunction

7.5Commonlyuseddistortion-limitedsourcecodingmethods

7.5.1Quantizationcoding

7.5.2Subbandcoding

7.5.3Predictivecoding

7.5.4Transformationcoding

Exercises7

AppendixAPreliminaryknowledgeofmathematics

A.1Probabilitytheoryandrandomprocesses

A.1.1Basicconceptsofprobabilitytheory

A.1.2Randomvariablesandtheirdistribution

A.1.3Multidimensionalrandomvariablesandtheirdistribution

A.1.4Numericalcharacteristicsofrandomvariables

A.1.5Randomprocess

A.2ConvexfunctionandJensen'sinequality

A.3Lemmaofchannelcapacitytheorem

A.4Progressiveequipartitionandtypicalsequence

AppendixBHands-onoperation

B.1Iterativealgorithmofchannelcapacity

B.2Theonlydecodabledecisioncriterion

B.3Shannoncoding

B.4Huffmancoding

B.5Fanocoding

B.6LZWcoding

B.7BSCsimulator

B.8Hamming(7,4)codec

B.9communicationSystemSimulation

BookInformationFive

BasicInformation

NameofChineseBook:BasicInformationTheory

Englishbooktitle:ElementsofInformationTheory

Author:ThomasM.Cover,JoyA,Thomas

Translator:RuanJishou,ZhangHua

Press:MachineryIndustryPress

Edition:1stedition(2008January1st)

Format:16open

Pages:439

ISBN:9787111220404

Introduction

Thisbookisaclassicandconcisetextbookinthefieldofinformationtheory.Themaincontentincludes:theintroductionofentropy,informationsource,channelcapacity,ratedistortion,datacompressionandcodingtheoryandcomplexitytheory.Thebookalsointroducesnetworkinformationtheoryandhypothesistesting,andtakesthehorseracingmodelasastartingpoint,andincorporatestheresearchofthesecuritiesmarketintotheframeworkofinformationtheory,bringingnewinvestmentconceptsandresearchtotheresearchofinvestmentportfoliosfromanewperspective.Skill.Thisbookissuitableasaninformationtheorybasiccoursetextbookforseniorundergraduatesandgraduatestudentsinelectronicengineering,statistics,andtelecommunications,andcanalsobeusedasareferenceforresearchersandprofessionals.

Contents

Chapter1IntroductionandOverview

Chapter2Entropy,RelativeEntropyandMutualInformation

Chapter3AsymptoticEquivalence

Chapter4EntropyRateofRandomProcesses

Chapter5DataCompression

Chapter6GameandDataCompression

Chapter7ChannelCapacity

Chapter8DifferentialEntropy

Chapter9GaussianChannel

Chapter10RateDistortionTheory

Chapter11InformationTheoryandStatistics

Chapter12MaximumEntropy

Chapter13GeneralSourceCoding

Chapter14KolmogorovComplexDegree

Chapter15NetworkInformationTheory

Chapter16InformationTheoryandPortfolioTheory

Chapter17InequalitiesinInformationTheory

ReferenceLiterature