[{"data":1,"prerenderedAt":11697},["ShallowReactive",2],{"post-\u002Fblog\u002F2026\u002F2026-04-06-nonlinearity-with-activations":3},{"id":4,"title":5,"body":6,"cardimage":11683,"description":11667,"draft":11684,"enableComment":11685,"extension":11686,"image":11667,"meta":11687,"navigation":11685,"onday":11688,"path":11689,"seo":11690,"stem":11691,"summary":11692,"tags":11693,"__hash__":11696},"blog\u002Fblog\u002F2026\u002F2026-04-06-nonlinearity-with-activations.md","Nonlinearity with Activations",{"type":7,"value":8,"toc":11666},"minimark",[9,14,26,29,144,149,155,314,1141,1145,1151,1310,2081,2084,2087,2154,2400,2403,2451,2956,2967,2971,3117,3215,3277,3280,3555,3776,3779,4508,4515,4518,4853,5119,5182,5186,5189,5192,5375,5378,5539,5542,5776,5781,5784,6005,6008,6157,6168,6171,6269,6439,6626,6662,6665,6905,6979,7013,7053,7056,7059,7065,7068,7071,7076,7080,7082,7305,7395,7709,7712,7715,8412,8470,8473,8680,8683,8686,9209,9212,9431,9434,10080,10083,10086,10286,10289,10348,10351,10592,10595,10598,11349,11352,11426,11429,11433,11436,11440,11443,11446,11639,11642,11645,11649,11656,11660,11663],[10,11,13],"h2",{"id":12},"linear-combinations","Linear Combinations",[15,16,17,18,22,23,25],"p",{},"The concept of ",[19,20,21],"strong",{},"linearity"," runs through mathematics, physics, engineering, and many other fields. Simply put, ",[19,24,21],{}," describes two fundamental operational properties: additivity and homogeneity.",[15,27,28],{},"A relationship, function, or system is said to be linear if it satisfies the following two conditions simultaneously.",[15,30,31,32,80,81,111,112,143],{},"Consider a transformation or function ",[33,34,38,60],"span",{"className":35,"translate":37},[36],"katex","no",[33,39,42],{"className":40},[41],"katex-mathml",[43,44,46],"math",{"xmlns":45},"http:\u002F\u002Fwww.w3.org\u002F1998\u002FMath\u002FMathML",[47,48,49,56],"semantics",{},[50,51,52],"mrow",{},[53,54,55],"mi",{},"f",[57,58,55],"annotation",{"encoding":59},"application\u002Fx-tex",[33,61,65],{"className":62,"ariaHidden":64},[63],"katex-html","true",[33,66,69,74],{"className":67},[68],"base",[33,70],{"className":71,"style":73},[72],"strut","height:0.8889em;vertical-align:-0.1944em;",[33,75,55],{"className":76,"style":79},[77,78],"mord","mathnormal","margin-right:0.1076em;"," acting on inputs ",[33,82,84,98],{"className":83,"translate":37},[36],[33,85,87],{"className":86},[41],[43,88,89],{"xmlns":45},[47,90,91,96],{},[50,92,93],{},[53,94,95],{},"x",[57,97,95],{"encoding":59},[33,99,101],{"className":100,"ariaHidden":64},[63],[33,102,104,108],{"className":103},[68],[33,105],{"className":106,"style":107},[72],"height:0.4306em;",[33,109,95],{"className":110},[77,78]," and ",[33,113,115,129],{"className":114,"translate":37},[36],[33,116,118],{"className":117},[41],[43,119,120],{"xmlns":45},[47,121,122,127],{},[50,123,124],{},[53,125,126],{},"y",[57,128,126],{"encoding":59},[33,130,132],{"className":131,"ariaHidden":64},[63],[33,133,135,139],{"className":134},[68],[33,136],{"className":137,"style":138},[72],"height:0.625em;vertical-align:-0.1944em;",[33,140,126],{"className":141,"style":142},[77,78],"margin-right:0.0359em;",".",[145,146,148],"h3",{"id":147},"additivity","Additivity",[15,150,151,154],{},[19,152,153],{},"The sum of parts equals the whole."," That is, processing two inputs separately and then adding the results is the same as adding the inputs first and then processing the sum.",[33,156,159],{"className":157,"translate":37},[158],"katex-display",[33,160,162,213],{"className":161,"translate":37},[36],[33,163,165],{"className":164},[41],[43,166,168],{"xmlns":45,"display":167},"block",[47,169,170,210],{},[50,171,172,174,179,181,184,186,189,192,194,196,198,200,202,204,206,208],{},[53,173,55],{},[175,176,178],"mo",{"stretchy":177},"false","(",[53,180,95],{},[175,182,183],{},"+",[53,185,126],{},[175,187,188],{"stretchy":177},")",[175,190,191],{},"=",[53,193,55],{},[175,195,178],{"stretchy":177},[53,197,95],{},[175,199,188],{"stretchy":177},[175,201,183],{},[53,203,55],{},[175,205,178],{"stretchy":177},[53,207,126],{},[175,209,188],{"stretchy":177},[57,211,212],{"encoding":59},"f(x + y) = f(x) + f(y)",[33,214,216,245,269,296],{"className":215,"ariaHidden":64},[63],[33,217,219,223,226,230,233,238,242],{"className":218},[68],[33,220],{"className":221,"style":222},[72],"height:1em;vertical-align:-0.25em;",[33,224,55],{"className":225,"style":79},[77,78],[33,227,178],{"className":228},[229],"mopen",[33,231,95],{"className":232},[77,78],[33,234],{"className":235,"style":237},[236],"mspace","margin-right:0.2222em;",[33,239,183],{"className":240},[241],"mbin",[33,243],{"className":244,"style":237},[236],[33,246,248,251,254,258,262,266],{"className":247},[68],[33,249],{"className":250,"style":222},[72],[33,252,126],{"className":253,"style":142},[77,78],[33,255,188],{"className":256},[257],"mclose",[33,259],{"className":260,"style":261},[236],"margin-right:0.2778em;",[33,263,191],{"className":264},[265],"mrel",[33,267],{"className":268,"style":261},[236],[33,270,272,275,278,281,284,287,290,293],{"className":271},[68],[33,273],{"className":274,"style":222},[72],[33,276,55],{"className":277,"style":79},[77,78],[33,279,178],{"className":280},[229],[33,282,95],{"className":283},[77,78],[33,285,188],{"className":286},[257],[33,288],{"className":289,"style":237},[236],[33,291,183],{"className":292},[241],[33,294],{"className":295,"style":237},[236],[33,297,299,302,305,308,311],{"className":298},[68],[33,300],{"className":301,"style":222},[72],[33,303,55],{"className":304,"style":79},[77,78],[33,306,178],{"className":307},[229],[33,309,126],{"className":310,"style":142},[77,78],[33,312,188],{"className":313},[257],[315,316,317,398,725,841],"blockquote",{},[15,318,319,322,323,397],{},[19,320,321],{},"Example",": Multiplication ",[33,324,326,354],{"className":325,"translate":37},[36],[33,327,329],{"className":328},[41],[43,330,331],{"xmlns":45},[47,332,333,351],{},[50,334,335,337,339,341,343,345,349],{},[53,336,55],{},[175,338,178],{"stretchy":177},[53,340,95],{},[175,342,188],{"stretchy":177},[175,344,191],{},[346,347,348],"mn",{},"5",[53,350,95],{},[57,352,353],{"encoding":59},"f(x) = 5x",[33,355,357,384],{"className":356,"ariaHidden":64},[63],[33,358,360,363,366,369,372,375,378,381],{"className":359},[68],[33,361],{"className":362,"style":222},[72],[33,364,55],{"className":365,"style":79},[77,78],[33,367,178],{"className":368},[229],[33,370,95],{"className":371},[77,78],[33,373,188],{"className":374},[257],[33,376],{"className":377,"style":261},[236],[33,379,191],{"className":380},[265],[33,382],{"className":383,"style":261},[236],[33,385,387,391,394],{"className":386},[68],[33,388],{"className":389,"style":390},[72],"height:0.6444em;",[33,392,348],{"className":393},[77],[33,395,95],{"className":396},[77,78]," satisfies additivity.",[399,400,401],"ul",{},[402,403,404,405,574,575,724],"li",{},"Left: ",[33,406,408,456],{"className":407,"translate":37},[36],[33,409,411],{"className":410},[41],[43,412,413],{"xmlns":45},[47,414,415,453],{},[50,416,417,419,421,424,426,429,431,433,435,438,440,442,444,446,448,450],{},[53,418,55],{},[175,420,178],{"stretchy":177},[346,422,423],{},"2",[175,425,183],{},[346,427,428],{},"3",[175,430,188],{"stretchy":177},[175,432,191],{},[346,434,348],{},[175,436,437],{},"×",[175,439,178],{"stretchy":177},[346,441,423],{},[175,443,183],{},[346,445,428],{},[175,447,188],{"stretchy":177},[175,449,191],{},[346,451,452],{},"25",[57,454,455],{"encoding":59},"f(2+3) = 5 \\times (2+3) = 25",[33,457,459,483,504,523,544,565],{"className":458,"ariaHidden":64},[63],[33,460,462,465,468,471,474,477,480],{"className":461},[68],[33,463],{"className":464,"style":222},[72],[33,466,55],{"className":467,"style":79},[77,78],[33,469,178],{"className":470},[229],[33,472,423],{"className":473},[77],[33,475],{"className":476,"style":237},[236],[33,478,183],{"className":479},[241],[33,481],{"className":482,"style":237},[236],[33,484,486,489,492,495,498,501],{"className":485},[68],[33,487],{"className":488,"style":222},[72],[33,490,428],{"className":491},[77],[33,493,188],{"className":494},[257],[33,496],{"className":497,"style":261},[236],[33,499,191],{"className":500},[265],[33,502],{"className":503,"style":261},[236],[33,505,507,511,514,517,520],{"className":506},[68],[33,508],{"className":509,"style":510},[72],"height:0.7278em;vertical-align:-0.0833em;",[33,512,348],{"className":513},[77],[33,515],{"className":516,"style":237},[236],[33,518,437],{"className":519},[241],[33,521],{"className":522,"style":237},[236],[33,524,526,529,532,535,538,541],{"className":525},[68],[33,527],{"className":528,"style":222},[72],[33,530,178],{"className":531},[229],[33,533,423],{"className":534},[77],[33,536],{"className":537,"style":237},[236],[33,539,183],{"className":540},[241],[33,542],{"className":543,"style":237},[236],[33,545,547,550,553,556,559,562],{"className":546},[68],[33,548],{"className":549,"style":222},[72],[33,551,428],{"className":552},[77],[33,554,188],{"className":555},[257],[33,557],{"className":558,"style":261},[236],[33,560,191],{"className":561},[265],[33,563],{"className":564,"style":261},[236],[33,566,568,571],{"className":567},[68],[33,569],{"className":570,"style":390},[72],[33,572,452],{"className":573},[77],", Right: ",[33,576,578,622],{"className":577,"translate":37},[36],[33,579,581],{"className":580},[41],[43,582,583],{"xmlns":45},[47,584,585,619],{},[50,586,587,589,591,593,595,597,599,601,603,605,607,610,612,615,617],{},[53,588,55],{},[175,590,178],{"stretchy":177},[346,592,423],{},[175,594,188],{"stretchy":177},[175,596,183],{},[53,598,55],{},[175,600,178],{"stretchy":177},[346,602,428],{},[175,604,188],{"stretchy":177},[175,606,191],{},[346,608,609],{},"10",[175,611,183],{},[346,613,614],{},"15",[175,616,191],{},[346,618,452],{},[57,620,621],{"encoding":59},"f(2) + f(3) = 10 + 15 = 25",[33,623,625,652,679,697,715],{"className":624,"ariaHidden":64},[63],[33,626,628,631,634,637,640,643,646,649],{"className":627},[68],[33,629],{"className":630,"style":222},[72],[33,632,55],{"className":633,"style":79},[77,78],[33,635,178],{"className":636},[229],[33,638,423],{"className":639},[77],[33,641,188],{"className":642},[257],[33,644],{"className":645,"style":237},[236],[33,647,183],{"className":648},[241],[33,650],{"className":651,"style":237},[236],[33,653,655,658,661,664,667,670,673,676],{"className":654},[68],[33,656],{"className":657,"style":222},[72],[33,659,55],{"className":660,"style":79},[77,78],[33,662,178],{"className":663},[229],[33,665,428],{"className":666},[77],[33,668,188],{"className":669},[257],[33,671],{"className":672,"style":261},[236],[33,674,191],{"className":675},[265],[33,677],{"className":678,"style":261},[236],[33,680,682,685,688,691,694],{"className":681},[68],[33,683],{"className":684,"style":510},[72],[33,686,609],{"className":687},[77],[33,689],{"className":690,"style":237},[236],[33,692,183],{"className":693},[241],[33,695],{"className":696,"style":237},[236],[33,698,700,703,706,709,712],{"className":699},[68],[33,701],{"className":702,"style":390},[72],[33,704,614],{"className":705},[77],[33,707],{"className":708,"style":261},[236],[33,710,191],{"className":711},[265],[33,713],{"className":714,"style":261},[236],[33,716,718,721],{"className":717},[68],[33,719],{"className":720,"style":390},[72],[33,722,452],{"className":723},[77],", equal.",[15,726,727,730,731,840],{},[19,728,729],{},"Counterexample",": The square function ",[33,732,734,763],{"className":733,"translate":37},[36],[33,735,737],{"className":736},[41],[43,738,739],{"xmlns":45},[47,740,741,760],{},[50,742,743,745,747,749,751,753],{},[53,744,55],{},[175,746,178],{"stretchy":177},[53,748,95],{},[175,750,188],{"stretchy":177},[175,752,191],{},[754,755,756,758],"msup",{},[53,757,95],{},[346,759,423],{},[57,761,762],{"encoding":59},"f(x) = x^2",[33,764,766,793],{"className":765,"ariaHidden":64},[63],[33,767,769,772,775,778,781,784,787,790],{"className":768},[68],[33,770],{"className":771,"style":222},[72],[33,773,55],{"className":774,"style":79},[77,78],[33,776,178],{"className":777},[229],[33,779,95],{"className":780},[77,78],[33,782,188],{"className":783},[257],[33,785],{"className":786,"style":261},[236],[33,788,191],{"className":789},[265],[33,791],{"className":792,"style":261},[236],[33,794,796,800],{"className":795},[68],[33,797],{"className":798,"style":799},[72],"height:0.8141em;",[33,801,803,806],{"className":802},[77],[33,804,95],{"className":805},[77,78],[33,807,810],{"className":808},[809],"msupsub",[33,811,814],{"className":812},[813],"vlist-t",[33,815,818],{"className":816},[817],"vlist-r",[33,819,822],{"className":820,"style":799},[821],"vlist",[33,823,825,830],{"style":824},"top:-3.063em;margin-right:0.05em;",[33,826],{"className":827,"style":829},[828],"pstrut","height:2.7em;",[33,831,837],{"className":832},[833,834,835,836],"sizing","reset-size6","size3","mtight",[33,838,423],{"className":839},[77,836]," does not satisfy additivity.",[399,842,843],{},[402,844,404,845,574,959,1140],{},[33,846,848,878],{"className":847,"translate":37},[36],[33,849,851],{"className":850},[41],[43,852,853],{"xmlns":45},[47,854,855,875],{},[50,856,857,859,861,863,865,871,873],{},[175,858,178],{"stretchy":177},[346,860,423],{},[175,862,183],{},[346,864,428],{},[754,866,867,869],{},[175,868,188],{"stretchy":177},[346,870,423],{},[175,872,191],{},[346,874,452],{},[57,876,877],{"encoding":59},"(2+3)^2 = 25",[33,879,881,902,950],{"className":880,"ariaHidden":64},[63],[33,882,884,887,890,893,896,899],{"className":883},[68],[33,885],{"className":886,"style":222},[72],[33,888,178],{"className":889},[229],[33,891,423],{"className":892},[77],[33,894],{"className":895,"style":237},[236],[33,897,183],{"className":898},[241],[33,900],{"className":901,"style":237},[236],[33,903,905,909,912,941,944,947],{"className":904},[68],[33,906],{"className":907,"style":908},[72],"height:1.0641em;vertical-align:-0.25em;",[33,910,428],{"className":911},[77],[33,913,915,918],{"className":914},[257],[33,916,188],{"className":917},[257],[33,919,921],{"className":920},[809],[33,922,924],{"className":923},[813],[33,925,927],{"className":926},[817],[33,928,930],{"className":929,"style":799},[821],[33,931,932,935],{"style":824},[33,933],{"className":934,"style":829},[828],[33,936,938],{"className":937},[833,834,835,836],[33,939,423],{"className":940},[77,836],[33,942],{"className":943,"style":261},[236],[33,945,191],{"className":946},[265],[33,948],{"className":949,"style":261},[236],[33,951,953,956],{"className":952},[68],[33,954],{"className":955,"style":390},[72],[33,957,452],{"className":958},[77],[33,960,962,1003],{"className":961,"translate":37},[36],[33,963,965],{"className":964},[41],[43,966,967],{"xmlns":45},[47,968,969,1000],{},[50,970,971,977,979,985,987,990,992,995,997],{},[754,972,973,975],{},[346,974,423],{},[346,976,423],{},[175,978,183],{},[754,980,981,983],{},[346,982,428],{},[346,984,423],{},[175,986,191],{},[346,988,989],{},"4",[175,991,183],{},[346,993,994],{},"9",[175,996,191],{},[346,998,999],{},"13",[57,1001,1002],{"encoding":59},"2^2 + 3^2 = 4+9=13",[33,1004,1006,1051,1095,1113,1131],{"className":1005,"ariaHidden":64},[63],[33,1007,1009,1013,1042,1045,1048],{"className":1008},[68],[33,1010],{"className":1011,"style":1012},[72],"height:0.8974em;vertical-align:-0.0833em;",[33,1014,1016,1019],{"className":1015},[77],[33,1017,423],{"className":1018},[77],[33,1020,1022],{"className":1021},[809],[33,1023,1025],{"className":1024},[813],[33,1026,1028],{"className":1027},[817],[33,1029,1031],{"className":1030,"style":799},[821],[33,1032,1033,1036],{"style":824},[33,1034],{"className":1035,"style":829},[828],[33,1037,1039],{"className":1038},[833,834,835,836],[33,1040,423],{"className":1041},[77,836],[33,1043],{"className":1044,"style":237},[236],[33,1046,183],{"className":1047},[241],[33,1049],{"className":1050,"style":237},[236],[33,1052,1054,1057,1086,1089,1092],{"className":1053},[68],[33,1055],{"className":1056,"style":799},[72],[33,1058,1060,1063],{"className":1059},[77],[33,1061,428],{"className":1062},[77],[33,1064,1066],{"className":1065},[809],[33,1067,1069],{"className":1068},[813],[33,1070,1072],{"className":1071},[817],[33,1073,1075],{"className":1074,"style":799},[821],[33,1076,1077,1080],{"style":824},[33,1078],{"className":1079,"style":829},[828],[33,1081,1083],{"className":1082},[833,834,835,836],[33,1084,423],{"className":1085},[77,836],[33,1087],{"className":1088,"style":261},[236],[33,1090,191],{"className":1091},[265],[33,1093],{"className":1094,"style":261},[236],[33,1096,1098,1101,1104,1107,1110],{"className":1097},[68],[33,1099],{"className":1100,"style":510},[72],[33,1102,989],{"className":1103},[77],[33,1105],{"className":1106,"style":237},[236],[33,1108,183],{"className":1109},[241],[33,1111],{"className":1112,"style":237},[236],[33,1114,1116,1119,1122,1125,1128],{"className":1115},[68],[33,1117],{"className":1118,"style":390},[72],[33,1120,994],{"className":1121},[77],[33,1123],{"className":1124,"style":261},[236],[33,1126,191],{"className":1127},[265],[33,1129],{"className":1130,"style":261},[236],[33,1132,1134,1137],{"className":1133},[68],[33,1135],{"className":1136,"style":390},[72],[33,1138,999],{"className":1139},[77],", not equal.",[145,1142,1144],{"id":1143},"homogeneity","Homogeneity",[15,1146,1147,1150],{},[19,1148,1149],{},"Scaling the input scales the output by the same factor."," That is, multiplying an input by a constant (scalar) yields the original output multiplied by the same constant.",[33,1152,1154],{"className":1153,"translate":37},[158],[33,1155,1157,1208],{"className":1156,"translate":37},[36],[33,1158,1160],{"className":1159},[41],[43,1161,1162],{"xmlns":45,"display":167},[47,1163,1164,1205],{},[50,1165,1166,1168,1170,1173,1176,1178,1180,1182,1184,1186,1188,1190,1192,1194,1197,1199,1203],{},[53,1167,55],{},[175,1169,178],{"stretchy":177},[53,1171,1172],{},"c",[175,1174,1175],{},"⋅",[53,1177,95],{},[175,1179,188],{"stretchy":177},[175,1181,191],{},[53,1183,1172],{},[175,1185,1175],{},[53,1187,55],{},[175,1189,178],{"stretchy":177},[53,1191,95],{},[175,1193,188],{"stretchy":177},[236,1195],{"width":1196},"1em",[175,1198,178],{"stretchy":177},[1200,1201,1202],"mtext",{},"c is any constant",[175,1204,188],{"stretchy":177},[57,1206,1207],{"encoding":59},"f(c \\cdot x) = c \\cdot f(x) \\quad (\\text{c is any constant})",[33,1209,1211,1235,1256,1275],{"className":1210,"ariaHidden":64},[63],[33,1212,1214,1217,1220,1223,1226,1229,1232],{"className":1213},[68],[33,1215],{"className":1216,"style":222},[72],[33,1218,55],{"className":1219,"style":79},[77,78],[33,1221,178],{"className":1222},[229],[33,1224,1172],{"className":1225},[77,78],[33,1227],{"className":1228,"style":237},[236],[33,1230,1175],{"className":1231},[241],[33,1233],{"className":1234,"style":237},[236],[33,1236,1238,1241,1244,1247,1250,1253],{"className":1237},[68],[33,1239],{"className":1240,"style":222},[72],[33,1242,95],{"className":1243},[77,78],[33,1245,188],{"className":1246},[257],[33,1248],{"className":1249,"style":261},[236],[33,1251,191],{"className":1252},[265],[33,1254],{"className":1255,"style":261},[236],[33,1257,1259,1263,1266,1269,1272],{"className":1258},[68],[33,1260],{"className":1261,"style":1262},[72],"height:0.4445em;",[33,1264,1172],{"className":1265},[77,78],[33,1267],{"className":1268,"style":237},[236],[33,1270,1175],{"className":1271},[241],[33,1273],{"className":1274,"style":237},[236],[33,1276,1278,1281,1284,1287,1290,1293,1297,1300,1307],{"className":1277},[68],[33,1279],{"className":1280,"style":222},[72],[33,1282,55],{"className":1283,"style":79},[77,78],[33,1285,178],{"className":1286},[229],[33,1288,95],{"className":1289},[77,78],[33,1291,188],{"className":1292},[257],[33,1294],{"className":1295,"style":1296},[236],"margin-right:1em;",[33,1298,178],{"className":1299},[229],[33,1301,1304],{"className":1302},[77,1303],"text",[33,1305,1202],{"className":1306},[77],[33,1308,188],{"className":1309},[257],[315,1311,1312,1388,1658,1757],{},[15,1313,1314,1316,1317,1387],{},[19,1315,321],{},": ",[33,1318,1320,1345],{"className":1319,"translate":37},[36],[33,1321,1323],{"className":1322},[41],[43,1324,1325],{"xmlns":45},[47,1326,1327,1343],{},[50,1328,1329,1331,1333,1335,1337,1339,1341],{},[53,1330,55],{},[175,1332,178],{"stretchy":177},[53,1334,95],{},[175,1336,188],{"stretchy":177},[175,1338,191],{},[346,1340,348],{},[53,1342,95],{},[57,1344,353],{"encoding":59},[33,1346,1348,1375],{"className":1347,"ariaHidden":64},[63],[33,1349,1351,1354,1357,1360,1363,1366,1369,1372],{"className":1350},[68],[33,1352],{"className":1353,"style":222},[72],[33,1355,55],{"className":1356,"style":79},[77,78],[33,1358,178],{"className":1359},[229],[33,1361,95],{"className":1362},[77,78],[33,1364,188],{"className":1365},[257],[33,1367],{"className":1368,"style":261},[236],[33,1370,191],{"className":1371},[265],[33,1373],{"className":1374,"style":261},[236],[33,1376,1378,1381,1384],{"className":1377},[68],[33,1379],{"className":1380,"style":390},[72],[33,1382,348],{"className":1383},[77],[33,1385,95],{"className":1386},[77,78]," satisfies homogeneity.",[399,1389,1390],{},[402,1391,404,1392,574,1526,724],{},[33,1393,1395,1433],{"className":1394,"translate":37},[36],[33,1396,1398],{"className":1397},[41],[43,1399,1400],{"xmlns":45},[47,1401,1402,1430],{},[50,1403,1404,1406,1408,1410,1412,1414,1416,1418,1420,1422,1425,1427],{},[53,1405,55],{},[175,1407,178],{"stretchy":177},[346,1409,989],{},[175,1411,437],{},[346,1413,428],{},[175,1415,188],{"stretchy":177},[175,1417,191],{},[346,1419,348],{},[175,1421,437],{},[346,1423,1424],{},"12",[175,1426,191],{},[346,1428,1429],{},"60",[57,1431,1432],{"encoding":59},"f(4 \\times 3) = 5 \\times 12 = 60",[33,1434,1436,1460,1481,1499,1517],{"className":1435,"ariaHidden":64},[63],[33,1437,1439,1442,1445,1448,1451,1454,1457],{"className":1438},[68],[33,1440],{"className":1441,"style":222},[72],[33,1443,55],{"className":1444,"style":79},[77,78],[33,1446,178],{"className":1447},[229],[33,1449,989],{"className":1450},[77],[33,1452],{"className":1453,"style":237},[236],[33,1455,437],{"className":1456},[241],[33,1458],{"className":1459,"style":237},[236],[33,1461,1463,1466,1469,1472,1475,1478],{"className":1462},[68],[33,1464],{"className":1465,"style":222},[72],[33,1467,428],{"className":1468},[77],[33,1470,188],{"className":1471},[257],[33,1473],{"className":1474,"style":261},[236],[33,1476,191],{"className":1477},[265],[33,1479],{"className":1480,"style":261},[236],[33,1482,1484,1487,1490,1493,1496],{"className":1483},[68],[33,1485],{"className":1486,"style":510},[72],[33,1488,348],{"className":1489},[77],[33,1491],{"className":1492,"style":237},[236],[33,1494,437],{"className":1495},[241],[33,1497],{"className":1498,"style":237},[236],[33,1500,1502,1505,1508,1511,1514],{"className":1501},[68],[33,1503],{"className":1504,"style":390},[72],[33,1506,1424],{"className":1507},[77],[33,1509],{"className":1510,"style":261},[236],[33,1512,191],{"className":1513},[265],[33,1515],{"className":1516,"style":261},[236],[33,1518,1520,1523],{"className":1519},[68],[33,1521],{"className":1522,"style":390},[72],[33,1524,1429],{"className":1525},[77],[33,1527,1529,1565],{"className":1528,"translate":37},[36],[33,1530,1532],{"className":1531},[41],[43,1533,1534],{"xmlns":45},[47,1535,1536,1562],{},[50,1537,1538,1540,1542,1544,1546,1548,1550,1552,1554,1556,1558,1560],{},[346,1539,989],{},[175,1541,437],{},[53,1543,55],{},[175,1545,178],{"stretchy":177},[346,1547,428],{},[175,1549,188],{"stretchy":177},[175,1551,191],{},[346,1553,989],{},[175,1555,437],{},[346,1557,614],{},[175,1559,191],{},[346,1561,1429],{},[57,1563,1564],{"encoding":59},"4 \\times f(3) = 4 \\times 15 = 60",[33,1566,1568,1586,1613,1631,1649],{"className":1567,"ariaHidden":64},[63],[33,1569,1571,1574,1577,1580,1583],{"className":1570},[68],[33,1572],{"className":1573,"style":510},[72],[33,1575,989],{"className":1576},[77],[33,1578],{"className":1579,"style":237},[236],[33,1581,437],{"className":1582},[241],[33,1584],{"className":1585,"style":237},[236],[33,1587,1589,1592,1595,1598,1601,1604,1607,1610],{"className":1588},[68],[33,1590],{"className":1591,"style":222},[72],[33,1593,55],{"className":1594,"style":79},[77,78],[33,1596,178],{"className":1597},[229],[33,1599,428],{"className":1600},[77],[33,1602,188],{"className":1603},[257],[33,1605],{"className":1606,"style":261},[236],[33,1608,191],{"className":1609},[265],[33,1611],{"className":1612,"style":261},[236],[33,1614,1616,1619,1622,1625,1628],{"className":1615},[68],[33,1617],{"className":1618,"style":510},[72],[33,1620,989],{"className":1621},[77],[33,1623],{"className":1624,"style":237},[236],[33,1626,437],{"className":1627},[241],[33,1629],{"className":1630,"style":237},[236],[33,1632,1634,1637,1640,1643,1646],{"className":1633},[68],[33,1635],{"className":1636,"style":390},[72],[33,1638,614],{"className":1639},[77],[33,1641],{"className":1642,"style":261},[236],[33,1644,191],{"className":1645},[265],[33,1647],{"className":1648,"style":261},[236],[33,1650,1652,1655],{"className":1651},[68],[33,1653],{"className":1654,"style":390},[72],[33,1656,1429],{"className":1657},[77],[15,1659,1660,1662,1663,1756],{},[19,1661,729],{},": Adding a constant ",[33,1664,1666,1696],{"className":1665,"translate":37},[36],[33,1667,1669],{"className":1668},[41],[43,1670,1671],{"xmlns":45},[47,1672,1673,1693],{},[50,1674,1675,1677,1679,1681,1683,1685,1687,1689,1691],{},[53,1676,55],{},[175,1678,178],{"stretchy":177},[53,1680,95],{},[175,1682,188],{"stretchy":177},[175,1684,191],{},[346,1686,348],{},[53,1688,95],{},[175,1690,183],{},[346,1692,423],{},[57,1694,1695],{"encoding":59},"f(x) = 5x + 2",[33,1697,1699,1726,1747],{"className":1698,"ariaHidden":64},[63],[33,1700,1702,1705,1708,1711,1714,1717,1720,1723],{"className":1701},[68],[33,1703],{"className":1704,"style":222},[72],[33,1706,55],{"className":1707,"style":79},[77,78],[33,1709,178],{"className":1710},[229],[33,1712,95],{"className":1713},[77,78],[33,1715,188],{"className":1716},[257],[33,1718],{"className":1719,"style":261},[236],[33,1721,191],{"className":1722},[265],[33,1724],{"className":1725,"style":261},[236],[33,1727,1729,1732,1735,1738,1741,1744],{"className":1728},[68],[33,1730],{"className":1731,"style":510},[72],[33,1733,348],{"className":1734},[77],[33,1736,95],{"className":1737},[77,78],[33,1739],{"className":1740,"style":237},[236],[33,1742,183],{"className":1743},[241],[33,1745],{"className":1746,"style":237},[236],[33,1748,1750,1753],{"className":1749},[68],[33,1751],{"className":1752,"style":390},[72],[33,1754,423],{"className":1755},[77]," does not satisfy homogeneity.",[399,1758,1759],{},[402,1760,404,1761,574,1916,1140],{},[33,1762,1764,1805],{"className":1763,"translate":37},[36],[33,1765,1767],{"className":1766},[41],[43,1768,1769],{"xmlns":45},[47,1770,1771,1802],{},[50,1772,1773,1775,1777,1779,1781,1783,1785,1787,1789,1791,1793,1795,1797,1799],{},[53,1774,55],{},[175,1776,178],{"stretchy":177},[346,1778,989],{},[175,1780,437],{},[346,1782,428],{},[175,1784,188],{"stretchy":177},[175,1786,191],{},[346,1788,348],{},[175,1790,437],{},[346,1792,1424],{},[175,1794,183],{},[346,1796,423],{},[175,1798,191],{},[346,1800,1801],{},"62",[57,1803,1804],{"encoding":59},"f(4 \\times 3) = 5 \\times 12 + 2 = 62",[33,1806,1808,1832,1853,1871,1889,1907],{"className":1807,"ariaHidden":64},[63],[33,1809,1811,1814,1817,1820,1823,1826,1829],{"className":1810},[68],[33,1812],{"className":1813,"style":222},[72],[33,1815,55],{"className":1816,"style":79},[77,78],[33,1818,178],{"className":1819},[229],[33,1821,989],{"className":1822},[77],[33,1824],{"className":1825,"style":237},[236],[33,1827,437],{"className":1828},[241],[33,1830],{"className":1831,"style":237},[236],[33,1833,1835,1838,1841,1844,1847,1850],{"className":1834},[68],[33,1836],{"className":1837,"style":222},[72],[33,1839,428],{"className":1840},[77],[33,1842,188],{"className":1843},[257],[33,1845],{"className":1846,"style":261},[236],[33,1848,191],{"className":1849},[265],[33,1851],{"className":1852,"style":261},[236],[33,1854,1856,1859,1862,1865,1868],{"className":1855},[68],[33,1857],{"className":1858,"style":510},[72],[33,1860,348],{"className":1861},[77],[33,1863],{"className":1864,"style":237},[236],[33,1866,437],{"className":1867},[241],[33,1869],{"className":1870,"style":237},[236],[33,1872,1874,1877,1880,1883,1886],{"className":1873},[68],[33,1875],{"className":1876,"style":510},[72],[33,1878,1424],{"className":1879},[77],[33,1881],{"className":1882,"style":237},[236],[33,1884,183],{"className":1885},[241],[33,1887],{"className":1888,"style":237},[236],[33,1890,1892,1895,1898,1901,1904],{"className":1891},[68],[33,1893],{"className":1894,"style":390},[72],[33,1896,423],{"className":1897},[77],[33,1899],{"className":1900,"style":261},[236],[33,1902,191],{"className":1903},[265],[33,1905],{"className":1906,"style":261},[236],[33,1908,1910,1913],{"className":1909},[68],[33,1911],{"className":1912,"style":390},[72],[33,1914,1801],{"className":1915},[77],[33,1917,1919,1964],{"className":1918,"translate":37},[36],[33,1920,1922],{"className":1921},[41],[43,1923,1924],{"xmlns":45},[47,1925,1926,1961],{},[50,1927,1928,1930,1932,1934,1936,1938,1940,1942,1944,1946,1948,1950,1952,1954,1956,1958],{},[346,1929,989],{},[175,1931,437],{},[53,1933,55],{},[175,1935,178],{"stretchy":177},[346,1937,428],{},[175,1939,188],{"stretchy":177},[175,1941,191],{},[346,1943,989],{},[175,1945,437],{},[175,1947,178],{"stretchy":177},[346,1949,614],{},[175,1951,183],{},[346,1953,423],{},[175,1955,188],{"stretchy":177},[175,1957,191],{},[346,1959,1960],{},"68",[57,1962,1963],{"encoding":59},"4 \\times f(3) = 4 \\times (15+2) = 68",[33,1965,1967,1985,2012,2030,2051,2072],{"className":1966,"ariaHidden":64},[63],[33,1968,1970,1973,1976,1979,1982],{"className":1969},[68],[33,1971],{"className":1972,"style":510},[72],[33,1974,989],{"className":1975},[77],[33,1977],{"className":1978,"style":237},[236],[33,1980,437],{"className":1981},[241],[33,1983],{"className":1984,"style":237},[236],[33,1986,1988,1991,1994,1997,2000,2003,2006,2009],{"className":1987},[68],[33,1989],{"className":1990,"style":222},[72],[33,1992,55],{"className":1993,"style":79},[77,78],[33,1995,178],{"className":1996},[229],[33,1998,428],{"className":1999},[77],[33,2001,188],{"className":2002},[257],[33,2004],{"className":2005,"style":261},[236],[33,2007,191],{"className":2008},[265],[33,2010],{"className":2011,"style":261},[236],[33,2013,2015,2018,2021,2024,2027],{"className":2014},[68],[33,2016],{"className":2017,"style":510},[72],[33,2019,989],{"className":2020},[77],[33,2022],{"className":2023,"style":237},[236],[33,2025,437],{"className":2026},[241],[33,2028],{"className":2029,"style":237},[236],[33,2031,2033,2036,2039,2042,2045,2048],{"className":2032},[68],[33,2034],{"className":2035,"style":222},[72],[33,2037,178],{"className":2038},[229],[33,2040,614],{"className":2041},[77],[33,2043],{"className":2044,"style":237},[236],[33,2046,183],{"className":2047},[241],[33,2049],{"className":2050,"style":237},[236],[33,2052,2054,2057,2060,2063,2066,2069],{"className":2053},[68],[33,2055],{"className":2056,"style":222},[72],[33,2058,423],{"className":2059},[77],[33,2061,188],{"className":2062},[257],[33,2064],{"className":2065,"style":261},[236],[33,2067,191],{"className":2068},[265],[33,2070],{"className":2071,"style":261},[236],[33,2073,2075,2078],{"className":2074},[68],[33,2076],{"className":2077,"style":390},[72],[33,2079,1960],{"className":2080},[77],[15,2082,2083],{},"In the definition above, satisfying both additivity and homogeneity is sufficient and necessary for linearity.",[15,2085,2086],{},"Many people mistakenly think that \"linear\" simply means \"the graph is a straight line.\" This understanding is inaccurate.",[15,2088,2089,2092,2093,2096,2097,2153],{},[19,2090,2091],{},"Truly linear",": A straight line that ",[19,2094,2095],{},"must pass through the origin",". For example, ",[33,2098,2100,2120],{"className":2099,"translate":37},[36],[33,2101,2103],{"className":2102},[41],[43,2104,2105],{"xmlns":45},[47,2106,2107,2117],{},[50,2108,2109,2111,2113,2115],{},[53,2110,126],{},[175,2112,191],{},[346,2114,423],{},[53,2116,95],{},[57,2118,2119],{"encoding":59},"y = 2x",[33,2121,2123,2141],{"className":2122,"ariaHidden":64},[63],[33,2124,2126,2129,2132,2135,2138],{"className":2125},[68],[33,2127],{"className":2128,"style":138},[72],[33,2130,126],{"className":2131,"style":142},[77,78],[33,2133],{"className":2134,"style":261},[236],[33,2136,191],{"className":2137},[265],[33,2139],{"className":2140,"style":261},[236],[33,2142,2144,2147,2150],{"className":2143},[68],[33,2145],{"className":2146,"style":390},[72],[33,2148,423],{"className":2149},[77],[33,2151,95],{"className":2152},[77,78],". It satisfies both additivity and homogeneity. If the input is 0, the output must also be 0.",[15,2155,2156,2157,2187,2188,2267,2268,2399],{},"Strictly speaking, if a bias term ",[33,2158,2160,2174],{"className":2159,"translate":37},[36],[33,2161,2163],{"className":2162},[41],[43,2164,2165],{"xmlns":45},[47,2166,2167,2172],{},[50,2168,2169],{},[53,2170,2171],{},"b",[57,2173,2171],{"encoding":59},[33,2175,2177],{"className":2176,"ariaHidden":64},[63],[33,2178,2180,2184],{"className":2179},[68],[33,2181],{"className":2182,"style":2183},[72],"height:0.6944em;",[33,2185,2171],{"className":2186},[77,78]," is added, it becomes affine. It is a straight line, but not necessarily passing through the origin. For example, ",[33,2189,2191,2216],{"className":2190,"translate":37},[36],[33,2192,2194],{"className":2193},[41],[43,2195,2196],{"xmlns":45},[47,2197,2198,2213],{},[50,2199,2200,2202,2204,2206,2208,2210],{},[53,2201,126],{},[175,2203,191],{},[346,2205,423],{},[53,2207,95],{},[175,2209,183],{},[346,2211,2212],{},"1",[57,2214,2215],{"encoding":59},"y = 2x + 1",[33,2217,2219,2237,2258],{"className":2218,"ariaHidden":64},[63],[33,2220,2222,2225,2228,2231,2234],{"className":2221},[68],[33,2223],{"className":2224,"style":138},[72],[33,2226,126],{"className":2227,"style":142},[77,78],[33,2229],{"className":2230,"style":261},[236],[33,2232,191],{"className":2233},[265],[33,2235],{"className":2236,"style":261},[236],[33,2238,2240,2243,2246,2249,2252,2255],{"className":2239},[68],[33,2241],{"className":2242,"style":510},[72],[33,2244,423],{"className":2245},[77],[33,2247,95],{"className":2248},[77,78],[33,2250],{"className":2251,"style":237},[236],[33,2253,183],{"className":2254},[241],[33,2256],{"className":2257,"style":237},[236],[33,2259,2261,2264],{"className":2260},[68],[33,2262],{"className":2263,"style":390},[72],[33,2265,2212],{"className":2266},[77],". It does not satisfy homogeneity (since ",[33,2269,2271,2302],{"className":2270,"translate":37},[36],[33,2272,2274],{"className":2273},[41],[43,2275,2276],{"xmlns":45},[47,2277,2278,2299],{},[50,2279,2280,2282,2284,2287,2289,2291,2293,2297],{},[53,2281,55],{},[175,2283,178],{"stretchy":177},[346,2285,2286],{},"0",[175,2288,188],{"stretchy":177},[175,2290,191],{},[346,2292,2212],{},[175,2294,2296],{"mathvariant":2295},"normal","≠",[346,2298,2286],{},[57,2300,2301],{"encoding":59},"f(0)=1 \\neq 0",[33,2303,2305,2332,2390],{"className":2304,"ariaHidden":64},[63],[33,2306,2308,2311,2314,2317,2320,2323,2326,2329],{"className":2307},[68],[33,2309],{"className":2310,"style":222},[72],[33,2312,55],{"className":2313,"style":79},[77,78],[33,2315,178],{"className":2316},[229],[33,2318,2286],{"className":2319},[77],[33,2321,188],{"className":2322},[257],[33,2324],{"className":2325,"style":261},[236],[33,2327,191],{"className":2328},[265],[33,2330],{"className":2331,"style":261},[236],[33,2333,2335,2338,2341,2344,2387],{"className":2334},[68],[33,2336],{"className":2337,"style":73},[72],[33,2339,2212],{"className":2340},[77],[33,2342],{"className":2343,"style":261},[236],[33,2345,2347,2380,2384],{"className":2346},[265],[33,2348,2350],{"className":2349},[265],[33,2351,2354],{"className":2352},[77,2353],"vbox",[33,2355,2358],{"className":2356},[2357],"thinbox",[33,2359,2362,2365,2376],{"className":2360},[2361],"rlap",[33,2363],{"className":2364,"style":73},[72],[33,2366,2369],{"className":2367},[2368],"inner",[33,2370,2372],{"className":2371},[77],[33,2373,2375],{"className":2374},[265],"",[33,2377],{"className":2378},[2379],"fix",[33,2381],{"className":2382},[236,2383],"nobreak",[33,2385,191],{"className":2386},[265],[33,2388],{"className":2389,"style":261},[236],[33,2391,2393,2396],{"className":2392},[68],[33,2394],{"className":2395,"style":390},[72],[33,2397,2286],{"className":2398},[77],"). Although it is often called a \"linear function,\" in the strict definition it is not linear, but \"linear + translation,\" also known as affine.",[15,2401,2402],{},"Now we can understand why it's called a \"linear\" combination. This combination allows only two types of \"linear\" operations:",[2404,2405,2406,2443],"ol",{},[402,2407,2408,2411,2412,2414,2415,143],{},[19,2409,2410],{},"Scalar multiplication"," (corresponding to ",[19,2413,1143],{},"): Stretching or compressing a vector by a factor ",[33,2416,2418,2431],{"className":2417,"translate":37},[36],[33,2419,2421],{"className":2420},[41],[43,2422,2423],{"xmlns":45},[47,2424,2425,2429],{},[50,2426,2427],{},[53,2428,1172],{},[57,2430,1172],{"encoding":59},[33,2432,2434],{"className":2433,"ariaHidden":64},[63],[33,2435,2437,2440],{"className":2436},[68],[33,2438],{"className":2439,"style":107},[72],[33,2441,1172],{"className":2442},[77,78],[402,2444,2445,2411,2448,2450],{},[19,2446,2447],{},"Addition",[19,2449,147],{},"): Adding the processed vectors together.",[15,2452,2453,2454,2706,2707,2955],{},"Nonlinear operations like squaring, taking absolute values, or logarithms are not allowed. Therefore, expressions like ",[33,2455,2457,2500],{"className":2456,"translate":37},[36],[33,2458,2460],{"className":2459},[41],[43,2461,2462],{"xmlns":45},[47,2463,2464,2497],{},[50,2465,2466,2473,2483,2485,2491],{},[2467,2468,2469,2471],"msub",{},[53,2470,1172],{},[346,2472,2212],{},[2474,2475,2476,2479,2481],"msubsup",{},[53,2477,2478],{},"v",[346,2480,2212],{},[346,2482,423],{},[175,2484,183],{},[2467,2486,2487,2489],{},[53,2488,1172],{},[346,2490,423],{},[2467,2492,2493,2495],{},[53,2494,2478],{},[346,2496,423],{},[57,2498,2499],{"encoding":59},"c_1 v_1^2 + c_2 v_2",[33,2501,2503,2618],{"className":2502,"ariaHidden":64},[63],[33,2504,2506,2510,2556,2609,2612,2615],{"className":2505},[68],[33,2507],{"className":2508,"style":2509},[72],"height:1.0622em;vertical-align:-0.2481em;",[33,2511,2513,2516],{"className":2512},[77],[33,2514,1172],{"className":2515},[77,78],[33,2517,2519],{"className":2518},[809],[33,2520,2523,2547],{"className":2521},[813,2522],"vlist-t2",[33,2524,2526,2542],{"className":2525},[817],[33,2527,2530],{"className":2528,"style":2529},[821],"height:0.3011em;",[33,2531,2533,2536],{"style":2532},"top:-2.55em;margin-left:0em;margin-right:0.05em;",[33,2534],{"className":2535,"style":829},[828],[33,2537,2539],{"className":2538},[833,834,835,836],[33,2540,2212],{"className":2541},[77,836],[33,2543,2546],{"className":2544},[2545],"vlist-s","​",[33,2548,2550],{"className":2549},[817],[33,2551,2554],{"className":2552,"style":2553},[821],"height:0.15em;",[33,2555],{},[33,2557,2559,2562],{"className":2558},[77],[33,2560,2478],{"className":2561,"style":142},[77,78],[33,2563,2565],{"className":2564},[809],[33,2566,2568,2600],{"className":2567},[813,2522],[33,2569,2571,2597],{"className":2570},[817],[33,2572,2574,2586],{"className":2573,"style":799},[821],[33,2575,2577,2580],{"style":2576},"top:-2.4519em;margin-left:-0.0359em;margin-right:0.05em;",[33,2578],{"className":2579,"style":829},[828],[33,2581,2583],{"className":2582},[833,834,835,836],[33,2584,2212],{"className":2585},[77,836],[33,2587,2588,2591],{"style":824},[33,2589],{"className":2590,"style":829},[828],[33,2592,2594],{"className":2593},[833,834,835,836],[33,2595,423],{"className":2596},[77,836],[33,2598,2546],{"className":2599},[2545],[33,2601,2603],{"className":2602},[817],[33,2604,2607],{"className":2605,"style":2606},[821],"height:0.2481em;",[33,2608],{},[33,2610],{"className":2611,"style":237},[236],[33,2613,183],{"className":2614},[241],[33,2616],{"className":2617,"style":237},[236],[33,2619,2621,2625,2665],{"className":2620},[68],[33,2622],{"className":2623,"style":2624},[72],"height:0.5806em;vertical-align:-0.15em;",[33,2626,2628,2631],{"className":2627},[77],[33,2629,1172],{"className":2630},[77,78],[33,2632,2634],{"className":2633},[809],[33,2635,2637,2657],{"className":2636},[813,2522],[33,2638,2640,2654],{"className":2639},[817],[33,2641,2643],{"className":2642,"style":2529},[821],[33,2644,2645,2648],{"style":2532},[33,2646],{"className":2647,"style":829},[828],[33,2649,2651],{"className":2650},[833,834,835,836],[33,2652,423],{"className":2653},[77,836],[33,2655,2546],{"className":2656},[2545],[33,2658,2660],{"className":2659},[817],[33,2661,2663],{"className":2662,"style":2553},[821],[33,2664],{},[33,2666,2668,2671],{"className":2667},[77],[33,2669,2478],{"className":2670,"style":142},[77,78],[33,2672,2674],{"className":2673},[809],[33,2675,2677,2698],{"className":2676},[813,2522],[33,2678,2680,2695],{"className":2679},[817],[33,2681,2683],{"className":2682,"style":2529},[821],[33,2684,2686,2689],{"style":2685},"top:-2.55em;margin-left:-0.0359em;margin-right:0.05em;",[33,2687],{"className":2688,"style":829},[828],[33,2690,2692],{"className":2691},[833,834,835,836],[33,2693,423],{"className":2694},[77,836],[33,2696,2546],{"className":2697},[2545],[33,2699,2701],{"className":2700},[817],[33,2702,2704],{"className":2703,"style":2553},[821],[33,2705],{}," or ",[33,2708,2710,2752],{"className":2709,"translate":37},[36],[33,2711,2713],{"className":2712},[41],[43,2714,2715],{"xmlns":45},[47,2716,2717,2749],{},[50,2718,2719,2725,2731,2733,2739,2745,2747],{},[2467,2720,2721,2723],{},[53,2722,1172],{},[346,2724,2212],{},[2467,2726,2727,2729],{},[53,2728,2478],{},[346,2730,2212],{},[175,2732,183],{},[2467,2734,2735,2737],{},[53,2736,1172],{},[346,2738,423],{},[2467,2740,2741,2743],{},[53,2742,2478],{},[346,2744,423],{},[175,2746,183],{},[346,2748,428],{},[57,2750,2751],{"encoding":59},"c_1 v_1 + c_2 v_2 + 3",[33,2753,2755,2851,2946],{"className":2754,"ariaHidden":64},[63],[33,2756,2758,2762,2802,2842,2845,2848],{"className":2757},[68],[33,2759],{"className":2760,"style":2761},[72],"height:0.7333em;vertical-align:-0.15em;",[33,2763,2765,2768],{"className":2764},[77],[33,2766,1172],{"className":2767},[77,78],[33,2769,2771],{"className":2770},[809],[33,2772,2774,2794],{"className":2773},[813,2522],[33,2775,2777,2791],{"className":2776},[817],[33,2778,2780],{"className":2779,"style":2529},[821],[33,2781,2782,2785],{"style":2532},[33,2783],{"className":2784,"style":829},[828],[33,2786,2788],{"className":2787},[833,834,835,836],[33,2789,2212],{"className":2790},[77,836],[33,2792,2546],{"className":2793},[2545],[33,2795,2797],{"className":2796},[817],[33,2798,2800],{"className":2799,"style":2553},[821],[33,2801],{},[33,2803,2805,2808],{"className":2804},[77],[33,2806,2478],{"className":2807,"style":142},[77,78],[33,2809,2811],{"className":2810},[809],[33,2812,2814,2834],{"className":2813},[813,2522],[33,2815,2817,2831],{"className":2816},[817],[33,2818,2820],{"className":2819,"style":2529},[821],[33,2821,2822,2825],{"style":2685},[33,2823],{"className":2824,"style":829},[828],[33,2826,2828],{"className":2827},[833,834,835,836],[33,2829,2212],{"className":2830},[77,836],[33,2832,2546],{"className":2833},[2545],[33,2835,2837],{"className":2836},[817],[33,2838,2840],{"className":2839,"style":2553},[821],[33,2841],{},[33,2843],{"className":2844,"style":237},[236],[33,2846,183],{"className":2847},[241],[33,2849],{"className":2850,"style":237},[236],[33,2852,2854,2857,2897,2937,2940,2943],{"className":2853},[68],[33,2855],{"className":2856,"style":2761},[72],[33,2858,2860,2863],{"className":2859},[77],[33,2861,1172],{"className":2862},[77,78],[33,2864,2866],{"className":2865},[809],[33,2867,2869,2889],{"className":2868},[813,2522],[33,2870,2872,2886],{"className":2871},[817],[33,2873,2875],{"className":2874,"style":2529},[821],[33,2876,2877,2880],{"style":2532},[33,2878],{"className":2879,"style":829},[828],[33,2881,2883],{"className":2882},[833,834,835,836],[33,2884,423],{"className":2885},[77,836],[33,2887,2546],{"className":2888},[2545],[33,2890,2892],{"className":2891},[817],[33,2893,2895],{"className":2894,"style":2553},[821],[33,2896],{},[33,2898,2900,2903],{"className":2899},[77],[33,2901,2478],{"className":2902,"style":142},[77,78],[33,2904,2906],{"className":2905},[809],[33,2907,2909,2929],{"className":2908},[813,2522],[33,2910,2912,2926],{"className":2911},[817],[33,2913,2915],{"className":2914,"style":2529},[821],[33,2916,2917,2920],{"style":2685},[33,2918],{"className":2919,"style":829},[828],[33,2921,2923],{"className":2922},[833,834,835,836],[33,2924,423],{"className":2925},[77,836],[33,2927,2546],{"className":2928},[2545],[33,2930,2932],{"className":2931},[817],[33,2933,2935],{"className":2934,"style":2553},[821],[33,2936],{},[33,2938],{"className":2939,"style":237},[236],[33,2941,183],{"className":2942},[241],[33,2944],{"className":2945,"style":237},[236],[33,2947,2949,2952],{"className":2948},[68],[33,2950],{"className":2951,"style":390},[72],[33,2953,428],{"className":2954},[77]," are not called linear combinations.",[15,2957,2958,2959,2962,2963,2966],{},"It can be said that \"linear\" is the simplest and most core relationship in mathematics and science because it allows us to ",[19,2960,2961],{},"decompose"," complex problems into a ",[19,2964,2965],{},"superposition"," of simple parts.",[10,2968,2970],{"id":2969},"affine","Affine",[15,2972,2973,2974,3116],{},"An affine function ",[33,2975,2977,3013],{"className":2976,"translate":37},[36],[33,2978,2980],{"className":2979},[41],[43,2981,2982],{"xmlns":45},[47,2983,2984,3010],{},[50,2985,2986,2988,2991,3000,3003],{},[53,2987,55],{},[175,2989,2990],{},":",[754,2992,2993,2997],{},[53,2994,2996],{"mathvariant":2995},"double-struck","R",[53,2998,2999],{},"n",[175,3001,3002],{},"→",[754,3004,3005,3007],{},[53,3006,2996],{"mathvariant":2995},[53,3008,3009],{},"m",[57,3011,3012],{"encoding":59},"f: \\mathbb{R}^n \\to \\mathbb{R}^m",[33,3014,3016,3034,3081],{"className":3015,"ariaHidden":64},[63],[33,3017,3019,3022,3025,3028,3031],{"className":3018},[68],[33,3020],{"className":3021,"style":73},[72],[33,3023,55],{"className":3024,"style":79},[77,78],[33,3026],{"className":3027,"style":261},[236],[33,3029,2990],{"className":3030},[265],[33,3032],{"className":3033,"style":261},[236],[33,3035,3037,3041,3072,3075,3078],{"className":3036},[68],[33,3038],{"className":3039,"style":3040},[72],"height:0.6889em;",[33,3042,3044,3048],{"className":3043},[77],[33,3045,2996],{"className":3046},[77,3047],"mathbb",[33,3049,3051],{"className":3050},[809],[33,3052,3054],{"className":3053},[813],[33,3055,3057],{"className":3056},[817],[33,3058,3061],{"className":3059,"style":3060},[821],"height:0.6644em;",[33,3062,3063,3066],{"style":824},[33,3064],{"className":3065,"style":829},[828],[33,3067,3069],{"className":3068},[833,834,835,836],[33,3070,2999],{"className":3071},[77,78,836],[33,3073],{"className":3074,"style":261},[236],[33,3076,3002],{"className":3077},[265],[33,3079],{"className":3080,"style":261},[236],[33,3082,3084,3087],{"className":3083},[68],[33,3085],{"className":3086,"style":3040},[72],[33,3088,3090,3093],{"className":3089},[77],[33,3091,2996],{"className":3092},[77,3047],[33,3094,3096],{"className":3095},[809],[33,3097,3099],{"className":3098},[813],[33,3100,3102],{"className":3101},[817],[33,3103,3105],{"className":3104,"style":3060},[821],[33,3106,3107,3110],{"style":824},[33,3108],{"className":3109,"style":829},[828],[33,3111,3113],{"className":3112},[833,834,835,836],[33,3114,3009],{"className":3115},[77,78,836]," is defined as:",[33,3118,3120],{"className":3119,"translate":37},[158],[33,3121,3123,3154],{"className":3122,"translate":37},[36],[33,3124,3126],{"className":3125},[41],[43,3127,3128],{"xmlns":45,"display":167},[47,3129,3130,3151],{},[50,3131,3132,3134,3136,3138,3140,3142,3145,3147,3149],{},[53,3133,55],{},[175,3135,178],{"stretchy":177},[53,3137,95],{},[175,3139,188],{"stretchy":177},[175,3141,191],{},[53,3143,3144],{},"A",[53,3146,95],{},[175,3148,183],{},[53,3150,2171],{},[57,3152,3153],{"encoding":59},"f(x) = Ax + b",[33,3155,3157,3184,3206],{"className":3156,"ariaHidden":64},[63],[33,3158,3160,3163,3166,3169,3172,3175,3178,3181],{"className":3159},[68],[33,3161],{"className":3162,"style":222},[72],[33,3164,55],{"className":3165,"style":79},[77,78],[33,3167,178],{"className":3168},[229],[33,3170,95],{"className":3171},[77,78],[33,3173,188],{"className":3174},[257],[33,3176],{"className":3177,"style":261},[236],[33,3179,191],{"className":3180},[265],[33,3182],{"className":3183,"style":261},[236],[33,3185,3187,3191,3194,3197,3200,3203],{"className":3186},[68],[33,3188],{"className":3189,"style":3190},[72],"height:0.7667em;vertical-align:-0.0833em;",[33,3192,3144],{"className":3193},[77,78],[33,3195,95],{"className":3196},[77,78],[33,3198],{"className":3199,"style":237},[236],[33,3201,183],{"className":3202},[241],[33,3204],{"className":3205,"style":237},[236],[33,3207,3209,3212],{"className":3208},[68],[33,3210],{"className":3211,"style":2183},[72],[33,3213,2171],{"className":3214},[77,78],[15,3216,3217,3218,3247,3248,3276],{},"where ",[33,3219,3221,3234],{"className":3220,"translate":37},[36],[33,3222,3224],{"className":3223},[41],[43,3225,3226],{"xmlns":45},[47,3227,3228,3232],{},[50,3229,3230],{},[53,3231,3144],{},[57,3233,3144],{"encoding":59},[33,3235,3237],{"className":3236,"ariaHidden":64},[63],[33,3238,3240,3244],{"className":3239},[68],[33,3241],{"className":3242,"style":3243},[72],"height:0.6833em;",[33,3245,3144],{"className":3246},[77,78]," is a matrix and ",[33,3249,3251,3264],{"className":3250,"translate":37},[36],[33,3252,3254],{"className":3253},[41],[43,3255,3256],{"xmlns":45},[47,3257,3258,3262],{},[50,3259,3260],{},[53,3261,2171],{},[57,3263,2171],{"encoding":59},[33,3265,3267],{"className":3266,"ariaHidden":64},[63],[33,3268,3270,3273],{"className":3269},[68],[33,3271],{"className":3272,"style":2183},[72],[33,3274,2171],{"className":3275},[77,78]," is a constant vector. It is a linear function plus a bias.",[15,3278,3279],{},"If you take a weighted linear combination (coefficients can be scalars) of affine functions:",[33,3281,3283],{"className":3282,"translate":37},[158],[33,3284,3286,3346],{"className":3285,"translate":37},[36],[33,3287,3289],{"className":3288},[41],[43,3290,3291],{"xmlns":45,"display":167},[47,3292,3293,3343],{},[50,3294,3295,3298,3300,3302,3304,3306,3324,3331,3337,3339,3341],{},[53,3296,3297],{},"g",[175,3299,178],{"stretchy":177},[53,3301,95],{},[175,3303,188],{"stretchy":177},[175,3305,191],{},[3307,3308,3309,3312,3321],"munderover",{},[175,3310,3311],{},"∑",[50,3313,3314,3317,3319],{},[53,3315,3316],{},"i",[175,3318,191],{},[346,3320,2212],{},[53,3322,3323],{},"k",[2467,3325,3326,3329],{},[53,3327,3328],{},"α",[53,3330,3316],{},[2467,3332,3333,3335],{},[53,3334,55],{},[53,3336,3316],{},[175,3338,178],{"stretchy":177},[53,3340,95],{},[175,3342,188],{"stretchy":177},[57,3344,3345],{"encoding":59},"g(x) = \\sum_{i=1}^k \\alpha_i f_i(x)",[33,3347,3349,3376],{"className":3348,"ariaHidden":64},[63],[33,3350,3352,3355,3358,3361,3364,3367,3370,3373],{"className":3351},[68],[33,3353],{"className":3354,"style":222},[72],[33,3356,3297],{"className":3357,"style":142},[77,78],[33,3359,178],{"className":3360},[229],[33,3362,95],{"className":3363},[77,78],[33,3365,188],{"className":3366},[257],[33,3368],{"className":3369,"style":261},[236],[33,3371,191],{"className":3372},[265],[33,3374],{"className":3375,"style":261},[236],[33,3377,3379,3383,3458,3462,3505,3546,3549,3552],{"className":3378},[68],[33,3380],{"className":3381,"style":3382},[72],"height:3.1138em;vertical-align:-1.2777em;",[33,3384,3388],{"className":3385},[3386,3387],"mop","op-limits",[33,3389,3391,3449],{"className":3390},[813,2522],[33,3392,3394,3446],{"className":3393},[817],[33,3395,3398,3420,3433],{"className":3396,"style":3397},[821],"height:1.8361em;",[33,3399,3401,3405],{"style":3400},"top:-1.8723em;margin-left:0em;",[33,3402],{"className":3403,"style":3404},[828],"height:3.05em;",[33,3406,3408],{"className":3407},[833,834,835,836],[33,3409,3411,3414,3417],{"className":3410},[77,836],[33,3412,3316],{"className":3413},[77,78,836],[33,3415,191],{"className":3416},[265,836],[33,3418,2212],{"className":3419},[77,836],[33,3421,3423,3426],{"style":3422},"top:-3.05em;",[33,3424],{"className":3425,"style":3404},[828],[33,3427,3428],{},[33,3429,3311],{"className":3430},[3386,3431,3432],"op-symbol","large-op",[33,3434,3436,3439],{"style":3435},"top:-4.3em;margin-left:0em;",[33,3437],{"className":3438,"style":3404},[828],[33,3440,3442],{"className":3441},[833,834,835,836],[33,3443,3323],{"className":3444,"style":3445},[77,78,836],"margin-right:0.0315em;",[33,3447,2546],{"className":3448},[2545],[33,3450,3452],{"className":3451},[817],[33,3453,3456],{"className":3454,"style":3455},[821],"height:1.2777em;",[33,3457],{},[33,3459],{"className":3460,"style":3461},[236],"margin-right:0.1667em;",[33,3463,3465,3469],{"className":3464},[77],[33,3466,3328],{"className":3467,"style":3468},[77,78],"margin-right:0.0037em;",[33,3470,3472],{"className":3471},[809],[33,3473,3475,3497],{"className":3474},[813,2522],[33,3476,3478,3494],{"className":3477},[817],[33,3479,3482],{"className":3480,"style":3481},[821],"height:0.3117em;",[33,3483,3485,3488],{"style":3484},"top:-2.55em;margin-left:-0.0037em;margin-right:0.05em;",[33,3486],{"className":3487,"style":829},[828],[33,3489,3491],{"className":3490},[833,834,835,836],[33,3492,3316],{"className":3493},[77,78,836],[33,3495,2546],{"className":3496},[2545],[33,3498,3500],{"className":3499},[817],[33,3501,3503],{"className":3502,"style":2553},[821],[33,3504],{},[33,3506,3508,3511],{"className":3507},[77],[33,3509,55],{"className":3510,"style":79},[77,78],[33,3512,3514],{"className":3513},[809],[33,3515,3517,3538],{"className":3516},[813,2522],[33,3518,3520,3535],{"className":3519},[817],[33,3521,3523],{"className":3522,"style":3481},[821],[33,3524,3526,3529],{"style":3525},"top:-2.55em;margin-left:-0.1076em;margin-right:0.05em;",[33,3527],{"className":3528,"style":829},[828],[33,3530,3532],{"className":3531},[833,834,835,836],[33,3533,3316],{"className":3534},[77,78,836],[33,3536,2546],{"className":3537},[2545],[33,3539,3541],{"className":3540},[817],[33,3542,3544],{"className":3543,"style":2553},[821],[33,3545],{},[33,3547,178],{"className":3548},[229],[33,3550,95],{"className":3551},[77,78],[33,3553,188],{"className":3554},[257],[15,3556,3557,3558,143],{},"where each ",[33,3559,3561,3603],{"className":3560,"translate":37},[36],[33,3562,3564],{"className":3563},[41],[43,3565,3566],{"xmlns":45},[47,3567,3568,3600],{},[50,3569,3570,3576,3578,3580,3582,3584,3590,3592,3594],{},[2467,3571,3572,3574],{},[53,3573,55],{},[53,3575,3316],{},[175,3577,178],{"stretchy":177},[53,3579,95],{},[175,3581,188],{"stretchy":177},[175,3583,191],{},[2467,3585,3586,3588],{},[53,3587,3144],{},[53,3589,3316],{},[53,3591,95],{},[175,3593,183],{},[2467,3595,3596,3598],{},[53,3597,2171],{},[53,3599,3316],{},[57,3601,3602],{"encoding":59},"f_i(x) = A_i x + b_i",[33,3604,3606,3670,3729],{"className":3605,"ariaHidden":64},[63],[33,3607,3609,3612,3652,3655,3658,3661,3664,3667],{"className":3608},[68],[33,3610],{"className":3611,"style":222},[72],[33,3613,3615,3618],{"className":3614},[77],[33,3616,55],{"className":3617,"style":79},[77,78],[33,3619,3621],{"className":3620},[809],[33,3622,3624,3644],{"className":3623},[813,2522],[33,3625,3627,3641],{"className":3626},[817],[33,3628,3630],{"className":3629,"style":3481},[821],[33,3631,3632,3635],{"style":3525},[33,3633],{"className":3634,"style":829},[828],[33,3636,3638],{"className":3637},[833,834,835,836],[33,3639,3316],{"className":3640},[77,78,836],[33,3642,2546],{"className":3643},[2545],[33,3645,3647],{"className":3646},[817],[33,3648,3650],{"className":3649,"style":2553},[821],[33,3651],{},[33,3653,178],{"className":3654},[229],[33,3656,95],{"className":3657},[77,78],[33,3659,188],{"className":3660},[257],[33,3662],{"className":3663,"style":261},[236],[33,3665,191],{"className":3666},[265],[33,3668],{"className":3669,"style":261},[236],[33,3671,3673,3677,3717,3720,3723,3726],{"className":3672},[68],[33,3674],{"className":3675,"style":3676},[72],"height:0.8333em;vertical-align:-0.15em;",[33,3678,3680,3683],{"className":3679},[77],[33,3681,3144],{"className":3682},[77,78],[33,3684,3686],{"className":3685},[809],[33,3687,3689,3709],{"className":3688},[813,2522],[33,3690,3692,3706],{"className":3691},[817],[33,3693,3695],{"className":3694,"style":3481},[821],[33,3696,3697,3700],{"style":2532},[33,3698],{"className":3699,"style":829},[828],[33,3701,3703],{"className":3702},[833,834,835,836],[33,3704,3316],{"className":3705},[77,78,836],[33,3707,2546],{"className":3708},[2545],[33,3710,3712],{"className":3711},[817],[33,3713,3715],{"className":3714,"style":2553},[821],[33,3716],{},[33,3718,95],{"className":3719},[77,78],[33,3721],{"className":3722,"style":237},[236],[33,3724,183],{"className":3725},[241],[33,3727],{"className":3728,"style":237},[236],[33,3730,3732,3736],{"className":3731},[68],[33,3733],{"className":3734,"style":3735},[72],"height:0.8444em;vertical-align:-0.15em;",[33,3737,3739,3742],{"className":3738},[77],[33,3740,2171],{"className":3741},[77,78],[33,3743,3745],{"className":3744},[809],[33,3746,3748,3768],{"className":3747},[813,2522],[33,3749,3751,3765],{"className":3750},[817],[33,3752,3754],{"className":3753,"style":3481},[821],[33,3755,3756,3759],{"style":2532},[33,3757],{"className":3758,"style":829},[828],[33,3760,3762],{"className":3761},[833,834,835,836],[33,3763,3316],{"className":3764},[77,78,836],[33,3766,2546],{"className":3767},[2545],[33,3769,3771],{"className":3770},[817],[33,3772,3774],{"className":3773,"style":2553},[821],[33,3775],{},[15,3777,3778],{},"Expanding:",[33,3780,3782],{"className":3781,"translate":37},[158],[33,3783,3785,3911],{"className":3784,"translate":37},[36],[33,3786,3788],{"className":3787},[41],[43,3789,3790],{"xmlns":45,"display":167},[47,3791,3792,3908],{},[50,3793,3794,3796,3798,3800,3802,3804,3818,3824,3826,3832,3834,3836,3842,3844,3846,3878,3880,3882,3896,3902],{},[53,3795,3297],{},[175,3797,178],{"stretchy":177},[53,3799,95],{},[175,3801,188],{"stretchy":177},[175,3803,191],{},[3307,3805,3806,3808,3816],{},[175,3807,3311],{},[50,3809,3810,3812,3814],{},[53,3811,3316],{},[175,3813,191],{},[346,3815,2212],{},[53,3817,3323],{},[2467,3819,3820,3822],{},[53,3821,3328],{},[53,3823,3316],{},[175,3825,178],{"stretchy":177},[2467,3827,3828,3830],{},[53,3829,3144],{},[53,3831,3316],{},[53,3833,95],{},[175,3835,183],{},[2467,3837,3838,3840],{},[53,3839,2171],{},[53,3841,3316],{},[175,3843,188],{"stretchy":177},[175,3845,191],{},[50,3847,3848,3850,3864,3870,3876],{},[175,3849,178],{"fence":64},[3307,3851,3852,3854,3862],{},[175,3853,3311],{},[50,3855,3856,3858,3860],{},[53,3857,3316],{},[175,3859,191],{},[346,3861,2212],{},[53,3863,3323],{},[2467,3865,3866,3868],{},[53,3867,3328],{},[53,3869,3316],{},[2467,3871,3872,3874],{},[53,3873,3144],{},[53,3875,3316],{},[175,3877,188],{"fence":64},[53,3879,95],{},[175,3881,183],{},[3307,3883,3884,3886,3894],{},[175,3885,3311],{},[50,3887,3888,3890,3892],{},[53,3889,3316],{},[175,3891,191],{},[346,3893,2212],{},[53,3895,3323],{},[2467,3897,3898,3900],{},[53,3899,3328],{},[53,3901,3316],{},[2467,3903,3904,3906],{},[53,3905,2171],{},[53,3907,3316],{},[57,3909,3910],{"encoding":59},"g(x) = \\sum_{i=1}^k \\alpha_i (A_i x + b_i) = \\left(\\sum_{i=1}^k \\alpha_i A_i \\right) x + \\sum_{i=1}^k \\alpha_i b_i",[33,3912,3914,3941,4109,4167,4355],{"className":3913,"ariaHidden":64},[63],[33,3915,3917,3920,3923,3926,3929,3932,3935,3938],{"className":3916},[68],[33,3918],{"className":3919,"style":222},[72],[33,3921,3297],{"className":3922,"style":142},[77,78],[33,3924,178],{"className":3925},[229],[33,3927,95],{"className":3928},[77,78],[33,3930,188],{"className":3931},[257],[33,3933],{"className":3934,"style":261},[236],[33,3936,191],{"className":3937},[265],[33,3939],{"className":3940,"style":261},[236],[33,3942,3944,3947,4011,4014,4054,4057,4097,4100,4103,4106],{"className":3943},[68],[33,3945],{"className":3946,"style":3382},[72],[33,3948,3950],{"className":3949},[3386,3387],[33,3951,3953,4003],{"className":3952},[813,2522],[33,3954,3956,4000],{"className":3955},[817],[33,3957,3959,3979,3989],{"className":3958,"style":3397},[821],[33,3960,3961,3964],{"style":3400},[33,3962],{"className":3963,"style":3404},[828],[33,3965,3967],{"className":3966},[833,834,835,836],[33,3968,3970,3973,3976],{"className":3969},[77,836],[33,3971,3316],{"className":3972},[77,78,836],[33,3974,191],{"className":3975},[265,836],[33,3977,2212],{"className":3978},[77,836],[33,3980,3981,3984],{"style":3422},[33,3982],{"className":3983,"style":3404},[828],[33,3985,3986],{},[33,3987,3311],{"className":3988},[3386,3431,3432],[33,3990,3991,3994],{"style":3435},[33,3992],{"className":3993,"style":3404},[828],[33,3995,3997],{"className":3996},[833,834,835,836],[33,3998,3323],{"className":3999,"style":3445},[77,78,836],[33,4001,2546],{"className":4002},[2545],[33,4004,4006],{"className":4005},[817],[33,4007,4009],{"className":4008,"style":3455},[821],[33,4010],{},[33,4012],{"className":4013,"style":3461},[236],[33,4015,4017,4020],{"className":4016},[77],[33,4018,3328],{"className":4019,"style":3468},[77,78],[33,4021,4023],{"className":4022},[809],[33,4024,4026,4046],{"className":4025},[813,2522],[33,4027,4029,4043],{"className":4028},[817],[33,4030,4032],{"className":4031,"style":3481},[821],[33,4033,4034,4037],{"style":3484},[33,4035],{"className":4036,"style":829},[828],[33,4038,4040],{"className":4039},[833,834,835,836],[33,4041,3316],{"className":4042},[77,78,836],[33,4044,2546],{"className":4045},[2545],[33,4047,4049],{"className":4048},[817],[33,4050,4052],{"className":4051,"style":2553},[821],[33,4053],{},[33,4055,178],{"className":4056},[229],[33,4058,4060,4063],{"className":4059},[77],[33,4061,3144],{"className":4062},[77,78],[33,4064,4066],{"className":4065},[809],[33,4067,4069,4089],{"className":4068},[813,2522],[33,4070,4072,4086],{"className":4071},[817],[33,4073,4075],{"className":4074,"style":3481},[821],[33,4076,4077,4080],{"style":2532},[33,4078],{"className":4079,"style":829},[828],[33,4081,4083],{"className":4082},[833,834,835,836],[33,4084,3316],{"className":4085},[77,78,836],[33,4087,2546],{"className":4088},[2545],[33,4090,4092],{"className":4091},[817],[33,4093,4095],{"className":4094,"style":2553},[821],[33,4096],{},[33,4098,95],{"className":4099},[77,78],[33,4101],{"className":4102,"style":237},[236],[33,4104,183],{"className":4105},[241],[33,4107],{"className":4108,"style":237},[236],[33,4110,4112,4115,4155,4158,4161,4164],{"className":4111},[68],[33,4113],{"className":4114,"style":222},[72],[33,4116,4118,4121],{"className":4117},[77],[33,4119,2171],{"className":4120},[77,78],[33,4122,4124],{"className":4123},[809],[33,4125,4127,4147],{"className":4126},[813,2522],[33,4128,4130,4144],{"className":4129},[817],[33,4131,4133],{"className":4132,"style":3481},[821],[33,4134,4135,4138],{"style":2532},[33,4136],{"className":4137,"style":829},[828],[33,4139,4141],{"className":4140},[833,834,835,836],[33,4142,3316],{"className":4143},[77,78,836],[33,4145,2546],{"className":4146},[2545],[33,4148,4150],{"className":4149},[817],[33,4151,4153],{"className":4152,"style":2553},[821],[33,4154],{},[33,4156,188],{"className":4157},[257],[33,4159],{"className":4160,"style":261},[236],[33,4162,191],{"className":4163},[265],[33,4165],{"className":4166,"style":261},[236],[33,4168,4170,4173,4340,4343,4346,4349,4352],{"className":4169},[68],[33,4171],{"className":4172,"style":3382},[72],[33,4174,4177,4187,4251,4254,4294,4334],{"className":4175},[4176],"minner",[33,4178,4182],{"className":4179,"style":4181},[229,4180],"delimcenter","top:0em;",[33,4183,178],{"className":4184},[4185,4186],"delimsizing","size4",[33,4188,4190],{"className":4189},[3386,3387],[33,4191,4193,4243],{"className":4192},[813,2522],[33,4194,4196,4240],{"className":4195},[817],[33,4197,4199,4219,4229],{"className":4198,"style":3397},[821],[33,4200,4201,4204],{"style":3400},[33,4202],{"className":4203,"style":3404},[828],[33,4205,4207],{"className":4206},[833,834,835,836],[33,4208,4210,4213,4216],{"className":4209},[77,836],[33,4211,3316],{"className":4212},[77,78,836],[33,4214,191],{"className":4215},[265,836],[33,4217,2212],{"className":4218},[77,836],[33,4220,4221,4224],{"style":3422},[33,4222],{"className":4223,"style":3404},[828],[33,4225,4226],{},[33,4227,3311],{"className":4228},[3386,3431,3432],[33,4230,4231,4234],{"style":3435},[33,4232],{"className":4233,"style":3404},[828],[33,4235,4237],{"className":4236},[833,834,835,836],[33,4238,3323],{"className":4239,"style":3445},[77,78,836],[33,4241,2546],{"className":4242},[2545],[33,4244,4246],{"className":4245},[817],[33,4247,4249],{"className":4248,"style":3455},[821],[33,4250],{},[33,4252],{"className":4253,"style":3461},[236],[33,4255,4257,4260],{"className":4256},[77],[33,4258,3328],{"className":4259,"style":3468},[77,78],[33,4261,4263],{"className":4262},[809],[33,4264,4266,4286],{"className":4265},[813,2522],[33,4267,4269,4283],{"className":4268},[817],[33,4270,4272],{"className":4271,"style":3481},[821],[33,4273,4274,4277],{"style":3484},[33,4275],{"className":4276,"style":829},[828],[33,4278,4280],{"className":4279},[833,834,835,836],[33,4281,3316],{"className":4282},[77,78,836],[33,4284,2546],{"className":4285},[2545],[33,4287,4289],{"className":4288},[817],[33,4290,4292],{"className":4291,"style":2553},[821],[33,4293],{},[33,4295,4297,4300],{"className":4296},[77],[33,4298,3144],{"className":4299},[77,78],[33,4301,4303],{"className":4302},[809],[33,4304,4306,4326],{"className":4305},[813,2522],[33,4307,4309,4323],{"className":4308},[817],[33,4310,4312],{"className":4311,"style":3481},[821],[33,4313,4314,4317],{"style":2532},[33,4315],{"className":4316,"style":829},[828],[33,4318,4320],{"className":4319},[833,834,835,836],[33,4321,3316],{"className":4322},[77,78,836],[33,4324,2546],{"className":4325},[2545],[33,4327,4329],{"className":4328},[817],[33,4330,4332],{"className":4331,"style":2553},[821],[33,4333],{},[33,4335,4337],{"className":4336,"style":4181},[257,4180],[33,4338,188],{"className":4339},[4185,4186],[33,4341],{"className":4342,"style":3461},[236],[33,4344,95],{"className":4345},[77,78],[33,4347],{"className":4348,"style":237},[236],[33,4350,183],{"className":4351},[241],[33,4353],{"className":4354,"style":237},[236],[33,4356,4358,4361,4425,4428,4468],{"className":4357},[68],[33,4359],{"className":4360,"style":3382},[72],[33,4362,4364],{"className":4363},[3386,3387],[33,4365,4367,4417],{"className":4366},[813,2522],[33,4368,4370,4414],{"className":4369},[817],[33,4371,4373,4393,4403],{"className":4372,"style":3397},[821],[33,4374,4375,4378],{"style":3400},[33,4376],{"className":4377,"style":3404},[828],[33,4379,4381],{"className":4380},[833,834,835,836],[33,4382,4384,4387,4390],{"className":4383},[77,836],[33,4385,3316],{"className":4386},[77,78,836],[33,4388,191],{"className":4389},[265,836],[33,4391,2212],{"className":4392},[77,836],[33,4394,4395,4398],{"style":3422},[33,4396],{"className":4397,"style":3404},[828],[33,4399,4400],{},[33,4401,3311],{"className":4402},[3386,3431,3432],[33,4404,4405,4408],{"style":3435},[33,4406],{"className":4407,"style":3404},[828],[33,4409,4411],{"className":4410},[833,834,835,836],[33,4412,3323],{"className":4413,"style":3445},[77,78,836],[33,4415,2546],{"className":4416},[2545],[33,4418,4420],{"className":4419},[817],[33,4421,4423],{"className":4422,"style":3455},[821],[33,4424],{},[33,4426],{"className":4427,"style":3461},[236],[33,4429,4431,4434],{"className":4430},[77],[33,4432,3328],{"className":4433,"style":3468},[77,78],[33,4435,4437],{"className":4436},[809],[33,4438,4440,4460],{"className":4439},[813,2522],[33,4441,4443,4457],{"className":4442},[817],[33,4444,4446],{"className":4445,"style":3481},[821],[33,4447,4448,4451],{"style":3484},[33,4449],{"className":4450,"style":829},[828],[33,4452,4454],{"className":4453},[833,834,835,836],[33,4455,3316],{"className":4456},[77,78,836],[33,4458,2546],{"className":4459},[2545],[33,4461,4463],{"className":4462},[817],[33,4464,4466],{"className":4465,"style":2553},[821],[33,4467],{},[33,4469,4471,4474],{"className":4470},[77],[33,4472,2171],{"className":4473},[77,78],[33,4475,4477],{"className":4476},[809],[33,4478,4480,4500],{"className":4479},[813,2522],[33,4481,4483,4497],{"className":4482},[817],[33,4484,4486],{"className":4485,"style":3481},[821],[33,4487,4488,4491],{"style":2532},[33,4489],{"className":4490,"style":829},[828],[33,4492,4494],{"className":4493},[833,834,835,836],[33,4495,3316],{"className":4496},[77,78,836],[33,4498,2546],{"className":4499},[2545],[33,4501,4503],{"className":4502},[817],[33,4504,4506],{"className":4505,"style":2553},[821],[33,4507],{},[15,4509,4510,4511,4514],{},"This is still an ",[19,4512,4513],{},"affine function",". Thus, weighted combinations do not destroy affinity.",[15,4516,4517],{},"If you combine affine functions using nonlinear polynomial operations, such as squaring or higher powers:",[33,4519,4521],{"className":4520,"translate":37},[158],[33,4522,4524,4599],{"className":4523,"translate":37},[36],[33,4525,4527],{"className":4526},[41],[43,4528,4529],{"xmlns":45,"display":167},[47,4530,4531,4596],{},[50,4532,4533,4536,4538,4540,4542,4544,4546,4552,4554,4556,4558,4564,4566,4568,4574,4576,4578,4580,4586,4588,4590,4592,4594],{},[53,4534,4535],{},"h",[175,4537,178],{"stretchy":177},[53,4539,95],{},[175,4541,188],{"stretchy":177},[175,4543,191],{},[175,4545,178],{"stretchy":177},[2467,4547,4548,4550],{},[53,4549,55],{},[346,4551,2212],{},[175,4553,178],{"stretchy":177},[53,4555,95],{},[175,4557,188],{"stretchy":177},[754,4559,4560,4562],{},[175,4561,188],{"stretchy":177},[346,4563,423],{},[175,4565,183],{},[346,4567,428],{},[2467,4569,4570,4572],{},[53,4571,55],{},[346,4573,423],{},[175,4575,178],{"stretchy":177},[53,4577,95],{},[175,4579,188],{"stretchy":177},[2467,4581,4582,4584],{},[53,4583,55],{},[346,4585,428],{},[175,4587,178],{"stretchy":177},[53,4589,95],{},[175,4591,188],{"stretchy":177},[175,4593,183],{},[346,4595,348],{},[57,4597,4598],{"encoding":59},"h(x) = (f_1(x))^2 + 3 f_2(x) f_3(x) + 5",[33,4600,4602,4629,4728,4844],{"className":4601,"ariaHidden":64},[63],[33,4603,4605,4608,4611,4614,4617,4620,4623,4626],{"className":4604},[68],[33,4606],{"className":4607,"style":222},[72],[33,4609,4535],{"className":4610},[77,78],[33,4612,178],{"className":4613},[229],[33,4615,95],{"className":4616},[77,78],[33,4618,188],{"className":4619},[257],[33,4621],{"className":4622,"style":261},[236],[33,4624,191],{"className":4625},[265],[33,4627],{"className":4628,"style":261},[236],[33,4630,4632,4636,4639,4679,4682,4685,4688,4719,4722,4725],{"className":4631},[68],[33,4633],{"className":4634,"style":4635},[72],"height:1.1141em;vertical-align:-0.25em;",[33,4637,178],{"className":4638},[229],[33,4640,4642,4645],{"className":4641},[77],[33,4643,55],{"className":4644,"style":79},[77,78],[33,4646,4648],{"className":4647},[809],[33,4649,4651,4671],{"className":4650},[813,2522],[33,4652,4654,4668],{"className":4653},[817],[33,4655,4657],{"className":4656,"style":2529},[821],[33,4658,4659,4662],{"style":3525},[33,4660],{"className":4661,"style":829},[828],[33,4663,4665],{"className":4664},[833,834,835,836],[33,4666,2212],{"className":4667},[77,836],[33,4669,2546],{"className":4670},[2545],[33,4672,4674],{"className":4673},[817],[33,4675,4677],{"className":4676,"style":2553},[821],[33,4678],{},[33,4680,178],{"className":4681},[229],[33,4683,95],{"className":4684},[77,78],[33,4686,188],{"className":4687},[257],[33,4689,4691,4694],{"className":4690},[257],[33,4692,188],{"className":4693},[257],[33,4695,4697],{"className":4696},[809],[33,4698,4700],{"className":4699},[813],[33,4701,4703],{"className":4702},[817],[33,4704,4707],{"className":4705,"style":4706},[821],"height:0.8641em;",[33,4708,4710,4713],{"style":4709},"top:-3.113em;margin-right:0.05em;",[33,4711],{"className":4712,"style":829},[828],[33,4714,4716],{"className":4715},[833,834,835,836],[33,4717,423],{"className":4718},[77,836],[33,4720],{"className":4721,"style":237},[236],[33,4723,183],{"className":4724},[241],[33,4726],{"className":4727,"style":237},[236],[33,4729,4731,4734,4737,4777,4780,4783,4786,4826,4829,4832,4835,4838,4841],{"className":4730},[68],[33,4732],{"className":4733,"style":222},[72],[33,4735,428],{"className":4736},[77],[33,4738,4740,4743],{"className":4739},[77],[33,4741,55],{"className":4742,"style":79},[77,78],[33,4744,4746],{"className":4745},[809],[33,4747,4749,4769],{"className":4748},[813,2522],[33,4750,4752,4766],{"className":4751},[817],[33,4753,4755],{"className":4754,"style":2529},[821],[33,4756,4757,4760],{"style":3525},[33,4758],{"className":4759,"style":829},[828],[33,4761,4763],{"className":4762},[833,834,835,836],[33,4764,423],{"className":4765},[77,836],[33,4767,2546],{"className":4768},[2545],[33,4770,4772],{"className":4771},[817],[33,4773,4775],{"className":4774,"style":2553},[821],[33,4776],{},[33,4778,178],{"className":4779},[229],[33,4781,95],{"className":4782},[77,78],[33,4784,188],{"className":4785},[257],[33,4787,4789,4792],{"className":4788},[77],[33,4790,55],{"className":4791,"style":79},[77,78],[33,4793,4795],{"className":4794},[809],[33,4796,4798,4818],{"className":4797},[813,2522],[33,4799,4801,4815],{"className":4800},[817],[33,4802,4804],{"className":4803,"style":2529},[821],[33,4805,4806,4809],{"style":3525},[33,4807],{"className":4808,"style":829},[828],[33,4810,4812],{"className":4811},[833,834,835,836],[33,4813,428],{"className":4814},[77,836],[33,4816,2546],{"className":4817},[2545],[33,4819,4821],{"className":4820},[817],[33,4822,4824],{"className":4823,"style":2553},[821],[33,4825],{},[33,4827,178],{"className":4828},[229],[33,4830,95],{"className":4831},[77,78],[33,4833,188],{"className":4834},[257],[33,4836],{"className":4837,"style":237},[236],[33,4839,183],{"className":4840},[241],[33,4842],{"className":4843,"style":237},[236],[33,4845,4847,4850],{"className":4846},[68],[33,4848],{"className":4849,"style":390},[72],[33,4851,348],{"className":4852},[77],[15,4854,4855,4856,4971,4972,5118],{},"Note: ",[33,4857,4859,4887],{"className":4858,"translate":37},[36],[33,4860,4862],{"className":4861},[41],[43,4863,4864],{"xmlns":45},[47,4865,4866,4884],{},[50,4867,4868,4874,4876,4878],{},[2467,4869,4870,4872],{},[53,4871,55],{},[346,4873,2212],{},[175,4875,178],{"stretchy":177},[53,4877,95],{},[754,4879,4880,4882],{},[175,4881,188],{"stretchy":177},[346,4883,423],{},[57,4885,4886],{"encoding":59},"f_1(x)^2",[33,4888,4890],{"className":4889,"ariaHidden":64},[63],[33,4891,4893,4896,4936,4939,4942],{"className":4892},[68],[33,4894],{"className":4895,"style":908},[72],[33,4897,4899,4902],{"className":4898},[77],[33,4900,55],{"className":4901,"style":79},[77,78],[33,4903,4905],{"className":4904},[809],[33,4906,4908,4928],{"className":4907},[813,2522],[33,4909,4911,4925],{"className":4910},[817],[33,4912,4914],{"className":4913,"style":2529},[821],[33,4915,4916,4919],{"style":3525},[33,4917],{"className":4918,"style":829},[828],[33,4920,4922],{"className":4921},[833,834,835,836],[33,4923,2212],{"className":4924},[77,836],[33,4926,2546],{"className":4927},[2545],[33,4929,4931],{"className":4930},[817],[33,4932,4934],{"className":4933,"style":2553},[821],[33,4935],{},[33,4937,178],{"className":4938},[229],[33,4940,95],{"className":4941},[77,78],[33,4943,4945,4948],{"className":4944},[257],[33,4946,188],{"className":4947},[257],[33,4949,4951],{"className":4950},[809],[33,4952,4954],{"className":4953},[813],[33,4955,4957],{"className":4956},[817],[33,4958,4960],{"className":4959,"style":799},[821],[33,4961,4962,4965],{"style":824},[33,4963],{"className":4964,"style":829},[828],[33,4966,4968],{"className":4967},[833,834,835,836],[33,4969,423],{"className":4970},[77,836]," is element-wise squaring, ",[33,4973,4975,5011],{"className":4974,"translate":37},[36],[33,4976,4978],{"className":4977},[41],[43,4979,4980],{"xmlns":45},[47,4981,4982,5008],{},[50,4983,4984,4990,4992,4994,4996,5002,5004,5006],{},[2467,4985,4986,4988],{},[53,4987,55],{},[346,4989,423],{},[175,4991,178],{"stretchy":177},[53,4993,95],{},[175,4995,188],{"stretchy":177},[2467,4997,4998,5000],{},[53,4999,55],{},[346,5001,428],{},[175,5003,178],{"stretchy":177},[53,5005,95],{},[175,5007,188],{"stretchy":177},[57,5009,5010],{"encoding":59},"f_2(x) f_3(x)",[33,5012,5014],{"className":5013,"ariaHidden":64},[63],[33,5015,5017,5020,5060,5063,5066,5069,5109,5112,5115],{"className":5016},[68],[33,5018],{"className":5019,"style":222},[72],[33,5021,5023,5026],{"className":5022},[77],[33,5024,55],{"className":5025,"style":79},[77,78],[33,5027,5029],{"className":5028},[809],[33,5030,5032,5052],{"className":5031},[813,2522],[33,5033,5035,5049],{"className":5034},[817],[33,5036,5038],{"className":5037,"style":2529},[821],[33,5039,5040,5043],{"style":3525},[33,5041],{"className":5042,"style":829},[828],[33,5044,5046],{"className":5045},[833,834,835,836],[33,5047,423],{"className":5048},[77,836],[33,5050,2546],{"className":5051},[2545],[33,5053,5055],{"className":5054},[817],[33,5056,5058],{"className":5057,"style":2553},[821],[33,5059],{},[33,5061,178],{"className":5062},[229],[33,5064,95],{"className":5065},[77,78],[33,5067,188],{"className":5068},[257],[33,5070,5072,5075],{"className":5071},[77],[33,5073,55],{"className":5074,"style":79},[77,78],[33,5076,5078],{"className":5077},[809],[33,5079,5081,5101],{"className":5080},[813,2522],[33,5082,5084,5098],{"className":5083},[817],[33,5085,5087],{"className":5086,"style":2529},[821],[33,5088,5089,5092],{"style":3525},[33,5090],{"className":5091,"style":829},[828],[33,5093,5095],{"className":5094},[833,834,835,836],[33,5096,428],{"className":5097},[77,836],[33,5099,2546],{"className":5100},[2545],[33,5102,5104],{"className":5103},[817],[33,5105,5107],{"className":5106,"style":2553},[821],[33,5108],{},[33,5110,178],{"className":5111},[229],[33,5113,95],{"className":5114},[77,78],[33,5116,188],{"className":5117},[257]," is element-wise multiplication or dot product.",[15,5120,5121,5122,5178,5179,143],{},"These operations introduce quadratic or higher-order terms and no longer maintain the ",[33,5123,5125,5145],{"className":5124,"translate":37},[36],[33,5126,5128],{"className":5127},[41],[43,5129,5130],{"xmlns":45},[47,5131,5132,5142],{},[50,5133,5134,5136,5138,5140],{},[53,5135,3144],{},[53,5137,95],{},[175,5139,183],{},[53,5141,2171],{},[57,5143,5144],{"encoding":59},"Ax+b",[33,5146,5148,5169],{"className":5147,"ariaHidden":64},[63],[33,5149,5151,5154,5157,5160,5163,5166],{"className":5150},[68],[33,5152],{"className":5153,"style":3190},[72],[33,5155,3144],{"className":5156},[77,78],[33,5158,95],{"className":5159},[77,78],[33,5161],{"className":5162,"style":237},[236],[33,5164,183],{"className":5165},[241],[33,5167],{"className":5168,"style":237},[236],[33,5170,5172,5175],{"className":5171},[68],[33,5173],{"className":5174,"style":2183},[72],[33,5176,2171],{"className":5177},[77,78]," form. Therefore, nonlinear polynomial combinations are ",[19,5180,5181],{},"generally not affine functions",[10,5183,5185],{"id":5184},"affine-composition","Affine Composition",[15,5187,5188],{},"This question is very important and key to understanding the nature of neural networks. Let's carefully analyze whether the composition of affine functions remains affine.",[15,5190,5191],{},"Suppose you have two affine functions:",[33,5193,5195],{"className":5194,"translate":37},[158],[33,5196,5198,5253],{"className":5197,"translate":37},[36],[33,5199,5201],{"className":5200},[41],[43,5202,5203],{"xmlns":45,"display":167},[47,5204,5205,5250],{},[50,5206,5207,5209,5211,5213,5215,5217,5219,5221,5223,5225,5228,5230,5232,5234,5236,5238,5240,5243,5245,5247],{},[53,5208,55],{},[175,5210,178],{"stretchy":177},[53,5212,95],{},[175,5214,188],{"stretchy":177},[175,5216,191],{},[53,5218,3144],{},[53,5220,95],{},[175,5222,183],{},[53,5224,2171],{},[175,5226,5227],{"separator":64},",",[236,5229],{"width":1196},[53,5231,3297],{},[175,5233,178],{"stretchy":177},[53,5235,95],{},[175,5237,188],{"stretchy":177},[175,5239,191],{},[53,5241,5242],{},"C",[53,5244,95],{},[175,5246,183],{},[53,5248,5249],{},"d",[57,5251,5252],{"encoding":59},"f(x) = A x + b, \\quad g(x) = C x + d",[33,5254,5256,5283,5304,5344,5366],{"className":5255,"ariaHidden":64},[63],[33,5257,5259,5262,5265,5268,5271,5274,5277,5280],{"className":5258},[68],[33,5260],{"className":5261,"style":222},[72],[33,5263,55],{"className":5264,"style":79},[77,78],[33,5266,178],{"className":5267},[229],[33,5269,95],{"className":5270},[77,78],[33,5272,188],{"className":5273},[257],[33,5275],{"className":5276,"style":261},[236],[33,5278,191],{"className":5279},[265],[33,5281],{"className":5282,"style":261},[236],[33,5284,5286,5289,5292,5295,5298,5301],{"className":5285},[68],[33,5287],{"className":5288,"style":3190},[72],[33,5290,3144],{"className":5291},[77,78],[33,5293,95],{"className":5294},[77,78],[33,5296],{"className":5297,"style":237},[236],[33,5299,183],{"className":5300},[241],[33,5302],{"className":5303,"style":237},[236],[33,5305,5307,5310,5313,5317,5320,5323,5326,5329,5332,5335,5338,5341],{"className":5306},[68],[33,5308],{"className":5309,"style":222},[72],[33,5311,2171],{"className":5312},[77,78],[33,5314,5227],{"className":5315},[5316],"mpunct",[33,5318],{"className":5319,"style":1296},[236],[33,5321],{"className":5322,"style":3461},[236],[33,5324,3297],{"className":5325,"style":142},[77,78],[33,5327,178],{"className":5328},[229],[33,5330,95],{"className":5331},[77,78],[33,5333,188],{"className":5334},[257],[33,5336],{"className":5337,"style":261},[236],[33,5339,191],{"className":5340},[265],[33,5342],{"className":5343,"style":261},[236],[33,5345,5347,5350,5354,5357,5360,5363],{"className":5346},[68],[33,5348],{"className":5349,"style":3190},[72],[33,5351,5242],{"className":5352,"style":5353},[77,78],"margin-right:0.0715em;",[33,5355,95],{"className":5356},[77,78],[33,5358],{"className":5359,"style":237},[236],[33,5361,183],{"className":5362},[241],[33,5364],{"className":5365,"style":237},[236],[33,5367,5369,5372],{"className":5368},[68],[33,5370],{"className":5371,"style":2183},[72],[33,5373,5249],{"className":5374},[77,78],[15,5376,5377],{},"Composition is:",[33,5379,5381],{"className":5380,"translate":37},[158],[33,5382,5384,5436],{"className":5383,"translate":37},[36],[33,5385,5387],{"className":5386},[41],[43,5388,5389],{"xmlns":45,"display":167},[47,5390,5391,5433],{},[50,5392,5393,5395,5397,5399,5401,5403,5405,5407,5409,5411,5413,5415,5417,5419,5421,5423,5425,5427,5429,5431],{},[53,5394,4535],{},[175,5396,178],{"stretchy":177},[53,5398,95],{},[175,5400,188],{"stretchy":177},[175,5402,191],{},[53,5404,55],{},[175,5406,178],{"stretchy":177},[53,5408,3297],{},[175,5410,178],{"stretchy":177},[53,5412,95],{},[175,5414,188],{"stretchy":177},[175,5416,188],{"stretchy":177},[175,5418,191],{},[53,5420,55],{},[175,5422,178],{"stretchy":177},[53,5424,5242],{},[53,5426,95],{},[175,5428,183],{},[53,5430,5249],{},[175,5432,188],{"stretchy":177},[57,5434,5435],{"encoding":59},"h(x) = f(g(x)) = f(Cx + d)",[33,5437,5439,5466,5500,5527],{"className":5438,"ariaHidden":64},[63],[33,5440,5442,5445,5448,5451,5454,5457,5460,5463],{"className":5441},[68],[33,5443],{"className":5444,"style":222},[72],[33,5446,4535],{"className":5447},[77,78],[33,5449,178],{"className":5450},[229],[33,5452,95],{"className":5453},[77,78],[33,5455,188],{"className":5456},[257],[33,5458],{"className":5459,"style":261},[236],[33,5461,191],{"className":5462},[265],[33,5464],{"className":5465,"style":261},[236],[33,5467,5469,5472,5475,5478,5481,5484,5487,5491,5494,5497],{"className":5468},[68],[33,5470],{"className":5471,"style":222},[72],[33,5473,55],{"className":5474,"style":79},[77,78],[33,5476,178],{"className":5477},[229],[33,5479,3297],{"className":5480,"style":142},[77,78],[33,5482,178],{"className":5483},[229],[33,5485,95],{"className":5486},[77,78],[33,5488,5490],{"className":5489},[257],"))",[33,5492],{"className":5493,"style":261},[236],[33,5495,191],{"className":5496},[265],[33,5498],{"className":5499,"style":261},[236],[33,5501,5503,5506,5509,5512,5515,5518,5521,5524],{"className":5502},[68],[33,5504],{"className":5505,"style":222},[72],[33,5507,55],{"className":5508,"style":79},[77,78],[33,5510,178],{"className":5511},[229],[33,5513,5242],{"className":5514,"style":5353},[77,78],[33,5516,95],{"className":5517},[77,78],[33,5519],{"className":5520,"style":237},[236],[33,5522,183],{"className":5523},[241],[33,5525],{"className":5526,"style":237},[236],[33,5528,5530,5533,5536],{"className":5529},[68],[33,5531],{"className":5532,"style":222},[72],[33,5534,5249],{"className":5535},[77,78],[33,5537,188],{"className":5538},[257],[15,5540,5541],{},"Expanding to see the form:",[33,5543,5545],{"className":5544,"translate":37},[158],[33,5546,5548,5614],{"className":5547,"translate":37},[36],[33,5549,5551],{"className":5550},[41],[43,5552,5553],{"xmlns":45,"display":167},[47,5554,5555,5611],{},[50,5556,5557,5559,5561,5563,5565,5567,5569,5571,5573,5575,5577,5579,5581,5583,5585,5587,5589,5591,5593,5595,5597,5599,5601,5603,5605,5607,5609],{},[53,5558,4535],{},[175,5560,178],{"stretchy":177},[53,5562,95],{},[175,5564,188],{"stretchy":177},[175,5566,191],{},[53,5568,3144],{},[175,5570,178],{"stretchy":177},[53,5572,5242],{},[53,5574,95],{},[175,5576,183],{},[53,5578,5249],{},[175,5580,188],{"stretchy":177},[175,5582,183],{},[53,5584,2171],{},[175,5586,191],{},[175,5588,178],{"stretchy":177},[53,5590,3144],{},[53,5592,5242],{},[175,5594,188],{"stretchy":177},[53,5596,95],{},[175,5598,183],{},[175,5600,178],{"stretchy":177},[53,5602,3144],{},[53,5604,5249],{},[175,5606,183],{},[53,5608,2171],{},[175,5610,188],{"stretchy":177},[57,5612,5613],{"encoding":59},"h(x) = A(Cx + d) + b = (AC)x + (Ad + b)",[33,5615,5617,5644,5671,5692,5710,5740,5764],{"className":5616,"ariaHidden":64},[63],[33,5618,5620,5623,5626,5629,5632,5635,5638,5641],{"className":5619},[68],[33,5621],{"className":5622,"style":222},[72],[33,5624,4535],{"className":5625},[77,78],[33,5627,178],{"className":5628},[229],[33,5630,95],{"className":5631},[77,78],[33,5633,188],{"className":5634},[257],[33,5636],{"className":5637,"style":261},[236],[33,5639,191],{"className":5640},[265],[33,5642],{"className":5643,"style":261},[236],[33,5645,5647,5650,5653,5656,5659,5662,5665,5668],{"className":5646},[68],[33,5648],{"className":5649,"style":222},[72],[33,5651,3144],{"className":5652},[77,78],[33,5654,178],{"className":5655},[229],[33,5657,5242],{"className":5658,"style":5353},[77,78],[33,5660,95],{"className":5661},[77,78],[33,5663],{"className":5664,"style":237},[236],[33,5666,183],{"className":5667},[241],[33,5669],{"className":5670,"style":237},[236],[33,5672,5674,5677,5680,5683,5686,5689],{"className":5673},[68],[33,5675],{"className":5676,"style":222},[72],[33,5678,5249],{"className":5679},[77,78],[33,5681,188],{"className":5682},[257],[33,5684],{"className":5685,"style":237},[236],[33,5687,183],{"className":5688},[241],[33,5690],{"className":5691,"style":237},[236],[33,5693,5695,5698,5701,5704,5707],{"className":5694},[68],[33,5696],{"className":5697,"style":2183},[72],[33,5699,2171],{"className":5700},[77,78],[33,5702],{"className":5703,"style":261},[236],[33,5705,191],{"className":5706},[265],[33,5708],{"className":5709,"style":261},[236],[33,5711,5713,5716,5719,5722,5725,5728,5731,5734,5737],{"className":5712},[68],[33,5714],{"className":5715,"style":222},[72],[33,5717,178],{"className":5718},[229],[33,5720,3144],{"className":5721},[77,78],[33,5723,5242],{"className":5724,"style":5353},[77,78],[33,5726,188],{"className":5727},[257],[33,5729,95],{"className":5730},[77,78],[33,5732],{"className":5733,"style":237},[236],[33,5735,183],{"className":5736},[241],[33,5738],{"className":5739,"style":237},[236],[33,5741,5743,5746,5749,5752,5755,5758,5761],{"className":5742},[68],[33,5744],{"className":5745,"style":222},[72],[33,5747,178],{"className":5748},[229],[33,5750,3144],{"className":5751},[77,78],[33,5753,5249],{"className":5754},[77,78],[33,5756],{"className":5757,"style":237},[236],[33,5759,183],{"className":5760},[241],[33,5762],{"className":5763,"style":237},[236],[33,5765,5767,5770,5773],{"className":5766},[68],[33,5768],{"className":5769,"style":222},[72],[33,5771,2171],{"className":5772},[77,78],[33,5774,188],{"className":5775},[257],[15,5777,5778,5779,143],{},"The result is still an ",[19,5780,4513],{},[15,5782,5783],{},"Thus, the composition of affine functions remains affine. This can be generalized: composing any number of affine functions:",[33,5785,5787],{"className":5786,"translate":37},[158],[33,5788,5790,5842],{"className":5789,"translate":37},[36],[33,5791,5793],{"className":5792},[41],[43,5794,5795],{"xmlns":45,"display":167},[47,5796,5797,5839],{},[50,5798,5799,5805,5807,5810,5816,5818,5824,5826,5828,5830,5832,5834,5837],{},[2467,5800,5801,5803],{},[53,5802,55],{},[53,5804,2999],{},[175,5806,178],{"stretchy":177},[175,5808,5809],{},"…",[2467,5811,5812,5814],{},[53,5813,55],{},[346,5815,423],{},[175,5817,178],{"stretchy":177},[2467,5819,5820,5822],{},[53,5821,55],{},[346,5823,2212],{},[175,5825,178],{"stretchy":177},[53,5827,95],{},[175,5829,188],{"stretchy":177},[175,5831,188],{"stretchy":177},[175,5833,5809],{},[1200,5835,5836],{}," ",[175,5838,188],{"stretchy":177},[57,5840,5841],{"encoding":59},"f_n(\\dots f_2(f_1(x))\\dots)",[33,5843,5845],{"className":5844,"ariaHidden":64},[63],[33,5846,5848,5851,5892,5895,5898,5901,5941,5944,5984,5987,5990,5993,5996,5999,6002],{"className":5847},[68],[33,5849],{"className":5850,"style":222},[72],[33,5852,5854,5857],{"className":5853},[77],[33,5855,55],{"className":5856,"style":79},[77,78],[33,5858,5860],{"className":5859},[809],[33,5861,5863,5884],{"className":5862},[813,2522],[33,5864,5866,5881],{"className":5865},[817],[33,5867,5870],{"className":5868,"style":5869},[821],"height:0.1514em;",[33,5871,5872,5875],{"style":3525},[33,5873],{"className":5874,"style":829},[828],[33,5876,5878],{"className":5877},[833,834,835,836],[33,5879,2999],{"className":5880},[77,78,836],[33,5882,2546],{"className":5883},[2545],[33,5885,5887],{"className":5886},[817],[33,5888,5890],{"className":5889,"style":2553},[821],[33,5891],{},[33,5893,178],{"className":5894},[229],[33,5896,5809],{"className":5897},[4176],[33,5899],{"className":5900,"style":3461},[236],[33,5902,5904,5907],{"className":5903},[77],[33,5905,55],{"className":5906,"style":79},[77,78],[33,5908,5910],{"className":5909},[809],[33,5911,5913,5933],{"className":5912},[813,2522],[33,5914,5916,5930],{"className":5915},[817],[33,5917,5919],{"className":5918,"style":2529},[821],[33,5920,5921,5924],{"style":3525},[33,5922],{"className":5923,"style":829},[828],[33,5925,5927],{"className":5926},[833,834,835,836],[33,5928,423],{"className":5929},[77,836],[33,5931,2546],{"className":5932},[2545],[33,5934,5936],{"className":5935},[817],[33,5937,5939],{"className":5938,"style":2553},[821],[33,5940],{},[33,5942,178],{"className":5943},[229],[33,5945,5947,5950],{"className":5946},[77],[33,5948,55],{"className":5949,"style":79},[77,78],[33,5951,5953],{"className":5952},[809],[33,5954,5956,5976],{"className":5955},[813,2522],[33,5957,5959,5973],{"className":5958},[817],[33,5960,5962],{"className":5961,"style":2529},[821],[33,5963,5964,5967],{"style":3525},[33,5965],{"className":5966,"style":829},[828],[33,5968,5970],{"className":5969},[833,834,835,836],[33,5971,2212],{"className":5972},[77,836],[33,5974,2546],{"className":5975},[2545],[33,5977,5979],{"className":5978},[817],[33,5980,5982],{"className":5981,"style":2553},[821],[33,5983],{},[33,5985,178],{"className":5986},[229],[33,5988,95],{"className":5989},[77,78],[33,5991,5490],{"className":5992},[257],[33,5994],{"className":5995,"style":3461},[236],[33,5997,5809],{"className":5998},[4176],[33,6000],{"className":6001,"style":3461},[236],[33,6003,188],{"className":6004},[257],[15,6006,6007],{},"can always be written as:",[33,6009,6011],{"className":6010,"translate":37},[158],[33,6012,6014,6043],{"className":6013,"translate":37},[36],[33,6015,6017],{"className":6016},[41],[43,6018,6019],{"xmlns":45,"display":167},[47,6020,6021,6040],{},[50,6022,6023,6030,6032,6034],{},[2467,6024,6025,6027],{},[53,6026,3144],{},[1200,6028,6029],{},"total",[53,6031,95],{},[175,6033,183],{},[2467,6035,6036,6038],{},[53,6037,2171],{},[1200,6039,6029],{},[57,6041,6042],{"encoding":59},"A_\\text{total} x + b_\\text{total}",[33,6044,6046,6108],{"className":6045,"ariaHidden":64},[63],[33,6047,6049,6052,6096,6099,6102,6105],{"className":6048},[68],[33,6050],{"className":6051,"style":3676},[72],[33,6053,6055,6058],{"className":6054},[77],[33,6056,3144],{"className":6057},[77,78],[33,6059,6061],{"className":6060},[809],[33,6062,6064,6088],{"className":6063},[813,2522],[33,6065,6067,6085],{"className":6066},[817],[33,6068,6071],{"className":6069,"style":6070},[821],"height:0.3361em;",[33,6072,6073,6076],{"style":2532},[33,6074],{"className":6075,"style":829},[828],[33,6077,6079],{"className":6078},[833,834,835,836],[33,6080,6082],{"className":6081},[77,1303,836],[33,6083,6029],{"className":6084},[77,836],[33,6086,2546],{"className":6087},[2545],[33,6089,6091],{"className":6090},[817],[33,6092,6094],{"className":6093,"style":2553},[821],[33,6095],{},[33,6097,95],{"className":6098},[77,78],[33,6100],{"className":6101,"style":237},[236],[33,6103,183],{"className":6104},[241],[33,6106],{"className":6107,"style":237},[236],[33,6109,6111,6114],{"className":6110},[68],[33,6112],{"className":6113,"style":3735},[72],[33,6115,6117,6120],{"className":6116},[77],[33,6118,2171],{"className":6119},[77,78],[33,6121,6123],{"className":6122},[809],[33,6124,6126,6149],{"className":6125},[813,2522],[33,6127,6129,6146],{"className":6128},[817],[33,6130,6132],{"className":6131,"style":6070},[821],[33,6133,6134,6137],{"style":2532},[33,6135],{"className":6136,"style":829},[828],[33,6138,6140],{"className":6139},[833,834,835,836],[33,6141,6143],{"className":6142},[77,1303,836],[33,6144,6029],{"className":6145},[77,836],[33,6147,2546],{"className":6148},[2545],[33,6150,6152],{"className":6151},[817],[33,6153,6155],{"className":6154,"style":2553},[821],[33,6156],{},[15,6158,6159,6160,6163,6164,6167],{},"This is also why: ",[19,6161,6162],{},"if each layer of a neural network is affine without activation functions",", no matter how many layers are stacked, the network's expressive power is ",[19,6165,6166],{},"the same as a single layer",". We can think of an affine layer as an operation of stretching, rotating, or translating, or a combination of these. Composing multiple affine transformations is equivalent to continuous stretching + rotation + translation → ultimately, a single overall stretching + rotation + translation. Therefore, composing affine transformations does not increase nonlinear degrees of freedom and cannot represent curves or complex functions.",[15,6169,6170],{},"Each layer of a neural network is an affine mapping:",[33,6172,6174],{"className":6173,"translate":37},[158],[33,6175,6177,6208],{"className":6176,"translate":37},[36],[33,6178,6180],{"className":6179},[41],[43,6181,6182],{"xmlns":45,"display":167},[47,6183,6184,6205],{},[50,6185,6186,6188,6190,6192,6194,6196,6199,6201,6203],{},[53,6187,55],{},[175,6189,178],{"stretchy":177},[53,6191,95],{},[175,6193,188],{"stretchy":177},[175,6195,191],{},[53,6197,6198],{},"W",[53,6200,95],{},[175,6202,183],{},[53,6204,2171],{},[57,6206,6207],{"encoding":59},"f(x) = Wx + b",[33,6209,6211,6238,6260],{"className":6210,"ariaHidden":64},[63],[33,6212,6214,6217,6220,6223,6226,6229,6232,6235],{"className":6213},[68],[33,6215],{"className":6216,"style":222},[72],[33,6218,55],{"className":6219,"style":79},[77,78],[33,6221,178],{"className":6222},[229],[33,6224,95],{"className":6225},[77,78],[33,6227,188],{"className":6228},[257],[33,6230],{"className":6231,"style":261},[236],[33,6233,191],{"className":6234},[265],[33,6236],{"className":6237,"style":261},[236],[33,6239,6241,6244,6248,6251,6254,6257],{"className":6240},[68],[33,6242],{"className":6243,"style":3190},[72],[33,6245,6198],{"className":6246,"style":6247},[77,78],"margin-right:0.1389em;",[33,6249,95],{"className":6250},[77,78],[33,6252],{"className":6253,"style":237},[236],[33,6255,183],{"className":6256},[241],[33,6258],{"className":6259,"style":237},[236],[33,6261,6263,6266],{"className":6262},[68],[33,6264],{"className":6265,"style":2183},[72],[33,6267,2171],{"className":6268},[77,78],[399,6270,6271,6436],{},[402,6272,6273,6274,6302,6303,6406,6407,6435],{},"If ",[33,6275,6277,6290],{"className":6276,"translate":37},[36],[33,6278,6280],{"className":6279},[41],[43,6281,6282],{"xmlns":45},[47,6283,6284,6288],{},[50,6285,6286],{},[53,6287,6198],{},[57,6289,6198],{"encoding":59},[33,6291,6293],{"className":6292,"ariaHidden":64},[63],[33,6294,6296,6299],{"className":6295},[68],[33,6297],{"className":6298,"style":3243},[72],[33,6300,6198],{"className":6301,"style":6247},[77,78]," is invertible (i.e., ",[33,6304,6306,6334],{"className":6305,"translate":37},[36],[33,6307,6309],{"className":6308},[41],[43,6310,6311],{"xmlns":45},[47,6312,6313,6331],{},[50,6314,6315,6318,6321,6323,6325,6327,6329],{},[53,6316,6317],{},"det",[175,6319,6320],{},"⁡",[175,6322,178],{"stretchy":177},[53,6324,6198],{},[175,6326,188],{"stretchy":177},[175,6328,2296],{"mathvariant":2295},[346,6330,2286],{},[57,6332,6333],{"encoding":59},"\\det(W) \\neq 0",[33,6335,6337,6397],{"className":6336,"ariaHidden":64},[63],[33,6338,6340,6343,6346,6349,6352,6355,6358,6394],{"className":6339},[68],[33,6341],{"className":6342,"style":222},[72],[33,6344,6317],{"className":6345},[3386],[33,6347,178],{"className":6348},[229],[33,6350,6198],{"className":6351,"style":6247},[77,78],[33,6353,188],{"className":6354},[257],[33,6356],{"className":6357,"style":261},[236],[33,6359,6361,6388,6391],{"className":6360},[265],[33,6362,6364],{"className":6363},[265],[33,6365,6367],{"className":6366},[77,2353],[33,6368,6370],{"className":6369},[2357],[33,6371,6373,6376,6385],{"className":6372},[2361],[33,6374],{"className":6375,"style":73},[72],[33,6377,6379],{"className":6378},[2368],[33,6380,6382],{"className":6381},[77],[33,6383,2375],{"className":6384},[265],[33,6386],{"className":6387},[2379],[33,6389],{"className":6390},[236,2383],[33,6392,191],{"className":6393},[265],[33,6395],{"className":6396,"style":261},[236],[33,6398,6400,6403],{"className":6399},[68],[33,6401],{"className":6402,"style":390},[72],[33,6404,2286],{"className":6405},[77],"), then ",[33,6408,6410,6423],{"className":6409,"translate":37},[36],[33,6411,6413],{"className":6412},[41],[43,6414,6415],{"xmlns":45},[47,6416,6417,6421],{},[50,6418,6419],{},[53,6420,55],{},[57,6422,55],{"encoding":59},[33,6424,6426],{"className":6425,"ariaHidden":64},[63],[33,6427,6429,6432],{"className":6428},[68],[33,6430],{"className":6431,"style":73},[72],[33,6433,55],{"className":6434,"style":79},[77,78]," is bijective and continuous.",[402,6437,6438],{},"The inverse mapping is:",[33,6440,6442],{"className":6441,"translate":37},[158],[33,6443,6445,6496],{"className":6444,"translate":37},[36],[33,6446,6448],{"className":6447},[41],[43,6449,6450],{"xmlns":45,"display":167},[47,6451,6452,6493],{},[50,6453,6454,6465,6467,6469,6471,6473,6483,6485,6487,6489,6491],{},[754,6455,6456,6458],{},[53,6457,55],{},[50,6459,6460,6463],{},[175,6461,6462],{},"−",[346,6464,2212],{},[175,6466,178],{"stretchy":177},[53,6468,126],{},[175,6470,188],{"stretchy":177},[175,6472,191],{},[754,6474,6475,6477],{},[53,6476,3144],{},[50,6478,6479,6481],{},[175,6480,6462],{},[346,6482,2212],{},[175,6484,178],{"stretchy":177},[53,6486,126],{},[175,6488,6462],{},[53,6490,2171],{},[175,6492,188],{"stretchy":177},[57,6494,6495],{"encoding":59},"f^{-1}(y) = A^{-1}(y - b)",[33,6497,6499,6558,6614],{"className":6498,"ariaHidden":64},[63],[33,6500,6502,6505,6540,6543,6546,6549,6552,6555],{"className":6501},[68],[33,6503],{"className":6504,"style":4635},[72],[33,6506,6508,6511],{"className":6507},[77],[33,6509,55],{"className":6510,"style":79},[77,78],[33,6512,6514],{"className":6513},[809],[33,6515,6517],{"className":6516},[813],[33,6518,6520],{"className":6519},[817],[33,6521,6523],{"className":6522,"style":4706},[821],[33,6524,6525,6528],{"style":4709},[33,6526],{"className":6527,"style":829},[828],[33,6529,6531],{"className":6530},[833,834,835,836],[33,6532,6534,6537],{"className":6533},[77,836],[33,6535,6462],{"className":6536},[77,836],[33,6538,2212],{"className":6539},[77,836],[33,6541,178],{"className":6542},[229],[33,6544,126],{"className":6545,"style":142},[77,78],[33,6547,188],{"className":6548},[257],[33,6550],{"className":6551,"style":261},[236],[33,6553,191],{"className":6554},[265],[33,6556],{"className":6557,"style":261},[236],[33,6559,6561,6564,6599,6602,6605,6608,6611],{"className":6560},[68],[33,6562],{"className":6563,"style":4635},[72],[33,6565,6567,6570],{"className":6566},[77],[33,6568,3144],{"className":6569},[77,78],[33,6571,6573],{"className":6572},[809],[33,6574,6576],{"className":6575},[813],[33,6577,6579],{"className":6578},[817],[33,6580,6582],{"className":6581,"style":4706},[821],[33,6583,6584,6587],{"style":4709},[33,6585],{"className":6586,"style":829},[828],[33,6588,6590],{"className":6589},[833,834,835,836],[33,6591,6593,6596],{"className":6592},[77,836],[33,6594,6462],{"className":6595},[77,836],[33,6597,2212],{"className":6598},[77,836],[33,6600,178],{"className":6601},[229],[33,6603,126],{"className":6604,"style":142},[77,78],[33,6606],{"className":6607,"style":237},[236],[33,6609,6462],{"className":6610},[241],[33,6612],{"className":6613,"style":237},[236],[33,6615,6617,6620,6623],{"className":6616},[68],[33,6618],{"className":6619,"style":222},[72],[33,6621,2171],{"className":6622},[77,78],[33,6624,188],{"className":6625},[257],[15,6627,6628,6629,6632,6633,6661],{},"which is also affine and continuous → a ",[19,6630,6631],{},"homeomorphism",". However, there is a core condition: ",[33,6634,6636,6649],{"className":6635,"translate":37},[36],[33,6637,6639],{"className":6638},[41],[43,6640,6641],{"xmlns":45},[47,6642,6643,6647],{},[50,6644,6645],{},[53,6646,6198],{},[57,6648,6198],{"encoding":59},[33,6650,6652],{"className":6651,"ariaHidden":64},[63],[33,6653,6655,6658],{"className":6654},[68],[33,6656],{"className":6657,"style":3243},[72],[33,6659,6198],{"className":6660,"style":6247},[77,78]," must be invertible.",[15,6663,6664],{},"Composing multiple affine transformations is equivalent to composing multiple homeomorphisms. Suppose you have multiple consecutive invertible affine maps:",[33,6666,6668],{"className":6667,"translate":37},[158],[33,6669,6671,6717],{"className":6670,"translate":37},[36],[33,6672,6674],{"className":6673},[41],[43,6675,6676],{"xmlns":45,"display":167},[47,6677,6678,6714],{},[50,6679,6680,6686,6689,6701,6703,6706,6708],{},[2467,6681,6682,6684],{},[53,6683,55],{},[53,6685,2999],{},[175,6687,6688],{},"∘",[2467,6690,6691,6693],{},[53,6692,55],{},[50,6694,6695,6697,6699],{},[53,6696,2999],{},[175,6698,6462],{},[346,6700,2212],{},[175,6702,6688],{},[175,6704,6705],{},"⋯",[175,6707,6688],{},[2467,6709,6710,6712],{},[53,6711,55],{},[346,6713,2212],{},[57,6715,6716],{"encoding":59},"f_n \\circ f_{n-1} \\circ \\cdots \\circ f_1",[33,6718,6720,6775,6841,6859],{"className":6719,"ariaHidden":64},[63],[33,6721,6723,6726,6766,6769,6772],{"className":6722},[68],[33,6724],{"className":6725,"style":73},[72],[33,6727,6729,6732],{"className":6728},[77],[33,6730,55],{"className":6731,"style":79},[77,78],[33,6733,6735],{"className":6734},[809],[33,6736,6738,6758],{"className":6737},[813,2522],[33,6739,6741,6755],{"className":6740},[817],[33,6742,6744],{"className":6743,"style":5869},[821],[33,6745,6746,6749],{"style":3525},[33,6747],{"className":6748,"style":829},[828],[33,6750,6752],{"className":6751},[833,834,835,836],[33,6753,2999],{"className":6754},[77,78,836],[33,6756,2546],{"className":6757},[2545],[33,6759,6761],{"className":6760},[817],[33,6762,6764],{"className":6763,"style":2553},[821],[33,6765],{},[33,6767],{"className":6768,"style":237},[236],[33,6770,6688],{"className":6771},[241],[33,6773],{"className":6774,"style":237},[236],[33,6776,6778,6782,6832,6835,6838],{"className":6777},[68],[33,6779],{"className":6780,"style":6781},[72],"height:0.9028em;vertical-align:-0.2083em;",[33,6783,6785,6788],{"className":6784},[77],[33,6786,55],{"className":6787,"style":79},[77,78],[33,6789,6791],{"className":6790},[809],[33,6792,6794,6823],{"className":6793},[813,2522],[33,6795,6797,6820],{"className":6796},[817],[33,6798,6800],{"className":6799,"style":2529},[821],[33,6801,6802,6805],{"style":3525},[33,6803],{"className":6804,"style":829},[828],[33,6806,6808],{"className":6807},[833,834,835,836],[33,6809,6811,6814,6817],{"className":6810},[77,836],[33,6812,2999],{"className":6813},[77,78,836],[33,6815,6462],{"className":6816},[241,836],[33,6818,2212],{"className":6819},[77,836],[33,6821,2546],{"className":6822},[2545],[33,6824,6826],{"className":6825},[817],[33,6827,6830],{"className":6828,"style":6829},[821],"height:0.2083em;",[33,6831],{},[33,6833],{"className":6834,"style":237},[236],[33,6836,6688],{"className":6837},[241],[33,6839],{"className":6840,"style":237},[236],[33,6842,6844,6847,6850,6853,6856],{"className":6843},[68],[33,6845],{"className":6846,"style":1262},[72],[33,6848,6705],{"className":6849},[4176],[33,6851],{"className":6852,"style":237},[236],[33,6854,6688],{"className":6855},[241],[33,6857],{"className":6858,"style":237},[236],[33,6860,6862,6865],{"className":6861},[68],[33,6863],{"className":6864,"style":73},[72],[33,6866,6868,6871],{"className":6867},[77],[33,6869,55],{"className":6870,"style":79},[77,78],[33,6872,6874],{"className":6873},[809],[33,6875,6877,6897],{"className":6876},[813,2522],[33,6878,6880,6894],{"className":6879},[817],[33,6881,6883],{"className":6882,"style":2529},[821],[33,6884,6885,6888],{"style":3525},[33,6886],{"className":6887,"style":829},[828],[33,6889,6891],{"className":6890},[833,834,835,836],[33,6892,2212],{"className":6893},[77,836],[33,6895,2546],{"className":6896},[2545],[33,6898,6900],{"className":6899},[817],[33,6901,6903],{"className":6902,"style":2553},[821],[33,6904],{},[15,6906,6907,6908,6978],{},"Each ",[33,6909,6911,6929],{"className":6910,"translate":37},[36],[33,6912,6914],{"className":6913},[41],[43,6915,6916],{"xmlns":45},[47,6917,6918,6926],{},[50,6919,6920],{},[2467,6921,6922,6924],{},[53,6923,55],{},[53,6925,3316],{},[57,6927,6928],{"encoding":59},"f_i",[33,6930,6932],{"className":6931,"ariaHidden":64},[63],[33,6933,6935,6938],{"className":6934},[68],[33,6936],{"className":6937,"style":73},[72],[33,6939,6941,6944],{"className":6940},[77],[33,6942,55],{"className":6943,"style":79},[77,78],[33,6945,6947],{"className":6946},[809],[33,6948,6950,6970],{"className":6949},[813,2522],[33,6951,6953,6967],{"className":6952},[817],[33,6954,6956],{"className":6955,"style":3481},[821],[33,6957,6958,6961],{"style":3525},[33,6959],{"className":6960,"style":829},[828],[33,6962,6964],{"className":6963},[833,834,835,836],[33,6965,3316],{"className":6966},[77,78,836],[33,6968,2546],{"className":6969},[2545],[33,6971,6973],{"className":6972},[817],[33,6974,6976],{"className":6975,"style":2553},[821],[33,6977],{}," is a homeomorphism, and the composition of homeomorphisms remains a homeomorphism. Hence, the final mapping is also a homeomorphism.",[15,6980,6981,6982,7012],{},"Thus, we have the following conclusion: Let ",[33,6983,6985,6999],{"className":6984,"translate":37},[36],[33,6986,6988],{"className":6987},[41],[43,6989,6990],{"xmlns":45},[47,6991,6992,6997],{},[50,6993,6994],{},[53,6995,6996],{},"M",[57,6998,6996],{"encoding":59},[33,7000,7002],{"className":7001,"ariaHidden":64},[63],[33,7003,7005,7008],{"className":7004},[68],[33,7006],{"className":7007,"style":3243},[72],[33,7009,6996],{"className":7010,"style":7011},[77,78],"margin-right:0.109em;"," be the original manifold:",[399,7014,7015,7018],{},[402,7016,7017],{},"An affine mapping that is continuous and bijective → the manifold structure is preserved (homeomorphic).",[402,7019,7020,7021,7024,7025,143],{},"Composing multiple such affine maps continuously → the resulting manifold is ",[19,7022,7023],{},"homeomorphic"," to ",[33,7026,7028,7041],{"className":7027,"translate":37},[36],[33,7029,7031],{"className":7030},[41],[43,7032,7033],{"xmlns":45},[47,7034,7035,7039],{},[50,7036,7037],{},[53,7038,6996],{},[57,7040,6996],{"encoding":59},[33,7042,7044],{"className":7043,"ariaHidden":64},[63],[33,7045,7047,7050],{"className":7046},[68],[33,7048],{"className":7049,"style":3243},[72],[33,7051,6996],{"className":7052,"style":7011},[77,78],[15,7054,7055],{},"In other words: \"Continuous linear + bias transformations do not change the topology of the manifold; they only stretch\u002Frotate\u002Ftranslate\u002Fshear it.\"",[15,7057,7058],{},"This is why using only linear layers cannot \"bend\" the data manifold: the data retains its \"original shape\" (topologically invariant) under nested linear layers. If a decision boundary needs to cut a complex manifold, nonlinearity (activation functions) must be introduced.",[15,7060,7061,7062,143],{},"This is also the goal in deep learning. If the original manifold's structure is inseparable for a given task, but separability is required for a learning objective, then the original manifold and the target manifold ",[19,7063,7064],{},"must not be homeomorphic",[15,7066,7067],{},"In the transformation from the original manifold to the target manifold, losing homeomorphism requires breaking away from linear, affine transformations. Because affine transformations (or more generally, homeomorphic transformations) alone are usually insufficient for disentangling classes. Deep networks often need to reconstruct the data manifold through non-homeomorphic nonlinear operations to make it ultimately linearly separable.",[15,7069,7070],{},"We need to consider how to make the composition of affine transformations nonlinear. This insight is the core of the geometric interpretation of neural networks:",[315,7072,7073],{},[15,7074,7075],{},"Linear layers can only perform topology-preserving affine transformations; nonlinear layers allow topological changes, enabling neural networks to represent complex functions.",[10,7077,7079],{"id":7078},"activation-for-affine-layers","Activation for Affine Layers",[15,7081,6170],{},[33,7083,7085],{"className":7084,"translate":37},[158],[33,7086,7088,7131],{"className":7087,"translate":37},[36],[33,7089,7091],{"className":7090},[41],[43,7092,7093],{"xmlns":45,"display":167},[47,7094,7095,7128],{},[50,7096,7097,7104,7106,7108,7110,7112,7118,7120,7122],{},[2467,7098,7099,7102],{},[53,7100,7101],{},"l",[53,7103,3316],{},[175,7105,178],{"stretchy":177},[53,7107,95],{},[175,7109,188],{"stretchy":177},[175,7111,191],{},[2467,7113,7114,7116],{},[53,7115,6198],{},[53,7117,3316],{},[53,7119,95],{},[175,7121,183],{},[2467,7123,7124,7126],{},[53,7125,2171],{},[53,7127,3316],{},[57,7129,7130],{"encoding":59},"l_i(x) = W_i x + b_i",[33,7132,7134,7200,7259],{"className":7133,"ariaHidden":64},[63],[33,7135,7137,7140,7182,7185,7188,7191,7194,7197],{"className":7136},[68],[33,7138],{"className":7139,"style":222},[72],[33,7141,7143,7147],{"className":7142},[77],[33,7144,7101],{"className":7145,"style":7146},[77,78],"margin-right:0.0197em;",[33,7148,7150],{"className":7149},[809],[33,7151,7153,7174],{"className":7152},[813,2522],[33,7154,7156,7171],{"className":7155},[817],[33,7157,7159],{"className":7158,"style":3481},[821],[33,7160,7162,7165],{"style":7161},"top:-2.55em;margin-left:-0.0197em;margin-right:0.05em;",[33,7163],{"className":7164,"style":829},[828],[33,7166,7168],{"className":7167},[833,834,835,836],[33,7169,3316],{"className":7170},[77,78,836],[33,7172,2546],{"className":7173},[2545],[33,7175,7177],{"className":7176},[817],[33,7178,7180],{"className":7179,"style":2553},[821],[33,7181],{},[33,7183,178],{"className":7184},[229],[33,7186,95],{"className":7187},[77,78],[33,7189,188],{"className":7190},[257],[33,7192],{"className":7193,"style":261},[236],[33,7195,191],{"className":7196},[265],[33,7198],{"className":7199,"style":261},[236],[33,7201,7203,7206,7247,7250,7253,7256],{"className":7202},[68],[33,7204],{"className":7205,"style":3676},[72],[33,7207,7209,7212],{"className":7208},[77],[33,7210,6198],{"className":7211,"style":6247},[77,78],[33,7213,7215],{"className":7214},[809],[33,7216,7218,7239],{"className":7217},[813,2522],[33,7219,7221,7236],{"className":7220},[817],[33,7222,7224],{"className":7223,"style":3481},[821],[33,7225,7227,7230],{"style":7226},"top:-2.55em;margin-left:-0.1389em;margin-right:0.05em;",[33,7228],{"className":7229,"style":829},[828],[33,7231,7233],{"className":7232},[833,834,835,836],[33,7234,3316],{"className":7235},[77,78,836],[33,7237,2546],{"className":7238},[2545],[33,7240,7242],{"className":7241},[817],[33,7243,7245],{"className":7244,"style":2553},[821],[33,7246],{},[33,7248,95],{"className":7249},[77,78],[33,7251],{"className":7252,"style":237},[236],[33,7254,183],{"className":7255},[241],[33,7257],{"className":7258,"style":237},[236],[33,7260,7262,7265],{"className":7261},[68],[33,7263],{"className":7264,"style":3735},[72],[33,7266,7268,7271],{"className":7267},[77],[33,7269,2171],{"className":7270},[77,78],[33,7272,7274],{"className":7273},[809],[33,7275,7277,7297],{"className":7276},[813,2522],[33,7278,7280,7294],{"className":7279},[817],[33,7281,7283],{"className":7282,"style":3481},[821],[33,7284,7285,7288],{"style":2532},[33,7286],{"className":7287,"style":829},[828],[33,7289,7291],{"className":7290},[833,834,835,836],[33,7292,3316],{"className":7293},[77,78,836],[33,7295,2546],{"className":7296},[2545],[33,7298,7300],{"className":7299},[817],[33,7301,7303],{"className":7302,"style":2553},[821],[33,7304],{},[15,7306,3217,7307,7336,7337,7365,7366,7394],{},[33,7308,7310,7323],{"className":7309,"translate":37},[36],[33,7311,7313],{"className":7312},[41],[43,7314,7315],{"xmlns":45},[47,7316,7317,7321],{},[50,7318,7319],{},[53,7320,3316],{},[57,7322,3316],{"encoding":59},[33,7324,7326],{"className":7325,"ariaHidden":64},[63],[33,7327,7329,7333],{"className":7328},[68],[33,7330],{"className":7331,"style":7332},[72],"height:0.6595em;",[33,7334,3316],{"className":7335},[77,78]," denotes the ",[33,7338,7340,7353],{"className":7339,"translate":37},[36],[33,7341,7343],{"className":7342},[41],[43,7344,7345],{"xmlns":45},[47,7346,7347,7351],{},[50,7348,7349],{},[53,7350,3316],{},[57,7352,3316],{"encoding":59},[33,7354,7356],{"className":7355,"ariaHidden":64},[63],[33,7357,7359,7362],{"className":7358},[68],[33,7360],{"className":7361,"style":7332},[72],[33,7363,3316],{"className":7364},[77,78],"-th layer. The nested composition of affine mappings across ",[33,7367,7369,7382],{"className":7368,"translate":37},[36],[33,7370,7372],{"className":7371},[41],[43,7373,7374],{"xmlns":45},[47,7375,7376,7380],{},[50,7377,7378],{},[53,7379,2999],{},[57,7381,2999],{"encoding":59},[33,7383,7385],{"className":7384,"ariaHidden":64},[63],[33,7386,7388,7391],{"className":7387},[68],[33,7389],{"className":7390,"style":107},[72],[33,7392,2999],{"className":7393},[77,78]," layers is:",[33,7396,7398],{"className":7397,"translate":37},[158],[33,7399,7401,7459],{"className":7400,"translate":37},[36],[33,7402,7404],{"className":7403},[41],[43,7405,7406],{"xmlns":45,"display":167},[47,7407,7408,7456],{},[50,7409,7410,7416,7418,7430,7432,7444,7446,7448,7450],{},[2467,7411,7412,7414],{},[53,7413,7101],{},[53,7415,2999],{},[175,7417,6688],{},[2467,7419,7420,7422],{},[53,7421,7101],{},[50,7423,7424,7426,7428],{},[53,7425,2999],{},[175,7427,6462],{},[346,7429,2212],{},[175,7431,6688],{},[2467,7433,7434,7436],{},[53,7435,7101],{},[50,7437,7438,7440,7442],{},[53,7439,2999],{},[175,7441,6462],{},[346,7443,423],{},[175,7445,6688],{},[175,7447,6705],{},[175,7449,6688],{},[2467,7451,7452,7454],{},[53,7453,7101],{},[346,7455,2286],{},[57,7457,7458],{"encoding":59},"l_n \\circ l_{n-1} \\circ l_{n-2} \\circ \\dots \\circ l_0",[33,7460,7462,7517,7581,7645,7663],{"className":7461,"ariaHidden":64},[63],[33,7463,7465,7468,7508,7511,7514],{"className":7464},[68],[33,7466],{"className":7467,"style":3735},[72],[33,7469,7471,7474],{"className":7470},[77],[33,7472,7101],{"className":7473,"style":7146},[77,78],[33,7475,7477],{"className":7476},[809],[33,7478,7480,7500],{"className":7479},[813,2522],[33,7481,7483,7497],{"className":7482},[817],[33,7484,7486],{"className":7485,"style":5869},[821],[33,7487,7488,7491],{"style":7161},[33,7489],{"className":7490,"style":829},[828],[33,7492,7494],{"className":7493},[833,834,835,836],[33,7495,2999],{"className":7496},[77,78,836],[33,7498,2546],{"className":7499},[2545],[33,7501,7503],{"className":7502},[817],[33,7504,7506],{"className":7505,"style":2553},[821],[33,7507],{},[33,7509],{"className":7510,"style":237},[236],[33,7512,6688],{"className":7513},[241],[33,7515],{"className":7516,"style":237},[236],[33,7518,7520,7523,7572,7575,7578],{"className":7519},[68],[33,7521],{"className":7522,"style":6781},[72],[33,7524,7526,7529],{"className":7525},[77],[33,7527,7101],{"className":7528,"style":7146},[77,78],[33,7530,7532],{"className":7531},[809],[33,7533,7535,7564],{"className":7534},[813,2522],[33,7536,7538,7561],{"className":7537},[817],[33,7539,7541],{"className":7540,"style":2529},[821],[33,7542,7543,7546],{"style":7161},[33,7544],{"className":7545,"style":829},[828],[33,7547,7549],{"className":7548},[833,834,835,836],[33,7550,7552,7555,7558],{"className":7551},[77,836],[33,7553,2999],{"className":7554},[77,78,836],[33,7556,6462],{"className":7557},[241,836],[33,7559,2212],{"className":7560},[77,836],[33,7562,2546],{"className":7563},[2545],[33,7565,7567],{"className":7566},[817],[33,7568,7570],{"className":7569,"style":6829},[821],[33,7571],{},[33,7573],{"className":7574,"style":237},[236],[33,7576,6688],{"className":7577},[241],[33,7579],{"className":7580,"style":237},[236],[33,7582,7584,7587,7636,7639,7642],{"className":7583},[68],[33,7585],{"className":7586,"style":6781},[72],[33,7588,7590,7593],{"className":7589},[77],[33,7591,7101],{"className":7592,"style":7146},[77,78],[33,7594,7596],{"className":7595},[809],[33,7597,7599,7628],{"className":7598},[813,2522],[33,7600,7602,7625],{"className":7601},[817],[33,7603,7605],{"className":7604,"style":2529},[821],[33,7606,7607,7610],{"style":7161},[33,7608],{"className":7609,"style":829},[828],[33,7611,7613],{"className":7612},[833,834,835,836],[33,7614,7616,7619,7622],{"className":7615},[77,836],[33,7617,2999],{"className":7618},[77,78,836],[33,7620,6462],{"className":7621},[241,836],[33,7623,423],{"className":7624},[77,836],[33,7626,2546],{"className":7627},[2545],[33,7629,7631],{"className":7630},[817],[33,7632,7634],{"className":7633,"style":6829},[821],[33,7635],{},[33,7637],{"className":7638,"style":237},[236],[33,7640,6688],{"className":7641},[241],[33,7643],{"className":7644,"style":237},[236],[33,7646,7648,7651,7654,7657,7660],{"className":7647},[68],[33,7649],{"className":7650,"style":1262},[72],[33,7652,6705],{"className":7653},[4176],[33,7655],{"className":7656,"style":237},[236],[33,7658,6688],{"className":7659},[241],[33,7661],{"className":7662,"style":237},[236],[33,7664,7666,7669],{"className":7665},[68],[33,7667],{"className":7668,"style":3735},[72],[33,7670,7672,7675],{"className":7671},[77],[33,7673,7101],{"className":7674,"style":7146},[77,78],[33,7676,7678],{"className":7677},[809],[33,7679,7681,7701],{"className":7680},[813,2522],[33,7682,7684,7698],{"className":7683},[817],[33,7685,7687],{"className":7686,"style":2529},[821],[33,7688,7689,7692],{"style":7161},[33,7690],{"className":7691,"style":829},[828],[33,7693,7695],{"className":7694},[833,834,835,836],[33,7696,2286],{"className":7697},[77,836],[33,7699,2546],{"className":7700},[2545],[33,7702,7704],{"className":7703},[817],[33,7705,7707],{"className":7706,"style":2553},[821],[33,7708],{},[15,7710,7711],{},"If no activation function is introduced, the result above is still an affine function.",[15,7713,7714],{},"Proof is simple:",[33,7716,7718],{"className":7717,"translate":37},[158],[33,7719,7721,7837],{"className":7720,"translate":37},[36],[33,7722,7724],{"className":7723},[41],[43,7725,7726],{"xmlns":45,"display":167},[47,7727,7728,7834],{},[50,7729,7730,7736,7738,7744,7746,7748,7750,7752,7754,7760,7762,7768,7770,7772,7778,7780,7782,7788,7790,7792,7798,7804,7806,7808,7810,7812,7818,7824,7826,7832],{},[2467,7731,7732,7734],{},[53,7733,7101],{},[346,7735,423],{},[175,7737,178],{"stretchy":177},[2467,7739,7740,7742],{},[53,7741,7101],{},[346,7743,2212],{},[175,7745,178],{"stretchy":177},[53,7747,95],{},[175,7749,188],{"stretchy":177},[175,7751,188],{"stretchy":177},[175,7753,191],{},[2467,7755,7756,7758],{},[53,7757,6198],{},[346,7759,423],{},[175,7761,178],{"stretchy":177},[2467,7763,7764,7766],{},[53,7765,6198],{},[346,7767,2212],{},[53,7769,95],{},[175,7771,183],{},[2467,7773,7774,7776],{},[53,7775,2171],{},[346,7777,2212],{},[175,7779,188],{"stretchy":177},[175,7781,183],{},[2467,7783,7784,7786],{},[53,7785,2171],{},[346,7787,423],{},[175,7789,191],{},[175,7791,178],{"stretchy":177},[2467,7793,7794,7796],{},[53,7795,6198],{},[346,7797,423],{},[2467,7799,7800,7802],{},[53,7801,6198],{},[346,7803,2212],{},[175,7805,188],{"stretchy":177},[53,7807,95],{},[175,7809,183],{},[175,7811,178],{"stretchy":177},[2467,7813,7814,7816],{},[53,7815,6198],{},[346,7817,423],{},[2467,7819,7820,7822],{},[53,7821,2171],{},[346,7823,2212],{},[175,7825,183],{},[2467,7827,7828,7830],{},[53,7829,2171],{},[346,7831,423],{},[175,7833,188],{"stretchy":177},[57,7835,7836],{"encoding":59},"l_2(l_1(x)) = W_2(W_1x+b_1)+b_2 = (W_2W_1)x+(W_2b_1+b_2)",[33,7838,7840,7947,8048,8106,8161,8265,8363],{"className":7839,"ariaHidden":64},[63],[33,7841,7843,7846,7886,7889,7929,7932,7935,7938,7941,7944],{"className":7842},[68],[33,7844],{"className":7845,"style":222},[72],[33,7847,7849,7852],{"className":7848},[77],[33,7850,7101],{"className":7851,"style":7146},[77,78],[33,7853,7855],{"className":7854},[809],[33,7856,7858,7878],{"className":7857},[813,2522],[33,7859,7861,7875],{"className":7860},[817],[33,7862,7864],{"className":7863,"style":2529},[821],[33,7865,7866,7869],{"style":7161},[33,7867],{"className":7868,"style":829},[828],[33,7870,7872],{"className":7871},[833,834,835,836],[33,7873,423],{"className":7874},[77,836],[33,7876,2546],{"className":7877},[2545],[33,7879,7881],{"className":7880},[817],[33,7882,7884],{"className":7883,"style":2553},[821],[33,7885],{},[33,7887,178],{"className":7888},[229],[33,7890,7892,7895],{"className":7891},[77],[33,7893,7101],{"className":7894,"style":7146},[77,78],[33,7896,7898],{"className":7897},[809],[33,7899,7901,7921],{"className":7900},[813,2522],[33,7902,7904,7918],{"className":7903},[817],[33,7905,7907],{"className":7906,"style":2529},[821],[33,7908,7909,7912],{"style":7161},[33,7910],{"className":7911,"style":829},[828],[33,7913,7915],{"className":7914},[833,834,835,836],[33,7916,2212],{"className":7917},[77,836],[33,7919,2546],{"className":7920},[2545],[33,7922,7924],{"className":7923},[817],[33,7925,7927],{"className":7926,"style":2553},[821],[33,7928],{},[33,7930,178],{"className":7931},[229],[33,7933,95],{"className":7934},[77,78],[33,7936,5490],{"className":7937},[257],[33,7939],{"className":7940,"style":261},[236],[33,7942,191],{"className":7943},[265],[33,7945],{"className":7946,"style":261},[236],[33,7948,7950,7953,7993,7996,8036,8039,8042,8045],{"className":7949},[68],[33,7951],{"className":7952,"style":222},[72],[33,7954,7956,7959],{"className":7955},[77],[33,7957,6198],{"className":7958,"style":6247},[77,78],[33,7960,7962],{"className":7961},[809],[33,7963,7965,7985],{"className":7964},[813,2522],[33,7966,7968,7982],{"className":7967},[817],[33,7969,7971],{"className":7970,"style":2529},[821],[33,7972,7973,7976],{"style":7226},[33,7974],{"className":7975,"style":829},[828],[33,7977,7979],{"className":7978},[833,834,835,836],[33,7980,423],{"className":7981},[77,836],[33,7983,2546],{"className":7984},[2545],[33,7986,7988],{"className":7987},[817],[33,7989,7991],{"className":7990,"style":2553},[821],[33,7992],{},[33,7994,178],{"className":7995},[229],[33,7997,7999,8002],{"className":7998},[77],[33,8000,6198],{"className":8001,"style":6247},[77,78],[33,8003,8005],{"className":8004},[809],[33,8006,8008,8028],{"className":8007},[813,2522],[33,8009,8011,8025],{"className":8010},[817],[33,8012,8014],{"className":8013,"style":2529},[821],[33,8015,8016,8019],{"style":7226},[33,8017],{"className":8018,"style":829},[828],[33,8020,8022],{"className":8021},[833,834,835,836],[33,8023,2212],{"className":8024},[77,836],[33,8026,2546],{"className":8027},[2545],[33,8029,8031],{"className":8030},[817],[33,8032,8034],{"className":8033,"style":2553},[821],[33,8035],{},[33,8037,95],{"className":8038},[77,78],[33,8040],{"className":8041,"style":237},[236],[33,8043,183],{"className":8044},[241],[33,8046],{"className":8047,"style":237},[236],[33,8049,8051,8054,8094,8097,8100,8103],{"className":8050},[68],[33,8052],{"className":8053,"style":222},[72],[33,8055,8057,8060],{"className":8056},[77],[33,8058,2171],{"className":8059},[77,78],[33,8061,8063],{"className":8062},[809],[33,8064,8066,8086],{"className":8065},[813,2522],[33,8067,8069,8083],{"className":8068},[817],[33,8070,8072],{"className":8071,"style":2529},[821],[33,8073,8074,8077],{"style":2532},[33,8075],{"className":8076,"style":829},[828],[33,8078,8080],{"className":8079},[833,834,835,836],[33,8081,2212],{"className":8082},[77,836],[33,8084,2546],{"className":8085},[2545],[33,8087,8089],{"className":8088},[817],[33,8090,8092],{"className":8091,"style":2553},[821],[33,8093],{},[33,8095,188],{"className":8096},[257],[33,8098],{"className":8099,"style":237},[236],[33,8101,183],{"className":8102},[241],[33,8104],{"className":8105,"style":237},[236],[33,8107,8109,8112,8152,8155,8158],{"className":8108},[68],[33,8110],{"className":8111,"style":3735},[72],[33,8113,8115,8118],{"className":8114},[77],[33,8116,2171],{"className":8117},[77,78],[33,8119,8121],{"className":8120},[809],[33,8122,8124,8144],{"className":8123},[813,2522],[33,8125,8127,8141],{"className":8126},[817],[33,8128,8130],{"className":8129,"style":2529},[821],[33,8131,8132,8135],{"style":2532},[33,8133],{"className":8134,"style":829},[828],[33,8136,8138],{"className":8137},[833,834,835,836],[33,8139,423],{"className":8140},[77,836],[33,8142,2546],{"className":8143},[2545],[33,8145,8147],{"className":8146},[817],[33,8148,8150],{"className":8149,"style":2553},[821],[33,8151],{},[33,8153],{"className":8154,"style":261},[236],[33,8156,191],{"className":8157},[265],[33,8159],{"className":8160,"style":261},[236],[33,8162,8164,8167,8170,8210,8250,8253,8256,8259,8262],{"className":8163},[68],[33,8165],{"className":8166,"style":222},[72],[33,8168,178],{"className":8169},[229],[33,8171,8173,8176],{"className":8172},[77],[33,8174,6198],{"className":8175,"style":6247},[77,78],[33,8177,8179],{"className":8178},[809],[33,8180,8182,8202],{"className":8181},[813,2522],[33,8183,8185,8199],{"className":8184},[817],[33,8186,8188],{"className":8187,"style":2529},[821],[33,8189,8190,8193],{"style":7226},[33,8191],{"className":8192,"style":829},[828],[33,8194,8196],{"className":8195},[833,834,835,836],[33,8197,423],{"className":8198},[77,836],[33,8200,2546],{"className":8201},[2545],[33,8203,8205],{"className":8204},[817],[33,8206,8208],{"className":8207,"style":2553},[821],[33,8209],{},[33,8211,8213,8216],{"className":8212},[77],[33,8214,6198],{"className":8215,"style":6247},[77,78],[33,8217,8219],{"className":8218},[809],[33,8220,8222,8242],{"className":8221},[813,2522],[33,8223,8225,8239],{"className":8224},[817],[33,8226,8228],{"className":8227,"style":2529},[821],[33,8229,8230,8233],{"style":7226},[33,8231],{"className":8232,"style":829},[828],[33,8234,8236],{"className":8235},[833,834,835,836],[33,8237,2212],{"className":8238},[77,836],[33,8240,2546],{"className":8241},[2545],[33,8243,8245],{"className":8244},[817],[33,8246,8248],{"className":8247,"style":2553},[821],[33,8249],{},[33,8251,188],{"className":8252},[257],[33,8254,95],{"className":8255},[77,78],[33,8257],{"className":8258,"style":237},[236],[33,8260,183],{"className":8261},[241],[33,8263],{"className":8264,"style":237},[236],[33,8266,8268,8271,8274,8314,8354,8357,8360],{"className":8267},[68],[33,8269],{"className":8270,"style":222},[72],[33,8272,178],{"className":8273},[229],[33,8275,8277,8280],{"className":8276},[77],[33,8278,6198],{"className":8279,"style":6247},[77,78],[33,8281,8283],{"className":8282},[809],[33,8284,8286,8306],{"className":8285},[813,2522],[33,8287,8289,8303],{"className":8288},[817],[33,8290,8292],{"className":8291,"style":2529},[821],[33,8293,8294,8297],{"style":7226},[33,8295],{"className":8296,"style":829},[828],[33,8298,8300],{"className":8299},[833,834,835,836],[33,8301,423],{"className":8302},[77,836],[33,8304,2546],{"className":8305},[2545],[33,8307,8309],{"className":8308},[817],[33,8310,8312],{"className":8311,"style":2553},[821],[33,8313],{},[33,8315,8317,8320],{"className":8316},[77],[33,8318,2171],{"className":8319},[77,78],[33,8321,8323],{"className":8322},[809],[33,8324,8326,8346],{"className":8325},[813,2522],[33,8327,8329,8343],{"className":8328},[817],[33,8330,8332],{"className":8331,"style":2529},[821],[33,8333,8334,8337],{"style":2532},[33,8335],{"className":8336,"style":829},[828],[33,8338,8340],{"className":8339},[833,834,835,836],[33,8341,2212],{"className":8342},[77,836],[33,8344,2546],{"className":8345},[2545],[33,8347,8349],{"className":8348},[817],[33,8350,8352],{"className":8351,"style":2553},[821],[33,8353],{},[33,8355],{"className":8356,"style":237},[236],[33,8358,183],{"className":8359},[241],[33,8361],{"className":8362,"style":237},[236],[33,8364,8366,8369,8409],{"className":8365},[68],[33,8367],{"className":8368,"style":222},[72],[33,8370,8372,8375],{"className":8371},[77],[33,8373,2171],{"className":8374},[77,78],[33,8376,8378],{"className":8377},[809],[33,8379,8381,8401],{"className":8380},[813,2522],[33,8382,8384,8398],{"className":8383},[817],[33,8385,8387],{"className":8386,"style":2529},[821],[33,8388,8389,8392],{"style":2532},[33,8390],{"className":8391,"style":829},[828],[33,8393,8395],{"className":8394},[833,834,835,836],[33,8396,423],{"className":8397},[77,836],[33,8399,2546],{"className":8400},[2545],[33,8402,8404],{"className":8403},[817],[33,8405,8407],{"className":8406,"style":2553},[821],[33,8408],{},[33,8410,188],{"className":8411},[257],[15,8413,8414,8415,143],{},"which remains in the form ",[33,8416,8418,8437],{"className":8417,"translate":37},[36],[33,8419,8421],{"className":8420},[41],[43,8422,8423],{"xmlns":45},[47,8424,8425,8435],{},[50,8426,8427,8429,8431,8433],{},[53,8428,3144],{},[53,8430,95],{},[175,8432,183],{},[53,8434,2171],{},[57,8436,5144],{"encoding":59},[33,8438,8440,8461],{"className":8439,"ariaHidden":64},[63],[33,8441,8443,8446,8449,8452,8455,8458],{"className":8442},[68],[33,8444],{"className":8445,"style":3190},[72],[33,8447,3144],{"className":8448},[77,78],[33,8450,95],{"className":8451},[77,78],[33,8453],{"className":8454,"style":237},[236],[33,8456,183],{"className":8457},[241],[33,8459],{"className":8460,"style":237},[236],[33,8462,8464,8467],{"className":8463},[68],[33,8465],{"className":8466,"style":2183},[72],[33,8468,2171],{"className":8469},[77,78],[15,8471,8472],{},"By induction:",[33,8474,8476],{"className":8475,"translate":37},[158],[33,8477,8479,8519],{"className":8478,"translate":37},[36],[33,8480,8482],{"className":8481},[41],[43,8483,8484],{"xmlns":45,"display":167},[47,8485,8486,8516],{},[50,8487,8488,8494,8496,8498,8500,8506,8508,8510,8512,8514],{},[2467,8489,8490,8492],{},[53,8491,7101],{},[53,8493,2999],{},[175,8495,6688],{},[175,8497,6705],{},[175,8499,6688],{},[2467,8501,8502,8504],{},[53,8503,7101],{},[346,8505,2286],{},[175,8507,191],{},[53,8509,6198],{},[53,8511,95],{},[175,8513,183],{},[53,8515,2171],{},[57,8517,8518],{"encoding":59},"l_n \\circ \\cdots \\circ l_0 = Wx + b",[33,8520,8522,8577,8595,8650,8671],{"className":8521,"ariaHidden":64},[63],[33,8523,8525,8528,8568,8571,8574],{"className":8524},[68],[33,8526],{"className":8527,"style":3735},[72],[33,8529,8531,8534],{"className":8530},[77],[33,8532,7101],{"className":8533,"style":7146},[77,78],[33,8535,8537],{"className":8536},[809],[33,8538,8540,8560],{"className":8539},[813,2522],[33,8541,8543,8557],{"className":8542},[817],[33,8544,8546],{"className":8545,"style":5869},[821],[33,8547,8548,8551],{"style":7161},[33,8549],{"className":8550,"style":829},[828],[33,8552,8554],{"className":8553},[833,834,835,836],[33,8555,2999],{"className":8556},[77,78,836],[33,8558,2546],{"className":8559},[2545],[33,8561,8563],{"className":8562},[817],[33,8564,8566],{"className":8565,"style":2553},[821],[33,8567],{},[33,8569],{"className":8570,"style":237},[236],[33,8572,6688],{"className":8573},[241],[33,8575],{"className":8576,"style":237},[236],[33,8578,8580,8583,8586,8589,8592],{"className":8579},[68],[33,8581],{"className":8582,"style":1262},[72],[33,8584,6705],{"className":8585},[4176],[33,8587],{"className":8588,"style":237},[236],[33,8590,6688],{"className":8591},[241],[33,8593],{"className":8594,"style":237},[236],[33,8596,8598,8601,8641,8644,8647],{"className":8597},[68],[33,8599],{"className":8600,"style":3735},[72],[33,8602,8604,8607],{"className":8603},[77],[33,8605,7101],{"className":8606,"style":7146},[77,78],[33,8608,8610],{"className":8609},[809],[33,8611,8613,8633],{"className":8612},[813,2522],[33,8614,8616,8630],{"className":8615},[817],[33,8617,8619],{"className":8618,"style":2529},[821],[33,8620,8621,8624],{"style":7161},[33,8622],{"className":8623,"style":829},[828],[33,8625,8627],{"className":8626},[833,834,835,836],[33,8628,2286],{"className":8629},[77,836],[33,8631,2546],{"className":8632},[2545],[33,8634,8636],{"className":8635},[817],[33,8637,8639],{"className":8638,"style":2553},[821],[33,8640],{},[33,8642],{"className":8643,"style":261},[236],[33,8645,191],{"className":8646},[265],[33,8648],{"className":8649,"style":261},[236],[33,8651,8653,8656,8659,8662,8665,8668],{"className":8652},[68],[33,8654],{"className":8655,"style":3190},[72],[33,8657,6198],{"className":8658,"style":6247},[77,78],[33,8660,95],{"className":8661},[77,78],[33,8663],{"className":8664,"style":237},[236],[33,8666,183],{"className":8667},[241],[33,8669],{"className":8670,"style":237},[236],[33,8672,8674,8677],{"className":8673},[68],[33,8675],{"className":8676,"style":2183},[72],[33,8678,2171],{"className":8679},[77,78],[15,8681,8682],{},"Therefore, network depth loses its meaning entirely.",[15,8684,8685],{},"To introduce nonlinearity, we can add activation functions to each layer:",[33,8687,8689],{"className":8688,"translate":37},[158],[33,8690,8692,8795],{"className":8691,"translate":37},[36],[33,8693,8695],{"className":8694},[41],[43,8696,8697],{"xmlns":45,"display":167},[47,8698,8699,8792],{},[50,8700,8701,8703,8705,8707,8709,8711,8713,8720,8722,8728,8730,8732,8734,8746,8748,8760,8762,8764,8766,8768,8770,8776,8778,8784,8786,8788,8790],{},[53,8702,55],{},[175,8704,178],{"stretchy":177},[53,8706,95],{},[175,8708,188],{"stretchy":177},[175,8710,191],{},[175,8712,178],{"stretchy":177},[2467,8714,8715,8718],{},[53,8716,8717],{},"σ",[53,8719,2999],{},[175,8721,6688],{},[2467,8723,8724,8726],{},[53,8725,7101],{},[53,8727,2999],{},[175,8729,188],{"stretchy":177},[175,8731,6688],{},[175,8733,178],{"stretchy":177},[2467,8735,8736,8738],{},[53,8737,8717],{},[50,8739,8740,8742,8744],{},[53,8741,2999],{},[175,8743,6462],{},[346,8745,2212],{},[175,8747,6688],{},[2467,8749,8750,8752],{},[53,8751,7101],{},[50,8753,8754,8756,8758],{},[53,8755,2999],{},[175,8757,6462],{},[346,8759,2212],{},[175,8761,188],{"stretchy":177},[175,8763,6688],{},[175,8765,6705],{},[175,8767,6688],{},[175,8769,178],{"stretchy":177},[2467,8771,8772,8774],{},[53,8773,8717],{},[346,8775,2286],{},[175,8777,6688],{},[2467,8779,8780,8782],{},[53,8781,7101],{},[346,8783,2286],{},[175,8785,188],{"stretchy":177},[175,8787,178],{"stretchy":177},[53,8789,95],{},[175,8791,188],{"stretchy":177},[57,8793,8794],{"encoding":59},"f(x) = (\\sigma_n \\circ l_n) \\circ (\\sigma_{n-1} \\circ l_{n-1}) \\circ \\cdots \\circ (\\sigma_0 \\circ l_0)(x)",[33,8796,8798,8825,8883,8941,9008,9075,9093,9151],{"className":8797,"ariaHidden":64},[63],[33,8799,8801,8804,8807,8810,8813,8816,8819,8822],{"className":8800},[68],[33,8802],{"className":8803,"style":222},[72],[33,8805,55],{"className":8806,"style":79},[77,78],[33,8808,178],{"className":8809},[229],[33,8811,95],{"className":8812},[77,78],[33,8814,188],{"className":8815},[257],[33,8817],{"className":8818,"style":261},[236],[33,8820,191],{"className":8821},[265],[33,8823],{"className":8824,"style":261},[236],[33,8826,8828,8831,8834,8874,8877,8880],{"className":8827},[68],[33,8829],{"className":8830,"style":222},[72],[33,8832,178],{"className":8833},[229],[33,8835,8837,8840],{"className":8836},[77],[33,8838,8717],{"className":8839,"style":142},[77,78],[33,8841,8843],{"className":8842},[809],[33,8844,8846,8866],{"className":8845},[813,2522],[33,8847,8849,8863],{"className":8848},[817],[33,8850,8852],{"className":8851,"style":5869},[821],[33,8853,8854,8857],{"style":2685},[33,8855],{"className":8856,"style":829},[828],[33,8858,8860],{"className":8859},[833,834,835,836],[33,8861,2999],{"className":8862},[77,78,836],[33,8864,2546],{"className":8865},[2545],[33,8867,8869],{"className":8868},[817],[33,8870,8872],{"className":8871,"style":2553},[821],[33,8873],{},[33,8875],{"className":8876,"style":237},[236],[33,8878,6688],{"className":8879},[241],[33,8881],{"className":8882,"style":237},[236],[33,8884,8886,8889,8929,8932,8935,8938],{"className":8885},[68],[33,8887],{"className":8888,"style":222},[72],[33,8890,8892,8895],{"className":8891},[77],[33,8893,7101],{"className":8894,"style":7146},[77,78],[33,8896,8898],{"className":8897},[809],[33,8899,8901,8921],{"className":8900},[813,2522],[33,8902,8904,8918],{"className":8903},[817],[33,8905,8907],{"className":8906,"style":5869},[821],[33,8908,8909,8912],{"style":7161},[33,8910],{"className":8911,"style":829},[828],[33,8913,8915],{"className":8914},[833,834,835,836],[33,8916,2999],{"className":8917},[77,78,836],[33,8919,2546],{"className":8920},[2545],[33,8922,8924],{"className":8923},[817],[33,8925,8927],{"className":8926,"style":2553},[821],[33,8928],{},[33,8930,188],{"className":8931},[257],[33,8933],{"className":8934,"style":237},[236],[33,8936,6688],{"className":8937},[241],[33,8939],{"className":8940,"style":237},[236],[33,8942,8944,8947,8950,8999,9002,9005],{"className":8943},[68],[33,8945],{"className":8946,"style":222},[72],[33,8948,178],{"className":8949},[229],[33,8951,8953,8956],{"className":8952},[77],[33,8954,8717],{"className":8955,"style":142},[77,78],[33,8957,8959],{"className":8958},[809],[33,8960,8962,8991],{"className":8961},[813,2522],[33,8963,8965,8988],{"className":8964},[817],[33,8966,8968],{"className":8967,"style":2529},[821],[33,8969,8970,8973],{"style":2685},[33,8971],{"className":8972,"style":829},[828],[33,8974,8976],{"className":8975},[833,834,835,836],[33,8977,8979,8982,8985],{"className":8978},[77,836],[33,8980,2999],{"className":8981},[77,78,836],[33,8983,6462],{"className":8984},[241,836],[33,8986,2212],{"className":8987},[77,836],[33,8989,2546],{"className":8990},[2545],[33,8992,8994],{"className":8993},[817],[33,8995,8997],{"className":8996,"style":6829},[821],[33,8998],{},[33,9000],{"className":9001,"style":237},[236],[33,9003,6688],{"className":9004},[241],[33,9006],{"className":9007,"style":237},[236],[33,9009,9011,9014,9063,9066,9069,9072],{"className":9010},[68],[33,9012],{"className":9013,"style":222},[72],[33,9015,9017,9020],{"className":9016},[77],[33,9018,7101],{"className":9019,"style":7146},[77,78],[33,9021,9023],{"className":9022},[809],[33,9024,9026,9055],{"className":9025},[813,2522],[33,9027,9029,9052],{"className":9028},[817],[33,9030,9032],{"className":9031,"style":2529},[821],[33,9033,9034,9037],{"style":7161},[33,9035],{"className":9036,"style":829},[828],[33,9038,9040],{"className":9039},[833,834,835,836],[33,9041,9043,9046,9049],{"className":9042},[77,836],[33,9044,2999],{"className":9045},[77,78,836],[33,9047,6462],{"className":9048},[241,836],[33,9050,2212],{"className":9051},[77,836],[33,9053,2546],{"className":9054},[2545],[33,9056,9058],{"className":9057},[817],[33,9059,9061],{"className":9060,"style":6829},[821],[33,9062],{},[33,9064,188],{"className":9065},[257],[33,9067],{"className":9068,"style":237},[236],[33,9070,6688],{"className":9071},[241],[33,9073],{"className":9074,"style":237},[236],[33,9076,9078,9081,9084,9087,9090],{"className":9077},[68],[33,9079],{"className":9080,"style":1262},[72],[33,9082,6705],{"className":9083},[4176],[33,9085],{"className":9086,"style":237},[236],[33,9088,6688],{"className":9089},[241],[33,9091],{"className":9092,"style":237},[236],[33,9094,9096,9099,9102,9142,9145,9148],{"className":9095},[68],[33,9097],{"className":9098,"style":222},[72],[33,9100,178],{"className":9101},[229],[33,9103,9105,9108],{"className":9104},[77],[33,9106,8717],{"className":9107,"style":142},[77,78],[33,9109,9111],{"className":9110},[809],[33,9112,9114,9134],{"className":9113},[813,2522],[33,9115,9117,9131],{"className":9116},[817],[33,9118,9120],{"className":9119,"style":2529},[821],[33,9121,9122,9125],{"style":2685},[33,9123],{"className":9124,"style":829},[828],[33,9126,9128],{"className":9127},[833,834,835,836],[33,9129,2286],{"className":9130},[77,836],[33,9132,2546],{"className":9133},[2545],[33,9135,9137],{"className":9136},[817],[33,9138,9140],{"className":9139,"style":2553},[821],[33,9141],{},[33,9143],{"className":9144,"style":237},[236],[33,9146,6688],{"className":9147},[241],[33,9149],{"className":9150,"style":237},[236],[33,9152,9154,9157,9197,9200,9203,9206],{"className":9153},[68],[33,9155],{"className":9156,"style":222},[72],[33,9158,9160,9163],{"className":9159},[77],[33,9161,7101],{"className":9162,"style":7146},[77,78],[33,9164,9166],{"className":9165},[809],[33,9167,9169,9189],{"className":9168},[813,2522],[33,9170,9172,9186],{"className":9171},[817],[33,9173,9175],{"className":9174,"style":2529},[821],[33,9176,9177,9180],{"style":7161},[33,9178],{"className":9179,"style":829},[828],[33,9181,9183],{"className":9182},[833,834,835,836],[33,9184,2286],{"className":9185},[77,836],[33,9187,2546],{"className":9188},[2545],[33,9190,9192],{"className":9191},[817],[33,9193,9195],{"className":9194,"style":2553},[821],[33,9196],{},[33,9198,188],{"className":9199},[257],[33,9201,178],{"className":9202},[229],[33,9204,95],{"className":9205},[77,78],[33,9207,188],{"className":9208},[257],[15,9210,9211],{},"where",[33,9213,9215],{"className":9214,"translate":37},[158],[33,9216,9218,9260],{"className":9217,"translate":37},[36],[33,9219,9221],{"className":9220},[41],[43,9222,9223],{"xmlns":45,"display":167},[47,9224,9225,9257],{},[50,9226,9227,9233,9235,9237,9239,9241,9247,9249,9251],{},[2467,9228,9229,9231],{},[53,9230,7101],{},[53,9232,3316],{},[175,9234,178],{"stretchy":177},[53,9236,95],{},[175,9238,188],{"stretchy":177},[175,9240,191],{},[2467,9242,9243,9245],{},[53,9244,6198],{},[53,9246,3316],{},[53,9248,95],{},[175,9250,183],{},[2467,9252,9253,9255],{},[53,9254,2171],{},[53,9256,3316],{},[57,9258,9259],{"encoding":59},"l_i(x)=W_i x+b_i",[33,9261,9263,9327,9385],{"className":9262,"ariaHidden":64},[63],[33,9264,9266,9269,9309,9312,9315,9318,9321,9324],{"className":9265},[68],[33,9267],{"className":9268,"style":222},[72],[33,9270,9272,9275],{"className":9271},[77],[33,9273,7101],{"className":9274,"style":7146},[77,78],[33,9276,9278],{"className":9277},[809],[33,9279,9281,9301],{"className":9280},[813,2522],[33,9282,9284,9298],{"className":9283},[817],[33,9285,9287],{"className":9286,"style":3481},[821],[33,9288,9289,9292],{"style":7161},[33,9290],{"className":9291,"style":829},[828],[33,9293,9295],{"className":9294},[833,834,835,836],[33,9296,3316],{"className":9297},[77,78,836],[33,9299,2546],{"className":9300},[2545],[33,9302,9304],{"className":9303},[817],[33,9305,9307],{"className":9306,"style":2553},[821],[33,9308],{},[33,9310,178],{"className":9311},[229],[33,9313,95],{"className":9314},[77,78],[33,9316,188],{"className":9317},[257],[33,9319],{"className":9320,"style":261},[236],[33,9322,191],{"className":9323},[265],[33,9325],{"className":9326,"style":261},[236],[33,9328,9330,9333,9373,9376,9379,9382],{"className":9329},[68],[33,9331],{"className":9332,"style":3676},[72],[33,9334,9336,9339],{"className":9335},[77],[33,9337,6198],{"className":9338,"style":6247},[77,78],[33,9340,9342],{"className":9341},[809],[33,9343,9345,9365],{"className":9344},[813,2522],[33,9346,9348,9362],{"className":9347},[817],[33,9349,9351],{"className":9350,"style":3481},[821],[33,9352,9353,9356],{"style":7226},[33,9354],{"className":9355,"style":829},[828],[33,9357,9359],{"className":9358},[833,834,835,836],[33,9360,3316],{"className":9361},[77,78,836],[33,9363,2546],{"className":9364},[2545],[33,9366,9368],{"className":9367},[817],[33,9369,9371],{"className":9370,"style":2553},[821],[33,9372],{},[33,9374,95],{"className":9375},[77,78],[33,9377],{"className":9378,"style":237},[236],[33,9380,183],{"className":9381},[241],[33,9383],{"className":9384,"style":237},[236],[33,9386,9388,9391],{"className":9387},[68],[33,9389],{"className":9390,"style":3735},[72],[33,9392,9394,9397],{"className":9393},[77],[33,9395,2171],{"className":9396},[77,78],[33,9398,9400],{"className":9399},[809],[33,9401,9403,9423],{"className":9402},[813,2522],[33,9404,9406,9420],{"className":9405},[817],[33,9407,9409],{"className":9408,"style":3481},[821],[33,9410,9411,9414],{"style":2532},[33,9412],{"className":9413,"style":829},[828],[33,9415,9417],{"className":9416},[833,834,835,836],[33,9418,3316],{"className":9419},[77,78,836],[33,9421,2546],{"className":9422},[2545],[33,9424,9426],{"className":9425},[817],[33,9427,9429],{"className":9428,"style":2553},[821],[33,9430],{},[15,9432,9433],{},"Expanded, this is:",[33,9435,9437],{"className":9436,"translate":37},[158],[33,9438,9440,9560],{"className":9439,"translate":37},[36],[33,9441,9443],{"className":9442},[41],[43,9444,9445],{"xmlns":45,"display":167},[47,9446,9447,9557],{},[50,9448,9449,9451,9453,9455,9457,9459,9465,9468,9474,9486,9489,9501,9503,9509,9511,9517,9519,9521,9527,9529,9531,9533,9545,9547,9549,9555],{},[53,9450,55],{},[175,9452,178],{"stretchy":177},[53,9454,95],{},[175,9456,188],{"stretchy":177},[175,9458,191],{},[2467,9460,9461,9463],{},[53,9462,8717],{},[53,9464,2999],{},[175,9466,178],{"fence":64,"stretchy":64,"minsize":9467,"maxsize":9467},"1.8em",[2467,9469,9470,9472],{},[53,9471,6198],{},[53,9473,2999],{},[2467,9475,9476,9478],{},[53,9477,8717],{},[50,9479,9480,9482,9484],{},[53,9481,2999],{},[175,9483,6462],{},[346,9485,2212],{},[175,9487,178],{"fence":64,"stretchy":64,"minsize":9488,"maxsize":9488},"1.2em",[2467,9490,9491,9493],{},[53,9492,6198],{},[50,9494,9495,9497,9499],{},[53,9496,2999],{},[175,9498,6462],{},[346,9500,2212],{},[175,9502,6705],{},[2467,9504,9505,9507],{},[53,9506,8717],{},[346,9508,2286],{},[175,9510,178],{"stretchy":177},[2467,9512,9513,9515],{},[53,9514,6198],{},[346,9516,2286],{},[53,9518,95],{},[175,9520,183],{},[2467,9522,9523,9525],{},[53,9524,2171],{},[346,9526,2286],{},[175,9528,188],{"stretchy":177},[175,9530,6705],{},[175,9532,183],{},[2467,9534,9535,9537],{},[53,9536,2171],{},[50,9538,9539,9541,9543],{},[53,9540,2999],{},[175,9542,6462],{},[346,9544,2212],{},[175,9546,188],{"fence":64,"stretchy":64,"minsize":9488,"maxsize":9488},[175,9548,183],{},[2467,9550,9551,9553],{},[53,9552,2171],{},[53,9554,2999],{},[175,9556,188],{"fence":64,"stretchy":64,"minsize":9467,"maxsize":9467},[57,9558,9559],{"encoding":59},"f(x) = \\sigma_n \\Bigl( W_n \\sigma_{n-1} \\bigl( W_{n-1} \\cdots \\sigma_0(W_0x+b_0) \\cdots +b_{n-1} \\bigr) + b_n \\Bigr)",[33,9561,9563,9590,9893,9957,10028],{"className":9562,"ariaHidden":64},[63],[33,9564,9566,9569,9572,9575,9578,9581,9584,9587],{"className":9565},[68],[33,9567],{"className":9568,"style":222},[72],[33,9570,55],{"className":9571,"style":79},[77,78],[33,9573,178],{"className":9574},[229],[33,9576,95],{"className":9577},[77,78],[33,9579,188],{"className":9580},[257],[33,9582],{"className":9583,"style":261},[236],[33,9585,191],{"className":9586},[265],[33,9588],{"className":9589,"style":261},[236],[33,9591,9593,9597,9637,9644,9684,9733,9740,9789,9792,9795,9798,9838,9841,9881,9884,9887,9890],{"className":9592},[68],[33,9594],{"className":9595,"style":9596},[72],"height:1.8em;vertical-align:-0.65em;",[33,9598,9600,9603],{"className":9599},[77],[33,9601,8717],{"className":9602,"style":142},[77,78],[33,9604,9606],{"className":9605},[809],[33,9607,9609,9629],{"className":9608},[813,2522],[33,9610,9612,9626],{"className":9611},[817],[33,9613,9615],{"className":9614,"style":5869},[821],[33,9616,9617,9620],{"style":2685},[33,9618],{"className":9619,"style":829},[828],[33,9621,9623],{"className":9622},[833,834,835,836],[33,9624,2999],{"className":9625},[77,78,836],[33,9627,2546],{"className":9628},[2545],[33,9630,9632],{"className":9631},[817],[33,9633,9635],{"className":9634,"style":2553},[821],[33,9636],{},[33,9638,9640],{"className":9639},[229],[33,9641,178],{"className":9642},[4185,9643],"size2",[33,9645,9647,9650],{"className":9646},[77],[33,9648,6198],{"className":9649,"style":6247},[77,78],[33,9651,9653],{"className":9652},[809],[33,9654,9656,9676],{"className":9655},[813,2522],[33,9657,9659,9673],{"className":9658},[817],[33,9660,9662],{"className":9661,"style":5869},[821],[33,9663,9664,9667],{"style":7226},[33,9665],{"className":9666,"style":829},[828],[33,9668,9670],{"className":9669},[833,834,835,836],[33,9671,2999],{"className":9672},[77,78,836],[33,9674,2546],{"className":9675},[2545],[33,9677,9679],{"className":9678},[817],[33,9680,9682],{"className":9681,"style":2553},[821],[33,9683],{},[33,9685,9687,9690],{"className":9686},[77],[33,9688,8717],{"className":9689,"style":142},[77,78],[33,9691,9693],{"className":9692},[809],[33,9694,9696,9725],{"className":9695},[813,2522],[33,9697,9699,9722],{"className":9698},[817],[33,9700,9702],{"className":9701,"style":2529},[821],[33,9703,9704,9707],{"style":2685},[33,9705],{"className":9706,"style":829},[828],[33,9708,9710],{"className":9709},[833,834,835,836],[33,9711,9713,9716,9719],{"className":9712},[77,836],[33,9714,2999],{"className":9715},[77,78,836],[33,9717,6462],{"className":9718},[241,836],[33,9720,2212],{"className":9721},[77,836],[33,9723,2546],{"className":9724},[2545],[33,9726,9728],{"className":9727},[817],[33,9729,9731],{"className":9730,"style":6829},[821],[33,9732],{},[33,9734,9736],{"className":9735},[229],[33,9737,178],{"className":9738},[4185,9739],"size1",[33,9741,9743,9746],{"className":9742},[77],[33,9744,6198],{"className":9745,"style":6247},[77,78],[33,9747,9749],{"className":9748},[809],[33,9750,9752,9781],{"className":9751},[813,2522],[33,9753,9755,9778],{"className":9754},[817],[33,9756,9758],{"className":9757,"style":2529},[821],[33,9759,9760,9763],{"style":7226},[33,9761],{"className":9762,"style":829},[828],[33,9764,9766],{"className":9765},[833,834,835,836],[33,9767,9769,9772,9775],{"className":9768},[77,836],[33,9770,2999],{"className":9771},[77,78,836],[33,9773,6462],{"className":9774},[241,836],[33,9776,2212],{"className":9777},[77,836],[33,9779,2546],{"className":9780},[2545],[33,9782,9784],{"className":9783},[817],[33,9785,9787],{"className":9786,"style":6829},[821],[33,9788],{},[33,9790],{"className":9791,"style":3461},[236],[33,9793,6705],{"className":9794},[4176],[33,9796],{"className":9797,"style":3461},[236],[33,9799,9801,9804],{"className":9800},[77],[33,9802,8717],{"className":9803,"style":142},[77,78],[33,9805,9807],{"className":9806},[809],[33,9808,9810,9830],{"className":9809},[813,2522],[33,9811,9813,9827],{"className":9812},[817],[33,9814,9816],{"className":9815,"style":2529},[821],[33,9817,9818,9821],{"style":2685},[33,9819],{"className":9820,"style":829},[828],[33,9822,9824],{"className":9823},[833,834,835,836],[33,9825,2286],{"className":9826},[77,836],[33,9828,2546],{"className":9829},[2545],[33,9831,9833],{"className":9832},[817],[33,9834,9836],{"className":9835,"style":2553},[821],[33,9837],{},[33,9839,178],{"className":9840},[229],[33,9842,9844,9847],{"className":9843},[77],[33,9845,6198],{"className":9846,"style":6247},[77,78],[33,9848,9850],{"className":9849},[809],[33,9851,9853,9873],{"className":9852},[813,2522],[33,9854,9856,9870],{"className":9855},[817],[33,9857,9859],{"className":9858,"style":2529},[821],[33,9860,9861,9864],{"style":7226},[33,9862],{"className":9863,"style":829},[828],[33,9865,9867],{"className":9866},[833,834,835,836],[33,9868,2286],{"className":9869},[77,836],[33,9871,2546],{"className":9872},[2545],[33,9874,9876],{"className":9875},[817],[33,9877,9879],{"className":9878,"style":2553},[821],[33,9880],{},[33,9882,95],{"className":9883},[77,78],[33,9885],{"className":9886,"style":237},[236],[33,9888,183],{"className":9889},[241],[33,9891],{"className":9892,"style":237},[236],[33,9894,9896,9899,9939,9942,9945,9948,9951,9954],{"className":9895},[68],[33,9897],{"className":9898,"style":222},[72],[33,9900,9902,9905],{"className":9901},[77],[33,9903,2171],{"className":9904},[77,78],[33,9906,9908],{"className":9907},[809],[33,9909,9911,9931],{"className":9910},[813,2522],[33,9912,9914,9928],{"className":9913},[817],[33,9915,9917],{"className":9916,"style":2529},[821],[33,9918,9919,9922],{"style":2532},[33,9920],{"className":9921,"style":829},[828],[33,9923,9925],{"className":9924},[833,834,835,836],[33,9926,2286],{"className":9927},[77,836],[33,9929,2546],{"className":9930},[2545],[33,9932,9934],{"className":9933},[817],[33,9935,9937],{"className":9936,"style":2553},[821],[33,9938],{},[33,9940,188],{"className":9941},[257],[33,9943],{"className":9944,"style":3461},[236],[33,9946,6705],{"className":9947},[4176],[33,9949],{"className":9950,"style":237},[236],[33,9952,183],{"className":9953},[241],[33,9955],{"className":9956,"style":237},[236],[33,9958,9960,9964,10013,10019,10022,10025],{"className":9959},[68],[33,9961],{"className":9962,"style":9963},[72],"height:1.2em;vertical-align:-0.35em;",[33,9965,9967,9970],{"className":9966},[77],[33,9968,2171],{"className":9969},[77,78],[33,9971,9973],{"className":9972},[809],[33,9974,9976,10005],{"className":9975},[813,2522],[33,9977,9979,10002],{"className":9978},[817],[33,9980,9982],{"className":9981,"style":2529},[821],[33,9983,9984,9987],{"style":2532},[33,9985],{"className":9986,"style":829},[828],[33,9988,9990],{"className":9989},[833,834,835,836],[33,9991,9993,9996,9999],{"className":9992},[77,836],[33,9994,2999],{"className":9995},[77,78,836],[33,9997,6462],{"className":9998},[241,836],[33,10000,2212],{"className":10001},[77,836],[33,10003,2546],{"className":10004},[2545],[33,10006,10008],{"className":10007},[817],[33,10009,10011],{"className":10010,"style":6829},[821],[33,10012],{},[33,10014,10016],{"className":10015},[257],[33,10017,188],{"className":10018},[4185,9739],[33,10020],{"className":10021,"style":237},[236],[33,10023,183],{"className":10024},[241],[33,10026],{"className":10027,"style":237},[236],[33,10029,10031,10034,10074],{"className":10030},[68],[33,10032],{"className":10033,"style":9596},[72],[33,10035,10037,10040],{"className":10036},[77],[33,10038,2171],{"className":10039},[77,78],[33,10041,10043],{"className":10042},[809],[33,10044,10046,10066],{"className":10045},[813,2522],[33,10047,10049,10063],{"className":10048},[817],[33,10050,10052],{"className":10051,"style":5869},[821],[33,10053,10054,10057],{"style":2532},[33,10055],{"className":10056,"style":829},[828],[33,10058,10060],{"className":10059},[833,834,835,836],[33,10061,2999],{"className":10062},[77,78,836],[33,10064,2546],{"className":10065},[2545],[33,10067,10069],{"className":10068},[817],[33,10070,10072],{"className":10071,"style":2553},[821],[33,10073],{},[33,10075,10077],{"className":10076},[257],[33,10078,188],{"className":10079},[4185,9643],[15,10081,10082],{},"This is the standard form of a neural network.",[15,10084,10085],{},"With activation functions added:",[33,10087,10089],{"className":10088,"translate":37},[158],[33,10090,10092,10134],{"className":10091,"translate":37},[36],[33,10093,10095],{"className":10094},[41],[43,10096,10097],{"xmlns":45,"display":167},[47,10098,10099,10131],{},[50,10100,10101,10103,10105,10107,10113,10115,10117,10119,10121,10123,10129],{},[175,10102,178],{"stretchy":177},[53,10104,8717],{},[175,10106,6688],{},[2467,10108,10109,10111],{},[53,10110,7101],{},[346,10112,423],{},[175,10114,188],{"stretchy":177},[175,10116,6688],{},[175,10118,178],{"stretchy":177},[53,10120,8717],{},[175,10122,6688],{},[2467,10124,10125,10127],{},[53,10126,7101],{},[346,10128,2212],{},[175,10130,188],{"stretchy":177},[57,10132,10133],{"encoding":59},"(\\sigma \\circ l_2) \\circ (\\sigma \\circ l_1)",[33,10135,10137,10158,10216,10237],{"className":10136,"ariaHidden":64},[63],[33,10138,10140,10143,10146,10149,10152,10155],{"className":10139},[68],[33,10141],{"className":10142,"style":222},[72],[33,10144,178],{"className":10145},[229],[33,10147,8717],{"className":10148,"style":142},[77,78],[33,10150],{"className":10151,"style":237},[236],[33,10153,6688],{"className":10154},[241],[33,10156],{"className":10157,"style":237},[236],[33,10159,10161,10164,10204,10207,10210,10213],{"className":10160},[68],[33,10162],{"className":10163,"style":222},[72],[33,10165,10167,10170],{"className":10166},[77],[33,10168,7101],{"className":10169,"style":7146},[77,78],[33,10171,10173],{"className":10172},[809],[33,10174,10176,10196],{"className":10175},[813,2522],[33,10177,10179,10193],{"className":10178},[817],[33,10180,10182],{"className":10181,"style":2529},[821],[33,10183,10184,10187],{"style":7161},[33,10185],{"className":10186,"style":829},[828],[33,10188,10190],{"className":10189},[833,834,835,836],[33,10191,423],{"className":10192},[77,836],[33,10194,2546],{"className":10195},[2545],[33,10197,10199],{"className":10198},[817],[33,10200,10202],{"className":10201,"style":2553},[821],[33,10203],{},[33,10205,188],{"className":10206},[257],[33,10208],{"className":10209,"style":237},[236],[33,10211,6688],{"className":10212},[241],[33,10214],{"className":10215,"style":237},[236],[33,10217,10219,10222,10225,10228,10231,10234],{"className":10218},[68],[33,10220],{"className":10221,"style":222},[72],[33,10223,178],{"className":10224},[229],[33,10226,8717],{"className":10227,"style":142},[77,78],[33,10229],{"className":10230,"style":237},[236],[33,10232,6688],{"className":10233},[241],[33,10235],{"className":10236,"style":237},[236],[33,10238,10240,10243,10283],{"className":10239},[68],[33,10241],{"className":10242,"style":222},[72],[33,10244,10246,10249],{"className":10245},[77],[33,10247,7101],{"className":10248,"style":7146},[77,78],[33,10250,10252],{"className":10251},[809],[33,10253,10255,10275],{"className":10254},[813,2522],[33,10256,10258,10272],{"className":10257},[817],[33,10259,10261],{"className":10260,"style":2529},[821],[33,10262,10263,10266],{"style":7161},[33,10264],{"className":10265,"style":829},[828],[33,10267,10269],{"className":10268},[833,834,835,836],[33,10270,2212],{"className":10271},[77,836],[33,10273,2546],{"className":10274},[2545],[33,10276,10278],{"className":10277},[817],[33,10279,10281],{"className":10280,"style":2553},[821],[33,10282],{},[33,10284,188],{"className":10285},[257],[15,10287,10288],{},"cannot be simplified to",[33,10290,10292],{"className":10291,"translate":37},[158],[33,10293,10295,10315],{"className":10294,"translate":37},[36],[33,10296,10298],{"className":10297},[41],[43,10299,10300],{"xmlns":45,"display":167},[47,10301,10302,10312],{},[50,10303,10304,10306,10308,10310],{},[53,10305,6198],{},[53,10307,95],{},[175,10309,183],{},[53,10311,2171],{},[57,10313,10314],{"encoding":59},"Wx+b",[33,10316,10318,10339],{"className":10317,"ariaHidden":64},[63],[33,10319,10321,10324,10327,10330,10333,10336],{"className":10320},[68],[33,10322],{"className":10323,"style":3190},[72],[33,10325,6198],{"className":10326,"style":6247},[77,78],[33,10328,95],{"className":10329},[77,78],[33,10331],{"className":10332,"style":237},[236],[33,10334,183],{"className":10335},[241],[33,10337],{"className":10338,"style":237},[236],[33,10340,10342,10345],{"className":10341},[68],[33,10343],{"className":10344,"style":2183},[72],[33,10346,2171],{"className":10347},[77,78],[15,10349,10350],{},"because generally:",[33,10352,10354],{"className":10353,"translate":37},[158],[33,10355,10357,10409],{"className":10356,"translate":37},[36],[33,10358,10360],{"className":10359},[41],[43,10361,10362],{"xmlns":45,"display":167},[47,10363,10364,10406],{},[50,10365,10366,10368,10370,10372,10374,10376,10378,10380,10382,10390,10392,10394,10396,10398,10400],{},[53,10367,8717],{},[175,10369,178],{"stretchy":177},[53,10371,3144],{},[53,10373,95],{},[175,10375,183],{},[53,10377,2171],{},[175,10379,188],{"stretchy":177},[175,10381,2296],{"mathvariant":2295},[754,10383,10384,10386],{},[53,10385,3144],{},[175,10387,10389],{"mathvariant":2295,"lspace":10388,"rspace":10388},"0em","′",[53,10391,8717],{},[175,10393,178],{"stretchy":177},[53,10395,95],{},[175,10397,188],{"stretchy":177},[175,10399,183],{},[754,10401,10402,10404],{},[53,10403,2171],{},[175,10405,10389],{"mathvariant":2295,"lspace":10388,"rspace":10388},[57,10407,10408],{"encoding":59},"\\sigma(Ax+b) \\neq A'\\sigma(x)+b'",[33,10410,10412,10439,10493,10554],{"className":10411,"ariaHidden":64},[63],[33,10413,10415,10418,10421,10424,10427,10430,10433,10436],{"className":10414},[68],[33,10416],{"className":10417,"style":222},[72],[33,10419,8717],{"className":10420,"style":142},[77,78],[33,10422,178],{"className":10423},[229],[33,10425,3144],{"className":10426},[77,78],[33,10428,95],{"className":10429},[77,78],[33,10431],{"className":10432,"style":237},[236],[33,10434,183],{"className":10435},[241],[33,10437],{"className":10438,"style":237},[236],[33,10440,10442,10445,10448,10451,10454,10490],{"className":10441},[68],[33,10443],{"className":10444,"style":222},[72],[33,10446,2171],{"className":10447},[77,78],[33,10449,188],{"className":10450},[257],[33,10452],{"className":10453,"style":261},[236],[33,10455,10457,10484,10487],{"className":10456},[265],[33,10458,10460],{"className":10459},[265],[33,10461,10463],{"className":10462},[77,2353],[33,10464,10466],{"className":10465},[2357],[33,10467,10469,10472,10481],{"className":10468},[2361],[33,10470],{"className":10471,"style":73},[72],[33,10473,10475],{"className":10474},[2368],[33,10476,10478],{"className":10477},[77],[33,10479,2375],{"className":10480},[265],[33,10482],{"className":10483},[2379],[33,10485],{"className":10486},[236,2383],[33,10488,191],{"className":10489},[265],[33,10491],{"className":10492,"style":261},[236],[33,10494,10496,10500,10533,10536,10539,10542,10545,10548,10551],{"className":10495},[68],[33,10497],{"className":10498,"style":10499},[72],"height:1.0519em;vertical-align:-0.25em;",[33,10501,10503,10506],{"className":10502},[77],[33,10504,3144],{"className":10505},[77,78],[33,10507,10509],{"className":10508},[809],[33,10510,10512],{"className":10511},[813],[33,10513,10515],{"className":10514},[817],[33,10516,10519],{"className":10517,"style":10518},[821],"height:0.8019em;",[33,10520,10521,10524],{"style":4709},[33,10522],{"className":10523,"style":829},[828],[33,10525,10527],{"className":10526},[833,834,835,836],[33,10528,10530],{"className":10529},[77,836],[33,10531,10389],{"className":10532},[77,836],[33,10534,8717],{"className":10535,"style":142},[77,78],[33,10537,178],{"className":10538},[229],[33,10540,95],{"className":10541},[77,78],[33,10543,188],{"className":10544},[257],[33,10546],{"className":10547,"style":237},[236],[33,10549,183],{"className":10550},[241],[33,10552],{"className":10553,"style":237},[236],[33,10555,10557,10560],{"className":10556},[68],[33,10558],{"className":10559,"style":10518},[72],[33,10561,10563,10566],{"className":10562},[77],[33,10564,2171],{"className":10565},[77,78],[33,10567,10569],{"className":10568},[809],[33,10570,10572],{"className":10571},[813],[33,10573,10575],{"className":10574},[817],[33,10576,10578],{"className":10577,"style":10518},[821],[33,10579,10580,10583],{"style":4709},[33,10581],{"className":10582,"style":829},[828],[33,10584,10586],{"className":10585},[833,834,835,836],[33,10587,10589],{"className":10588},[77,836],[33,10590,10389],{"className":10591},[77,836],[15,10593,10594],{},"The activation function breaks the closure property of affine mappings under composition. In other words, it breaks the two key characteristics of additivity and homogeneity. This is essentially the source of the nonlinear expressive power of neural networks.",[15,10596,10597],{},"From the manifold perspective discussed earlier, this can be further written as:",[33,10599,10601],{"className":10600,"translate":37},[158],[33,10602,10604,10703],{"className":10603,"translate":37},[36],[33,10605,10607],{"className":10606},[41],[43,10608,10609],{"xmlns":45,"display":167},[47,10610,10611,10700],{},[50,10612,10613,10615,10632,10638,10650,10658,10670,10676,10688,10696,10698],{},[53,10614,6996],{},[10616,10617,10618,10621],"mover",{},[175,10619,3002],{"stretchy":64,"minsize":10620},"3.0em",[10622,10623,10626],"mpadded",{"width":10624,"lspace":10625},"+0.6em","0.3em",[2467,10627,10628,10630],{},[53,10629,7101],{},[346,10631,2286],{},[2467,10633,10634,10636],{},[53,10635,6996],{},[346,10637,2212],{},[10616,10639,10640,10642],{},[175,10641,3002],{"stretchy":64,"minsize":10620},[10622,10643,10644],{"width":10624,"lspace":10625},[2467,10645,10646,10648],{},[53,10647,8717],{},[346,10649,2286],{},[2474,10651,10652,10654,10656],{},[53,10653,6996],{},[346,10655,2212],{},[175,10657,10389],{"mathvariant":2295,"lspace":10388,"rspace":10388},[10616,10659,10660,10662],{},[175,10661,3002],{"stretchy":64,"minsize":10620},[10622,10663,10664],{"width":10624,"lspace":10625},[2467,10665,10666,10668],{},[53,10667,7101],{},[346,10669,2212],{},[2467,10671,10672,10674],{},[53,10673,6996],{},[346,10675,423],{},[10616,10677,10678,10680],{},[175,10679,3002],{"stretchy":64,"minsize":10620},[10622,10681,10682],{"width":10624,"lspace":10625},[2467,10683,10684,10686],{},[53,10685,8717],{},[346,10687,2212],{},[2474,10689,10690,10692,10694],{},[53,10691,6996],{},[346,10693,423],{},[175,10695,10389],{"mathvariant":2295,"lspace":10388,"rspace":10388},[175,10697,3002],{},[175,10699,6705],{},[57,10701,10702],{"encoding":59},"M \\xrightarrow{l_0} M_1 \\xrightarrow{\\sigma_0} M_1' \\xrightarrow{l_1} M_2 \\xrightarrow{\\sigma_1} M_2' \\rightarrow \\cdots",[33,10704,10706,10831,10974,11130,11269,11339],{"className":10705,"ariaHidden":64},[63],[33,10707,10709,10713,10716,10719,10828],{"className":10708},[68],[33,10710],{"className":10711,"style":10712},[72],"height:1.1191em;vertical-align:-0.011em;",[33,10714,6996],{"className":10715,"style":7011},[77,78],[33,10717],{"className":10718,"style":261},[236],[33,10720,10723],{"className":10721},[265,10722],"x-arrow",[33,10724,10726,10819],{"className":10725},[813,2522],[33,10727,10729,10816],{"className":10728},[817],[33,10730,10733,10791],{"className":10731,"style":10732},[821],"height:1.1081em;",[33,10734,10736,10739],{"style":10735},"top:-3.322em;",[33,10737],{"className":10738,"style":829},[828],[33,10740,10743],{"className":10741},[833,834,835,836,10742],"x-arrow-pad",[33,10744,10746],{"className":10745},[77,836],[33,10747,10749,10752],{"className":10748},[77,836],[33,10750,7101],{"className":10751,"style":7146},[77,78,836],[33,10753,10755],{"className":10754},[809],[33,10756,10758,10782],{"className":10757},[813,2522],[33,10759,10761,10779],{"className":10760},[817],[33,10762,10765],{"className":10763,"style":10764},[821],"height:0.3173em;",[33,10766,10768,10772],{"style":10767},"top:-2.357em;margin-left:-0.0197em;margin-right:0.0714em;",[33,10769],{"className":10770,"style":10771},[828],"height:2.5em;",[33,10773,10776],{"className":10774},[833,10775,9739,836],"reset-size3",[33,10777,2286],{"className":10778},[77,836],[33,10780,2546],{"className":10781},[2545],[33,10783,10785],{"className":10784},[817],[33,10786,10789],{"className":10787,"style":10788},[821],"height:0.143em;",[33,10790],{},[33,10792,10796,10799],{"className":10793,"style":10795},[10794],"svg-align","top:-2.689em;",[33,10797],{"className":10798,"style":829},[828],[33,10800,10804],{"className":10801,"style":10803},[10802],"hide-tail","height:0.522em;min-width:1.469em;",[10805,10806,10812],"svg",{"xmlns":10807,"width":10808,"height":10809,"viewBox":10810,"preserveAspectRatio":10811},"http:\u002F\u002Fwww.w3.org\u002F2000\u002Fsvg","400em","0.522em","0 0 400000 522","xMaxYMin slice",[10813,10814],"path",{"d":10815},"M0 241v40h399891c-47.3 35.3-84 78-110 128\n-16.7 32-27.7 63.7-33 95 0 1.3-.2 2.7-.5 4-.3 1.3-.5 2.3-.5 3 0 7.3 6.7 11 20\n 11 8 0 13.2-.8 15.5-2.5 2.3-1.7 4.2-5.5 5.5-11.5 2-13.3 5.7-27 11-41 14.7-44.7\n 39-84.5 73-119.5s73.7-60.2 119-75.5c6-2 9-5.7 9-11s-3-9-9-11c-45.3-15.3-85\n-40.5-119-75.5s-58.3-74.8-73-119.5c-4.7-14-8.3-27.3-11-40-1.3-6.7-3.2-10.8-5.5\n-12.5-2.3-1.7-7.5-2.5-15.5-2.5-14 0-21 3.7-21 11 0 2 2 10.3 6 25 20.7 83.3 67\n 151.7 139 205zm0 0v40h399900v-40z",[33,10817,2546],{"className":10818},[2545],[33,10820,10822],{"className":10821},[817],[33,10823,10826],{"className":10824,"style":10825},[821],"height:0.011em;",[33,10827],{},[33,10829],{"className":10830,"style":261},[236],[33,10832,10834,10838,10879,10882,10971],{"className":10833},[68],[33,10835],{"className":10836,"style":10837},[72],"height:1.0734em;vertical-align:-0.15em;",[33,10839,10841,10844],{"className":10840},[77],[33,10842,6996],{"className":10843,"style":7011},[77,78],[33,10845,10847],{"className":10846},[809],[33,10848,10850,10871],{"className":10849},[813,2522],[33,10851,10853,10868],{"className":10852},[817],[33,10854,10856],{"className":10855,"style":2529},[821],[33,10857,10859,10862],{"style":10858},"top:-2.55em;margin-left:-0.109em;margin-right:0.05em;",[33,10860],{"className":10861,"style":829},[828],[33,10863,10865],{"className":10864},[833,834,835,836],[33,10866,2212],{"className":10867},[77,836],[33,10869,2546],{"className":10870},[2545],[33,10872,10874],{"className":10873},[817],[33,10875,10877],{"className":10876,"style":2553},[821],[33,10878],{},[33,10880],{"className":10881,"style":261},[236],[33,10883,10885],{"className":10884},[265,10722],[33,10886,10888,10963],{"className":10887},[813,2522],[33,10889,10891,10960],{"className":10890},[817],[33,10892,10895,10947],{"className":10893,"style":10894},[821],"height:0.9234em;",[33,10896,10897,10900],{"style":10735},[33,10898],{"className":10899,"style":829},[828],[33,10901,10903],{"className":10902},[833,834,835,836,10742],[33,10904,10906],{"className":10905},[77,836],[33,10907,10909,10912],{"className":10908},[77,836],[33,10910,8717],{"className":10911,"style":142},[77,78,836],[33,10913,10915],{"className":10914},[809],[33,10916,10918,10939],{"className":10917},[813,2522],[33,10919,10921,10936],{"className":10920},[817],[33,10922,10924],{"className":10923,"style":10764},[821],[33,10925,10927,10930],{"style":10926},"top:-2.357em;margin-left:-0.0359em;margin-right:0.0714em;",[33,10928],{"className":10929,"style":10771},[828],[33,10931,10933],{"className":10932},[833,10775,9739,836],[33,10934,2286],{"className":10935},[77,836],[33,10937,2546],{"className":10938},[2545],[33,10940,10942],{"className":10941},[817],[33,10943,10945],{"className":10944,"style":10788},[821],[33,10946],{},[33,10948,10950,10953],{"className":10949,"style":10795},[10794],[33,10951],{"className":10952,"style":829},[828],[33,10954,10956],{"className":10955,"style":10803},[10802],[10805,10957,10958],{"xmlns":10807,"width":10808,"height":10809,"viewBox":10810,"preserveAspectRatio":10811},[10813,10959],{"d":10815},[33,10961,2546],{"className":10962},[2545],[33,10964,10966],{"className":10965},[817],[33,10967,10969],{"className":10968,"style":10825},[821],[33,10970],{},[33,10972],{"className":10973,"style":261},[236],[33,10975,10977,10981,11037,11040,11127],{"className":10976},[68],[33,10978],{"className":10979,"style":10980},[72],"height:1.3551em;vertical-align:-0.247em;",[33,10982,10984,10987],{"className":10983},[77],[33,10985,6996],{"className":10986,"style":7011},[77,78],[33,10988,10990],{"className":10989},[809],[33,10991,10993,11028],{"className":10992},[813,2522],[33,10994,10996,11025],{"className":10995},[817],[33,10997,10999,11011],{"className":10998,"style":10518},[821],[33,11000,11002,11005],{"style":11001},"top:-2.453em;margin-left:-0.109em;margin-right:0.05em;",[33,11003],{"className":11004,"style":829},[828],[33,11006,11008],{"className":11007},[833,834,835,836],[33,11009,2212],{"className":11010},[77,836],[33,11012,11013,11016],{"style":4709},[33,11014],{"className":11015,"style":829},[828],[33,11017,11019],{"className":11018},[833,834,835,836],[33,11020,11022],{"className":11021},[77,836],[33,11023,10389],{"className":11024},[77,836],[33,11026,2546],{"className":11027},[2545],[33,11029,11031],{"className":11030},[817],[33,11032,11035],{"className":11033,"style":11034},[821],"height:0.247em;",[33,11036],{},[33,11038],{"className":11039,"style":261},[236],[33,11041,11043],{"className":11042},[265,10722],[33,11044,11046,11119],{"className":11045},[813,2522],[33,11047,11049,11116],{"className":11048},[817],[33,11050,11052,11103],{"className":11051,"style":10732},[821],[33,11053,11054,11057],{"style":10735},[33,11055],{"className":11056,"style":829},[828],[33,11058,11060],{"className":11059},[833,834,835,836,10742],[33,11061,11063],{"className":11062},[77,836],[33,11064,11066,11069],{"className":11065},[77,836],[33,11067,7101],{"className":11068,"style":7146},[77,78,836],[33,11070,11072],{"className":11071},[809],[33,11073,11075,11095],{"className":11074},[813,2522],[33,11076,11078,11092],{"className":11077},[817],[33,11079,11081],{"className":11080,"style":10764},[821],[33,11082,11083,11086],{"style":10767},[33,11084],{"className":11085,"style":10771},[828],[33,11087,11089],{"className":11088},[833,10775,9739,836],[33,11090,2212],{"className":11091},[77,836],[33,11093,2546],{"className":11094},[2545],[33,11096,11098],{"className":11097},[817],[33,11099,11101],{"className":11100,"style":10788},[821],[33,11102],{},[33,11104,11106,11109],{"className":11105,"style":10795},[10794],[33,11107],{"className":11108,"style":829},[828],[33,11110,11112],{"className":11111,"style":10803},[10802],[10805,11113,11114],{"xmlns":10807,"width":10808,"height":10809,"viewBox":10810,"preserveAspectRatio":10811},[10813,11115],{"d":10815},[33,11117,2546],{"className":11118},[2545],[33,11120,11122],{"className":11121},[817],[33,11123,11125],{"className":11124,"style":10825},[821],[33,11126],{},[33,11128],{"className":11129,"style":261},[236],[33,11131,11133,11136,11176,11179,11266],{"className":11132},[68],[33,11134],{"className":11135,"style":10837},[72],[33,11137,11139,11142],{"className":11138},[77],[33,11140,6996],{"className":11141,"style":7011},[77,78],[33,11143,11145],{"className":11144},[809],[33,11146,11148,11168],{"className":11147},[813,2522],[33,11149,11151,11165],{"className":11150},[817],[33,11152,11154],{"className":11153,"style":2529},[821],[33,11155,11156,11159],{"style":10858},[33,11157],{"className":11158,"style":829},[828],[33,11160,11162],{"className":11161},[833,834,835,836],[33,11163,423],{"className":11164},[77,836],[33,11166,2546],{"className":11167},[2545],[33,11169,11171],{"className":11170},[817],[33,11172,11174],{"className":11173,"style":2553},[821],[33,11175],{},[33,11177],{"className":11178,"style":261},[236],[33,11180,11182],{"className":11181},[265,10722],[33,11183,11185,11258],{"className":11184},[813,2522],[33,11186,11188,11255],{"className":11187},[817],[33,11189,11191,11242],{"className":11190,"style":10894},[821],[33,11192,11193,11196],{"style":10735},[33,11194],{"className":11195,"style":829},[828],[33,11197,11199],{"className":11198},[833,834,835,836,10742],[33,11200,11202],{"className":11201},[77,836],[33,11203,11205,11208],{"className":11204},[77,836],[33,11206,8717],{"className":11207,"style":142},[77,78,836],[33,11209,11211],{"className":11210},[809],[33,11212,11214,11234],{"className":11213},[813,2522],[33,11215,11217,11231],{"className":11216},[817],[33,11218,11220],{"className":11219,"style":10764},[821],[33,11221,11222,11225],{"style":10926},[33,11223],{"className":11224,"style":10771},[828],[33,11226,11228],{"className":11227},[833,10775,9739,836],[33,11229,2212],{"className":11230},[77,836],[33,11232,2546],{"className":11233},[2545],[33,11235,11237],{"className":11236},[817],[33,11238,11240],{"className":11239,"style":10788},[821],[33,11241],{},[33,11243,11245,11248],{"className":11244,"style":10795},[10794],[33,11246],{"className":11247,"style":829},[828],[33,11249,11251],{"className":11250,"style":10803},[10802],[10805,11252,11253],{"xmlns":10807,"width":10808,"height":10809,"viewBox":10810,"preserveAspectRatio":10811},[10813,11254],{"d":10815},[33,11256,2546],{"className":11257},[2545],[33,11259,11261],{"className":11260},[817],[33,11262,11264],{"className":11263,"style":10825},[821],[33,11265],{},[33,11267],{"className":11268,"style":261},[236],[33,11270,11272,11276,11330,11333,11336],{"className":11271},[68],[33,11273],{"className":11274,"style":11275},[72],"height:1.0489em;vertical-align:-0.247em;",[33,11277,11279,11282],{"className":11278},[77],[33,11280,6996],{"className":11281,"style":7011},[77,78],[33,11283,11285],{"className":11284},[809],[33,11286,11288,11322],{"className":11287},[813,2522],[33,11289,11291,11319],{"className":11290},[817],[33,11292,11294,11305],{"className":11293,"style":10518},[821],[33,11295,11296,11299],{"style":11001},[33,11297],{"className":11298,"style":829},[828],[33,11300,11302],{"className":11301},[833,834,835,836],[33,11303,423],{"className":11304},[77,836],[33,11306,11307,11310],{"style":4709},[33,11308],{"className":11309,"style":829},[828],[33,11311,11313],{"className":11312},[833,834,835,836],[33,11314,11316],{"className":11315},[77,836],[33,11317,10389],{"className":11318},[77,836],[33,11320,2546],{"className":11321},[2545],[33,11323,11325],{"className":11324},[817],[33,11326,11328],{"className":11327,"style":11034},[821],[33,11329],{},[33,11331],{"className":11332,"style":261},[236],[33,11334,3002],{"className":11335},[265],[33,11337],{"className":11338,"style":261},[236],[33,11340,11342,11346],{"className":11341},[68],[33,11343],{"className":11344,"style":11345},[72],"height:0.313em;",[33,11347,6705],{"className":11348},[4176],[15,11350,11351],{},"where: affine layers are responsible for stretching, rotating, translating, shearing; activation layers are responsible for folding, cropping, compressing, and restructuring the manifold.",[15,11353,11354,11355,11425],{},"If all ",[33,11356,11358,11376],{"className":11357,"translate":37},[36],[33,11359,11361],{"className":11360},[41],[43,11362,11363],{"xmlns":45},[47,11364,11365,11373],{},[50,11366,11367],{},[2467,11368,11369,11371],{},[53,11370,6198],{},[53,11372,3316],{},[57,11374,11375],{"encoding":59},"W_i",[33,11377,11379],{"className":11378,"ariaHidden":64},[63],[33,11380,11382,11385],{"className":11381},[68],[33,11383],{"className":11384,"style":3676},[72],[33,11386,11388,11391],{"className":11387},[77],[33,11389,6198],{"className":11390,"style":6247},[77,78],[33,11392,11394],{"className":11393},[809],[33,11395,11397,11417],{"className":11396},[813,2522],[33,11398,11400,11414],{"className":11399},[817],[33,11401,11403],{"className":11402,"style":3481},[821],[33,11404,11405,11408],{"style":7226},[33,11406],{"className":11407,"style":829},[828],[33,11409,11411],{"className":11410},[833,834,835,836],[33,11412,3316],{"className":11413},[77,78,836],[33,11415,2546],{"className":11416},[2545],[33,11418,11420],{"className":11419},[817],[33,11421,11423],{"className":11422,"style":2553},[821],[33,11424],{}," are invertible, then the affine layers themselves are only homeomorphic transformations and do not change the topology of the manifold. What gradually makes the manifold separable is often precisely the nonlinear deformation introduced by activation functions.",[15,11427,11428],{},"Therefore, from a geometric perspective, a neural network can be understood as an alternating combination of affine transformations and nonlinear manifold operations acting on the data manifold. This perspective aligns with many interpretations of modern manifold learning and geometric deep learning.",[10,11430,11432],{"id":11431},"activations","Activations",[15,11434,11435],{},"There is no \"single best\" choice for an activation function. However, from the perspective of trainability and expressive power of neural networks, a good activation function generally should satisfy several core characteristics:",[145,11437,11439],{"id":11438},"nonlinearity","Nonlinearity",[15,11441,11442],{},"As discussed earlier, nonlinearity is needed to break the linear closure of stacked affine layers, allowing multi-layer networks to approximate arbitrary functions.",[15,11444,11445],{},"Mathematical requirement: It must not satisfy linear combination closure:",[33,11447,11449],{"className":11448,"translate":37},[158],[33,11450,11452,11505],{"className":11451,"translate":37},[36],[33,11453,11455],{"className":11454},[41],[43,11456,11457],{"xmlns":45,"display":167},[47,11458,11459,11502],{},[50,11460,11461,11463,11465,11468,11470,11472,11474,11476,11478,11480,11482,11484,11486,11488,11490,11492,11494,11496,11498,11500],{},[53,11462,8717],{},[175,11464,178],{"stretchy":177},[53,11466,11467],{},"a",[53,11469,95],{},[175,11471,183],{},[53,11473,2171],{},[53,11475,126],{},[175,11477,188],{"stretchy":177},[175,11479,2296],{"mathvariant":2295},[53,11481,11467],{},[53,11483,8717],{},[175,11485,178],{"stretchy":177},[53,11487,95],{},[175,11489,188],{"stretchy":177},[175,11491,183],{},[53,11493,2171],{},[53,11495,8717],{},[175,11497,178],{"stretchy":177},[53,11499,126],{},[175,11501,188],{"stretchy":177},[57,11503,11504],{"encoding":59},"\\sigma(a x + b y) \\neq a \\sigma(x) + b \\sigma(y)",[33,11506,11508,11535,11592,11620],{"className":11507,"ariaHidden":64},[63],[33,11509,11511,11514,11517,11520,11523,11526,11529,11532],{"className":11510},[68],[33,11512],{"className":11513,"style":222},[72],[33,11515,8717],{"className":11516,"style":142},[77,78],[33,11518,178],{"className":11519},[229],[33,11521,11467],{"className":11522},[77,78],[33,11524,95],{"className":11525},[77,78],[33,11527],{"className":11528,"style":237},[236],[33,11530,183],{"className":11531},[241],[33,11533],{"className":11534,"style":237},[236],[33,11536,11538,11541,11544,11547,11550,11553,11589],{"className":11537},[68],[33,11539],{"className":11540,"style":222},[72],[33,11542,2171],{"className":11543},[77,78],[33,11545,126],{"className":11546,"style":142},[77,78],[33,11548,188],{"className":11549},[257],[33,11551],{"className":11552,"style":261},[236],[33,11554,11556,11583,11586],{"className":11555},[265],[33,11557,11559],{"className":11558},[265],[33,11560,11562],{"className":11561},[77,2353],[33,11563,11565],{"className":11564},[2357],[33,11566,11568,11571,11580],{"className":11567},[2361],[33,11569],{"className":11570,"style":73},[72],[33,11572,11574],{"className":11573},[2368],[33,11575,11577],{"className":11576},[77],[33,11578,2375],{"className":11579},[265],[33,11581],{"className":11582},[2379],[33,11584],{"className":11585},[236,2383],[33,11587,191],{"className":11588},[265],[33,11590],{"className":11591,"style":261},[236],[33,11593,11595,11598,11602,11605,11608,11611,11614,11617],{"className":11594},[68],[33,11596],{"className":11597,"style":222},[72],[33,11599,11601],{"className":11600,"style":142},[77,78],"aσ",[33,11603,178],{"className":11604},[229],[33,11606,95],{"className":11607},[77,78],[33,11609,188],{"className":11610},[257],[33,11612],{"className":11613,"style":237},[236],[33,11615,183],{"className":11616},[241],[33,11618],{"className":11619,"style":237},[236],[33,11621,11623,11626,11630,11633,11636],{"className":11622},[68],[33,11624],{"className":11625,"style":222},[72],[33,11627,11629],{"className":11628,"style":142},[77,78],"bσ",[33,11631,178],{"className":11632},[229],[33,11634,126],{"className":11635,"style":142},[77,78],[33,11637,188],{"className":11638},[257],[15,11640,11641],{},"Examples: ReLU, Tanh, Sigmoid, GELU.",[15,11643,11644],{},"Linear functions are not suitable because the composition of multiple linear layers remains linear, which would degenerate back to affine composition and limit expressive power.",[145,11646,11648],{"id":11647},"differentiability","Differentiability",[15,11650,11651,11652,11655],{},"Because the core of neural networks relies on backpropagation to determine weights, the activation function must be differentiable over most of its domain to facilitate gradient calculation during backpropagation. For non-differentiable points (e.g., ReLU at 0), ",[19,11653,11654],{},"subgradients"," are typically used. Moreover, the derivative properties affect gradient flow: gradients that are too small can cause vanishing gradients (Sigmoid), while gradients that are too large can cause exploding gradients.",[145,11657,11659],{"id":11658},"continuity","Continuity",[15,11661,11662],{},"Ensures that the network output changes continuously with the input; otherwise, training becomes unstable. Having few and controllable discontinuities is usually acceptable (e.g., ReLU has only one point of discontinuity, which has little practical impact).",[15,11664,11665],{},"Among the properties above, meeting the conditions is merely an \"entry ticket,\" not a \"guarantee of success.\"",{"title":11667,"searchDepth":11668,"depth":11668,"links":11669},"",2,[11670,11675,11676,11677,11678],{"id":12,"depth":11668,"text":13,"children":11671},[11672,11674],{"id":147,"depth":11673,"text":148},3,{"id":1143,"depth":11673,"text":1144},{"id":2969,"depth":11668,"text":2970},{"id":5184,"depth":11668,"text":5185},{"id":7078,"depth":11668,"text":7079},{"id":11431,"depth":11668,"text":11432,"children":11679},[11680,11681,11682],{"id":11438,"depth":11673,"text":11439},{"id":11647,"depth":11673,"text":11648},{"id":11658,"depth":11673,"text":11659},null,false,true,"md",{},"2026-04-06","\u002Fblog\u002F2026\u002F2026-04-06-nonlinearity-with-activations",{"title":5,"description":11667},"blog\u002F2026\u002F2026-04-06-nonlinearity-with-activations","Linearity requires additivity and scaling, but neural networks need nonlinearity to break free from simple affine transformations. While affine layers only stretch and rotate data without changing its topology, activation functions enable complex reshaping—allowing networks to untangle data manifolds and learn intricate patterns effectively.",[11694,11695],"mathematics","machine-learning","j7YjKDl87OP8DsUL-Ffg3fHhw_D6D21HRWTrG_OexDw",1781192434023]